SVN UUID Mismatch and SVN Data Synchronization

Introduction

This document describes how to synchronize the Subversion control (SVN) repository between PCRFClient01 and PCRFClient02.

SVN UUID MISMATCH

All the policy configurations in policy builder (PB) are stored as an SVN repository. So, in order to have high availability (HA) you need to have the same SVN Universal Unique Identifier (UUID) across both of the PCRFClient Virtual Machines (VMs). However, Cisco Policy Suite (CPS) Version 7.X does not have the same UUID across PCRFClient VMs. That is, higher CPS versions have the same UUID. An enhancement bug has been entered, Cisco bug ID CSCuu85319. You could achieve HA if you have the same revision number in both PCRFClient0X VMs.

Verify SVN Synchronization Across PCRFClient VMs

Since the UUID is not the same across both PCRFClient VMs, enter these commands:

svn info http://pcrfclient01/repos/configuration
svn info http://pcrfclient02/repos/configuration

Note: With these commands, you have to give the exact host name of the PCRFClient01 and PCRFClient02 VMs. You can also use the script check_svn_sync.sh in order to check whether the SVN repository is synchronized across PCRFClient VMs. This script is provided at the end of this article.

You might notice these  Quantum Network suite (QNS) error logs. These error messages can be checked with the script svn_uuid_mismatch.sh. This script is provided at the end of this article.

==qns-1.log==
2015-06-10 00:51:15,058 [pool-4-thread-1] WARN
  c.b.r.impl.ReferenceDataManager - SVN directory failed in updating
2015-06-10 01:51:44,050 [pool-4-thread-1] WARN
  c.b.r.impl.ReferenceDataManager - SVN directory failed in updating
2015-06-10 03:52:41,061 [pool-4-thread-1] WARN
  c.b.r.impl.ReferenceDataManager - SVN directory failed in updating

==Service-qns-1.log==
svn: Repository UUID '70e98d90-b9c7-4c5b-a3db-681dd68c62a6'
 doesn't match expected UUID 'ed2f2f76-f588-443e-9bb8-3fb05abd903b'
svn: Repository UUID '70e98d90-b9c7-4c5b-a3db-681dd68c62a6'
 doesn't match expected UUID 'ed2f2f76-f588-443e-9bb8-3fb05abd903b'
svn: Repository UUID '70e98d90-b9c7-4c5b-a3db-681dd68c62a6'
 doesn't match expected UUID 'ed2f2f76-f588-443e-9bb8-3fb05abd903b'

As shown below you can observe the UUID is different across VMs, but the versions have to remain the same. If the SVN version in both the PCRFClient VMs are same, then the SVN repository synchronization is there and subsequently you have high availability.

===PCRFClient01===

[root@pcrfclient01 ~]# svn info http://pcrfclient01/repos/configuration         
Path: configuration
URL: http://pcrfclient01/repos/configuration
Repository Root: http://pcrfclient01/repos
Repository UUID: ed2f2f76-f588-443e-9bb8-3fb05abd903b
Revision: 392    <<<<  revision number 
Node Kind: directory
Last Changed Author: broadhop
Last Changed Rev: 392
Last Changed Date: 2015-06-01 15:52:12 -0600 (Mon, 01 Jun 2015)


===PCRFClient02===

[root@pcrfclient01 ~]# svn info http://pcrfclient02/repos/configuration         
Path: configuration
URL: http://pcrfclient02/repos/configuration
Repository Root: http://pcrfclient02/repos
Repository UUID: 70e98d90-b9c7-4c5b-a3db-681dd68c62a6
Revision: 392   <<<< revision number 	
Node Kind: directory
Last Changed Author: broadhop
Last Changed Rev: 392
Last Changed Date: 2015-06-01 15:52:12 -0600 (Mon, 01 Jun 2015)

Fix the SVN Repository Synchronization

The previous section explained how to check the SVN repository synchronization. This section discusses how to fix the SVN synchronization. Assume that PCRFClient01 is primary and PCRFClient02 is secondary and that the PCRFClient02 SVN repository is out of sync.

Complete these steps in order to make PCRFClient02 SVN repository sync up with PCRFClient01:

1. On PCRFClient01, disable the SVN sync job:

   crontab -e
   
   Comment this cron job entry:
   /usr/bin/svnsync sync http:// pcrfclient02/repos-proxy-sync

   Note: Here, PCRFClient02 is the customer host name of the PCRFClient02 VM.

2. Log in to PCRFClient02 and edit "/etc/httpd/conf.d/subversion.conf" and update it with this bolded text:

   <Location /repos-proxy-sync>
         DAV svn
         SVNPath /var/www/svn/repos
         Order deny,allow
         Deny from all
             Allow from pcrfclient01     <<< # customer host name of PCRFClient01 VM
   </Location>

3. On the PCRFClient02 VM , enter this command in order to restart the HTTPD service.

   service httpd restart

4. On PCRFClient01, enter this command in order to synchronize the SVN data. The SVN repository synchronization might take time dependent upon the number and size of revisions.

   /usr/bin/svnsync sync http://pcrfclient02/repos-proxy-sync

   Note: Here, PCRFClient02 is the customer host name of PCRFClient02 VM.

   If lock errors are reported, then enter this command and then retry the svnsync command:

   /usr/bin/svn propdel svn:sync-lock --revprop -r 0 http:// pcrfclient02/repos-proxy-sync

   If the errors are still seen, proceed to Reinitialize SVN Synchronization.

5. In order to verify that the revision number for the SVN repository is the same in PCRFClient01 and PCRFClient02, use the check_svn_sync.sh script or the CLI command mentioned in Verify SVN Synchronization Across PCRFClient VM.
6. On PCRFClient01, enable the SVN sync job:

   crontab -e
   /usr/bin/svnsync sync http://pcrfclient02/repos-proxy-sync

Reinitialize SVN Synchronization

Perform these steps only if you encounter an error in step 4 in section Fix the SVN Repository Synchronization. If you do not encounter any errors in the previous section, you can ignore these steps.

Note: Reach out to the Cisco Technical Support team before you perform these steps.

1. Before you reinitialize the SVN sync, first disable PCRFClient02 from svn_proxy in the haproxy configuration:
   1. Go to standby lb (lb02 in this example) and modify "/etc/haproxy/haproxy.cfg" and comment out the bolded line shown here:

      listen svn_proxy lbvip02:80
              mode http
              balance roundrobin
              option httpchk
              option httpclose
              option abortonclose
              server pcrfclient01 pcrfclient01:80 check
              #server pcrfclient02 pcrfclient02:80 check backup
      

   2. Go to active lb (lb01 in this example) and enter this command in order to do a HA failover:

      /etc/init.d/heartbeat restart

   3. Verify that "haproxy/memcached/VIP" shifted to the other lb (lb02 in this example).
   4. Go to standby lb (now lb01 is standby) and modify "/etc/haproxy/haproxy.cfg" and comment out the bolded line:

      listen svn_proxy lbvip02:80
              mode http
              balance roundrobin
              option httpchk
              option httpclose
              option abortonclose
              server pcrfclient01 pcrfclient01:80 check
              #server pcrfclient02 pcrfclient02:80 check backup

   5. Go to active lb (now lb02 is active) and enter this command in order to do an HA failover:

      /etc/init.d/heartbeat restart

   6. Verify "haproxy/memcached/VIP" shifted to the other lb (lb01 in this example).
2. On PCRFClient02, enter these commands in order to reinitialize the SVN repository directory:

   tar -czf /var/tmp/repos.tar.gz /var/www/svn/repos
   mkdir -p /var/www/svn/repos
   rmdir /var/www/svn/repos
   /usr/bin/svnadmin create /var/www/svn/repos
   chown -R apache:apache /var/www/svn/repos

3. On PCRFClient02, create a file named "pre-revprop-change" in the path "/var/www/svn/repos/hooks/" with this data:

   #!/bin/sh
   #/var/www/svn/repos/hooks/pre-revprop-change
   exit 0

4. On PCRFClient02, enter these commands in order to change the permissions of the previously created file and also perform HTTPD service restarts:

   chmod 700 /var/www/svn/repos/hooks/pre-revprop-change
   chown apache:apache /var/www/svn/repos/hooks/pre-revprop-change
   service httpd restart

5. On PCRFClient01, enter these commands in order to perform SVN data synchronization:

   /usr/bin/svnsync init http://pcrfclient02/repos-proxy-sync
    http:// pcrfclient01/repos-proxy-sync
   /usr/bin/svnsync sync http:// pcrfclient02/repos-proxy-sync

   Note: In place of PCRFClient01 and PCRFClient02, provide the appropriate host-name of PCRFClient01 and PCRFClient02 as per your setup.

6. In order to verify whether your SVN repositories are synchronized, complete the steps in Verify SVN Synchronization Across PCRFClient VM.
7. Enable PCRFClient02 from the SVN proxy in the haproxy configuration. Basically, uncomment the lines which you commented out in step 1.
   1. Go to standby lb (lb02 in this example) and modify "/etc/haproxy/haproxy.cfg" and uncomment the bolded line:

      listen svn_proxy lbvip02:80
              mode http
              balance roundrobin
              option httpchk
              option httpclose
              option abortonclose
              server pcrfclient01 pcrfclient01:80 check
              server pcrfclient02 pcrfclient02:80 check backup
      

   2. Verify "haproxy/memcached/VIP" shifted to the other lb (lb02 in this example).

      /etc/init.d/heartbeat restart 

   3. Verify "haproxy/memcached/VIP" shifted to the other lb (lb02 in this example).
   4. Go to the standby lb (now lb01 is standby) and modify "/etc/haproxy/haproxy.cfg" and uncomment the bolded line shown here:

      listen svn_proxy lbvip02:80
              mode http
              balance roundrobin
              option httpchk
              option httpclose
              option abortonclose
              server pcrfclient01 pcrfclient01:80 check
              server pcrfclient02 pcrfclient02:80 check backup
      

   5. Go to active lb (nowlb02 is active) and enter this command in order to complete the HA failover:

      /etc/init.d/heartbeat restart

   6. Verify that "haproxy/memcached/VIP" shifted to the other lb (lb01 in this example).
8. On PCRFClient01, enable the SVN sync job:

   crontab -e/usr/bin/svnsync sync http://pcrfclient02/repos-proxy-sync

Scripts

check_svn_sync.sh

Please put this script in your CPS system in order to check the SVN synchronization across PCRFClient VMs.

#!/bin/bash
a=$(svn info http://pcrfclient01/repos/configuration | grep -i Revision | grep -o '[0-9]*')
b=$(svn info http://pcrfclient02/repos/configuration | grep -i Revision | grep -o '[0-9]*')
if [ "$a" == "$b" ]; then
	echo -e  "SVN repository  Synchronization:\e[0;32m PASS \e[0m"
	echo -e  "PCRFClient01 SVN repository revision number is : \e[1;33m $a \e[0m"
	echo  -e  "PCRFClient02 SVN repository revision number is : \e[1;33m $b \e[0m"
else
echo -e "SVN repository Synchronization :\e[0:31m FAIL \e[0m"
echo -e  "PCRFClient01 SVN repository revision number is : \e[1;33m $a \e[0m"
echo -e  "PCRFClient02 SVN repository revision number is : \e[1;33m $b \e[0m"
fi

svn_uuid_mismatch.sh

Please put this script in your CPS system in order to monitor the SVN UUID related error messages from the individual QNS VMs.

#!/bin/bash
for HN in `hosts.sh |grep qns`; 
do echo -e "\e[1;32m $HN\e[0m";
echo -e "\e[1;33m===qns-1.log===\e[0m" 
ssh $HN grep -wh "SVN directory failed" 2>/dev/null  /var/log/broadhop/qns-1.log | tail -10;
echo -e  "\e[1;33m===service-qns-1.log===\e[0m"
ssh $HN grep -wh "match expected UUID" 2>/dev/null  /var/log/broadhop/service-qns-1.log | tail -10;
done