This is an old problem with Citrix ADC / NetScaler: You should test all changes in test-site first and move them to production, or synchronise production site and disaster recovery site. Unfortunately, there is no built-in mechanism to synchronize configuration. So, over time, these sites will start differing, a very unwelcome state. This is especially true for WAF, as applications tend to change frequently and force admins, to adapt their WAF profiles.
So what can we do?
I created a set of scripts, allowing to copy parts of a configuration to another ADC. My use-case has been WAF profiles, but you may use the script for any other configuration you want to synchronize.
Prerequisites:
- Both sites need the same object with the same names.
- SSH ports (TCP/22) has to be open between these two sites.
- It takes SSH keys to log on to the other site.
- netscaler config must be saved, my script just coppies the saved config.
There are 3 scripts. These scripts should be located in /nsconfig on the source system. They are:
- extractconfig.pl, a pearl-script extracting the config
- runexport.sh the bash script, starting the export
- remotebatch.ns, a Citrix NetScaler script running via ssh on the remote system
Extracting the configuration:
My script is a simple pearl-script running on the active site. Its name is extractconfig.pl (you may use a different name, however, you will have to adapt the shell script accordingly):
#!/usr/bin/perl
# (C) Johannes Norz, September 2020
# extracts lines from an ns.conf file
# Description:
# The searchfor variable holds an array of strings to search for in the ns.conf file.
# Any line containing one of these strings get copied into the output-file.
# Example: searchfor = ("opject1", "object2", "object3")
# the output-file is /var/tmp/export.txt
@searchfor = ("PROF_APPFW_SAP", "PROF_APPFW_CRM");
use feature qw/say/;
# opening the ns.conf-file
open(FHin, '<', "/flash/nsconfig/ns.conf") or die $!; # creating the output file open(FHout, '>', "/var/tmp/export.txt") or die $!;
while (){
# searching for the search pattern
for (my $i = 0; $i < scalar(@searchfor); $i++){
if (index($_, $searchfor[$i]) != -1) {
# copying current line to the output file
print FHout $_;
last;
}
}
The name of this script should be extractconfig.pl
This script will extract all lines of code, containing the words PROF_APPFW_SAP or PROF_APPFW_CRM. These lines get written into a file called /var/tmp/export.txt
The bash-script
This script, of course, has to get triggered somehow, and there is more to do than just extract some lines of code from the ns.conf file: It also has to get copied to the other site and imported into the remote system. This is done by the next script:
#!/usr/bin/bash
#name of the script: runexport.sh
# copy parts of ns.conf to other Citrix ADCs
# (C) Johannes Norz, September 2020
# consists of 3 files:
# runexport.sh, a bash script to be called by chron (needs execution permission)
# extractconfig.pl, a perl-script extracting config changes (needs execution permission)
# remotebatch.ns, a script running on remote system
# this script copies to 2 ADCs, 10.0.0.1, 10.0.0.2 (active/passive HA).
# we never know, which one is the active one!
# it logs to a syslog server 10.10.10.10:5519
# function to copy the configuration
docopy () {
scp -i /nsconfig/ssh/SSH_key /var/tmp/export.txt nsroot@$1:/var/tmp/import.txt
ssh -i /nsconfig/ssh/SSH_key nsroot@$1 < /nsconfig/remotebatch.ns > /var/tmp/export_config.err
}
# call the pearl script to extract the configuration
/nsconfig/extractconfig.pl
# see if something had been changed
# we compare to the previous version of export.txt
DIFF=$(diff /var/tmp/export.txt /var/tmp/export.txt.0)
# has there been any difference?
if [ "$DIFF" != "" ]
then
# copy it to rempte-netscalers
docopy 10.0.0.1 # replace with your IPs
docopy 10.0.0.2 # replace with your IPs
#logging
echo "Changes copied $(date)" >> /var/log/Config-export.log
echo "Changes copied $(date)" | logger -h 10.10.10.10 -P 5519
# grooming backups, we keep a maximum of 4
rm /var/tmp/export.txt.4
mv /var/tmp/export.txt.3 /var/tmp/export.txt.4
mv /var/tmp/export.txt.2 /var/tmp/export.txt.3
mv /var/tmp/export.txt.1 /var/tmp/export.txt.2
mv /var/tmp/export.txt.0 /var/tmp/export.txt.1
mv /var/tmp/export.txt /var/tmp/export.txt.0
else
#logging
echo "No changes $(date)" >> /var/log/Config-export.log
echo "No changes $(date)" | logger -h 10.10.10.10 -P 5519
fi
The name of this script should be runexport.sh
Don’t forget to change the IP addresses!
You will need a pair of SSH keys, called SSH_key and SSH_key.pub. SSH_key.pub has to be in the remote ADC’s authorized_keys (/nsconfig/ssh/authorized_keys)! The script will logon to the remote system as nsroot using these SSH keys. You may use any other user, as long as this user has superuser privileges because it needs to copy files to the remote system. Users need to be local users, there is no support for TACACS, LDAP or RADIUS users! I have written a blog about SSH keys.
The script leaves a set of 5 “old” exported configurations behind. You may reduce this to only one, or have way more of them. I know, I didn’t do it in an elegant way, don’t blame me for this. At least one old configuration is needed.
The script running on the remote system
Like always, we have to import the configuration into the remote system. In many cases, it’s not possible to just overwrite parts of the Citrix ADC / NetScaler configuration. That’s also true for WAF signatures. So I change the policies first, assigning the APPFW_BYPASS profile.
There is a minor risk about it: There won’t be any protection for some seconds. I would permit this to happen, however, this is something you have to discuss with your security department. This is at a predictable time, as the script normally gets triggered via cron, always at the same time. Mitigating factor: The script won’t run often, so the chance is quite rare.
# (C) Johannes.norz1@citrix.com, September 2020 # first we assing a default profile to the policies set appfw policy POL_APPFW_SAP -profileName APPFW_BYPASS set appfw policy POL_APPFW_CRM -profileName APPFW_BYPASS # remove the existing profiles to maks sure, all changes get applied rm appfw profile PROF_APPFW_SAP rm appfw profile PROF_APPFW_CRM # import new data (profiles, ...) batch -f /var/tmp/import.txt # remove tmp-files shell rm /var/tmp/import.txt # bind the newly created profile again, so everything is OK. set appfw policy POL_APPFW_SAP -profileName PROF_APPFW_SAP set appfw policy POL_APPFW_CRM -profileName PROF_APPFW_CRM # save the configuration to be on the save side save config #that's it, exit, so we return to the source system exit
the name of this scrip should be remotebatch.ns
So you see: We assign default profiles to the policies, delete the existing profiles and recreate them.
Make it all happen automatically
Last step: create a cron job. I want a cron entry like that to be created in /ect/crontab:
0 3 * * * * /nsconfig/runexport.sh
Unfortunately, all the /etc directory is in RAM only, so it won’t survive a reboot. Luckily, there is a file in /nsconfig called rc.netscaler. This file gets executed after the complete boot, both BSD and NetScaler OS, is completed. We can use it to put data into the crontab.
so we need to add the following line into rc.netscaler:
echo 0 3 * * * * /nsconfig/runexport.sh >> /etc/crontab
Finished! The script will run every day by 3 AM. There will be a log in /var/log/Config-export.log. It will either:
- No changes 2020-09-09
- Changes copied 2020-09-09
Logging can be done to a syslog server in addition, using the logger-command. In my example, the syslog server is 10.10.10.10 port 5519. Fix the IP address!
I hope you like my little script. Any comments and suggestions are highly welcome!
