last update: March 2nd 2020
Well, there are many guides. So why do I write a blog about it? Just to have one more?
Bull shit!
The truth is: I don’t like them at all!
What’s wrong about all these guides?
They all focus on how to remove malware currently installed on our Citrix ADCs (NetScalers). And, to be honest, it does not make the least little bit of sense. How can you ever be 100% sure, you removed all malware? I’m a trained and certified “white hat”, I had been able to write my own exploit on December 17th. But I can’t remove all malware and therefore can’t be 100% sure I removed everything. There is malware out there, currently behaiving absolutely silent. And no one out there is able to say, “it’s 100% sure I removed everything”. That’s the truth.
You didn’t patch your box immediately after December 17th, but you are not affected? Sure? How do you know? Consider yourself to be affected!
What now?
Well, it’s a VPX, isn’t it? Do you have an old snap shot (prior to December 12th)? Revert to it. You don’t have a snap shot like that? You think, it’s time fro a fresh start? Recreate your VPX!
We have to know about attack vectors in CVE-2019-19781, to know, how to deal with a potentially infected VM.
Attack vectors in CVE-2019-19781
A Citrix ADC has a very simple file system. Let’s have a look at the mount tab:
You can see easily:
/flash and /var are on disks. Root (/) is on /dev/md0. And /dev/md0 is a RAM drive. Because of this, nothing can survive a reboot, but things located in /var and /flash. This is also true for /etc.
Usually, daemons get started from /etc/cron. How can malware survive a reboot? There are several methods. The most popular one had been used by Citrix to patch CVE-2019-19781: They put data into /flash/nsconfig/rc.netscaler. Unfortunately that’s not the only method, there may be several other ones, including, but not limited too, infecting Citrix ADC / NetScaler core files (located in /var).
rc.netscaler, by default, is empty, or does not exist. Therefore, we won’t include rc.netscaler during our migration.
Migrating the current configuration to a new VPX
Of course, we don’t want to start from scratch. Who would? So we will backup the current configuration first. To do so, we open winscp and copy all /flash/nsconfig directory to our local machine. This will also copy certificates from /flash/nsconfig/ssl, license files form /flash/nsconfig/license and (if you use admin partitions) partition configuration from /flash/nsconfig/partitions.
In /var we need all the /var/log/db/ subdirectory.
WAF-Signatures are located in /var/download/custom/ and /var/ns_system_backup/backup. There are also some configuration data bases called /var/log/db/default/appfwprofile/appfwprofile_*.pdb. Signatures are in /var/newnslog/asl. You also have to backup and migrate these files, or the profiles won’t get migrated, and because of this, the corresponding WAF policies won’t be there! This is a horrible problem. You won’t see any issues, but there is protection from WAF no more. Double check the existence if all signatures!
If you costumized the GUI in a Citrix Gateway, AAA vServer, Citrix Unified Gateway, …, you also fave to back up files in /var/ns_gui_custom/ and /var/netscaler/logon/themes/. Make sure you just copied your own files and no binaries, coming from somewhere else (even though, they won’t get invoked, as there is nothing left to start these binaries)
That’s it!
Now let’s delete the rc.netscaler file (located in nsconfig directory). Different to mal-ware, we have no need for this file. I usually also check presence of suspicious files, but never found any.
You would probably want to also migrate log files and counters (what for???), in case, you would do the same to /var/log and /var/nslog. Be sure to copy nothing but this, we don’t want to migrate malware!
Setting up the new VM
Download and import a new vm. This vm will be a “virgin”, so it won’t have any configuration and the nsroot password will be nsroot.
First time run-wizard
Next, we have to start this newly created VM. We have to do a first time run setup, including a new NSIP (only used during migration. We will restore the original one). My screen shots show this first time run for one of my customer’s sites.
Than, press 4, save and quit and the box will boot up.
Restoring the configuration
Now, the box is booted up. Username / password both is nsroot. We don’t need to change anything about it, that’s just fine. The machine is ready to get restored.
Again, I open winscp and copy my backed up files to /flash/nsconfig, overwriting the existing ones. Do the same if you wanted to migrate logs and counters as well. Shut down this box, don’t save anything!
Shut down your current Citrix ADC / NetScaler.
Now set the mac address of the newly created ADC to the MAC address the original one had. Doing so allows us to use the original license files. The procedure varies on type of hypervisor you’re using, so I won’t provide screen shots.
Shut down the original Citrix ADC / NetScaler. At the same time, boot up the new one.
Your complete configuration should be there now, nothing should have changed, but there is no malware on the box for sure. The password is the original one, AD, … will work perfectly well. Give it a try, and delete the old, potentially compromised box, if you’re sure everything is there!
What to check after you migrated successfully?
Check, if all licenses are there. Give all the vServers a try, if you use admin partitions, especially check the ones inside partitions. Even though SSL configuration should not have changes, it would be time for a new SSL-labs SSL test. The requirements changed, and you should fine-tune your settings.
Migrate with HA
I thought of using HA to refresh the box (destroy HA, create a new VPX, put it in HA, delete the former active and create the other VPX). But I don’t suggest this procedure, as these boxes share the same nsinternal user/password and are aware of each other. A skilfully written malware may be able to infect a new passive box within seconds.
You may probably not agree to my concerns, but please consider my arguments …
THE END
Write to me, if you run into troubles, but it worked fine for me several times, so I guess, it will also work fine for you. Special thanks to
Hi Johannes,
Just a quick question on the below bit :
WAF-Signatures are located in /var/download/custom/ and /var/ns_system_backup/backup. There are also some configuration data bases called /var/log/db/default/appfwprofile/appfwprofile_*.pdb. Signatures are in /var/newnslog/asl. You also have to backup and migrate these files, or the profiles won’t get migrated, and because of this, the corresponding WAF policies won’t be there!
If you are upgrading the FW on a Netscaler VPX do you manually have to backup these locations ?
If you perform the upgrade say from 11 to 13 would these locations and files still be there after the FW upgrade or would they be gone for good ? If they are gone and you performed a full backup via NS gui could you get them back ?
Thanks,
Mada, unfortunately, I have seen all WAF configuration go away completely after upgrading from 11.? to 12.1. I can’t explain how this had been possible (as I found out way too late), nor could I reproduce this behaviour in my lab, but I have seen. 🙁
I think it’s always a wise idea to back up all these files, all together with your ns.conf. And after upgrade: Go to system -> diagnostics -> “Pre v/s post upgrade”. I would always do this, as it will also point out changes in syntax and more.