Last update: 2018/03/27
Sometimes we have to schedule commands in a Citrix NetScaler. A good example would be:
force HA failover
It’s obvious, we don’t want to fail over during day time to not disconnect TCP connections, to not interrupt users. The best time would be something like 3:30 AM. It’s obvious, we don’t want to set an alarm for 3:00 to get up, take a shower, brush teeth, just to force an HA fail over. At least I don’t want!
Scheduling an HA fail over for off peak hours is important for both, Citrix NetScalers proxying big files for download and for NetScaler Gateways: During HA fail-over we will loose TCP-sessions, so downloads will break and HDX (ICA) sessions will get disconnected.
Starting to dig into Citrix NetScaler
Inside a NetScaler there are two operating systems working at the same time and therefore two different shells:
- the Citrix NetScaler shell, the first one you connect too using putty (or even better: smarTTY)
- the BSD shell. It can be reached typing
shell
into NetScaler’s command line
There is no chance to schedule commands in NetScaler OS. But BSD is just an ordinary UNIX (please don’t call BSD a Linux, it is not). My first guess would be to use at, however at is not there. So we need to use crontab.
Crontab in UNIX is used to schedule commands on a regular base. So crontab would be great to schedule a backup of Citrix NetScaler configuration, it’s not perfect for one time commands.
We could install at into BSD, but I never install software into a NetScaler and I would strongly advise you to keep away from doing this. So we need to use crontab.
How to execute a NetScaler shell command from BSD?
That’s a big question. BSD shell just allows to execute BSD commands. So what now?
nscli
nscli is a UNIX command on a NetScaler, allowing users to execute NetScaler commands from BSD
root@82e3d3135738# man mscli
No manual entry for mscli
shit.
root@82e3d3135738# nscli --help
Usage: nscli [-norc]
[-U []:]
[-D ] [-s]
[[-k]
where:
-norc causes the personal initialization file, ~/.nsclirc,
to be skipped
is the IP of the target NetScaler
is used to log in to the target NetScaler
is an integer between 0 and 9
-s stifles "exec:" and "Done" messages
much better! so we have to execute a command like that:
nscli -U 127.0.0.1:nsroot show ns runningconfig
so we specify a NetScaler IP (no SNIP, sorry guys, we’re dealing with BSD!), an user name and NetScaler commands after this.
It works fine, unfortunately we get prompted for a password. So we can’t easily use this command in a batch file? Yes we can. There is some information missing: we may specify a password as well. No too beautiful, as this batch file will also contain the password in plain text, but possible. The command would look like that:
nscli -U 127.0.0.1:nsroot:your_Password_goes_here show ns runningconfig
easy? Yes, it is! You may even skip the IP using this command locally:
nscli -U :nsroot:your_Password_goes_here show ns runningconfig
This leading : assumes an IP of 127.0.0.1.
Using crontab on a NetScaler
Using crontab on a NetSaler would be more than just easy. Just add a standard crontab entry into /etc/crontab.
30 3 * * * root nscli -U 127.0.0.1:nsroot:your_Password_goes_here force ha failover -force
That’s simple.
Next we’ll have to kill cron and start cron (cron start) again, so it will reread crontab.
root@82e3d3135738# cron start
cron: cron already running, pid: 965
root@82e3d3135738# kill 965
root@82e3d3135738# cron start
Unfortunately this entry won’t disappear after executing, so it will get executed tomorrow and the day after tomorrow as well. So you have to remove this entry tomorrow morning. Still by far better than getting up in the middle of the night, isn’t it?
What else could we do?
We could also use this for daily tasks, such as backing up ns.conf, purging log files and many more!
BUT
never reboot your NetScaler! Why? All content in /etc gets discarded. /etc is just RAM, no disk based file system.
What to do?
Well we need to rewrite /etc/crontab with every reboot! I’m pretty sure you won’t like to do this. There has to be an other way, a more automatic way, to write data into crontab!
We could use /etc/rc.conf to fill crontab after reboot. Unfortunately we face the same problems here: It will get discarded during boot. However there is a file called /flash/nsconfig/rc.netscaler (see CTX122271). This is the template for the /etc/rc.conf.
There is a good description in Citrix forums by Rob Harp about how to use it. Rob’s example is about doing daily backups. I’d suggest reading his article.
An important note in the end
Keep in mind: Changes to BSD shell is executed on this very Citrix NetScaler only. It will never get executed on the other node of a HA or cluster! You’ll probably have to do these changes with all nodes!
Hello!
And what if the command requires confirmation of “yes / no”?
root@ns2-vpx-esx:~$ nscli -U :nsroot:nsroot force ha failover
Done
[WARNING]:Force Failover may cause configuration loss, peer health not optimum. Reason(s):
– HA peer node DOWN/NOT-UP/STAYSECONDARY
Please confirm whether you want force-failover (Y/N)? [N]:
Pups. I didn’t run into this, because I specified a -force parameter. It forces them to fail over without asking. It was latest version 11.1 and 12.0 firmware.
I admit: force something -force sounds spooky. But actually that’s what you have to do: Force twice. The right comment is:
force ha failover -force
Yes, thanks! And on NS9.3: Build 65.8.cl version is similar (the version can not be updated)