Citrix NetScalerVPX out of disk space


My NetScaler VPX ran out of disk space. I did several tricks like adding a second disk and mounting ít into the update directors, to be able to upgrade my NetScaler.

Today I found time to investigate this issue. I had no success flooowing CTX133588. I so I started looking for the largest files and directories in /var.

root@82e6de130138# du -a /var | sort -n -r | head -n 10
13033540        /var
11023956        /var/ns_system_backup
466274  /var/log
461740  /var/log/db
459822  /var/wi
434188  /var/netscaler
305070  /var/log/db/default
248492  /var/nslog
183824  /var/ns_system_backup/backup_7b829416-9965-475b-8dfb-d57d18ce9ba0.tgz
183824  /var/ns_system_backup/backup_1676b077-6132-47f5-88a5-aab4832e5c93.tgz
root@82e6de130138# df
Filesystem  1K-blocks    Used    Avail Capacity  Mounted on
/dev/md0       358830  344602     7052    98%    /
devfs               1       1        0   100%    /dev
procfs              4       4        0   100%    /proc
/dev/ad0s1a   1482638  941530   422498    69%    /flash
/dev/ad0s1e  14348494 2009796 11190820    15%    /var

What a big surprise! 11 GB in /var/ns_system_backup? How comes? I looked into it.

root@82e6de130138# ls -l
total 11019220
drwxr-xr-x  2 root  wheel       3584 Mar  8 17:13 WebServer
-rw-r--r--  1 root  wheel  188077595 Mar  5 17:14 backup_083f6140-a660-41d5-83f2-389403f29334.tgz
-rw-r--r--  1 root  wheel  187946493 Feb 20 17:13 backup_0bd31ba5-fa66-4e40-a912-d6c1a9ffdbdc.tgz
-rw-r--r--  1 root  wheel  187814966 Feb 14 05:13 backup_11653391-2dff-4c22-ba22-3cd9212cd7c2.tgz
-rw-r--r--  1 root  wheel  187751996 Feb  8 05:13 backup_15904e38-2e8f-49f1-ad8c-e153d9daf980.tgz
-rw-r--r--  1 root  wheel  188106376 Mar  8 17:15 backup_1676b077-6132-47f5-88a5-aab4832e5c93.tgz
-rw-r--r--  1 root  wheel  187932332 Feb 19 05:13 backup_189cc344-e721-40ae-81af-55935a0ee7ac.tgz
-rw-r--r--  1 root  wheel  187961509 Feb 22 05:13 backup_195f2b0a-8b21-4be9-b277-50220247b553.tgz
-rw-r--r--  1 root  wheel  188000564 Feb 26 05:14 backup_26ac5a11-0c8c-4034-b79b-64a94fee1ac1.tgz
-rw-r--r--  1 root  wheel  188098718 Mar  7 05:14 backup_3379fb1a-068e-4bc5-acfc-091108cece18.tgz
-rw-r--r--  1 root  wheel  187761902 Feb  9 05:13 backup_34fd7671-fad1-43b4-b662-0e3b7f221475.tgz

a total of 60 files, round about 183 MB each. In addition there was a subdirectory with the name of my admin partition, containing 60 more files, 7 kb each.

All these files had been relatively new. They had been created at 5:13 AM and PM, so my first guess was a cron job.

These tgz files are a full copy of /flash/nsconfig and selected subdirectories of /var. I could not find a cron job doing this backup, and I could not find any documentation about a /var/ns_system_backup folder.

I kept playing round, but did not find. I posted this question on and Carl Stalhood asked do you have MAS in place? Yes I do!

It’s MAS to be blamed for!

So I opened MAS, SystemSystem Administration and clicked on Instance Backup Settings. MAS does backup on managed devices twice a day by default. I changed to one backup in 10 days. It should keep just 1 backup, but it failed to delete. I also upgraded MAS and hope problem is solved by now …

Versions involved:

MAS: 11.1 52.26
NetScaler: 11.1 Build 52.13 and 51.26

About the author

Johannes Norz

Johannes Norz is a Citrix Certified Citrix Technology Advocate (CTA), Citrix Certified Instructor (CCI) and Citrix Certified Expert on Application Delivery and Security.

He frequently works for Citrix international Consulting Services and several education centres all around the globe.

Add comment

Recent Posts

Recent Comments