last update: October 22nd  2021

Citrix NetScaler Logging and policy troubleshooting

Some times it’s quite hard to understand what’s going on. There is a lot of mystics about policies. And it’s even harder to understand what went on (past tense). “Johannes, there had been several problems connecting to <any blabla application here>” “I’m sorry, I can’t know. What did they do? The request could have been blocked by a responder policy or by WAF”. Did you ever get an answer to a “What did you do?” question?

So we have to do reverse engineering. The only thing we know is: There had been something going on, and it failed. Maybe we know some of the symptoms.

That’s a mess. And we don’t need to act blindly like that. There are logs in a Citrix NetScaler!

Where do logs too?

All logging goes to /var/log/ns.log. This log gets periodically archived and recreated. That’s the source of information if our Citrix NetScaler web application firewall (WAF) blocked!

How do WAF-Logs look like?

Let’s have a look at some entries:

Mar  9 09:26:28 <local0.info> 192.168.0.25 03/09/2017:09:26:28 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_STARTURL 5852 0 :  81.153.82.18 40102-PPE1 /PctEsh
SVh4Uv9sK/sBQOR5GSdg0001 app_fw_blog_data Disallow Illegal URL: https://blog.nor
z.at/category/common-citrix-themes/page/3/?as_url_id=AAAAAAW5jxPwYiwdw2xu-S6V6uY
6IgElFoDo1HPM_1hQWBYABzbf-StjgFD_ZnsEPbr_nDarC6Y-P6y_CgyS0u_ubJBmqVmA7df3rvHA5Io
PE2Hq6w%3D%3D <blocked>

So we see it’s a Citrix NetScaler Web Application Firewall (WAF) log (APPFW). It’s the start URL function blocking (APPFW_STARTURL). The policy is called app_fw_blog_data. The vServer is at IP 192.168.0.25. The URL, my attacker (IP address 81.153.82.18, British Telecom, if IP location finder is right) wanted to access, was https://blog.norz.at/category/common-citrix-themes/page/3/ with some parameters attached to it. Taking a closer look at it I can see: It’s a non existing page, it’s a forceful browsing attack!

Let’s see a second one:

Mar  9 09:36:07 <local0.info> 192.168.0.25 03/09/2017:09:36:07 GMT 82e6de130138 
0-PPE-0 : default APPFW AF_400_RESP 193 0 :  36.84.91.100 14386-PPE0 - norz_df h
ttp:/// Bad request headers.No host header <blocked>

This one is also by my WAF (APPFW). It’s the AF_400_RESP function, the HTTP request syntax checker. vServer, again, 192.168.0.25. The request came from 36.84.91.100, an IP from Indonesia (PT Telkom Indonesia). The WAF policy was norz_df. There had been bad request headers, so the WAF did not let this request pass through. I guess it’s a request generated by a scan- bot.

Mar  9 10:08:28 <local0.info> 192.168.0.25 03/09/2017:10:08:28 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_DENYURL 6195 0 :  56.181.27.117 43870-PPE1 - norz_
df Disallow Deny URL: https://192.168.200.109/test/ for rule pattern="/test/?" <blocked>

This one: also by my WAF (APPFW). It’s the APPFW_DENYURL feature, so i’s a denied URL. The rule triggered was 6195, blocking access to /test/?. The rest like the ones above. Easy like that.

Mar  9 10:13:09 <local0.info> 192.168.0.25 03/09/2017:10:13:09 GMT 82e6de130138 
0-PPE-1 : default APPFW APPFW_BUFFEROVERFLOW_URL 6241 0 :  56.181.27.117 44091-P
PE1 - norz.at URL length(1420) is greater than maximum allowed(1024): https://no
rz.at/agb?1234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123 <blocked>

Again the WAF (APPFW), the subystem was APPFW_BUFFEROVERFLOW_URL. So there was a buffer overflow about the URL. If you take a look at it you see: there is an enormous query string. (the attacker was me, so it’s easy to find out)

What about other policies like responders?

<put 4 letter word of your choice here> there is no logging capability built into responder policies.

Is this true?

Sorry to say: yes. So we can’t reproduce what’s going on.

But?

You long for a but? OK. I have a but!

But you can make them logging! They don’t log, unless you make them logging. Great news, isn’t it?

How to make (responder) policies log

Let’s have a look at a policy:

This is a Citrix NetScaler responder policy dropping requests originating from well known malicious IPs. (IP reputation is a platinum feature).

Let’s take a closer look: There is an action, very well known to all of us (drop in this case) and there are two more actions: a Log Action and an AppFow Action. The appflow action would send data about this request to an IPFIX (NetFlow) receiver of choice, let’s say, Splunk or Solarwinds, or Citrix tools like Insight Center or MAS. Could be a great idea, but I don’t go that way. Instead I wanted this logs to appear at /var/log/ns.log

So we click to the + near to Log Action.

Before doing so we need to enable logging of “user configurable Log messages” in our syslog definition. So go to System → Auditing and click “Change Auditing Syslog Settings”. Same has to be done with every external syslog server (System → Auditing → Syslog, click Servers)

We now ´may create our own log actions. I’ll create a sample log:

This action will write an informational message into /var/log/ns.log. This message will look like that:

Mar  9 12:01:58 <local0.info> 172.31.1.31 03/09/2017:11:01:58 GMT ns01 0-PPE-0 : 
 default RESPONDER Message 50652 0 :  "A request for / from: 36.84.91.107 was dr
opped by res_pol_drop_malicious"

So, my responder policy logs! You see, I included the URL requested and the IP address of the attacker. You may create a string of your choice, using any policy expression you like.

Log destination will be /var/log/ns.log (optional /var/newnslog/ns.log)

Where do I find these log actions? From command-line it’s show audit messageaction. Creating my policy would be add audit messageaction log_drop_malicious INFORMATIONAL "\"A request for\" + HTTP.REQ.URL.URL_RESERVED_CHRRS_SAFE + \" from\ " + CLIENT.IP.SRC + \" was dropped by res_pol_drop_malicious\""

In GUI I can find all these policies in System → Audit → Message Actions.

---

I hope you find this article is beneficial. Have fun!

Johannes