last update: February 21^st 2022

Recently, I had been asked, how to protect a gateway from threads. It’s easy, I thought, Citrix ADC has everything needed in good quality: A Bot Management, Web Application Firewall (WAF), and AppQoE (Application quality of experience, a DOS protection feature). So nothing easier than that: Create the policies desired and bind them to the gateway.

Shortly after starting, I found out, it’s not possible to bind policies like that. They can only get bound to a load-balancing or content-switching vServer. (remark: WAF works with Citrix Gateway from version 14.1 build 8.50, thanks, Jochen Hoffmann, I forgot to update this article)

---

Binding the Policies

It takes a lot of tricks to bind these policies. Some dirty tricks, to be honest.

Have you ever tried to use the WAF wizard? No? That’s great. Never do so. You won’t get rid of the policy you created using the GUI. You won’t even find the bind point. It’s bound to a bind-point called None(?). GUI will not allow unbinding from there (13.0 built 82.42). Command-line helps to find, what’s been going on:

show ns runningConfig | grep appfw_pol_WAF-Wizzard
bind appfw global appfw_pol_WAF-Wizzard 62990 END -type REQ_DEFAULT

It’s bound globally, but the type is REQ_DEFAULT. So we can unbind it easily from the command-line:

unbind appfw global appfw_pol_WAF-Wizzard

We need to use this trick to proceed. Some of our policies need to be bound globally, with -type REQ_DEFAULT. This can only be done from the command line. And of course, the policy expression has to contain the hostname of the gateway (HTTP.REQ.HOSTNAME.EQ("gw.mycompany.com")), so these policies don’t interfere with other vServers hosted on this ADC. But that’s not true for the WAF. Unfortunately, the WAF is of rather limited help here, as it can only be used with logon policies, not with the gateway itself.

---

Bot Management

Maybe I’ll later create a more detailed blog about Bot Management. Bot Management deals with crawlers like Google bot, Bing bot, and several more. It allows you to select which of the well-known bots are desired and which ones are not. Have you ever tried to search Google for the term “Citrix Gateway”? You’ll find billions of them. Some branded, some not, but in most cases, the URL allows to guess, to whom it belongs. Even more: in the case, example.lab got its gateway indexed by Google, the term "citrix gateway" site:example.lab will give you the URL of all of example.lab’s gateways. In most cases, this is not desired (to say the least). So it would be a good idea to drop all connections from each and every crawler.

Citrix Bot Management comes with a more or less complete list of well-known bots. There are thousands of bots. And if we need to hide a site, we don’t like any of them to go there. So our first task would be: Create a new bot definition, denying access to all of them. Go to Bot Management -> signatures, check the checkbox for *Default Bot Signatures and click clone. This will create a JSON file containing all well-known bots. Before actually creating it, it will show the JSON file to you. I found out: The fastest way to disable all well-known bots is to copy this file into notepad++ and proceed there.
...
{
"version": "2.1",
"name": "AddThis.com",
"drop": "FALSE",
"redirect": "FALSE",
"reset": "FALSE",
"id": "2",
"type": "Good Bot",
"category": "Crawler",
"log": "TRUE",
"enabled": "TRUE",
"developer": "Oracle"
},eloper": "Mirafox"
...

This is just a small snippet of this file. You see, the AddThis.com bot is a “good one” (at least, that’s what Citrix thought, I won’t express my thoughts about it as they might institute proceedings against me), so connections would not drop. To make them drop, I have to change the term "reset": "FALSE" into "reset": "TRUE". I will copy/paste the content into Notepad++. Search and replace will set all “reset” to true within seconds for me. I’ll just have to paste the content back in, create a profile and a policy, and assign the policy to your gateway: All bots get reset, so you’ll never see your gateway on Google or any other search engine.

Usually, we don’t have just a gateway on the Citrix ADC (NetScaler), but also web servers. And we want crawlers to crawl most of them. However, we can’t bind the bot policy to the gateway, so we need to bind it globally, however, filter the policy based on hostname. That’s an example to filter the policy for gateway.example.lab:

HTTP.REQ.HOSTNAME.EQ("gateway.example.lab)

So that’s what my Bot-Profile would look like:

And that’s the policy

I am using the User-Agent switcher plugin for FireFox. If I change the user-agent to “Google-Bot”, I see a message, telling me, a secured connection can’t get established.

Bot protection could even do more, but it’s sufficient to block just well-known search engines and crawlers

If you are very concerned about bots, you could also block two tools: curl and wget. Curl is a windows tool to download websites, wget is a Linux tool with a similar purpose. Many tools simply use wget to fetch data. The respective user-agent strings would be:

• curl/7.54
• Wget/1.19.4 (linux-gnu)

(version numbers may vary of course)

So you would add a responder policy, action drop, with expression HTTP.REQ.HEADER("User-Agent").SET_TEXT_MODE(IGNORECASE).CONTAINS("curl") || HTTP.REQ.HEADER("User-Agent").SET_TEXT_MODE(IGNORECASE).CONTAINS("wget")

---

Web Application Firewall

The first question has to be: Do we trust authenticated users?

While this is usually the case with small to medium-sized businesses, it might not be true for a huge enterprise with thousands of users connecting: employees, contractors, and probably even some probably completely untrusted ones. Logically, we would just protect the login process, if we may trust authenticated users, while we would also protect everything behind in case we can’t.

---

Protecting the authentication process

Protecting an authentication process using the WAF is surprisingly difficult. The reason is, the WAF is processed only after the logon had been successful. So binding a WAF policy the way we bound the bot-protection policy, will not work as expected.

So we need a different trick. And the trick is binding an HTTP callout into the authentication policy. We will forward the original HTTP request to a callout vServer and bind a WAF policy to this server. This will not protect the AAA vServer, but the logon process gets secured.

This method has a very unpleasant side effect: it requires an HTTP load balancing vServer. This server must be active all the time. The authentication fails if this server is down. All requests are sent to this vServer and forwarded to the backend system, including username and password. The server must therefore not be of type HTTP , but must use TLS. Using the so-called “Always-On” service does not help, as the Always-On service can not respond in any way. Because of this, every callout that does not trigger the WAF policy times out and the policy returns an undefined.

I follow a Citrix white-paper, however, I did some changes, i.e. with the callout.

---

I don’t go into creating this vServer. I simply called it lb_vs_colors. I’m pretty sure, you can do this yourself. So my first step is:

Creating the HTTP callout.

add policy httpCallout callout_waf -vServer http_vs_callout -returnType BOOL -fullReqExpr "HTTP.REQ.FULL_HEADER + HTTP.REQ.BODY(2048)" -scheme https -resultExpr true

You see, we do an expression-based callout to this vServer. The expression we use for this callout is the full HTTP request (so all the HTTP headers plus the first 2k of the body of the HTTP request). Schema, I already mentioned it, has to be HTTPS for security reasons (don’t follow my screenshot, it’s highly insecure!). The return type will be a boolean, a true. So, it returns true, if the callout does not get blocked by the waf policy.

If the WAF is hit, the callout will not return a true, instead, a non-boolean return type. Because of this, the policy expression neither evaluates to true, nor to false but to undefined. If the policy expression evaluates to undefined, the policy won’t get invoked.

If we use this trick with an authentication policy, authentication is only possible, if the WAF policy does not block the request. And that’s what a user will see: “No active policy during authentication”. The user’s logon will fail.

The authentication policy

For my test environment, I just used a simple LDAP policy. Creating LDAP profiles is simple, so I’ll just show the policy expression:

add authentication Policy auth_ldap -rule "SYS.HTTP_CALLOUT(callout_waf) " -action act_ldap

The original Citrix policy also evaluates the HTTP.REQ.URL, but it does not make sense to me, as the LDAP policy only gets invoked if someone posted to the login process.

---

If you can’t trust your users

It’s not enough to protect the login process if you are not allowed to trust authenticated users. You’ll have to protect the gateway from attacks occurring post login. You’ll have to follow my guidelines for bot detection, to bind a WAF policy to a gateway. You may evaluate the logon cookies to distinguish between authenticated and non-authenticated users. The policy expression to select authenticated users would look like that:

HTTP.REQ.HOSTNAME.EQ("gateway.example.lab") && HTTP.REQ.COOKIE.CONTAINS("NSC_AAAC").NOT

---

Which security checks would you use?

The profile has to be of type HTML and XML. HTML is used for browser-based login, XML for apps like Citrix Workspace App.

Protecting the logon point is not that hard. I would define all the input fields (Allow all characters for the password field, but demand at least 6 and a maximum of 128 .{6,128} and only 2 to 128 word characters for the username \w{2,128}). Turn on learning and off blocking for all security checks possible. Don’t forget to set Buffer Overflow! Buffer overflow might be an attack vector. Be careful about SQL injection and Cross Site Scripting checks, they may cause false positives with the password field.

If you are scared of your authenticated users, the profile has to get way more complex.

Security checks I’d suggest to use as a minimum:

• StartURL
• DenyURL
• Cookie protection (to protect the session cookie)
• Field Formats (to make sure, just allowed characters appear in username and password fields. Keep in mind: Your users may use foreign language keyboards, so passwords may contain characters you didn’t think of)
• Buffer overflow
• XML format (workspace app widely uses XML)
• XML deny of service

Never use cookie theft, as this one does not work with all browsers and TLS 1.3 (see limitations here)

---

DDOS Protection

Probably, AppQoE would be the most important one of all these features. However, we can’t bind it to a gateway. No matter, which tricks I tried, it didn’t work. I could also not bind it globally, so I had to use something different. That’s why I had to use Rate Limiting and responder policies. Responder policies can get bound to a gateway directly, so no tricks are needed.d.

The first thing you have to do: Select a Stream Selector. The stream selector defines, on what counter you build your DOS (denial of service) protection. The easiest, and not the worst one, is just a client IP. Too many requests from a client IP to a certain URL will get blocked. The “certain URL”, of course, would be the login process of the gateway. This method would block requests from a single IP, and is helpful, however, not alone. Most DOS attacks today are DDOS attacks, distributed denial of service attacks, so they originate from various IP addresses.

The Stream Selector

Stream selector to protect from DOS attacks

add stream selector Selector_DoS_RateLimit HTTP.REQ.URL CLIENT.IP.SRC
This stream selector is based both, on IP address and URL. So a certain IP attacking a certain URL will hit the identifier.

Stream selector to protect from DDOS attacks

We want to protect from requests originating from various IPs to a single URL. So the criteria would just be HTTP.REQ.URL, that’s the built-in selector Top_URL. Nothing to do here.

Stream Identifiers

Stream Identifier to protect from DOS attacks

While the selector just selects the object, we are interested in, the stream identifier defines, how many requests in which time we would tolerate to this object.

add ns limitIdentifier Limit_Identifier_DoS -threshold 10 -timeSlice 300000 -mode REQUEST_RATE -selectorName Selector_DoS_RateLimit
-threshold: The number of requests you would be able to tolerate. Given, a user may add 5 wrong passwords and will then be locked out for 5 minutes, I would suggest 10. 10 is the double of 5, this would be more than sufficient. We don’t need to be too strict, we just have to throttle down attacks.
-timeSlice: The time we keep this list in milliseconds. Keep in mind: If there is an attack, these Lists can get huge, so they are on the cost of memory. The longer you keep these lists, the more memory you waste on lists. I would suggest not keeping it longer than 5 minutes, that’s 300,000 ms.
-mode: REQUEST_RATE (number of requests) or CONNECTION (number of connections). HTML reuses connections, so “number of requests” is perfect.
-limitType: SMOOTH or BURSTY. With smooth, all requests have to be distributed evenly, so 10 / 5 minutes would mean, we permit one every 30 seconds. That’s not good, as users may be quicker. So I would suggest using BURSTY. Bursty is the default and does not need to be specified.
-maximumBandwidth: similar to “threshold”, used if you limit on bandwidth.
-traps: how many traps to be sent in “timeSlize”. There will be additional traps, if the attack does not stop and a new time slice started.

Stream identifier to protect from DDOS attacks

Values here are way more complicated to find. You have to take care to not block your end-of-quarter or Christmas business! So monitor carefully! I don’t go into details about the settings, as it’s similar to the DOS identifier described above.

add ns limitIdentifier Limit_Identifier_DDoS -threshold 500 -timeSlice 300000 -selectorName Top_URL in my example, I allow a total of 500 requests to a certain URL (for all users together) in 5 minutes. It’s not that easy to find out, how many requests we need. I created a script, that can get run from BASH on your Citrix ADC / NetScaler:

#!/usr/bin/bash
# ---------------------------------------------------------------------------
# (C) Johannes Norz January 2021 https://192.168.200.109
# tested on: Citrix ADC 13
# ---------------------------------------------------------------------------
# Functional description:
# outputs the highest value of a given limit identifier.
# the program will run until you press [ctrl][C]
# purpose: find maximum values for DOS protection and similar use-cases
# known issues: output messed up if limit identifier does not contain data
# ---------------------------------------------------------------------------
# setting the parameters for this script
#Name of the limit identifier
Limit=Limit_Identifier_DDoS;
#Frequency of logging (in seconds)
Frequency=10
#username (user needs read only privilegs at least)
username=nsroot
password=Password1
#timestamp for the log. %T is just the time in 24h format, %r is 12 hour, %Y is year, %m=month, %d is day, %H hours, %M minutes, %S seconds
dateFormat=%T
while :;
do echo -n $(date +"$dateFormat") \#:; # output current time.
nscli -U 127.0.0.1:$username:$password show ns limitsessions $Limit | grep Hits | awk -F 'Hits:' '{print $2}' | awk -F 'Action' '{print $1}' | tr -d ' ' | sort -n | tail -n 1
sleep $Frequency;
done
Attention: This script does not output the number of requests within the time silce, but the value increases (so it’s the summary of all requests since the last boot of the appliance)

The responder policy

The responder policies are simple. However, like all security-related policies, there should be Syslog entry, if it gets hit. So we need a logging action as well.

Logging

Logging action, DOS

add audit messageaction Act_Log_Resp_DoS WARNING "\"DOS detected from \" + CLIENT.IP.SRC + \" to URL \" + HTTP.REQ.URL + \" User Agent: \" + HTTP.REQ.HEADER(\"User-Agent\")"
These logs will contain both, the IP and the URL affected. I don’t take it that seriously, so I set the severity to WARNING.

Logging action, DDOS

add audit messageaction Act_Log_Resp_DDoS CRITICAL "\"DDOS detected: URL \" + HTTP.REQ.URL"
These logs will not contain IPs (they usually originate from zombie-PCs, so no need to know the IPs),  but just the URL. I take this one very seriously, so I set the severity to CRITICAL.

Allowing user-configurable logs:

These logging actions won’t appear in a log, except, you allow user-configurable log messages for the syslog server.

set audit syslogParams -userDefinedAuditlog YES
(this is for the local Syslog policy, you have to do the same to all Syslog servers you defined)

Policies

I would suggest simply dropping malicious requests silently. That’s why I don’t use sophisticated responder actions. Feel free to create thoughtful responder actions, if you don’t agree. A normal browser would re-send dropped requests several times, so the policy will get several hits.

DOS protection

add responder policy Pol_Resp_DoS "HTTP.REQ.COOKIE.CONTAINS(\"NSC_AAAC\").NOT && SYS.CHECK_LIMIT(\"Limit_Identifier_DoS\")" DROP -logAction Act_Log_Resp_DoS
This policy checks, if the user is not logged on (so he does not own a cookie called NSC_AAAC), and if he is not, it checks the limit.

DDOS protection

add responder policy Pol_Resp_DDoS "HTTP.REQ.COOKIE.CONTAINS(\"NSC_AAAC\").NOT && SYS.CHECK_LIMIT(\"Limit_Identifier_DoS\")" DROP -logAction Act_Log_Resp_DoS
This policy is quite similar to the DOS protection policy, however, it uses a different limit identifier. See there.

---

As always, I’d be happy to read about your thoughts and concerns. Dust drop me a comment!