Yesterday I published a blog about DDOS- protection. I used the Citrix NetScaler AppQoE feature to do so. That’s nice, but not enough. I still could beat my server to a pulp easily. Just 10 clients launching a DDOS attack using HULK had been enough. I can’t throttle down the number of users to just 10!

WTF?

Basically AppQoE will just limit the number of users (actually it’s the number of source IPs) being able to connect, not the number of requests. No matter how deep you set the limit, even a single user may screw up a web server, given his connection is fast enough. No protection from NetScaler AppQoE here!

What to do next?

What’s wrong? We limit the number of users. That’s a damn good story as it is possible to do serious attacks with distributed users. But we did not limit the number of requests per user! How can we limit the number of requests per user? Or, the number of requests to a certain URL?

---

Think you are the boss of a police station. One nice and sunny morning you would tell one of your officers to check cars (that’s the Selector, he has to select cars). Unfortunately a single officer would not be able to check all cars, so you tell him to stop every 10th car. This would be the Idenitfyer. Both together give a good job description: “check every 10th car!”. But it takes an officer to do the job. That’s what our (responder) policy would do.

So we have to set up a stream selector, a stream identifier and a responder policy.

---

Rate Limiting is the feature or choice!

The very best thing: Rate limiting is standard edition, so everyone may use it. How could Citrix NetScaler Rate Limiting look like to protect from a DOS attack?

The NetScaler Rate Limiting Stream Selector

The first ting to do: select the kind of traffic we want to limit. There are two kinds of approach: the client IP and the URL; or a combination of both. There are built in stream selectors we could use: Top_URL (HTTP.REQ.URL) and Top_CLIENTS (CLIENT.IP.SRC). I did a trace of an attack done by HULK. HULK always appends some random strings like ?KCDPT=SRKGT. So Top_URL is not useful, instead we would need to create our own stream selector based on HTTP.REQ.URL.PATH to strip the query string.

I created a stream selector using both, URL and IP. This is a list of URLs per IP.

add stream selector rate_sel_URL_IP CLIENT.IP.SRC HTTP.REQ.URL.PATH

The stream selector “does not do anything” but selects streams. So our next step would be:

Create a NetScaler Rate Limiting Stream Idenitfyer

The stream identifier checks weather or not to pick a stream. There are several parameters:

• The Selector is the stream selector we created before.
• We have two Modes: Request Rate and Connection. Request Rate tracks requests (that’s what we need), Conection counts connections. That’s not useful in here, we want to block requests, not limit connections!
• The Limit Type may be bursty or smooth. Smooth wants requests to come in evenly, while bursty would allow much higher rates as long as the total number does not exceed the limit for the given time slice (so if you define a limit of 10 and a time slice of 10s there may be 10 requests within a second, but no more during the next nine seconds if you go with bursty and 1 per second for smooth). I go with bursty as this is how requests usually come in.
• The Threshold is what you have been looking for: The number of requests (or connections). Select them wisely! If you permit too many you may kill your server, if you don’t allow enough you will block legitimate requests of your users.
  
• The Time Slice (msec) is the amount of time we use to count our requests, in milliseconds. So 5,000 would be 5 seconds, 60,000 a minute.
• Maximum Bandwidth (Kbps) is an additional limit you may specify for connections. We may leave it blank as we are not interested in connections.
• Traps is the number of traps to be sent in a time slice. 0 means no traps, I guess 1 would be fine, even if your time slice is several minutes.

Usually a single user won’t request a certain URL more often than 10 to 100 times per 5 seconds (be aware of spacer GIFs, some web site programmers use them excessively! I have seen hundreds of requests within a second. And do you know what I think of the web site programmer? What a bloody idiot! But he’s just a web site programmer, no engineer, maybe that’s why).

add ns limitIdentifier rate_id_URL_IP -threshold 10 -timeSlice 5000 -selectorName rate_sel_URL_IP -maxBandwidth 0 -trapsInTimeSlice

---

The responder policy

So we could find out to malicious requests. But in the end we will need to react. Our stream selector, our stream identifier don’t respond to client requests on their own. That’s why we need to create a responder policy.

The responder action

What to do? There is no best practice. I tend to drop all malicious requests, but you also could answer with a 404, not found, a 500, server error, redirect users to a religious site of your choice (to force him to think about his sins), or with a custom error page. It’s up to you. I will simply drop.

The responder policy

add responder policy res_pol_drop_DOS "SYS.CHECK_LIMIT(\"rate_id_URL_IP\")" DROP

So you can see, my action is a DROP. The point of interest for you will be the expression: SYS.CHECK_LIMIT(“rate_id_URL_IP”)

I tried to do some logging. Logging helped with trouble shooting.

---

I hope you like it. Leave a comment if you find something you don’t understand (or don’t agree). I also accept comments telling me I did a good job