last update: December 10^th 2020

I recently came across a problem, that had been hard to resolve. Active/passive load-balancing typically is easy to do: You create a load-balancing vServer for the active service, and another one, intended to be passive, for desaster recovery. Then you set the disaster recovery vServer as a protection vServer for the active vServer. It will automatically switch to the desaster recovery server, if the active services go down. As soon as the active service comes back, the system will swichtz back and the system will be back to normal. In electronics, we call this a monostable flip/flop.

This administrator, however, wanted another solution: Their application does not like users to switch from vServers t the other, so users should stay on the other server after a fail-over occurred and should only failback in case the other service went down. This is what we call a bi-stable flip/flop.

---

The solution

is based on content switching. I need 3 non-addressable load-balancing vServers, a red, a green and a red-and_green vServer. I also need a content-switching vServer.

There will be a NetScaler variable, storing the state. Depending on the state of this variable, we will either use the red or the green vServer.

I used responder policies to trigger a failover. These responder policies are bound to the content-switching vServer. So switching is triggered, whenever a request is received. Unfortunately, the first request after a service-down will fail, as traffic still has to be directed to the failed vServer, however, it’s not there any more. I had to use a bit more complex content-switching policies.

The variable

Variables in Citrix ADC / NetScaler are a little bit different from “normal” variables. They have to be defined, and it takes assignments to assign values. Assignments can be used as responder actions. I named my variable var_cs_vserver. The type is ulong, stores numbers: 0 for red, 1 for green.

---

The setup

I use Wonderkitchen’s test environment, published to the internet, so you could simply use my NetScaler commands to set up your testing environment.

Loadbalancing requirement

Load-balancing is straightforward and does not need any comments (and you are wrong here if you don’t understand load-balancing)

The two services

add service sc_red 93.83.148.43 HTTP 80
add service sc_green 93.83.148.45 HTTP 80

The green vServer

add lb vserver lb_vs_green HTTP 0.0.0.0 0 -persistenceType NONE -cltTimeout 180
bind lb vserver lb_vs_red_green sc_green

The red vServer

add lb vserver lb_vs_green HTTP 0.0.0.0 0 -persistenceType NONE -cltTimeout 180
bind lb vserver lb_vs_red_green sc_red

Definition of the variable

So the first step about the variables would be, creating the variable. I assume you are not familiar with NetScaler variables.

add ns variable var_cs_vserver -type ulong -init 0
this adds a variable var_cs_vserver, type is ulong (ulong is numeric) and assigns an initial value of 0.

The two assignments

add ns assignment ass_set_green -variable "$var_cs_vserver" -set 1
add ns assignment ass_set_red -variable "$var_cs_vserver" -set 0

This will change the value of the variable. Assignments can be used as policy expressions. That’s what I plan to do.

Responder policies

add responder policy res_pol_set_red "SYS.SERVICE(\"sc_green\").STATE.EQ(UP).NOT" ass_set_red
add responder policy res_pol_set_green "SYS.SERVICE(\"sc_red\").STATE.EQ(UP).NOT" ass_set_green

The first one sets the variable to 0 to send traffic to the red vServer, in case the green service is down, the second one to 1, to send traffic to the green one in case the red service is down.

Content-switching

Content-switching is straightforward and does not need any comments.

The content switching actions

add cs action cs_act_red -targetLBVserver lb_sv_red
add cs action cs_act_green -targetLBVserver lb_vs_green

The content switching policies

add cs policy cs_pol_red -rule "$var_cs_vserver.EQ(0) && SYS.SERVICE(\"sc_red\").STATE.EQ(UP) || SYS.SERVICE(\"sc_green\").STATE.EQ(UP).NOT" -action cs_act_red

Basically, the expression $var_cs_vserver.EQ(0) would be sufficient. But in case of the red service just came down (that is our a fail-over condition), the variable would still be 0 (target: red vServer) for the very first request. So content-switching is not able to apply this policy, so the policy gets skipped. Content-switching will respond with a 500 internal server error.

So I check if the service is not up (so it is DOWN or OUT OF SERVICE). "$var_cs_vserver.EQ(0) && SYS.SERVICE(\"sc_red\").STATE.EQ(UP). In case, the flag is set to 0, but red service is down, the policy is false.

If the flag is set to 1, but the green service is down, we will check the policy, "$var_cs_vserver.EQ(0) && SYS.SERVICE(\"sc_red\").STATE.EQ(UP) returns false, but SYS.SERVICE(\"sc_green\").STATE.EQ(UP).NOT gets true, so we switch to red.

add cs policy cs_pol_green -rule "$var_cs_vserver.EQ(1) || SYS.SERVICE(\"sc_red\").STATE.EQ(UP).NOT" -action cs_act_green
This one is similar to the previous, but I skip the check if the green service is up. This policy is bound second, so the first one already checked the status of the green service.

I thank Jan Tytgat. He inspired me to this solution. My previous solution needed a red/green default server to switch flawlessly, however, the cs-policy just checked the status of the variable.

Binding all together

add cs vserver cs_vs_act-pass HTTP 192.168.229.34 80
bind cs vserver cs_vs_act-pass -policyName cs_pol_red -priority 16
bind cs vserver cs_vs_act-pass -policyName cs_pol_green -priority 32
bind cs vserver cs_vs_act-pass -lbvserver lb_vs_red_green

Triggering a fail-over

To trigger a failover, these policies have to get hit. Therefore, I also have to bind my responder policies to the content-switching vServer.

bind cs vserver cs_vs_act-pass -policyName res_pol_set_red -priority 16 -gotoPriorityExpression END -type REQUEST
bind cs vserver cs_vs_act-pass -policyName res_pol_set_green -priority 32 -gotoPriorityExpression END -type REQUEST  

So if a request comes in, the cs-vserver will check, if the red service is down, and if it is down, it will set the variable to 1. Similar, if the green service is down, the variable will be set to 0.

I use .UP.NOT, as there are three states: Up, Down and OUT OF SERVICE. Not up is either down or out of service.

---

I hope, this has been helpful for you. I would be happy to get a short message about your concerns and suggestions