Trouble shooting Citrix NetScaler Gateway ICA/HDX connection issues

last update: February 10 2023

One of the most annoying issues in Citrix NetScaler is ICA / HDX connection issues. The reason for this is the way connection issues are reported.

There are two potential sources of trouble: Citrix StoreFront and Citrix NetScaler Gateway. So I will divide my blog into three sections: How to find the source of trouble, Troubleshooting Citrix StoreFront and Troubleshooting Citrix NetScaler Gateway.

How to find the source of trouble

It seems to be annoying and hardly possible. I am one of the moderators of a Facebook group about Citrix. Questions about connection issues come up quite often. Most of the answers don’t focus to the right source. They hardly ever ask: Which component is guilty? Instead, people give misleading tips (check STA is the most frequent one, probably, because people don’t understand the concept of STA). I want to stay away from misleading tips, instead, guide you through a well-structured troubleshooting guide.

Let’s try to understand what’s going on:

The stages of a Citrix NetScaler Gateway connection

I talk about using the Citrix StoreFront website, there is not so much difference to a ‘receiver for the web’ site. If you still use Citrix WebInterface: not much difference there, but my screenshots won’t be of any help.

1. a user connects to the NetScaler Gateway website and is prompted with a login page
2. the user enters his credentials. These credentials are checked against login providers like LDAP and RADIUS-based sources (Active Directory, RSA, Safe Word, SMS Token and many more).
3. The user will see applications only after logging on successfully. So logon is done and without any issue as soon as we see applications!
   We now know: NetScaler Gateway was able to authenticate the user, it also connected to Citrix StoreFront (or Web Interface) successfully and StoreFront was successfully connecting to XML broker service.
   So no need to check here, it’s already checked: Logon works perfectly fine, the connection to StoreFront / Web Interface worked fine, and its connection to XML broker service is tested (we would not see any application if any of them fails)
4. The user clicks an application. This click is proxied via NetScaler Gateway and StoreFront (WI) to XML broker service. XML broker service selects a resource, a desktop or an application, connects to this resource’s IP vis HTTP(s) (XenDesktop) or IMA (XenApp up to version 6.5), and stores this user’s credentials inside this machine. The machine returns a so-called NFuse ticket (NFuse is the old name of Citrix Web Interface). The IP address together with this NFuse ticket is returned to StoreFront (Web Interface).
5. Getting an STA ticket: This is the first source of the problem I want to go into: We have to store the target’s IP address inside our secure environment. The store we use is called STA, and it’s usually one of the XenApp servers or XenDesktop DDCs (desktop delivery controller). The STA returns a so-called STA ticket.
6. We now create an ICA file. The ICA file will contain the name of the NetScaler Gateway (FQDN), the NFuse ticket and the STA ticket (don’t mix these up!) together with some information about the screen resolution, clipboard mapping and so on. I attached a sample ICA file:

   [ApplicationServers]
   Notepad=
   ...
   [Notepad]
   Address=;40;STA324731891;832A84599E0B7449B8578DCB8DBA95  this is STA ID and STA ticket
   AutologonAllowed=ON
   BrowserProtocol=HTTPonTCP
   CGPSecurityTicket=On
   ClearPassword=E16458A6937769  This is the 1st half of the NFuse ticket
   ...
   Domain=\C48CC641E8301B33  This is the 2nd half of the NFuse ticket
   ...
   InitialProgram=#Notepad
   ...
   Launcher=WI
   ...
   LogonTicket=E16458A6937769C48CC641E8301B33  this is the NFuse ticket
   LogonTicketType=CTXS1
   ...
   SSLProxyHost=gateway.norz.at:443  The FQDN of the NetScaler Gateway used by Receiver
   ...
   TransportDriver=TCP/IP

7. This ICA file is returned to the client via NetScaler Gateway. We don’t need to consider this connection to be guilty for our problems as it has already proved to work OK: it’s job is to send an ICA file, and we received it!
8. The browser forwards this ICA-File to the Citrix receiver. (Begin of second part!) Citrix receiver will read the ICA file and …
9. … connect to the NetScaler Gateway. We can see this, as we will see a progress bar.
10. The receiver will send the STA ticket to the NetScaler Gateway. NetScaler Gateway will connect to the STA and try to resolve this ticket.
11. NetScaler Gateway will try to connect to the target device (XenApp server, VDI devices), as soon as it could resolve the STA ticket and get a VDA’s IP in exchange.
12. the application/desktop launches.

It’s essential to understand the connection process you want to troubleshoot!

So, where does it break into parts?

I have already mentioned: as soon as the ICA file is created and returned to the client the second part starts. How can we find out? Easy like that: The Citrix receiver (former names: ICA- client, ICA plugin, Citrix client, and approx 1.742.946 names more) is started, we successfully passed the first stage. So this is my first question: did it download the ICA file?

You are not sure if you received an ICA file or not?

• Firefox and Chrome: The ICA file goes into your download area, typically %username%\AppData\Local\Temp (or %tmp%). However, it usually gets deleted immediately after launching the receiver.
• InternetExplorer: There is a file created in %tmp%, but it is not accessible, the extension is not .ICA. And same as with other browsers, it usually gets deleted immediately.

As the ICA file usually gets deleted immediately you may use Microsoft’s Process Monitor to be 100% sure! You could also edit the ICA file in StoreFront (C:\inetpub\wwwroot\Citrix\Store\App_Data\default.ica). It is a windows INI file. You may change RemoveICAFile=yes to no in [WFClient] section, so it will stay forever (and spam the %tmp% directory).

More methods to find the stage of the connection process

Usually, you will see an error message. It’s stage 1 (StoreFront alone to blame for your issue) if this error message is displayed inside your browser, it’s stage 2 if it’s a windows (Mac, Linux, Windows, …) message box.

---

Troubleshooting Citrix StoreFront

If you got stuck within the first portion of the connection process, your issue is not directly related to NetScaler, you don’t even need to log on to NetScaler!

1. Log on to your StoreFront server and check NetScaler Gateway settings:
   • Your authentication methods have to contain Pass-Through from NetScaler Gateway (right-hand side, lower section, Manage Authentication Methods)

     

   • You need to define a NetScaler Gateway (right-hand side, upper section, Manage NetScaler Gateways)

     

     

     Don’t check authentication settings: Authentication worked fine, so there is nothing to do in here!

   • Also, check the STAs. The STAs have to be resolvable! (same dialogue as above)

     

     Use telnet (or putty) to connect to the desired port. So in my example, I would do a telnet XD7-DC.norz.local 80. The screen will turn black if it is able to connect. If I enter “something” I will see some HTML output. I won’t see anything if I connect to an HTTPS based server: telnet XD7-DC.norz.local 443 as I won’t be able to do an SSL handshake. If you mistyped the name of the STA, or the STA is not reachable you will see:
     telnet XD7-DC.norz.lokal 443
     connecting to XD7-DC.norz.lokal….
     The connection attempt will time out. Always do these tests from your StoreFront servers!
     Reasons for an STA not being reachable may be a miss-typed STA name or the (application) firewall blocking connections.

   • Enable remote access! (right-hand side, lower section, Configure Remote Access Settings).

     

2. There should not be the need to mention, as this is a very basic windows administration strategy, however, I see tons of people not being aware of it: Check the event log of your StoreFront servers!

Events and their meanings

If something goes wrong in StoreFront you usually see this message:

you will know: We never downloaded an ICA file. We are in trouble with StoreFront. Never check Citrix NetScaler Gateway, it was not involved, check events in StoreFront server. It may be hard to locate an event if you load balance your StoreFront servers, so I tend to disable all services but one.

Events pointing to STA problems:

The events can be found, both in administrative events or in “Application and Service Logs” -> “Citrix Delivery Services”

There will be a set of events: Citrix Store Service, Error 0, Citrix Store Service, Error 1003, Citrix Store Service, Warning 28.

Store Service Error 0: The server name <your server’s name> cannot be resolved. The specified Secure Ticket Authority could not be contacted and has been temporarily removed from the list of active services.

I think, the meaning of this event is more than clear: Citrix StoreFront could not connect to at least one of the STA servers you specified. There might be a chance to connect if there is more than a single STA server. Anyway: You should fix this problem!

Citrix Store Service, Error 1003. All the configured Secure Ticket Authorities failed to respond to this XML transaction: https://<yor server mane>/scripts/ctxsta.dll.

This event will always follow one or more Citrix Store Service, Error 0 events. This is a serious event, it means: It’s absolutely impossible to launch an application or desktop: There is no STA server available. Citrix Store Service, Error 1003 has to be fixed, it’s the reason for your connection problems! No way: You have to fix this problem!

Citrix Store Service, Warning 28: Failed to launch the resource ‘Local.<your application/desktop name>’, unable to obtain a ticket from the configured Secure Ticket Authorities.

This is the final result. We could not launch the application. It’s just a summary, fix Citrix Store Service Error 0 above and you’ll get rid of the 1003 and this one at the same time!

---

Troubleshooting Citrix NetScaler Gateway

Our problem is related to NetScaler Gateway if we successfully mastered part 1. So let’s troubleshoot problems here.

Before we see an error like this we will see the progress bar indicating: Citrix Receiver received an STA file. This progress bar is of some interest! Unfortunately, this message may disappear way too fast, so you will probably just see the message above.

That’s absolutely thrilling information for all of you! Click on “more information” and you’ll see where we got stuck!

So this picture shows the receiver establishing a connection to Citrix NetScaler Gateway. To be 100% clear: we still are not connected! We are just establishing a connection to NetScaler Gateway, so a TCP Sync packet is sent, but the TCP/IP connection is either still not established, or the SSL connection is not established yet!

Reasons for connections failing during this stage:

There may be several reasons for connections failing during this stage:

1. the name of the gateway can’t get resolved. Check the name in StoreFront.
   StoreFront: set a NetScaler Gateway

   

2. The Citrix NetScaler Gateway server certificate is not trusted, or the certificate chain is broken. So as the first step: download NetScaler Gateway’s certificate and open it at your workstation (not in a browser, just from OS). Resolve all problems with this certificate. Don’t even think of continuing without solving this problem, it doesn’t make any sense at all.
3. If you miss the intermediate CA certificate you have to download it and import it into NetScaler and link it.
   NetScaler 11.1: Go to Traffic Management → SSL → CA Certificates. Import the certificate. Next, go to Traffic Management → SSL → Server Certificates. Click the NetScaler Gateway server certificate. Than Action and Link. It should display the certificate of the intermediate CA. Click OK.

STA Tickets

So we successfully connected to Citrix NetScaler Gateway. Connection in progress disappeared. The current state is connected: There is an SSL connection from Client to NetScaler Gateway.

During the next stage, the Citrix receiver will send the STA ticket to NetScaler Gateway, and it will try to resolve the STA ticket. To do so it has to connect the configured STA.

STAs don’t replicate (actually they don’t even know about each other), so we need to specify exactly the same STA to NetScaler Gateway as we did in StoreFront. We will have to check StoreFront for STAs (see here). We then will check Citrix NetScaler Gateway for STA settings.

We navigate to NetScaler Gateway → Global Settings:

As you see: the bound STA appears to be down. There are 3 reasons for this:

1. the name is wrong, or can’t get resolved. I would put the name into the clipboard and then navigate to System → Diagnostics and start the ping utility. Paste the hostname into the clipboard and see if it is ping-able. You will see, at least, if the hostname is resolvable
2. the hostname is not resolvable. So the DNS server you configured for your NetScaler gateway is unable to resolve the hostname. In both cases the result of this ping will look like that:

   

3. a firewall is blocking the STA communication.

After resolving all of these issues the STA settings in NetScaler Gateway should look like this:

You will notice the STA IDs, indicating NetScaler Gateway could connect to this STA at least once, and the green light (it may be missing with some elder versions of NetScaler) indicates actual connections.

No more problems about NetScaler Gateway and StoreFront as soon as you are fine until here!

It takes too much time to establish connections from outside, compared to inside?

Don’t blame NetScaler for this:

So NetScaler knows where to connect. NetScaler will use TCP/2598 for this connection: CGP (Citrix Gateway Protocol, former name: Common Gateway Protocol). At least as long as you did not turn off session reliability. I bet my life, you did not. NetScaler Gateway will try to connect via CGP for 30 seconds then give up and try plain HDX (formerly known as ICA) on TCP/1494. So open up TCP/2598 on your firewall, it will safe you 30 valuable seconds!

Do your connections still fail?

Let’s keep thinking: we successfully connected to NetScaler Gateway. We successfully resolved the ticket, so NetScaler Gateway now connects to the target device: a Citrix XenApp server or a Citrix XenDesktop VDI device.

So there are 2 reasons for this issue:

1. a firewall blocks the connection
2. NetScaler Gateway does not know a route to this IP

Just resolve these issues by opening up the firewall ports, or add the route to the desired network.

---

I hope this helped! Feel free to ask if you see additional problems not covered in here, I’ll answer your question and add the solution here.

Unfortunately, I was unable to capture screenshots from Citrix Receiver connection stages due to my (relatively) fast environment. I’d be glad to get your screenshots 😉