last update: February 10 2023
One of the most annoying issues in Citrix NetScaler is ICA / HDX connection issues. The reason for this is the way connection issues are reported.
There are two potential sources of trouble: Citrix StoreFront and Citrix NetScaler Gateway. So I will divide my blog into three sections: How to find the source of trouble, Troubleshooting Citrix StoreFront and Troubleshooting Citrix NetScaler Gateway.
How to find the source of trouble
It seems to be annoying and hardly possible. I am one of the moderators of a Facebook group about Citrix. Questions about connection issues come up quite often. Most of the answers don’t focus to the right source. They hardly ever ask: Which component is guilty? Instead, people give misleading tips (check STA is the most frequent one, probably, because people don’t understand the concept of STA). I want to stay away from misleading tips, instead, guide you through a well-structured troubleshooting guide.
Let’s try to understand what’s going on:
The stages of a Citrix NetScaler Gateway connection
I talk about using the Citrix StoreFront website, there is not so much difference to a ‘receiver for the web’ site. If you still use Citrix WebInterface: not much difference there, but my screenshots won’t be of any help.
- a user connects to the NetScaler Gateway website and is prompted with a login page
- the user enters his credentials. These credentials are checked against login providers like LDAP and RADIUS-based sources (Active Directory, RSA, Safe Word, SMS Token and many more).
- The user will see applications only after logging on successfully. So logon is done and without any issue as soon as we see applications!
We now know: NetScaler Gateway was able to authenticate the user, it also connected to Citrix StoreFront (or Web Interface) successfully and StoreFront was successfully connecting to XML broker service.
So no need to check here, it’s already checked: Logon works perfectly fine, the connection to StoreFront / Web Interface worked fine, and its connection to XML broker service is tested (we would not see any application if any of them fails)
- The user clicks an application. This click is proxied via NetScaler Gateway and StoreFront (WI) to XML broker service. XML broker service selects a resource, a desktop or an application, connects to this resource’s IP vis HTTP(s) (XenDesktop) or IMA (XenApp up to version 6.5), and stores this user’s credentials inside this machine. The machine returns a so-called NFuse ticket (NFuse is the old name of Citrix Web Interface). The IP address together with this NFuse ticket is returned to StoreFront (Web Interface).
- Getting an STA ticket: This is the first source of the problem I want to go into: We have to store the target’s IP address inside our secure environment. The store we use is called STA, and it’s usually one of the XenApp servers or XenDesktop DDCs (desktop delivery controller). The STA returns a so-called STA ticket.
- We now create an ICA file. The ICA file will contain the name of the NetScaler Gateway (FQDN), the NFuse ticket and the STA ticket (don’t mix these up!) together with some information about the screen resolution, clipboard mapping and so on. I attached a sample ICA file:
[ApplicationServers] Notepad= ... [Notepad] Address=;40;STA324731891;832A84599E0B7449B8578DCB8DBA95 this is STA ID and STA ticket AutologonAllowed=ON BrowserProtocol=HTTPonTCP CGPSecurityTicket=On ClearPassword=E16458A6937769 This is the 1st half of the NFuse ticket ... Domain=\C48CC641E8301B33 This is the 2nd half of the NFuse ticket ... InitialProgram=#Notepad ... Launcher=WI ... LogonTicket=E16458A6937769C48CC641E8301B33 this is the NFuse ticket LogonTicketType=CTXS1 ... SSLProxyHost=gateway.norz.at:443 The FQDN of the NetScaler Gateway used by Receiver ... TransportDriver=TCP/IP
- This ICA file is returned to the client via NetScaler Gateway. We don’t need to consider this connection to be guilty for our problems as it has already proved to work OK: it’s job is to send an ICA file, and we received it!
- The browser forwards this ICA-File to the Citrix receiver. (Begin of second part!) Citrix receiver will read the ICA file and …
- … connect to the NetScaler Gateway. We can see this, as we will see a progress bar.
- The receiver will send the STA ticket to the NetScaler Gateway. NetScaler Gateway will connect to the STA and try to resolve this ticket.
- NetScaler Gateway will try to connect to the target device (XenApp server, VDI devices), as soon as it could resolve the STA ticket and get a VDA’s IP in exchange.
- the application/desktop launches.
It’s essential to understand the connection process you want to troubleshoot!
So, where does it break into parts?
I have already mentioned: as soon as the ICA file is created and returned to the client the second part starts. How can we find out? Easy like that: The Citrix receiver (former names: ICA- client, ICA plugin, Citrix client, and approx 1.742.946 names more) is started, we successfully passed the first stage. So this is my first question: did it download the ICA file?
I did not download an ICA file
so let’s continue trouble shooting Storefront!
(don’t even thing of touching your NetScaler!)
I downloaded an ICA file
so let’s continue trouble shooting NetScaler gateway!
(no point in looking into StoreFront)
You are not sure if you received an ICA file or not?
- Firefox and Chrome: The ICA file goes into your download area, typically %username%\AppData\Local\Temp (or %tmp%). However, it usually gets deleted immediately after launching the receiver.
- InternetExplorer: There is a file created in %tmp%, but it is not accessible, the extension is not .ICA. And same as with other browsers, it usually gets deleted immediately.
As the ICA file usually gets deleted immediately you may use Microsoft’s Process Monitor to be 100% sure! You could also edit the ICA file in StoreFront (C:\inetpub\wwwroot\Citrix\Store\App_Data\default.ica). It is a windows INI file. You may change RemoveICAFile=yes to no in [WFClient] section, so it will stay forever (and spam the %tmp% directory).
More methods to find the stage of the connection process
Usually, you will see an error message. It’s stage 1 (StoreFront alone to blame for your issue) if this error message is displayed inside your browser, it’s stage 2 if it’s a windows (Mac, Linux, Windows, …) message box.
Troubleshooting Citrix StoreFront
If you got stuck within the first portion of the connection process, your issue is not directly related to NetScaler, you don’t even need to log on to NetScaler!
- Log on to your StoreFront server and check NetScaler Gateway settings:
- Your authentication methods have to contain Pass-Through from NetScaler Gateway (right-hand side, lower section, Manage Authentication Methods)
- You need to define a NetScaler Gateway (right-hand side, upper section, Manage NetScaler Gateways)
Don’t check authentication settings: Authentication worked fine, so there is nothing to do in here!
- Also, check the STAs. The STAs have to be resolvable! (same dialogue as above)
Use telnet (or putty) to connect to the desired port. So in my example, I would do a telnet XD7-DC.norz.local 80. The screen will turn black if it is able to connect. If I enter “something” I will see some HTML output. I won’t see anything if I connect to an HTTPS based server: telnet XD7-DC.norz.local 443 as I won’t be able to do an SSL handshake. If you mistyped the name of the STA, or the STA is not reachable you will see:
telnet XD7-DC.norz.lokal 443
connecting to XD7-DC.norz.lokal….
The connection attempt will time out. Always do these tests from your StoreFront servers!
Reasons for an STA not being reachable may be a miss-typed STA name or the (application) firewall blocking connections.
- Enable remote access! (right-hand side, lower section, Configure Remote Access Settings).
- There should not be the need to mention, as this is a very basic windows administration strategy, however, I see tons of people not being aware of it: Check the event log of your StoreFront servers!
Events and their meanings
If something goes wrong in StoreFront you usually see this message:
you will know: We never downloaded an ICA file. We are in trouble with StoreFront. Never check Citrix NetScaler Gateway, it was not involved, check events in StoreFront server. It may be hard to locate an event if you load balance your StoreFront servers, so I tend to disable all services but one.
Events pointing to STA problems:
The events can be found, both in administrative events or in “Application and Service Logs” -> “Citrix Delivery Services”
There will be a set of events: Citrix Store Service, Error 0, Citrix Store Service, Error 1003, Citrix Store Service, Warning 28.
Store Service Error 0: The server name <your server’s name> cannot be resolved. The specified Secure Ticket Authority could not be contacted and has been temporarily removed from the list of active services.
I think, the meaning of this event is more than clear: Citrix StoreFront could not connect to at least one of the STA servers you specified. There might be a chance to connect if there is more than a single STA server. Anyway: You should fix this problem!
Citrix Store Service, Error 1003. All the configured Secure Ticket Authorities failed to respond to this XML transaction: https://<yor server mane>/scripts/ctxsta.dll.
This event will always follow one or more Citrix Store Service, Error 0 events. This is a serious event, it means: It’s absolutely impossible to launch an application or desktop: There is no STA server available. Citrix Store Service, Error 1003 has to be fixed, it’s the reason for your connection problems! No way: You have to fix this problem!
Citrix Store Service, Warning 28: Failed to launch the resource ‘Local.<your application/desktop name>’, unable to obtain a ticket from the configured Secure Ticket Authorities.
This is the final result. We could not launch the application. It’s just a summary, fix Citrix Store Service Error 0 above and you’ll get rid of the 1003 and this one at the same time!
Troubleshooting Citrix NetScaler Gateway
Our problem is related to NetScaler Gateway if we successfully mastered part 1. So let’s troubleshoot problems here.
Before we see an error like this we will see the progress bar indicating: Citrix Receiver received an STA file. This progress bar is of some interest! Unfortunately, this message may disappear way too fast, so you will probably just see the message above.
That’s absolutely thrilling information for all of you! Click on “more information” and you’ll see where we got stuck!
So this picture shows the receiver establishing a connection to Citrix NetScaler Gateway. To be 100% clear: we still are not connected! We are just establishing a connection to NetScaler Gateway, so a TCP Sync packet is sent, but the TCP/IP connection is either still not established, or the SSL connection is not established yet!
Reasons for connections failing during this stage:
There may be several reasons for connections failing during this stage:
- the name of the gateway can’t get resolved. Check the name in StoreFront.
StoreFront: set a NetScaler Gateway
- The Citrix NetScaler Gateway server certificate is not trusted, or the certificate chain is broken. So as the first step: download NetScaler Gateway’s certificate and open it at your workstation (not in a browser, just from OS). Resolve all problems with this certificate. Don’t even think of continuing without solving this problem, it doesn’t make any sense at all.
- If you miss the intermediate CA certificate you have to download it and import it into NetScaler and link it.
NetScaler 11.1: Go to Traffic Management → SSL → CA Certificates. Import the certificate. Next, go to Traffic Management → SSL → Server Certificates. Click the NetScaler Gateway server certificate. Than Action and Link. It should display the certificate of the intermediate CA. Click OK.
So we successfully connected to Citrix NetScaler Gateway. Connection in progress disappeared. The current state is connected: There is an SSL connection from Client to NetScaler Gateway.
During the next stage, the Citrix receiver will send the STA ticket to NetScaler Gateway, and it will try to resolve the STA ticket. To do so it has to connect the configured STA.
STAs don’t replicate (actually they don’t even know about each other), so we need to specify exactly the same STA to NetScaler Gateway as we did in StoreFront. We will have to check StoreFront for STAs (see here). We then will check Citrix NetScaler Gateway for STA settings.
We navigate to NetScaler Gateway → Global Settings:
As you see: the bound STA appears to be down. There are 3 reasons for this:
- the name is wrong, or can’t get resolved. I would put the name into the clipboard and then navigate to System → Diagnostics and start the ping utility. Paste the hostname into the clipboard and see if it is ping-able. You will see, at least, if the hostname is resolvable
- the hostname is not resolvable. So the DNS server you configured for your NetScaler gateway is unable to resolve the hostname. In both cases the result of this ping will look like that:
- a firewall is blocking the STA communication.
After resolving all of these issues the STA settings in NetScaler Gateway should look like this:
You will notice the STA IDs, indicating NetScaler Gateway could connect to this STA at least once, and the green light (it may be missing with some elder versions of NetScaler) indicates actual connections.
No more problems about NetScaler Gateway and StoreFront as soon as you are fine until here!
It takes too much time to establish connections from outside, compared to inside?
Don’t blame NetScaler for this:
So NetScaler knows where to connect. NetScaler will use TCP/2598 for this connection: CGP (Citrix Gateway Protocol, former name: Common Gateway Protocol). At least as long as you did not turn off session reliability. I bet my life, you did not. NetScaler Gateway will try to connect via CGP for 30 seconds then give up and try plain HDX (formerly known as ICA) on TCP/1494. So open up TCP/2598 on your firewall, it will safe you 30 valuable seconds!
Do your connections still fail?
Let’s keep thinking: we successfully connected to NetScaler Gateway. We successfully resolved the ticket, so NetScaler Gateway now connects to the target device: a Citrix XenApp server or a Citrix XenDesktop VDI device.
So there are 2 reasons for this issue:
- a firewall blocks the connection
- NetScaler Gateway does not know a route to this IP
Just resolve these issues by opening up the firewall ports, or add the route to the desired network.
I hope this helped! Feel free to ask if you see additional problems not covered in here, I’ll answer your question and add the solution here.
Unfortunately, I was unable to capture screenshots from Citrix Receiver connection stages due to my (relatively) fast environment. I’d be glad to get your screenshots 😉
>Chrome: I don’t know where chrome puts these files too! (tell me if you know)
I use up-to-date-Chrome on up-to-date Win10. And I connect to StoreFront. My ICA files go into %TEMP%. I manually delete them from time to time. Sample file name : “ica5048260177296.ica”
Thanks, Hendrik, so they all do more or less the same. Yes, some times they don’t get deleted nowadays. I guess it’s a bug in Citrix Receiver. The template ICA file is in C:\inetpub\wwwroot\Citrix\Store\App_Data\default.ica. It is a windows .ini file and contains (per default) a RemoveICAFile=yes in [WFClient] section, so it should definitely get deleted :-).
I get through all the stages until I click on an application and then I get a message that tells me it took too long to connect? and asks me to check Proxy settings and firewall settings.
The connection worked perfectly a few days prior, so why have things changed?
And what can I do about it. I”m an Australian working in London.
looks like this client can’t connect to your NG. There are several reasons. Did you update your certificate to SHA2? if so you will have to upgrade your receiver.
We are having an issue similar to this in our environment, except it is when pulling an active internal session to external. It seems to work intermittently. Applications can be successfully launched by clicking the icon but this opens a seperate connection to another server.
This may happen if you did not add a call back URL in StoreFront. Check if you added a callback URL. An other potential reason would be XML Service Trust
Thanks for the quick response! Call back url is present and has been working. We have had this issue for a long time. Citrix recommended we enable ica proxy migration on gateway vip but this has made no difference. I have tested in both basic and smart access mode. It”s strange that it works randomly, i can see reciever start to load the active apps but it disconnects quickly or hangs when negotiating.
I am not sure what the xml service trust is. Where can i find this setting? I appreciate the help 🙂
Hm, Andrew, seems to be not that easy 🙂
I don”t think it”s a matter of trust, (https://docs.citrix.com/en-us/xenapp-and-xendesktop/xenapp-6-5/xenapp65-w2k8-wrapper/xenapp65-admin-wrapper/ps-maintain-wrapper-v3-tsk/ps-sf-citrix-xml-service-port-set-v2-tsk.html) any more.
The session is detected (and therefore disconnected from current Client, if it was active before). Next it should get reconnected, it gets reconnected, but it fails during the “negotiating catabilities” state? Is this true?
To be honest, I have no idea. I would do a Network trace, just to see what”s going on. But I”m pretty sure I would not find out.
[…] Trouble shooting Citrix NetScaler Gateway connection issues. – Jan 6, 2017. One of the most annoying issues in Citrix NetScaler are ICA / HDX connection issues. connection to StoreFront / Web Interface worked fine, and it's connection to XML. SSLProxyHost=gateway.norz.at:443 The FQDN of the NetScaler Gateway. Citrix Store Service, Error 1003: No STA server available! […]
Intermittently we are getting the below error. Is it NetScaler/StoreFront or Receiver that we should be looking at?
The published resource is not available currently. Please contact your system administrator for further assistance
Eventlogs on delivery controller and in StoreFront. It’s not related to NetScaler. This message came from StoreFront.
Thank you I had Netscaler installed for me and had some growing pains. Your troubleshooting method really helped me put it all together.
Thank you for sharing.
Great to hear, great you liekd it!
General Error: Too many HTTP redirects
Have you seen this error before? If so, how do I troubleshoot this message?
This happens, if there is a redirect (HTTP 301, 302 or 303), triggering an other redirect. Usually this is due to wrong policy expressions.
I usually press F12 on my browser, this will show me these redirects. Next I go to NetScaler and look into policies. The policy with most redirects is guilty 🙂
Hi Johannes, We have been plagued constantly in our environment where users are getting cannot start app message when using citrix receiver to connect to their virtual machine. We have to reboot the virtual in order for them to access it successfully again. This continues to happen regularly. when I look in the system logs on the virtual machine there are a ton of TDICA event id’s 1003 and 1003-The Citrix ICA Transport Driver is no longer waiting for connectionas on port 1494. it checks on several ports. specifiaclly there are exactly 3ea event id’s 1002 followed by 12ea event id 1003 evrytime I try to connect to the virtual and then it will fail with “cannot start app. It doesn’t seem to matter which citrix receiver client we use either. Can you Help?
Kelly, it’s a bit hard to answer. There may be many reasons for this behaivour. But there is one thing I see from here: You should open TCP/UDP 2598 from DMZ to your XA/XP environment, as receiver usually tunnels ICA inside of CGP (Citrix Gateway Protocol), as long as you don’t disable session reliability. But this is not the main reason, this may just slow down things by 30 seconds.
About the events: You saw these events on the VDA? Unfortunately I’m not that good about VDA. It seems not to be a gateway issue?
we have a regular problem with our netscaler. Everything just works fine, But then the login page wont be reachable and the connected sessions lose their connection. From the cli of the netscaler i cant ping the XD broker, until i restart the netscaler VM.
Do you have any ideas why this can happen? We have also configured Citrix Mobile on the same netscaler. This connection is working fine, even if the Xen Desktop gateway is not reachable.
Karsten, I would need some more information (i.E., firmware version). There are SERIOUS bugs in 13 built 36.27. I have seen several issues like that with this built. Even a down-date to 12.1 won’t bring the appliance to a stable state. There is a new version, 13 built 41:20. Unfortunately I have no idea if it fixes these issues. See here: https://blog.norz.at/citrix-adc-13-crash-dumps-filling-up-var-directory/
wow… That’s are very interesting and amazing posts, I just loved it,
thank you for sharing these posts your posts is Very impressive and motivational!
I get this error when trying to integrate storefront with netscaler. “storefront could not be contacted. the status of the server is down”.
How can i resolve this?
I would need to know some things. Did you load balance StoreFront? If so, are the services up?
If not, create a service pointing to your StoreFront server. Is it up? No? Check, the reason phrase. I’m pretty sure, a firewall is blocking (“Failure: Time out during TCP connection establishment stage”). It’s up? So connection is possible, it has to be something else.
Thank you for your post. I have gone through the steps but I am unable to find where we are having issues. Our system works fine with some ISPs but some others (like using a Verizon Hotspot on a phone) will not go past the connection in progress step. It will just eventually time out. When we first started having this issue we were upgrading to a new XenApp 7 farm. Connections were working fine on applications that were on the 6.5 farm but the applications on the 7 farm would not work correctly. It is a very strange issue and I can’t yet find a clue where to start looking.
Another Nice Post,
I completed enumeration, resolve and launched process…
My issue is.. that.. my applications and desktop did launched and error out “unknown client error 1110” following a warning “The citrix SSL server you have selected is not accepting Connections.
My launch.ica contain the following setting
i still unable to get pass this “unknown client error 1110”. Any thought??
Let me see. So you received a .ICA file. You are in the 2nd stage. This error 1110 is a rather generic one (for a good reason, we don’t want to give information to hackers).
It may come from SSL:
a) your certificate is expired
b) you didn’t link the intermediate certificate
c) Citrix Workspace App doesn’t trust this certificate (it uses Windows’s certificate store, not the one of your browser)
It may be an issue with the STA:
a) Shows your STA Up in Citrix NetScaler? (open the gateway, scroll down to Published Applications, and check the STA)
b) do you use the same STA as StoreFront does?
c) is StoreFront set up correctly (I miss a line like Address=;40;STA528037612;540868FE462B5E36AA6D510F45BA2A in your ICA file)
a) could potentially block traffic from SNIP to the VDA
b) block traffic from SNIP to STA (desktop controller)
See here: https://support.citrix.com/article/CTX200890/error-failed-with-status-1110-when-launching-desktops-or-apps-through-netscaler-gateway