You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Issue where the websocket connection is closed before it is succseefully opened.
Observed in Firefox, Edge and Chrome so far with the following messages:
WebSocket connection to '' failed: WebSocket is closed before the connection is established.
Firefox can’t establish a connection to the server at ws://.
Observed on Linux and Windows.
The websocket gets closed with error code 1006 and a blank reason. This is the same result as killing the UIS.
There is a related timeout issue which can present similarly, add debugging into the subscriptions-transport-ws code to tell the difference.
These failed requests do not reach the Tornado websocket implementation.
OP
Sometimes it takes a long time for the UI to connect with the backend.
It seems to get there in the end but it often takes in excess of 45 seconds to establish the subscription connection. The issue occurs randomly when the page is loaded and causes the no-connection error box hangs around.
Whilst we are waiting for the connection this message appears multiple times (Firefox):
Firefox can’t establish a connection to the server at ws://localhost:8888/cylc/subscriptions.
Which suggests multiple failed attempts at connection, however, after jamming some logging into the UIS (backend server) I can see no evidence of connection attempts (at least none which actually get into the websocket code). The UIS responds quickly once it receives the request.
It's not clear what's causing this, however, it is worth noting that we are using a deprecated framework for subscriptions, we should try moving to something more modern and see if the problem goes away by itself - #1028.
From manual inspection, the websocket error code is 1006 which is:
Reserved for situations where the connection was closed abnormally without sending a close frame when a status code was expected.
This appears to mean that the connection was closed unexpectedly (no close message), but I'm not sure how it's determining that, is ping-pong established this early on.
Note, the reason field is blank. This exactly matches the response you get when you stop a running server.
I was reading an article on JS and performance for Autosubmit UI, when I noticed the author commented something interesting about browser settings that control the maximum number of connections to a server. The author also mentioned about this being a problem to WebSockets, which is why some people used a service workers to have a single client connecting to the server.
Anyhoo, I opened Firefox, then about:config, and searched for websocket. It returned only 12, with a couple of entries for devtools (these can be discarded).
Everything was working fine in the UI, and network.websocket.max-connections was set to 200 by default. Then I opened a few tabs, reduced it to 30, opened a few more tabs with the Cylc GUI, and voilà
I started closing my tabs and... eventually it connected.
I can't spend much more time on this issue, but I think
this is a plausible explanation, as that reflects the scenario I had when the issue happened.... I was working on several tabs in a browser,
the bug happened with complex and simple experiments
the bug would go away in some seconds, or if I closed other tabs or the browser and opened it again
Open questions (that I wish I had time to keep digging 🤓)
I tried monitoring the connections with lsof and netstat but I got different numbers than 30. I was expecting to find a way to filter by process + server (socket, domain name, etc.).
When a WebSocket is closed, does it mean the browser connection (or its object in some pool?) is discarded? When the maxCount variable is decreased?
Any way to monitor that number and find a better error message, maybe somewhere in the stack trace it says "Reached Maximum Connections" or something?
There are other settings like delays for reconnect, timeouts, would any of that help?
Is there anything else on the server side (tornado/jupyter/graphql-ws that could be throttling the max connections as well? i.e. firefox/browser causes the issue, but maybe other parts of the system do that too?, or even linux 🤷♂️ )
I only know how to manage websockets connections in Ffox, but there should be a similar setting in Chrome/Chromium/Edge/Safari/Opera/etc.
So if anyone is experiencing this a lot, and if you have a lot of tabs, go to about:config and increase that number (assuming you consider it safe as the server & client memory would be affected).
Possible solution:
Use a SharedWorker (and this SO) in the UI to reuse the WebSockets connection
Does it work with current Cylc arch? (I'm using Cylc for small experiments but not following the dev closely)
Any security issues when two tabs in the same browser have different users logged in (e.g. Firefox container tabs)?
Use some other type of Service Worker?
Ask users to manage the number of max connections, especially if they monitor multi experiments in multi tabs, maybe update the UI error message too...
Replace WebSocket by something else? (LongPolling, WebRTC, something from HTTP/3? 🤷♂️ )
I don't have this issue as we are not using WebSockets in the Autosubmit UI, so this is as far as my curiosity took me 🙂 , hope that helps.
Just commenting (and following) that I see this a lot too. But, I haven't seen this error message mentioned, so I'll add it (it always happens after the "can't establish a connection to the server" line.
The connection to wss://..../cylc/subscriptions was interrupted while the page was loading.
So if anyone is experiencing this a lot, and if you have a lot of tabs, go to about:config and increase that number (assuming you consider it safe as the server & client memory would be affected).
It isn't just if you have many tabs. I have a single tab open and this happens. The page eventually loads, but it is slow.
The precise problem this issue was created for (slow websocket connection as opposed to slow page loading) was a major problem at one time (and extremely difficult to debug), however, seemed to settle down and doesn't appear to be causing issues any more (although slow page load via other mechanisms, e.g. slow scan may still be present).
As a result, I've dropped this from the high priority list. Will continue to monitor for now...
Activity
MetRonnie commentedon Apr 28, 2023
I don't think I have ever encountered this on Chrome
oliver-sanders commentedon May 2, 2023
Try FF/VDI, the lag can be up to one minute.
oliver-sanders commentedon May 11, 2023
I've replicated this with offline mode on the Vue 3 branch :(
MetRonnie commentedon May 11, 2023
Assuming this is just an issue on VDI, what could be the cause?
oliver-sanders commentedon May 11, 2023
I'm assuming this is a general issue exacerbated by a laggy platform, you can see multiple connection attempts in the console.
hjoliver commentedon May 16, 2023
(What's VDI - virtual desktop?)
21 remaining items
oliver-sanders commentedon Nov 20, 2023
From manual inspection, the websocket error code is 1006 which is:
This appears to mean that the connection was closed unexpectedly (no close message), but I'm not sure how it's determining that, is ping-pong established this early on.
Note, the reason field is blank. This exactly matches the response you get when you stop a running server.
kinow commentedon Mar 30, 2024
@oliver-sanders ' in this issue show what I can now easily reproduce locally 🎉
I was reading an article on JS and performance for Autosubmit UI, when I noticed the author commented something interesting about browser settings that control the maximum number of connections to a server. The author also mentioned about this being a problem to WebSockets, which is why some people used a service workers to have a single client connecting to the server.
Anyhoo, I opened Firefox, then
about:config, and searched forwebsocket. It returned only 12, with a couple of entries for devtools (these can be discarded).Everything was working fine in the UI, and
network.websocket.max-connectionswas set to200by default. Then I opened a few tabs, reduced it to30, opened a few more tabs with the Cylc GUI, and voilàI started closing my tabs and... eventually it connected.
I can't spend much more time on this issue, but I think
Open questions (that I wish I had time to keep digging 🤓)
lsofandnetstatbut I got different numbers than30. I was expecting to find a way to filter by process + server (socket, domain name, etc.).So if anyone is experiencing this a lot, and if you have a lot of tabs, go to about:config and increase that number (assuming you consider it safe as the server & client memory would be affected).
Possible solution:
SharedWorker(and this SO) in the UI to reuse the WebSockets connectionI don't have this issue as we are not using WebSockets in the Autosubmit UI, so this is as far as my curiosity took me 🙂 , hope that helps.
Bruno
ColemanTom commentedon Aug 5, 2024
Just commenting (and following) that I see this a lot too. But, I haven't seen this error message mentioned, so I'll add it (it always happens after the "can't establish a connection to the server" line.
ColemanTom commentedon Aug 5, 2024
It isn't just if you have many tabs. I have a single tab open and this happens. The page eventually loads, but it is slow.
oliver-sanders commentedon Mar 20, 2026
The precise problem this issue was created for (slow websocket connection as opposed to slow page loading) was a major problem at one time (and extremely difficult to debug), however, seemed to settle down and doesn't appear to be causing issues any more (although slow page load via other mechanisms, e.g. slow scan may still be present).
As a result, I've dropped this from the high priority list. Will continue to monitor for now...