Skip to content

slow initial connection #1200

Open
Open

Description

@oliver-sanders

Updates

  • Issue where the websocket connection is closed before it is succseefully opened.
  • Observed in Firefox, Edge and Chrome so far with the following messages:
    • WebSocket connection to '' failed: WebSocket is closed before the connection is established.
    • Firefox can’t establish a connection to the server at ws://.
  • Observed on Linux and Windows.
  • The websocket gets closed with error code 1006 and a blank reason. This is the same result as killing the UIS.
  • There is a related timeout issue which can present similarly, add debugging into the subscriptions-transport-ws code to tell the difference.
  • These failed requests do not reach the Tornado websocket implementation.

OP

Sometimes it takes a long time for the UI to connect with the backend.

It seems to get there in the end but it often takes in excess of 45 seconds to establish the subscription connection. The issue occurs randomly when the page is loaded and causes the no-connection error box hangs around.

Whilst we are waiting for the connection this message appears multiple times (Firefox):

 Firefox can’t establish a connection to the server at ws://localhost:8888/cylc/subscriptions.

Which suggests multiple failed attempts at connection, however, after jamming some logging into the UIS (backend server) I can see no evidence of connection attempts (at least none which actually get into the websocket code). The UIS responds quickly once it receives the request.

It's not clear what's causing this, however, it is worth noting that we are using a deprecated framework for subscriptions, we should try moving to something more modern and see if the problem goes away by itself - #1028.

Activity

  1. added this to the 1.5.0 milestone on Jan 10, 2023
  2. modified the milestones: 1.5.0, 1.6.0 on Jan 25, 2023
  3. modified the milestones: 1.6.0, 2.0.0 on Apr 24, 2023
  4. MetRonnie commented on Apr 28, 2023

    @MetRonnie
    Member

    I don't think I have ever encountered this on Chrome

  5. oliver-sanders commented on May 2, 2023

    @oliver-sanders
    MemberAuthor

    Try FF/VDI, the lag can be up to one minute.

  6. oliver-sanders commented on May 11, 2023

    @oliver-sanders
    MemberAuthor

    I've replicated this with offline mode on the Vue 3 branch :(

  7. MetRonnie commented on May 11, 2023

    @MetRonnie
    Member

    Assuming this is just an issue on VDI, what could be the cause?

  8. oliver-sanders commented on May 11, 2023

    @oliver-sanders
    MemberAuthor

    I'm assuming this is a general issue exacerbated by a laggy platform, you can see multiple connection attempts in the console.

  9. hjoliver commented on May 16, 2023

    @hjoliver
    Member

    (What's VDI - virtual desktop?)

  10. pinned this issue on Jun 13, 2023
  11. unpinned this issue on Jun 14, 2023
  12. 21 remaining items

  13. oliver-sanders commented on Nov 20, 2023

    @oliver-sanders
    MemberAuthor

    From manual inspection, the websocket error code is 1006 which is:

    Reserved for situations where the connection was closed abnormally without sending a close frame when a status code was expected.

    This appears to mean that the connection was closed unexpectedly (no close message), but I'm not sure how it's determining that, is ping-pong established this early on.

    Note, the reason field is blank. This exactly matches the response you get when you stop a running server.

  14. modified the milestones: 2.4.0, 2.5.0 on Mar 27, 2024
  15. kinow commented on Mar 30, 2024

    @kinow
    Member

    @oliver-sanders ' in this issue show what I can now easily reproduce locally 🎉

    I was reading an article on JS and performance for Autosubmit UI, when I noticed the author commented something interesting about browser settings that control the maximum number of connections to a server. The author also mentioned about this being a problem to WebSockets, which is why some people used a service workers to have a single client connecting to the server.

    Anyhoo, I opened Firefox, then about:config, and searched for websocket. It returned only 12, with a couple of entries for devtools (these can be discarded).

    Everything was working fine in the UI, and network.websocket.max-connections was set to 200 by default. Then I opened a few tabs, reduced it to 30, opened a few more tabs with the Cylc GUI, and voilà

    image

    I started closing my tabs and... eventually it connected.

    image

    I can't spend much more time on this issue, but I think

    • this is a plausible explanation, as that reflects the scenario I had when the issue happened.... I was working on several tabs in a browser,
    • the bug happened with complex and simple experiments
    • the bug would go away in some seconds, or if I closed other tabs or the browser and opened it again

    Open questions (that I wish I had time to keep digging 🤓)

    • I tried monitoring the connections with lsof and netstat but I got different numbers than 30. I was expecting to find a way to filter by process + server (socket, domain name, etc.).
    • When a WebSocket is closed, does it mean the browser connection (or its object in some pool?) is discarded? When the maxCount variable is decreased?
    • Any way to monitor that number and find a better error message, maybe somewhere in the stack trace it says "Reached Maximum Connections" or something?
    • There are other settings like delays for reconnect, timeouts, would any of that help?
    • Is there anything else on the server side (tornado/jupyter/graphql-ws that could be throttling the max connections as well? i.e. firefox/browser causes the issue, but maybe other parts of the system do that too?, or even linux 🤷‍♂️ )
    • I only know how to manage websockets connections in Ffox, but there should be a similar setting in Chrome/Chromium/Edge/Safari/Opera/etc.

    So if anyone is experiencing this a lot, and if you have a lot of tabs, go to about:config and increase that number (assuming you consider it safe as the server & client memory would be affected).

    Possible solution:

    • Use a SharedWorker (and this SO) in the UI to reuse the WebSockets connection
      • Does it work with current Cylc arch? (I'm using Cylc for small experiments but not following the dev closely)
      • Any security issues when two tabs in the same browser have different users logged in (e.g. Firefox container tabs)?
    • Use some other type of Service Worker?
    • Ask users to manage the number of max connections, especially if they monitor multi experiments in multi tabs, maybe update the UI error message too...
    • Replace WebSocket by something else? (LongPolling, WebRTC, something from HTTP/3? 🤷‍♂️ )

    I don't have this issue as we are not using WebSockets in the Autosubmit UI, so this is as far as my curiosity took me 🙂 , hope that helps.

    Bruno

  16. modified the milestones: 2.5.0, 2.6.0 on Apr 24, 2024
  17. ColemanTom commented on Aug 5, 2024

    @ColemanTom

    Just commenting (and following) that I see this a lot too. But, I haven't seen this error message mentioned, so I'll add it (it always happens after the "can't establish a connection to the server" line.

    The connection to wss://..../cylc/subscriptions was interrupted while the page was loading.

  18. ColemanTom commented on Aug 5, 2024

    @ColemanTom

    So if anyone is experiencing this a lot, and if you have a lot of tabs, go to about:config and increase that number (assuming you consider it safe as the server & client memory would be affected).

    It isn't just if you have many tabs. I have a single tab open and this happens. The page eventually loads, but it is slow.

  19. modified the milestones: 2.6.0, 2.7.0 on Oct 9, 2024
  20. modified the milestones: 2.7.0, 2.8.0 on Dec 3, 2024
  21. modified the milestones: 2.8.0, 2.x on Feb 17, 2025
  22. oliver-sanders commented on Mar 20, 2026

    @oliver-sanders
    MemberAuthor

    The precise problem this issue was created for (slow websocket connection as opposed to slow page loading) was a major problem at one time (and extremely difficult to debug), however, seemed to settle down and doesn't appear to be causing issues any more (although slow page load via other mechanisms, e.g. slow scan may still be present).

    As a result, I've dropped this from the high priority list. Will continue to monitor for now...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions