Python 3.14 forkserver workers can use a stale GraphServer address

[KonradPilch]

What is the issue?

Python 3.14 changed the default multiprocessing start method on non-macOS POSIX systems from fork to forkserver. The fork server is a persistent process: workers created later inherit its environment rather than subsequent changes made in the original parent process.

This is documented here: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

ezmsg currently relies on backend processes resolving the GraphServer address from EZMSG_GRAPHSERVER_ADDR when no explicit graph_address was supplied. This can produce a stale address when all of the following occur:

1. A process starts the multiprocessing fork server while EZMSG_GRAPHSERVER_ADDR points to address A.
2. The parent later changes EZMSG_GRAPHSERVER_ADDR to address B.
3. The parent calls ez.run() without an explicit graph_address.
4. The parent attaches to B, but backend workers inherit A from the fork server.

The backend workers then connect to the wrong GraphServer. If the old server is no longer running, startup fails with ConnectionRefusedError. The remaining processes may wait indefinitely at their startup barrier instead of propagating the failure.

I have seen this on Linux (WSL2) with python 3.14. I have not seen this with earlier Linux python versions since there, the multiprocessing start method defaults to fork. On Windows and macOS it defaults to spawn - also no issues.

Steps to reproduce (minimal example)

import multiprocessing as mp
import os


def report_address():
    print(os.environ["EZMSG_GRAPHSERVER_ADDR"], flush=True)


if __name__ == "__main__":
    print(mp.get_start_method())  # forkserver on Linux/Python 3.14

    os.environ["EZMSG_GRAPHSERVER_ADDR"] = "127.0.0.1:10001"
    first = mp.Process(target=report_address)
    first.start()
    first.join()

    os.environ["EZMSG_GRAPHSERVER_ADDR"] = "127.0.0.1:10002"
    second = mp.Process(target=report_address)
    second.start()
    second.join()

On Linux/Python 3.14, both workers report 127.0.0.1:10001.

Observed ezmsg reproduction

A test run that first starts multiprocessing with one GraphServer address and then runs a multi-process graph with another address reproduces the issue:

First GraphServer:            127.0.0.1:38187
Current parent GraphServer:   127.0.0.1:35067
Backend connection attempt:   127.0.0.1:38187
ConnectionRefusedError

When working on fixing tests attaching to existing GraphServers in this PR, after the first two commits, the GitHub ubuntu-latest python 3.14 test-runner failed. It failed at the first test_shm_grow case then hung until GitHub Actions cancelled the job after six hours! This failure case is not limited to tests. It could affect notebooks, daemons, orchestration services, or other long-lived applications that reconfigure the GraphServer address after multiprocessing has already started.

Solution criteria

• Parent and backend processes must use the same resolved GraphServer address.
• Correctness must not depend on mutable environment variables propagating to existing fork-server processes.
• Default, environment-provided, and explicitly supplied addresses must continue to work.
• The solution should work with fork, spawn, and forkserver.
• Python 3.10 through 3.14 should remain supported.
• A regression test should verify that a concrete address is passed to backend processes when the parent attaches through the default address.
• Ideally, backend startup failures should propagate instead of leaving other processes waiting indefinitely.

Possible solution ideas

1. Resolve and pass the address explicitly

Resolve the effective address in the parent before creating backend processes:

address = graph_context.graph_address
if address is None:
    address = GraphService.default_address()

execution_context.create_processes(
    graph_address=address,
    backend_process=backend_process,
)

This is the preferred solution because workers no longer need to re-resolve the address from their environment.

2. Store the resolved address in GraphContext

When GraphContext attaches to an existing default GraphServer, update it with the resolved address. graph_context.graph_address would then always return a concrete address after initialization.

This may provide a stronger invariant, but it has a wider behavioral impact than resolving the address at process creation.

3. Use a fixed multiprocessing start method

Forcing spawn would avoid the persistent fork-server environment, but libraries generally should not override an application's multiprocessing context. It would also hide rather than remove the dependency on environment inheritance.

4. Improve startup failure handling

Add a timeout or failure-propagation mechanism around backend startup barriers. This would not fix the stale address, but it would turn connection failures into actionable errors instead of indefinite hangs.

[KonradPilch]

Solution 1 implemented in commit Fix Python 3.14 error