Deliberately un-pin and upgrade dask/distributed (long-frozen at <2023.0)

[wshlavacek]

Background

pyproject.toml currently caps:

dask>=2021.5.0,<2023.0
distributed>=2021.6.2,<2023.0
tornado>=6.1,<7

We are effectively frozen on dask/distributed ~2022.12.1 — roughly three years behind upstream. This freeze just produced a real bug: #388, where custom_as_completed subclassed as_completed and reimplemented a dask internal method (track_future) by copying its body. dask later renamed that method to _track_future (the tornado→asyncio migration), the override silently became dead code, and errored futures leaked into the run loop as raw (type, exc, traceback) tuples — crashing fits with AttributeError: 'tuple' object has no attribute 'score'.

That specific coupling is now removed (PR for #388 + follow-up cleanup), but the frozen pin itself is unaddressed technical debt. The longer we stay pinned, the larger and riskier the eventual jump (Python-version compatibility, security fixes, and API drift all accumulate).

Why this is its own effort (not part of the #388 fix)

Un-pinning is a substantial, risky change touching the distributed-execution core:

• The tornado coupling (tornado<7) likely needs to go; modern dask is asyncio-native.
• as_completed semantics, Client/Future behavior, scatter/gather, and serialization have all evolved across ~10 releases.
• cluster.py (SSH/paramiko cluster bring-up) and the scheduler/worker lifecycle need revalidation against a current distributed.
• Needs validation across the cluster/HPC execution paths, not just local runs.

Scope / acceptance criteria

• Audit for any remaining reliance on dask/distributed (or tornado) internals — private modules, underscored methods/attrs, copied method bodies. (Scatter search crashes with "'tuple' object has no attribute 'score'" when a fit evaluation fails #388 cleanup removed the known one.)
• Bump dask/distributed to a current release; drop or update the tornado<7 pin accordingly.
• Re-validate local, multimodel/smoothing, and SSH-cluster execution paths.
• Full test suite green on the new versions.
• Add a dask-version regression check (option 2 from the Scatter search crashes with "'tuple' object has no attribute 'score'" when a fit evaluation fails #388 post-mortem). Intent: run the relevant tests against the newest in-range dask — and ideally one above the cap — so the next silent internals break surfaces in automation, not in a user's multi-hour fit. This does not need to wait for GitHub Actions (restored June 1): the repo already has a local-CI substitute that covers this gap — the pre-commit pre-push hook (pytest-bngsim in .pre-commit-config.yaml, the same gate that ran on the Scatter search crashes with "'tuple' object has no attribute 'score'" when a fit evaluation fails #388 PRs). Two gaps to close there: (a) the hook currently exercises only the pinned dask, so add a step that installs/runs against a newer dask; and (b) the Scatter search crashes with "'tuple' object has no attribute 'score'" when a fit evaluation fails #388 contract test (tests/test_failed_sim_handling.py, which pins the as_completed(with_results=True, raise_errors=False) behavior we now depend on) isn't in the hook's file list yet — add it so a future dask bump that breaks that contract fails at git push.

Reference

See #388 for the failure this freeze caused and the post-mortem reasoning.