Context
PR #349 (thank you for that fix) raised the shared transport's ResponseHeaderTimeout from 60s
to 120s specifically to tolerate a slow cloud proxy (ollama *:cloud) that withholds its 200
response header until the upstream model emits a first token, without wrongly aborting an
otherwise-alive request.
The gap
A throttled local Ollama deployment (low num_gpu/num_thread on constrained hardware, not
a cloud proxy) can see real time-to-first-token of 1-5 minutes on a cold model load. That still
exceeds the 120s ceiling PR #349 established, on a request that is alive, not hung — the same
class of problem #349 fixed, just a slower real-world case than the cloud-proxy scenario it was
tuned for.
Proposal
Rather than raising the shared default again for every deployment (which would just shift the
same trade-off), make it configurable via an opt-in env var, ZERO_RESPONSE_HEADER_TIMEOUT
(same accepted forms as the existing ZERO_STREAM_IDLE_TIMEOUT: a Go duration string or bare
seconds). The default stays exactly 120s — nothing changes for anyone who doesn't set it.
Validated locally against a throttled local Ollama deployment:
ZERO_RESPONSE_HEADER_TIMEOUT=5s fails at ~6.4s (expected — proves the override takes effect)ZERO_RESPONSE_HEADER_TIMEOUT=300s succeeds at ~223s (a request that would have hit the old
120s ceiling)
I have a patch + regression test ready locally (mirrors the existing
TestResolveStreamIdleTimeout pattern — fmt-check/vet/package tests all green). Per
AGENTS.md, I understand a community PR needs an issue-approved label first — happy to open
the PR as soon as this is approved.
Context
PR #349 (thank you for that fix) raised the shared transport's
ResponseHeaderTimeoutfrom 60sto 120s specifically to tolerate a slow cloud proxy (
ollama *:cloud) that withholds its 200response header until the upstream model emits a first token, without wrongly aborting an
otherwise-alive request.
The gap
A throttled local Ollama deployment (low
num_gpu/num_threadon constrained hardware, nota cloud proxy) can see real time-to-first-token of 1-5 minutes on a cold model load. That still
exceeds the 120s ceiling PR #349 established, on a request that is alive, not hung — the same
class of problem #349 fixed, just a slower real-world case than the cloud-proxy scenario it was
tuned for.
Proposal
Rather than raising the shared default again for every deployment (which would just shift the
same trade-off), make it configurable via an opt-in env var,
ZERO_RESPONSE_HEADER_TIMEOUT(same accepted forms as the existing
ZERO_STREAM_IDLE_TIMEOUT: a Go duration string or bareseconds). The default stays exactly 120s — nothing changes for anyone who doesn't set it.
Validated locally against a throttled local Ollama deployment:
ZERO_RESPONSE_HEADER_TIMEOUT=5sfails at ~6.4s (expected — proves the override takes effect)ZERO_RESPONSE_HEADER_TIMEOUT=300ssucceeds at ~223s (a request that would have hit the old120s ceiling)
I have a patch + regression test ready locally (mirrors the existing
TestResolveStreamIdleTimeoutpattern —fmt-check/vet/package tests all green). PerAGENTS.md, I understand a community PR needs anissue-approvedlabel first — happy to openthe PR as soon as this is approved.