fix: prevent prolonged ClickHouse read hangs

[disintegrator]

Summary

• Set the ClickHouse native socket read timeout to 70 seconds and bound each read operation to 75 seconds, below the edge proxy's 180-second request limit.
• Retry Query and QueryRow once when a transport failure occurs before any result is exposed. The retry opens a separate ClickHouse client so it dials a fresh connection.
• Keep unsafe operations out of the retry path: writes pass through unchanged, Select is deadline-bounded but not retried because it may partially mutate its destination, and failures after row consumption begins are returned directly.
• Keep the read deadline active while rows are consumed and close temporary retry connections when rows are exhausted or closed.

Motivation

An alert identified a burst of six telemetry API failures. Every request ended as an Nginx 504 after almost exactly 180 seconds, indicating that work was remaining in flight until the proxy deadline rather than failing promptly in the application.

The investigation correlated the failures with a ClickHouse Cloud rolling node replacement around a scheduled backup. Server logs showed native ClickHouse connections receiving EOF while reading the first result block from connections established to nodes leaving service. The client configured a 60-second server-side execution limit, but left clickhouse-go's socket read timeout unset; the driver therefore used its 300-second default. A stalled socket could consequently outlive the 180-second HTTP request limit and hold request capacity until Nginx terminated it.

The mitigation establishes an application-owned failure budget below the proxy deadline and gives idempotent reads one chance to recover on a newly dialed connection. Persistent or non-transport failures still return within that budget instead of accumulating as 180-second requests.

---

Summary by cubic

Fixes ClickHouse read hangs that caused telemetry API 504s by bounding read timeouts and retrying transport failures before results are exposed.

• Sets the socket read timeout to 70 seconds and bounds read operations to 75 seconds, below the 180-second proxy deadline.
• Retries Query and QueryRow once on transport failure using a freshly dialed connection.
• Writes and Select are not retried; Select can partially mutate its destination and failures after rows are consumed are returned directly.
• Keeps the read deadline active while rows are consumed and closes retry connections when exhausted.

^{Written for commit 3248e1d. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 3248e1d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR