feat: support zstd response encoding based on Accept-Encoding#62
feat: support zstd response encoding based on Accept-Encoding#62
Conversation
There was a problem hiding this comment.
Pull request overview
Adds per-request response compression negotiation so the hotblocks service can return zstd or gzip depending on the client’s Accept-Encoding.
Changes:
- Introduces
ContentEncodingand anAccept-Encodingparser with unit tests. - Plumbs selected encoding from the HTTP layer through the query pipeline.
- Adds a runtime compressor abstraction to emit either gzip or zstd streams.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
crates/hotblocks/src/encoding.rs |
New ContentEncoding type and Accept-Encoding parsing + tests. |
crates/hotblocks/src/api.rs |
Extracts Accept-Encoding, parses request body manually, sets content-encoding. |
crates/hotblocks/src/query/service.rs |
Adds encoding parameter to query entrypoints and passes it to QueryResponse. |
crates/hotblocks/src/query/response.rs |
Passes encoding into RunningQuery::new. |
crates/hotblocks/src/query/running.rs |
Implements Compressor enum wrapping gzip/zstd encoders. |
crates/hotblocks/src/main.rs |
Registers new encoding module. |
crates/hotblocks/Cargo.toml |
Adds zstd dependency. |
Cargo.lock |
Locks zstd dependency. |
crates/hotblocks/src/query/mod.rs |
Minor formatting-only change. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if let Some(enc) = encoding { | ||
| if q > best_q || (q == best_q && enc == ContentEncoding::Zstd) { | ||
| best_q = q; | ||
| best = Some(enc); | ||
| } |
There was a problem hiding this comment.
Tie-breaking uses q == best_q on f32, which can be unreliable due to floating point representation (two decimal strings that should be equal may compare unequal after parsing). To make tie-breaking deterministic, consider parsing q into an integer scale (e.g. 0..=1000) or using an epsilon/ordered representation.
There was a problem hiding this comment.
RFC q-values have at most 3 decimal digits. f32 represents all values with ≤3 decimal digits exactly. Not a practical concern.
crates/hotblocks/src/api.rs
Outdated
| async fn extract_encoding_and_body<T: serde::de::DeserializeOwned>(req: Request) -> Result<(ContentEncoding, T), Response> { | ||
| let accept = req.headers() | ||
| .get("accept-encoding") | ||
| .and_then(|v| v.to_str().ok()); | ||
| let encoding = ContentEncoding::from_accept_encoding(accept) | ||
| .unwrap_or(ContentEncoding::Gzip); | ||
| let body = axum::body::to_bytes(req.into_body(), 1024 * 1024).await | ||
| .map_err(|_| text!(StatusCode::BAD_REQUEST, "failed to read body"))?; | ||
| let query: T = serde_json::from_slice(&body) | ||
| .map_err(|e| text!(StatusCode::BAD_REQUEST, "{}", e))?; |
There was a problem hiding this comment.
extract_encoding_and_body hardcodes a 1 MiB body limit and no longer enforces Content-Type: application/json (unlike Axum’s Json extractor). This is a behavioral regression for large queries and can accept unexpected content types. Consider keeping the previous JSON extractor semantics (or at least matching its default limit / adding an explicit Content-Type check).
There was a problem hiding this comment.
Fixed — reverted to Axum's Json extractor. Headers parsed separately via HeaderMap.
mo4islona
force-pushed
the
fix/zstd-response-encoding
branch
from
d7e958d to
84fa936
Compare
Respect client Accept-Encoding header per RFC 7231: - If client sends Accept-Encoding: zstd → respond with zstd compression - If client sends Accept-Encoding: gzip → respond with gzip (backward compat) - Quality values respected (e.g. "zstd;q=1.0, gzip;q=0.5" → zstd) - At equal q, prefer zstd over gzip - Default to gzip when no Accept-Encoding header present Related: subsquid/squid-sdk#456
mo4islona
force-pushed
the
fix/zstd-response-encoding
branch
from
84fa936 to
cfbd04c
Compare
Summary
Accept-Encodingheader per RFC 7231 §5.3.4Accept-Encoding: zstd→ respond with zstd compressionAccept-Encoding: gzip→ respond with gzip (backward compatible)zstd;q=1.0, gzip;q=0.5→ zstd winsAccept-Encodingheader presentChanges
crates/hotblocks/src/encoding.rs:ContentEncodingenum with RFC-compliant parser (13 unit tests)api.rs: extractAccept-Encodingfrom request, pass through to query pipelinerunning.rs:Compressorenum wrappingGzEncoder/ZstdEncoder, selected per-requestresponse.rs,service.rs: plumb encoding through query pipeline🤖 Generated with Claude Code