ByteStream ZSTD reads ignore read_offset, corrupting resumed downloads

[snakethatlovesstaticlibs]

Full disclosure, AI was used to find this in our dev system, but I looked at the finding before filing

Summary

The ZSTD implementation of ByteStream.Read does not honor ReadRequest.read_offset.

When a compressed download is interrupted, Bazel retries with a nonzero read_offset representing the uncompressed bytes already received. Buildbarn restarts the ZSTD stream from the beginning of the blob. Bazel then appends the restarted content to its partial output, producing an oversized file and a digest mismatch.

Affected version

Observed with:

• bb-storage commit: 10acc76a8295d86f64baa8485c8e616bce0d53ca
• Bazel: 9.2.0
• --remote_cache_compression enabled
• Buildbarn configured with supportedCompressors: ['ZSTD']

The affected path appears to have been introduced by PR #258.

Relevant code

The uncompressed path correctly passes in.ReadOffset to ToChunkReader:

bb-storage/pkg/blobstore/grpcservers/byte_stream_server.go

Lines 47 to 62 in 10acc76

	case remoteexecution.Compressor_IDENTITY:
	r := s.blobAccess.Get(ctx, digest).ToChunkReader(in.ReadOffset, s.readChunkSize)
	defer r.Close()
	for {
	readBuf, readErr := r.Read()
	if readErr == io.EOF {
	return nil
	}
	if readErr != nil {
	return readErr
	}
	if writeErr := out.Send(&bytestream.ReadResponse{Data: readBuf}); writeErr != nil {
	return writeErr
	}
	}

The ZSTD path does not use in.ReadOffset and passes the complete blob directly to the encoder:

bb-storage/pkg/blobstore/grpcservers/byte_stream_server.go

Lines 64 to 72 in 10acc76

	case remoteexecution.Compressor_ZSTD:
	b := s.blobAccess.Get(ctx, digest)
	encoder, err := s.zstdPool.NewEncoder(ctx, &readStreamWriter{out: out})
	if err != nil {
	b.Discard()
	return status.Errorf(codes.ResourceExhausted, "Failed to acquire ZSTD encoder: %v", err)
	}
	defer encoder.Close()
	return b.IntoWriter(encoder)

This behavior was introduced as part of: #258

Production evidence

A remote cache download expected:

  e31efa8db7403d522c31aa12a9be8c1803a76fe8968ce0acad5aa389476bcbb8/709569128

Retries for that same digest produced several different oversized outputs:

  811c07177cada0ca514ba52b1de17b14f8a397ce45389ebef8956fcd6f7a1778/1918315112
  b2af.../1943874152
  a9f6.../2397776488

The varying sizes are consistent with partial downloads followed by one or more complete copies being appended after retries.

Bazel reported:

  Remote Cache: Output download failed:
  Expected digest 'e31efa8d.../709569128'
  does not match received digest '811c0717.../1918315112'.

Bazel rejected the cache result and rebuilt the output, so the build completed successfully, but this causes excessive network transfer and lost cache hits.

Reproduction outline

1. Configure Buildbarn with supportedCompressors: ['ZSTD'].
2. Store a sufficiently large CAS blob.
3. Start a compressed ByteStream.Read with read_offset = 0.
4. Consume and decompress the first N bytes, then interrupt the RPC.
5. Retry the same resource with read_offset = N.
6. Decompress the retry response and append it to the first N bytes.
7. Observe that the retry contains the full blob from offset zero, producing N + blob size bytes and a digest mismatch.

Multiple interruptions can append multiple partial or complete copies.

Expected behavior

For a compressed resource, a request with nonzero read_offset should return a new ZSTD stream containing the logical uncompressed blob starting at that offset.

Actual behavior

The response contains a ZSTD stream for the entire blob starting at offset zero.

Suggested fix

Apply in.ReadOffset to the uncompressed blob before passing data into the ZSTD encoder.

Please also add regression coverage for:

• ZSTD reads with read_offset = 0
• ZSTD reads with a nonzero offset
• interrupted downloads resumed through multiple requests
• verifying the concatenated decompressed output matches the original digest and size

Workaround

Disabling Bazel remote-cache compression avoids the affected branch: --noremote_cache_compression