apollo_committer,apollo_batcher: compress state commitment infos at the committer

[itamar-starkware]

Compress each block's StateCommitmentInfos to a base64(zstd) String at the committer (in ReadPathsAndCommitBlockResponse) instead of at the consensus orchestrator, so the witness travels compressed across the committer->batcher RPC (capped at 8MB max_response_body_bytes, which wedged the chain on large state diffs), storage, and cende. The compressed String propagates end-to-end; the orchestrator- and storage-layer compression are removed.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

[reviewable-StarkWare]

This change is 

[itamar-starkware]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• apollo_consensus_orchestrator: send only the commitment-info delta the recorder is missing #14831
• apollo_consensus_orchestrator: add recorder commitment_infos_height_offset query #14830
• apollo_batcher_types,apollo_batcher: get_state_commitment_infos returns Option #14829
• apollo_committer,apollo_batcher: compress state commitment infos at the committer #14670 👈 (View in Graphite)
• apollo_consensus_orchestrator: factor recorder queries into a generic fetch helper #14810 : 1 other dependent PR (#14820 )
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Artifacts upload workflows:

• Started at 2026-06-30T13:44:28Z
• Started at 2026-07-15T09:49:55Z
• Started at 2026-07-15T10:17:47Z
• Started at 2026-07-15T10:45:15Z

[yoavGrs]

@yoavGrs reviewed 15 files and made 12 comments.
Reviewable status: 15 of 19 files reviewed, 12 unresolved discussions (waiting on itamar-starkware).

---

crates/apollo_committer/src/committer.rs line 154 at r1 (raw file):

/// Compresses the state commitment infos to `base64(zstd(serde_json(..)))`, once at the source, so
/// the witness stays compact across the committer->batcher RPC (which has an 8 MB response cap),

Instead of determining the size, please point to the constant with this value.

Code quote:

which has an 8 MB response cap

---

crates/apollo_committer/src/committer.rs line 157 at r1 (raw file):

/// batcher storage, and the cende blob.
#[cfg(feature = "os_input")]
fn compress_state_commitment_infos(infos: &StateCommitmentInfos) -> CommitterResult<String> {

We already have StateCommitmentInfos::compress,

Code quote:

fn compress_state_commitment_infos(infos: &StateCommitmentInfos) -> CommitterResult<String> {

---

crates/apollo_committer/src/committer.rs line 162 at r1 (raw file):

    let compressed = zstd::encode_all(json.as_slice(), STATE_COMMITMENT_INFOS_ZSTD_LEVEL)
        .map_err(|err| CommitterError::StateCommitmentInfosCompression(err.to_string()))?;
    Ok(base64::encode(compressed))

Can't do without it?

Code quote:

base64::encode

---

crates/apollo_committer/Cargo.toml line 25 at r1 (raw file):

async-trait.workspace = true
base64 = { workspace = true, optional = true }
serde_json.workspace = true

How is it not optional?

Code quote:

serde_json.workspace = true

---

crates/apollo_batcher_types/src/communication.rs line 57 at r1 (raw file):

    /// Gets the block hash for a given block number.
    async fn get_block_hash(&self, block_number: BlockNumber) -> BatcherClientResult<BlockHash>;
    /// Gets the compressed (`base64(zstd(serde_json(..)))`) state commitment infos for a block.

The compression method should be documented by the functionality of the compressed type

Code quote:

(`base64(zstd(serde_json(..)))`)

---

crates/apollo_committer/src/request_paths_and_commit_block_tests.rs line 167 at r1 (raw file):

/// Decompresses the response witness (`base64(zstd(serde_json(..)))`) so the assertions below can
/// inspect the trie commitment infos.
fn decompress_response_commitment_infos(

We already have StateCommitmentInfos::decompress,

---

crates/apollo_storage/src/state_commitment_infos_test.rs line 10 at r1 (raw file):

// Storage persists the witness verbatim, so a plain string stands in for the real payload.
fn sample_state_commitment_infos() -> String {

Consider

Suggestion:

dummy_state_commitment_infos

---

crates/apollo_consensus_orchestrator/src/cende/mod.rs line 81 at r1 (raw file):

#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)]
pub struct StateCommitmentInfosAndNumber {
    /// Compressed (`base64(zstd(serde_json(..)))`) commitment infos from the committer, forwarded

The compression method should be documented by the functionality of the compressed type,

Code quote:

Compressed (`base64(zstd(serde_json(..)))`) 

---

crates/apollo_committer_types/src/errors.rs line 66 at r1 (raw file):

    MissingPatriciaPaths { height: BlockNumber },
    // Holds the error message rather than `#[from] serde_json::Error` because `CommitterError`
    // crosses the RPC boundary and must stay `Clone`/`Serialize`/`Deserialize`.

Consider delete the comments.

Code quote:

    // Holds the error message rather than `#[from] serde_json::Error` because `CommitterError`
    // crosses the RPC boundary and must stay `Clone`/`Serialize`/`Deserialize`.

---

crates/apollo_committer_types/src/committer_types.rs line 53 at r1 (raw file):

    /// The OS-input commitment infos, compressed by the committer into a
    /// `base64(zstd(serde_json(StateCommitmentInfos)))` string (kept compact across this RPC).
    pub state_commitment_infos: String,

Please define struct CompressedStateCommitmentInfos(String) (or with Vec<u8> instead of String) , and impl converters from / to StateCommitmentInfos.

Suggestion:

state_commitment_infos: CompressedStateCommitmentInfos,

---

crates/apollo_storage/src/lib.rs line 1174 at r1 (raw file):

    accessed_keys: FileHandler<VersionZeroWrapper<AccessedKeys>, Mode>,
    #[cfg(feature = "os_input")]
    state_commitment_infos: FileHandler<VersionZeroWrapper<String>, Mode>,

Same here, a dedicated type.

Suggestion:

FileHandler<VersionZeroWrapper<StateCommitmentInfosCompressed>

---

crates/apollo_storage/src/lib.rs line 1329 at r1 (raw file):

    #[cfg(feature = "os_input")]
    // Returns the commitment infos at the given location or an error in case they don't exist.

Please don't delete it.

Code quote:

or an error in case they don't exist

[cursor]

PR Summary

Medium Risk
Changes the OS-input data shape across committer RPC, batcher storage, and cende; wrong compression handling would break OS blobs, but scope is limited to os_input and existing compress/decompress codecs.

Overview
OS-input Patricia witnesses now stay compressed from the committer outward, using a new CompressedStateCommitmentInfos byte wrapper instead of full StateCommitmentInfos on RPC, in the batcher commitment path, and in mmap storage.

The committer calls compress() when building ReadPathsAndCommitBlockResponse (commit and replay paths). The batcher stores and serves that blob unchanged; logging reports compressed byte length rather than trie counts. Storage serialization drops compress-on-write—it persists the raw compressed bytes only. Consensus orchestrator / cende blobs carry the compressed form verbatim.

apollo_batcher no longer depends on starknet_committer directly (types come via apollo_committer_types / apollo_storage). Tests that assert witness structure decompress locally.

^{Reviewed by Cursor Bugbot for commit 8cd1880. Bugbot is set up for automated code reviews on this repo. Configure here.}