Skip to content

fix(observability): bound the MPC metrics hot path; refuse mock-crypto builds on real networks#1816

Merged
omersadika merged 2 commits into
mainfrom
fix/metrics-hot-path
Jul 12, 2026
Merged

fix(observability): bound the MPC metrics hot path; refuse mock-crypto builds on real networks#1816
omersadika merged 2 commits into
mainfrom
fix/metrics-hot-path

Conversation

@omersadika

Copy link
Copy Markdown
Contributor

What

Item 7 of the 1.2.0 pre-tag checklist (release review) — the cheap fixes worth taking before the tag.

1. MPC metrics hot path (hand-verified findings)

refresh_observability_metrics ran on every ~20ms service iteration and re-emitted ~11 per-seq gauge series (plus a per-committee-member matrix) for every session tracked this epoch — terminal sessions stay in self.sessions until epoch end (zero remove/retain), so per-tick work grew with the epoch's cumulative session count, on the same loop that advances MPC sessions. The in-code "bounded by max_active_sessions_buffer" claim was false.

  • Refresh rate-limited to 1/s (OBSERVABILITY_REFRESH_INTERVAL).
  • Terminal (Completed/Failed) sessions get exactly one final emission recording the terminal state, then are skipped; re-armed if a session ever leaves terminal state. Their seqs stay in current_seqs so the existing stale-zeroing block doesn't wipe the terminal record.
  • The sessions_with_self_output_no_quorum counter moved into the cheap unconditional scan — semantics unchanged.

2. Checkpoint per-session gauge label leak

ika_dwallet_checkpoint_user_session_written_at_seq accumulated one labeled series per user session for the process lifetime. The builder is per-epoch, so its constructor now reset()s the gauge — label set bounded to the current epoch.

3. unsafe-mock tripwire

A binary compiled with dwallet-mpc-unsafe-mock now refuses to boot against ika mainnet/testnet (chain derived from the deployed ika system object, exactly like #1810's minimum-CPU gate; predicate unit-tested in the same style) and logs a grep-able insecure_mock_crypto=true error banner on any other network. CLAUDE.md documents the verified footgun that workspace-scope cargo test silently mocks ika-core's crypto via ika-test-cluster's self-dev-dependency feature unification (verified both directions with cargo tree: dev-graph resolution enables it; plain ika-node builds do not).

4. Deployed-bytes decode tool (G1 evidence residual)

New #[ignore]d decode_deployed_network_key_bytes (sharing the deployed-envs table with the existing NetworkKeyId tool): fetches the real mainnet/testnet network-key bytes and decodes them on the exact types the v3 backward-compatible reconfiguration arm consumes — V1 anchor inner → class-groups DKG output (+ byte-identical re-encode check), V2 reconfiguration output → bwd-compat PublicOutput. This is the real-chain-bytes check the synthesized-anchor regression tests can't provide; run results will be posted on this PR.

Verification

  • cargo clippy -p ika-core --lib --tests clean; cargo fmt --all.
  • runtime::tests 4/4 green (new mock_crypto_forbidden_exactly_on_real_networks included).
  • Behavior notes: rate limit and once-per-terminal emission change gauge freshness (1s) only; series values and meanings are unchanged. Registry cardinality within an epoch is unchanged (per-seq series still exist); steady-state per-tick work drops from O(epoch sessions × 50/s) to O(in-flight sessions × 1/s).

🤖 Generated with Claude Code

…o builds on real networks

Three pre-tag items from the 1.2.0 release review:

1. refresh_observability_metrics ran on every ~20ms service iteration
   and re-emitted per-seq gauge series for EVERY session tracked this
   epoch — terminal (Completed/Failed) sessions stay in the map until
   epoch end, so per-tick work and allocations grew with the epoch's
   cumulative session count, on the same loop that advances MPC
   sessions (its in-code "bounded by max_active_sessions_buffer" claim
   was false). Refreshes are now rate-limited to once a second and a
   terminal session gets exactly one final emission recording its
   terminal state (re-armed if it ever leaves terminal state); the
   waiting-on-quorum counter moved to the cheap full-scan loop so its
   semantics are unchanged.

2. ika_dwallet_checkpoint_user_session_written_at_seq accumulated one
   labeled series per user session for the process lifetime (nothing
   ever removed label values). The checkpoint builder is constructed
   once per epoch, so it now resets the gauge there, bounding the label
   set to the current epoch.

3. A binary compiled with dwallet-mpc-unsafe-mock now refuses to boot
   against the ika mainnet/testnet networks (chain derived from the
   deployed ika system object, as the minimum-CPU gate does) and logs a
   grep-able insecure_mock_crypto=true error banner elsewhere, so a
   mocked binary can never pass as real-crypto evidence. CLAUDE.md now
   documents the verified footgun that WORKSPACE-scope cargo test
   silently enables the mock across the graph via ika-test-cluster's
   self-dev-dependency (feature unification) — real-crypto runs need
   package scope.

Also adds decode_deployed_network_key_bytes, an #[ignore]d one-off tool
(next to the existing NetworkKeyId derivation tool, sharing its
deployed-envs table) that fetches the real mainnet/testnet network-key
bytes and decodes them on the exact types the v3 backward-compatible
reconfiguration arm consumes — the real-chain-bytes evidence the
synthesized-anchor regression tests cannot provide.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fullnode.testnet.sui.io currently rejects JSON-RPC with a 404, which
sank the whole multi-env run at the first unwrap. Each env now carries
an ordered endpoint list (canonical Mysten first, public fallback), and
connection failure skips the env with a printed reason instead of
panicking — one downed endpoint no longer hides the other network's
result.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@omersadika

Copy link
Copy Markdown
Contributor Author

Deployed-bytes decode evidence (G1 residual: CLOSED)

Ran decode_deployed_network_key_bytes (release build, inkrypto pin 1d8b5a9, 2026-07-12) against both live chains:

CONNECT testnet: https://fullnode.testnet.sui.io:443 failed: Request rejected `404`
CONNECT testnet: using https://sui-testnet-rpc.publicnode.com
DECODE testnet: key 0xe7c79a60931299e110297554fc02e0a0e095e96778775092c97f07a1bd1337cc: V1 anchor OK (175302 bytes, re-encode byte-identical)
DECODE testnet: key 0xe7c79a60931299e110297554fc02e0a0e095e96778775092c97f07a1bd1337cc: V2 reconfiguration output OK (151034 bytes)
CONNECT mainnet: using https://fullnode.mainnet.sui.io:443
DECODE mainnet: key 0x0a9c0b88a5c729378bce1a98a8c285f1dde26f89e53d07164dba8a059dc15587: V1 anchor OK (180458 bytes, re-encode byte-identical)
DECODE mainnet: key 0x0a9c0b88a5c729378bce1a98a8c285f1dde26f89e53d07164dba8a059dc15587: V2 reconfiguration output OK (197504 bytes)
test ... ok (finished in 40.15s)

What this proves, on the actual on-chain bytes (not synthesized test vectors):

  1. Both deployed anchors are V1-tagged — the premise of the G1 fix (fix(dwallet-mpc): handle the deployed V1 network-DKG anchor on the v3 reconfiguration path (G1) #1798/feat(dwallet-mpc): support the deployed V1 network-DKG anchor on the main (v4) reconfiguration path + one-time V1->V3 anchor migration #1799) holds on both networks.
  2. The exact decode the v3 backward-compatible reconfiguration arm performs (V1 inner → class_groups::dkg::PublicOutput) succeeds on the real bytes, and re-encodes byte-identically — no bcs layout drift across the crypto-source migration and both inkrypto bumps (0572d6c → c407ad4 → 1d8b5a9).
  3. The prior V2 reconfiguration outputs decode as the bwd-compat PublicOutput — the second input the arm consumes on every reconfiguration of the deployed keys.

This closes the last G1 residual from the release review (the regression tests use synthesized anchors re-encoded under the current crate, so they could not have caught a layout drift against the deployed bytes).

Side note captured in the second commit: fullnode.testnet.sui.io:443 currently 404s JSON-RPC, so the tools now carry an ordered endpoint list with a public fallback and skip-with-reason instead of panicking.

🤖 Generated with Claude Code

@omersadika omersadika merged commit 3061d1f into main Jul 12, 2026
10 checks passed
@omersadika omersadika deleted the fix/metrics-hot-path branch July 12, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant