feat(autobahn): implement new lane ID for epoch (CON-358) - #3862
feat(autobahn): implement new lane ID for epoch (CON-358)#3862wen-coding wants to merge 9 commits into
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
8d254d4 to
4f52a00
Compare
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3862 +/- ##
==========================================
- Coverage 61.61% 60.73% -0.88%
==========================================
Files 2348 2254 -94
Lines 200852 190212 -10640
==========================================
- Hits 123755 115531 -8224
+ Misses 66044 64480 -1564
+ Partials 11053 10201 -852
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
The LaneID = (validator, e_join) refactor is coherent and well-tested at the type/proto layer, but the new lane-deletion path (pruneInactiveLanes + DeleteLane) breaks the "lanes are only added, never removed" invariant that three unguarded map reads in avail still rely on, and the leave/rejoin handling in SubscribeLaneProposals and ProduceLocalBlock has races that panic or permanently kill the producer. These are latent until epoch transitions are wired up (#3736), but they are defects in this PR's own feature and are not covered by the added tests.
Findings: 6 blocking | 13 non-blocking | 9 posted inline
Blockers
pruneInactiveLanesdeletes entries frominner.blocks/inner.votes, but three call sites still index those maps without anokcheck and will nil-deref once a leaver is pruned:avail/state.go:654(headers:q := inner.votes[lr.Lane()]thenq.first),avail/state.go:815(PushQC loop:inner.blocks[lr.Lane()].q[n]), andavail/inner.go:189(laneQC:i.votes[lane].q[n]). All three iterate the committee of the QC's epoch, which can be an older epoch that still contains the leaver. The comment removed frompersist/blocks.go("lanes are only added, never removed") was load-bearing for these too — every reader needs an ok-check (or lanes must be retained until the prune anchor passes them).- No test covers the interaction between
tryPruneLeaveLanesand a lagging reader.TestApplyEpoch_AddsJoinerDefersLeaverUntilCommitQCWatermarkverifies the leaver's maps/WAL disappear, but nothing exercisesheaders()/fullCommitQC/ thes.data.PushQCloop against a previous-epoch CommitQC after the prune, which is exactly the crash path. Please add one. - Prune watermark choice needs justification: normal block retention is gated on the durable prune anchor (AppQC-derived,
advancePersistedBlockStart), but a leaver's in-memory queues and WAL are dropped as soon as any durable CommitQC lands in the new epoch. Blocks that are committed but not yet executed/served can still be needed at that point. Either reuse the prune-anchor watermark or document why CommitQC-epoch is sufficient. - 3 blocking issue(s) flagged inline on specific lines.
Non-blocking
- The Cursor review file (
cursor-review.md) is empty — that pass produced no output, so this review merges only Claude's and Codex's findings. - Codex P1 #1 (legacy block WALs / wire format): the lane WAL directory name changes from
hex(pubkey)(32B) tohex(pubkey||e_join)(40B), andBlockHeader.lane/LaneRange.lanechange proto type fromPublicKeytoLaneIDon the same field number. Both are hard breaks — existing WAL dirs are silently skipped on restart, and peers on the old binary cannot decode. SinceApplyEpoch/ActivateEpochhave no production callers yet, this is presumably pre-launch and acceptable; please confirm explicitly in the PR description rather than leaving it implicit. ApplyEpochnever returns a non-nil error. Either drop the return value or note that it is reserved for the follow-up wiring.tryPruneLeaveLanesre-Stores the identicallatestCommitQCvalue after the disk delete purely to wake waiters. It is safe today only becausemarkCommitQCsPersistedandtryPruneLeaveLanesare both on therunPersistgoroutine — worth stating that in the comment, since a concurrent writer would make this a watermark regression.ctrl.Updated()alone may be enough.markBlockPersistedwritesinner.nextBlockToPersist[lane] = nextunconditionally, so a pruned lane can be resurrected as a stale map entry (small leak, and it makes the map key sets diverge fromblocks/votes).LaneProposalsRecv.Recvallocates an errgroup and two goroutines per block received, plus a freshLocalLaneUpdates()subscription per iteration. On the hot proposal path this is meaningful churn; consider hoisting the lane-change watcher out of the per-block loop.alignMempoolForLanereadsNextBlock(lane)before taking the mempool lock, so the tip can be stale by the time it is applied; and a rejoin silently discards all bufferedevmTxs/evmNonces. Both are probably intended, but neither is documented.- 6 suggestion(s)/nit(s) flagged inline on specific lines.
There was a problem hiding this comment.
Solid, well-documented reshaping of LaneID into (validator, e_join) with good test coverage of the stay/leave/rejoin state machine, but two correctness problems in the persistence/pruning paths are blocking: restored leave-lane queues are never positioned at the prune anchor (restart fails), and the leave-lane retention watermark is keyed on e_join rather than the leave epoch, so long-tenured leavers are dropped before their final tips are committed. Cursor's second-opinion pass produced no output; Codex's two findings are both confirmed and included.
Findings: 3 blocking | 12 non-blocking | 7 posted inline
Blockers
avail: no test covers the restart path that actually breaks — a persisted leave-lane WAL whose surviving blocks start above 0 (i.e. a prune anchor with a non-emptyLaneRangefor the leaver).TestApplyEpoch_AddsJoinerDefersLeaverUntilAppQCWatermarkandTestTryPruneLeaveLanes_OrphanWALWithoutMapsboth persist laneB at block 0 with no anchor, so they pass over the bug innewInner. Please add a restart test with an anchor whose leaverLaneRange.First() > 0.- 2 blocking issue(s) flagged inline on specific lines.
Non-blocking
- Cursor's review file (
cursor-review.md) is empty — that pass produced no output, so this review merges only Claude's and Codex's findings. - Epoch-boundary peer teardown for remote lanes is unhandled.
Registry.VerifyInWindowstill only checks the latest committee (existing TODO), andPushBlock/PushVotenow returnErrBadLanewheninner.blocks/inner.voteslacks the lane. A peer that applies epoch N+1 slightly before us and pushes a proposal on its new rejoin LaneID makesclientStreamLaneProposalsreturn an error (giga/avail.go:194), tearing down that peer's RPC stream. The PR carefully protects the local lane viaErrLaneIdentityChanged; remote lanes need the equivalent soft-failure (or a widened verification window) beforeApplyEpochis wired up. ApplyEpochswapsinner.epochwholesale, soinner.laneQCandPushVoteimmediately evaluate already-accumulated old-epoch votes against the new committee'sLaneQuorum()and weights. Votes from departed validators are still in the queues and would be assembled into aLaneQCthat fails verification against the new committee. The existing// TODO: filter votes per-epoch committeebecomes load-bearing onceApplyEpochis called in production — worth stating explicitly in the #3736 follow-up.- The
BlockHeader.lane/LaneRange.laneproto type change (PublicKey→LaneID) alters block header hashes and makes previously persisted WAL entries undecodable, so any node with existing autobahn state needs a coordinated state reset. Autobahn is opt-in viaautobahn-config-file, so this is likely fine, but the PR description doesn't mention it and there's no migration note. producer.mempoolFirst()(producer/mempool.go:83) is now dead —alignMempoolForLanereplaced its only caller.unusedisn't enabled in.golangci.ymlso it won't fail lint, but it should be removed.types.GenCommittee(testonly.go) now bypasses the public constructors and callsnormalizeWeights+finalizeCommitteedirectly to inject randome_joinvalues. Randomizinge_joinin tests is a genuine improvement (it catches code assuminge_join == 0), but duplicating the constructor body in test-only code means future changes toNewCommitteewon't be reflected. Consider anewCommitteeWithEJoins(weights, func(PublicKey) EpochIndex)helper shared by all three.State.ApplyEpochalways returnsnil. The comment explains this is retained for #3736, which is reasonable, but every current caller has to handle an error that cannot occur.- 5 suggestion(s)/nit(s) flagged inline on specific lines.
There was a problem hiding this comment.
No new bugs found this run. The latest commit (b3a591f, "harden LaneID leave/rejoin lifecycle") appears to resolve the checkBound panic, nil-map-deref, orphan-WAL, and producer-race issues flagged in earlier reviews on this PR (cursor, seidroid, and my own prior comments). Given the size and consensus-critical scope of this change, I'd still like a human to take a look before merge rather than shadow-approving.
What was reviewed:
checkBound/Recvno longer panics on leave+rejoin coalescing - now returnsErrLaneIdentityChanged(avail/subscriptions.go)headers(), thePushQCblock-collection loop, andlaneQC()now guard missing lane maps instead of nil-dereferencing after a leave prune (avail/state.go, avail/inner.go)produceLocalBlock/streakOpErrnow bind to an explicit LaneID per streak, so a stale rejoin race resolves toErrBadLane/context.Canceledinstead of killingproducer.Run(producer/state.go)- Checked the ruled-out mempool-reset-on-rejoin candidate -
alignMempoolForLaneonly resets state when the streak's LaneID actually changes, so a same-lane stay keeps its tip and txs
Extended reasoning...
This run's bug hunter found no new issues. Cross-referencing the current HEAD (b3a591f, ahead of the PR diff shown) against the earlier cursor-bot, seidroid-bot, and my own prior claude[bot] comments on this thread, the latest commit appears to specifically address nearly every previously flagged blocker: the LaneID-coalescing panic in checkBound (now returns ErrLaneIdentityChanged), the nil-map-dereference in headers()/PushQC/laneQC() after a leave-lane prune (now guarded with ok-checks), the orphan leave-WAL-survives-restart issue (inner.go now reattaches leave-lane WALs into maps so tryPruneLeaveLanes/DeleteLane can reap them), the 'leave tears down peer RPC' issue (giga/avail.go now retries/resubscribes instead of propagating a fatal error out of the multiplexed RunServer scope), the producer lane-rederivation race (produceLocalBlock now takes an explicit bound LaneID and checks HasLane against it), and the HasLane O(n) regression (committee.go now keeps a byValidator map for O(1) lookup). I did not find a case where these fixes are incomplete.
This is nonetheless a large (36-file), consensus-critical change to how lanes are identified across the availability plane, persistence layer, producer, and wire protocol (BlockHeader.lane and LaneRange.lane both change proto type), including a protobuf wire-format change and WAL directory-naming change. ApplyEpoch/ActivateEpoch have no production callers yet (explicitly deferred to #3736), which reduces blast radius today, but the design decisions here (e.g. epoch-scoped LaneID reuse, AppQC-floor-gated lane pruning, coalescing-safe subscription semantics) are exactly the kind of judgment calls that warrant a human's sign-off before the epoch-transition wiring lands on top of them.
No security-sensitive auth/crypto/permission logic is touched beyond the existing signature verification already in place; the main risk surface is correctness/liveness of consensus (panics, stuck goroutines, dropped blocks) rather than exploitable vulnerabilities. Test coverage is substantial (new tests for committee activation, lane WAL orphan pruning, coalesced leave/rejoin subscription behavior, and a producer lifecycle test), which supports confidence in the fixes but doesn't substitute for a maintainer familiar with the epoch-transition roadmap reviewing the design.
There was a problem hiding this comment.
Solid, well-tested refactor of LaneID from PublicKey to (validator, e_join) with careful prune/leave/rejoin bookkeeping. The blocking concern is that it silently breaks the on-disk autobahn WAL format (and the P2P wire format) with no migration or documented state-dir reset, so an in-place upgrade of an autobahn-enabled node fails to start; several smaller robustness/hot-path issues are noted below.
Findings: 2 blocking | 11 non-blocking | 7 posted inline
Blockers
- On-disk WAL format break with no migration path.
BlockHeader.laneandLaneRange.lanechange fromPublicKeyto the newLaneIDmessage.PublicKey{ed25519: <32 bytes>}andLaneID{validator, e_join}are not wire-compatible, so on an in-place upgrade of an autobahn-enabled node a persisted prune anchor decodes throughPruneAnchorConv.Decode->types.CommitQCConv.Decode->LaneRangeConv.Decodeand fails on the embeddedLaneRange.loadPersistedState(sei-tendermint/internal/autobahn/avail/state.go:267) treats that as fatal, soNewStateerrors and the node will not start until<persistent_state_dir>is wiped. Same for the peer wire format: old and new binaries cannot exchangeBlockHeader/LaneRange, andBlockHeader.Hash()changes. Autobahn is opt-in and off by default, so this is likely acceptable in substance -- but it needs to be explicit: either handle/skip the old encoding, or state the required state-dir reset and the no-mixed-version constraint in the PR description / release notes. Right now the only breakage signal is thenon-app-hash-breakinglabel, which reads as the opposite. - 1 blocking issue(s) flagged inline on specific lines.
Non-blocking
- Cursor's second-opinion pass produced no output (
cursor-review.mdis empty), so this review merges only Claude's and Codex's findings. - Codex's point, confirmed but downgraded:
avail.State.ApplyEpoch,epoch.Registry.ActivateEpoch, andtypes.ActivateCommitteehave no non-test callers, so the entire leave/rejoin/prune path this PR adds is unreachable in production until theadvanceEpoch/onAdvancewiring in #3736 lands. That is consistent with the PR being explicitly incremental (theApplyEpochdoc comment says so), so it is not a defect -- but it does mean the new gates get no production coverage in this PR, and the correctness oftryPruneLeaveLanesrests entirely on unit tests plus a chain of non-local invariants (see the inline notes on state.go:122 and blocks.go:347). Worth an integration test that drives a real epoch transition throughproducer.Run+runPersistbefore the wiring PR flips it on. Registry.ActivateEpochchangingRWMutex[registryState]toRWMutex[*registryState]is the right fix (s.latest = nexton a value copy would have been lost) -- good catch. Unrelated nit:ActivateEpochdoesn't touchprev'sRoadRange, so callers passingOpenRoadRange()for every epoch (as all the new tests do) end up with overlapping open ranges. The doc comment acknowledges this; consider rejecting aroadsthat overlapsprevso the invalid state is unrepresentable rather than a caller contract.- The PR description says leavers are pruned at the "AppQC prune floor (
lane.EJoin() < appQC.EpochIndex())", but the implemented gate is tipcut-committee membership (!current.HasLane(lane) && !tipcut.HasLane(lane)), with noEJoin()comparison anywhere. The code is fine; the description should match it so future readers don't go looking for theEJoincheck. serverStreamLaneProposalscan now silently switch the stream to a differentLaneIDmid-connection (rejoin resetsfirst = 0).clientStreamLaneProposalshappens to tolerate this becausePushBlockresolves the lane from the header, but the commented-out sanity check at giga/avail.go:191 (proposal lane == peer key) can no longer be a simple equality once it's uncommented -- it has to compareLane().Validator(). Worth updating that comment now while the context is fresh.- 6 suggestion(s)/nit(s) flagged inline on specific lines.
There was a problem hiding this comment.
Solid, well-documented refactor of LaneID from a bare pubkey to (validator, e_join), with good test coverage for the stay/leave/rejoin state machine and correctly regenerated wireguard/proto size tables. Two blocking issues remain: existing autobahn persistent state becomes unreadable with no migration or version gate (commitQC WAL hard-fails, block WAL dirs are silently orphaned), and a restart inside a leave window can make the local PushQC loop permanently skip road indices whose retained CommitQCs still reference the departed lane.
Findings: 5 blocking | 13 non-blocking | 9 posted inline
Blockers
- Upgrade path for an existing autobahn
persistent_state_diris undefined and handled inconsistently: the commitQC WAL / prune anchor hard-fail to decode (startup abort), while legacy block WAL dirs are warn-and-skipped (silent data loss + orphaned dirs thattryPruneLeaveLanescan never reach, since they are not inbp.lanes). Pick one policy — a state-dir version marker that fails fast with an actionable message, or an explicit migration/cleanup — and state it in the PR description. See the two inline comments onblocks.go:234andautobahn.proto:131. - No test covers the upgrade path at all: there is no case that opens a
BlockPersisterover a 32-byte-hex lane dir, and none that feeds a pre-change (field-1lane)LaneRange/CommitQCWAL entry throughloadAllCommitQCs. Whatever policy is chosen for the item above should be pinned by a test, since this is exactly the failure that only shows up on a real operator's disk. - 3 blocking issue(s) flagged inline on specific lines.
Non-blocking
- The Cursor pass (
cursor-review.md) produced no output — the file is empty. Only Codex's three findings were available to merge; all three are reflected above (its P1 #1 as a blocker, P1 #2 as a blocker, P2 #3 as an inline suggestion). - PR description drift: it states the prune floor is the "in-memory AppQC prune floor (
lane.EJoin() < appQC.EpochIndex())", but the implemented gate intryPruneLeaveLanesis "the first retained CommitQC's committee no longer names the lane". The code comments are accurate; the description is not. Worth fixing since this is the subtlest invariant in the change. producer.mempoolFirst()(producer/mempool.go:83) is now dead —Runwas its only caller and it went throughalignMempoolForLane.unusedis not in.golangci.yml, so lint will not catch it.tryPruneLeaveLanesallocates a fullKnownLanes()snapshot and re-scans it after every persist batch, even when nothing has left the committee. Cheap fix: skip the whole body unless the epoch changed since the last check orlen(inner.blocks) > current.Lanes().Len()(the guard already used for the in-memory half).LaneID.Bytes()callsl.validator.Bytes()twice (once for the capacity, once for the append). Hoist it into a local.epoch.GenRegistrystill builds its genesis committee viaNewCommittee(alle_join = 0), so the avail/producer/giga tests only ever exercisee_join = 0lanes end-to-end; the random-e_joinGenCommitteechange is confined to thetypespackage. AnActivateEpoch-based fixture with a non-zeroe_joinstayer would give the new WAL naming and map keying real coverage.ApplyEpochreturns anerrorthat is documented as always nil. Fine as a forward-compat hook for #3736, but every current caller has torequire.NoErroron something that cannot fail.- 6 suggestion(s)/nit(s) flagged inline on specific lines.
wen-coding
changed the title
wen-coding
added
the
ai: skip-review
There was a problem hiding this comment.
This PR is a large, consensus-critical refactor (LaneID now (validator, e_join) instead of a bare pubkey, touching committee membership, avail WAL persistence, producer lifecycle, and the wire format) — well outside the scope for auto-approval regardless of this run's findings.
No new bugs were found this run. Beyond the reported findings, I also checked three additional candidates and ruled them out: a hang in Block/PushBlock/WaitForLocalCapacity if a lane is deleted mid-wait (the deployment model in this repo means ApplyEpoch/epoch transitions have no production callers yet, so this is dormant); tryPruneLeaveLanes turning a missing registry epoch into a NewState startup-abort (registry only holds genesis today, so unreachable until epoch persistence lands); and unbounded epoch registry growth (bounded in practice since nothing currently activates new epochs).
Extended reasoning...
Given the volume of prior findings on this thread (several genuine correctness gaps around leave-lane pruning windows, restart handling of in-flight leave WALs, and WAL/wire format compatibility flagged by other reviewers) and the fact that this changes core consensus lane-identity semantics, a human should review this rather than relying on automated approval.
3c335fa to
fcee025
Compare
LaneID pairs validator with join epoch so leave/rejoin does not collide with a recycled validator slot. TipEpoch leave prunes maps on the same tick as DeleteLane; Subscribe fails with ErrLanePruned after drop. Co-authored-by: Cursor <cursoragent@cursor.com>
26a7c0b to
d37679b
Compare
wen-coding
removed
the
ai: skip-review
There was a problem hiding this comment.
Solid, well-documented reworking of LaneID into (validator, e_join) with good coverage on the persist/DeleteLane paths, but two blocking issues remain: PushBlock/PushVote don't re-check lane presence after waiting (the exact hazard the rest of the PR carefully fixes elsewhere), and four unrelated QC-verification tests were deleted with no replacement, leaving PrepareQC.Verify and AppQC.Verify weight/epoch-binding coverage at zero.
Findings: 4 blocking | 11 non-blocking | 8 posted inline
Blockers
- Unexplained test deletions in
sei-tendermint/autobahn/types/committee_test.go:TestPrepareQCVerifyChecksWeight,TestPrepareQCVerifyChecksEpochBinding,TestAppQCVerifyChecksWeight, andTestNewCommittee_RejectsEmptyWeightsare removed with no replacement.TestCommitQCVerifyChecksWeightwas not added — the old one was deleted and the PrepareQC test renamed into its place. After this PR there is no test anywhere inautobahn/typesexercisingPrepareQC.Verify, and none exercisingAppQC.Verifyweight thresholds. None of these depend onLaneID, so nothing in this change requires dropping them. Please restore them (mechanically updating tocommittee.Lane(...)where needed). - 3 blocking issue(s) flagged inline on specific lines.
Non-blocking
cursor-review.mdis empty — the Cursor pass produced no output, so this review reflects only Claude + Codex findings.inner.prune(c *types.Committee, ...)(inner.go:228) never referencesc— its body iteratesi.votes. This PR now threads a carefully computedanchorCommitteeinto that ignored parameter, which reads as if the anchor committee scopes the prune when it does not. Either use it or drop the parameter.- Stale-leave pruning only runs when
collectPersistBatch'sWaitUntilpredicate fires (new blocks / new commitQC / new appQC). A tip-stale leave lane with nothing pending and an otherwise idle chain lingers ininner.blocksand on disk until unrelated activity wakes the loop. Consider addingstaleLaneDisposableto the wake predicate. PushCommitQCstill hard-rejects QCs whoseEpochIndexdiffers from the applied epoch (state.go:435). OnceApplyEpochhas production callers, in-flight prior-epoch QCs will surface as stream errors to peers during a transition. The PR body defers this to #3736 — worth aTODO(#3736)at that comparison so it isn't lost.NewBlockPersisterlogs and skips lane dirs whose names don't parse as the new 80-hexLaneID(old 64-hex pubkey dirs). Those directories are then never reclaimed. Given the PR requires a state wipe this is acceptable, but the skip path should say the entry leaks rather than implying it's handled.LaneVotesRecv.next(subscriptions.go:55) is keyed byLaneIDand never has entries removed whendropLanesfires, so it accumulates one entry per historical lane over the node's lifetime. Negligible in size, but easy to clean up alongsidedropLanes.- 5 suggestion(s)/nit(s) flagged inline on specific lines.
| } | ||
|
|
||
| // finalizeCommittee sorts lanes and rejects duplicate validators (multiple e_join). | ||
| func finalizeCommittee(lanes []LaneID, weights map[PublicKey]uint64, totalWeight uint64) (*Committee, error) { |
There was a problem hiding this comment.
newCommittee? It is effectively just a constructor.
| rng := utils.TestRng() | ||
| ep, keys := makeEpoch(rng) | ||
| vote := NewLaneVote(NewBlock(keys[0].Public(), 0, GenBlockHeaderHash(rng), GenPayload(rng)).Header()) | ||
| vote := NewLaneVote(NewBlock(NewLaneID(keys[0].Public(), 0), 0, GenBlockHeaderHash(rng), GenPayload(rng)).Header()) |
There was a problem hiding this comment.
shouldn't it be a laneID from ep.Committee() instead? Check other tests as well.
| type LaneID struct { | ||
| utils.ReadOnly | ||
| validator PublicKey | ||
| eJoin EpochIndex |
There was a problem hiding this comment.
Since LaneID's purpose is to be passed by value, make it a plain struct with public fields.
| type LaneID struct { | ||
| utils.ReadOnly | ||
| validator PublicKey | ||
| eJoin EpochIndex |
There was a problem hiding this comment.
nit: imo "eJoin" doesn't feel very informative. How about calling it sth like "Start"/"Begin"/"First"? I.e. epoch at which the lane started? The fact that this is an epoch index is implied by the type, so "e" prefix is redundant. Or perhaps simply "Joined"?
|
|
||
| // Compare orders by validator, then e_join. | ||
| func (l LaneID) Compare(other LaneID) int { | ||
| if c := l.validator.Compare(other.validator); c != 0 { |
|
|
||
| func newInner(epoch *types.Epoch, loaded utils.Option[*loadedAvailState]) (*inner, error) { | ||
| func newInner(registry *epoch.Registry, loaded utils.Option[*loadedAvailState]) (*inner, error) { | ||
| ep := registry.LatestEpoch() |
There was a problem hiding this comment.
epoch of Inner is not "Latest". It is the epoch of the next CommitQC.
| } | ||
|
|
||
| // WaitProduce waits until LocalLane is Some (produce session start). | ||
| func (s *State) WaitProduce(ctx context.Context) (types.LaneID, error) { |
There was a problem hiding this comment.
WaitForLocalLane? We are simply waiting for local lane to be available here.
| // is separate; production wiring is #3736. | ||
| func (s *State) ApplyEpoch(ep *types.Epoch) { | ||
| for inner, ctrl := range s.inner.Lock() { | ||
| inner.addCommitteeLanes(ep.Committee()) |
There was a problem hiding this comment.
why not pruning lanes here as well?
There was a problem hiding this comment.
persister can catch up based on the diff
| } | ||
|
|
||
| // tipEpochOf is the registry epoch of the first retained CommitQC. | ||
| func tipEpochOf(inner *inner, registry *epoch.Registry) (utils.Option[*types.Epoch], error) { |
|
|
||
| // staleLaneDisposable: tipEpoch omits lane and e_join < tip (joiners at/after tip stay). | ||
| // None tipEpoch → false. | ||
| func staleLaneDisposable(lane types.LaneID, tipEpoch utils.Option[*types.Epoch]) bool { |
There was a problem hiding this comment.
"Epoch.IsClosed(lane) bool"? Also move it to types. You can add IsClosedOpt function if you want it to work on Option[Epoch], although I don't understand what are the conditions under which you don't have epoch available. It doesn't make sense to evaluate IsClosed without epoch.
|
|
||
| // deleteStaleLaneWAL Deletes WALs for tip-stale leave maps. | ||
| // DeleteLane no-ops if a lane never opened a WAL (empty leave). | ||
| func (s *State) deleteStaleLaneWAL(lanes []types.LaneID) error { |
There was a problem hiding this comment.
this should be method of blocks. Or just inline it, since this is a trivial loop over existing method.
|
|
||
| // pruneStaleLeave Deletes WALs then drops maps for tip-stale leave LaneIDs | ||
| // (same tick as runPersist after Parallel). | ||
| func (s *State) pruneStaleLeave(staleLeave []types.LaneID) error { |
There was a problem hiding this comment.
dropLanes/pruneLanes? "pruneStaleLeave" doesn't tell me what we are pruning, what is stale, and whether "prune" or "leave" is the verb here.
| // Block returns block n of the given lane. | ||
| // Waits until the block is available. | ||
| // Returns ErrPruned if the block has been already pruned. | ||
| // Returns ErrBadLane if the lane map is gone (tipEpoch leave prune). |
| func (s *State) Block(ctx context.Context, lane types.LaneID, n types.BlockNumber) (*types.Signed[*types.LaneProposal], error) { | ||
| for inner, ctrl := range s.inner.Lock() { | ||
| if err := ctrl.WaitUntil(ctx, func() bool { | ||
| q, ok := inner.blocks[lane] |
There was a problem hiding this comment.
it might be missing, because it is from the future though. Use Epoch.IsClosed(lane).
| } | ||
| if err := ctrl.WaitUntil(ctx, func() bool { | ||
| return h.BlockNumber() <= min(q.next, inner.persistedBlockStart[h.Lane()]+BlocksPerLane-1) | ||
| q, ok := inner.blocks[lane] |
There was a problem hiding this comment.
ditto, you should use IsClosed() here to distinguish future lanes from closed lanes.
|
|
||
| // headers collects headers for the given range. | ||
| // Missing vote queue (leave map dropped past AppQC floor) → ErrPruned so PushQC can skip; | ||
| // ErrBadLane would kill avail.Run. |
There was a problem hiding this comment.
do we really need to distinguish BadLane from Pruned in general? We technically can distinguish pruned lanes from bad lanes in case eJoin == current epoch, but this is a special case.
Again, this should be blocking in case LaneRange.ID is from the future.
| // ErrBadLane if the lane left committee or its map was tipEpoch-pruned while waiting. | ||
| // Presence is keyed off blocks (always set for live lanes); persistedBlockStart may | ||
| // be absent on a fresh start (zero start), which is not a prune. | ||
| func (s *State) WaitForLocalCapacity(ctx context.Context, lane types.LaneID, toProduce types.BlockNumber) error { |
There was a problem hiding this comment.
just "WaitForCapacity", if you are providing the LaneID anyway
There was a problem hiding this comment.
again, block if LaneID is from the future.
| blocksByLane[lane] = append(blocksByLane[lane], proposal) | ||
| } | ||
|
|
||
| active := s.epoch.Load().Committee() |
There was a problem hiding this comment.
"committee"/"activeCommittee"?
| // TODO: when epoch transitions land, also union in lanes from all | ||
| // epochs that appear in batch.commitQCs so new-epoch lanes are | ||
| // never skipped in a cross-epoch batch. | ||
| // TODO(#3736): only lanes of the latest CommitQC's epoch are |
| pruneAnchor utils.Option[*PruneAnchor] | ||
| // staleLeave: tipEpoch-disposable map keys skipped for append this tick. | ||
| // WAL deleted then maps dropped after Parallel (same iteration). | ||
| staleLeave []types.LaneID |
There was a problem hiding this comment.
IMO this should be derived from blocks and persister state within runPersist
| // runPersist's Parallel batch returns for tip-stale leave maps. | ||
| // | ||
| // No-op if the lane WAL is not open (never created, or already deleted). | ||
| func (bp *BlockPersister) DeleteLane(lane types.LaneID) error { |
There was a problem hiding this comment.
nit: IMO it would be a cleaner API to have a separate idempotent (so that most of the time it is a fast noop) method BlockPersister.UpdateLanes(epoch) which would prune old/add new WALs. And only then do persisting of the lanes. One quirk will be that the CommitQCs of the previous epoch need to be persisted BEFORE old lanes are pruned.
|
|
||
| type mempool struct { | ||
| capacity uint64 | ||
| lane utils.Option[types.LaneID] |
There was a problem hiding this comment.
wouldn't it be better for the whole mempool to be optional instead of the lane inside? Closed lane invalidates the mempool anyway.
| } | ||
|
|
||
| // clearMempool wipes pending txs and session lane so InsertTx rejects until alignMempool. | ||
| func (s *State) clearMempool() { |
There was a problem hiding this comment.
that's what I'm talking about. Mempool should just not exist in epochs without local lane.
| // sessionOpErr maps leave ErrBadLane → Canceled so Run can WaitProduce again. | ||
| func (s *State) sessionOpErr(lane types.LaneID, op string, err error) error { | ||
| if errors.Is(err, avail.ErrBadLane) { | ||
| if got, ok := s.consensus.Avail().LocalLane().Get(); !ok || got != lane { |
There was a problem hiding this comment.
imo you can just do Avail().epoch.Iter(func() { produceSession() }) here
| return ctx.Err() | ||
| } | ||
|
|
||
| func (s *State) produceSession(ctx context.Context, availState *avail.State, lane types.LaneID) error { |
| option (hashable.hashable) = true; | ||
| option (wireguard.sized) = true; | ||
| optional PublicKey lane = 1; // required | ||
| // Field 1 was PublicKey "lane"; LaneID is additive on a new number/name so |
| optional uint64 first = 2; // required | ||
| optional uint64 next = 3; // required | ||
| optional bytes last_hash = 4 [(wireguard.max_size) = 32]; // required | ||
| optional LaneID lane_id = 5; // required |
There was a problem hiding this comment.
lane is it the top level selector, despite it having highest tag. I'd put it on top (where "lane" used to be)
| sub := x.validatorState().Avail().SubscribeLaneProposals(req.FirstBlockNumber) | ||
| // ErrLanePruned ends the stream; leave alone keeps serving. Do not bubble — | ||
| // wait and resubscribe (rejoin tip is 0; back-leash prunes before rejoin). | ||
| first := req.FirstBlockNumber |
There was a problem hiding this comment.
make request include full LaneID
There was a problem hiding this comment.
then there won't be no need for specifically waiting for SOME local lane: first verify that LaneID is actually local (by comparing the key), then avail.Block() will take care of distinguishing closed lanes from future lanes
There was a problem hiding this comment.
RPC should gracefully terminate (without error) in case the lane has been closed. Client should wait for a new lane of this producer in this case (i.e. observe its avail.epoch)
Document replica vs lane order on Committee. Use Option.Or and cmp.Or, document normalizeWeights, take test lanes from the committee, and put lane_id first in proto source (tag unchanged). Co-authored-by: Cursor <cursoragent@cursor.com>
CI lint regenerates protos and diffs; field declaration order in autobahn.proto changed the generated Go descriptors. Co-authored-by: Cursor <cursoragent@cursor.com>
2 participants
Summary
LaneID = PublicKeywithLaneID = (validator, e_join): stay keepse_join, leave is terminal for that identity, rejoin allocates a new LaneID (tip fromNextBlock, typically 0 for a fresh map).ApplyEpochseeds joiner lane maps; leavers stay in memory/WAL until tipEpoch (first retained CommitQC) omits them (staleLaneDisposable:e_join < tipand not in tip committee), thenDeleteLane+ map drop. Persist still flushes leave tips when proposals are non-empty before the first WAL open (allowCreatedecided in avail, not persist).WaitProduce/WaitMustStop; leave clears mempool and rejects inserts (ErrNotProducing).SubscribeLaneProposalsbinds lane at subscribe and keeps serving until tipEpoch prune (ErrLanePruned); giga pauses and resubscribes without tearing down peer RPC.Compatibility / ops
persistent_state_dir: WAL dirs arehex(pubkey||e_join)(washex(pubkey)), andBlockHeader/LaneRangewire the newLaneIDmessage (field 5; old field 1lanereserved). Pre-LaneID state does not migrate — wipe or coordinated reset before upgrade. Autobahn is opt-in / pre-launch; no mixed-version peers.Multi-epoch (#3736)
Production
ApplyEpoch/ActivateEpochwiring, neighborVerifyInWindow, and accepting prior-epochCommitQCwhile tip lags land in #3736. This PR ships LaneID + leave/rejoin scaffolding and unit coverage only; do not expect end-to-end multi-epoch production paths here.Made with Cursor