fix(publisher): resolve held lift jobs from chain proof — the check-chain dispatcher (GH#2270, part 3)

[Jurij89]

Summary

• Part 3 of the feat(publisher): automatically execute resolution-aware retries for all retryable lift failures #2270 chain (base: part 2 — chained; retarget as the chain merges). Part 2 holds every job whose transaction is unaccounted for and refuses to re-publish it; this part builds the lane that RESOLVES those holds from chain state — the proof-first dispatcher. PR-2's blockedPendingRecovery population is exactly this lane's work queue.
• The chain can now tell five truths about a publish transaction. resolvePublishTransaction (a second adapter entry point beside the receipt-only resolvePublishByTxHash, which keeps its non-recovery consumer in agent sync) returns confirmed / reverted / unrecognized / pending / not-found. Its not-found is earned three times over (review rounds 1-2): with no receipt it asks the node for the TRANSACTION, so a mempool tx answers pending — and even then, a null lookup is point-in-time, backend-local evidence, so the runner releases nothing on it alone. Release by absence needs (1) the transaction missing, (2) its NONCE — recorded at signing via a typed, awaited onBeforeBroadcast({txHash, nonce}) write-ahead callback (zero extra RPC; onPhase stays pure instrumentation) — spent at a FINALIZED block by something else, so the recorded hash can never mine, and (3) the knowledge-asset id the job would mint provably absent on chain (one ownerOf eth_call against the id the job's seal pins), because a replacement transaction on the same nonce slot could have performed the very same publish and the nonce alone cannot tell. A job that would ALLOCATE a fresh id on re-run — unsealed raw lift, where a re-run would double-mint — has nothing to ask and is never released by absence. Every gap (no recorded nonce, no pinned id, no adapter support for finalized reads, a throw) fails closed to inconclusive. The runner's createChainProofResolver adds a sixth, fail-closed member: inconclusive, where every unknown collapses (RPC error, unmapped result, and — deliberately — a null from a legacy adapter that cannot see the mempool, which may therefore never authorise a resend).
• The dispatcher runs on the recover() cadence and acts only on answers. Eligibility for the named lane is isHeldForChainProof — the same predicate that holds the job everywhere else, now with five surfaces: four that refuse to move a held job and one that resolves it. Per verdict: recovered → finalize the SAME job (both evidence carriers rebuild through one path — a recovery-carrier-only job gets its broadcast record rebuilt from the evidence that held it, a bug our own matrix caught before review); not-found → evidence-preserving reset on the same jobId (the checked hash rides along, so a later failure is still held); reverted → WHOSE transaction decides: a job that sent it itself is re-recorded tx_reverted (the registry's proven-ineffective verdict — the KA is publishable again, the job is not blindly re-run on this node's money), while an inherited hash (this job failed pre-send; an earlier attempt sent the reverted tx) takes the same release as proven absence; pending/unrecognized/inconclusive → stay held, ask again next tick.
• No time-based escape — deliberately. The raw-lift recoveryLookupTimeoutMs gate bounds how long a LIVE broadcast may stay unresolved while holding its wallet; this lane starts after that declaration, where an expiry could only mean "no proof, resend anyway" — the double publish this chain exists to prevent. A held job is chased indefinitely at one chain read per tick; the exits are proof or the operator's by-id clear.
• pause() now gates the dispatcher (it re-queues work and spends chain reads). The interrupted-job half of recover() deliberately keeps running while paused — it reconciles transactions this node already sent — and that asymmetry is pinned by its own row.
• The 503 keeps its round-8 promise: with a real resolution lane in the build, LIFT_JOB_PENDING_CHAIN_PROOF responses flip back to retryable: true, and the message says chain recovery is chasing the job (by-id clear stays the impatient-operator exit).
• Round 4 taught the dispatcher about UPDATE jobs — and where absence proof must refuse to reach: the lookup carries the operation kind and intended update root; a mined update is recognized canonically through verifyKAUpdate (receipt + root chain-of-custody) and finalizes, while release-by-ABSENCE stays CREATE-ONLY — an update has no monotone register to prove absence against ("intended root is not current" also describes our update landing and being superseded, and a release would re-apply a stale root over newer state), so unproven updates hold with the by-id exit, enforced at both the resolver and the disposition table. Round 4 also made verdicts survive concurrency (the disposition re-reads the job under the queue's claim lock, so a verdict resolved mid-clear-job is dropped, never resurrected) and made the absence pair atomic (readFinalizedChainProofSnapshot: nonce + minted state read together at ONE pinned finalized block on ONE provider — a lagging fallback yields its own consistent pair, never a splice).
• Review round 3 found and fixed a live phantom-class bug on the named-KA UPDATE path: the queued update branch dropped the write-ahead callback at two hops (the publish branch spreads its option bag and carried it for free; the update branch names every field and silently lost it, and agent.update dropped it again), so every named-KA update transaction went out with NO durable pre-send record. Both hops now thread it, pinned end to end. Two more round-3 fixes: an ACCOUNTED hash (one the dispatcher itself proved absent or reverted) no longer re-holds the job after a later pre-send failure (txHashAccounted audit-preserving mark, written only by the dispatcher's release paths), and no fallible phase instrumentation can run between the durable write-ahead and the send (phases first, durable callback last — a rejecting listener can abort a send, never fabricate a broadcast).
• Raw lift takes the same pre-send write-ahead KA-VM publish has taken since publisher: model the KA pre-send broadcast boundary as a typed write-ahead outcome #1864 (one shared recorder, no second copy): the signed txHash is durably recorded before the send, so a crash in the send window leaves a held job with evidence instead of a phantom that recovery would re-broadcast under a fresh hash. Named behavior delta: a raw-lift job that fails after the write-ahead now carries broadcast.txHash, so PR-2 holds it — intended, and the dispatcher is what resolves it.
• tx_reverted is now reachable from included (registry allowed-states widened, pre-send states still excluded): a reorg can replace an included job's transaction with a failing one, and the dispatcher records that truthfully.

Related

• GH#2270 (part 3 of 3; part 1 = the automatic allow-list lane fix(publisher): execute automatic retries for provably pre-send lift failures (GH#2270, part 1) #2291, part 2 = evidence-safe manual paths fix(publisher): never republish a lift job that may have sent a transaction (GH#2270, part 2) #2293)
• Supersedes fix(publisher): resolve held lift jobs from chain proof — the check-chain dispatcher (GH#2270, part 3) #2297 (accidentally merged into its base before review converged; the merge was unwound and review rounds 1-4 remain readable there — all four rounds are APPLIED in this branch, round 4 in the two newest commits)
• GH#1864 (the KA-VM pre-send write-ahead this PR extends to raw lift)
• GH#1620 (the recover() cadence the dispatcher rides)

Diagrams

A held job whose transaction the chain can account for

Before:

sequenceDiagram
    participant Job as Held failed job with txHash
    participant Recover as the recover loop
    participant Chain
    participant Operator
    Recover->>Job: canRetryFailedRecovery is false, never asked
    Note over Job: waits forever
    Operator->>Job: clear-job by id is the only exit

Loading

After:

sequenceDiagram
    participant Job as Held failed job with txHash
    participant Recover as the recover loop
    participant Chain
    Recover->>Chain: resolvePublishTransaction for the held txHash
    alt confirmed
        Chain-->>Recover: publish found
        Recover->>Job: finalize the SAME job
    else proven absent by all three, missing tx and spent nonce and unminted id
        Chain-->>Recover: proven absence
        Recover->>Job: evidence-preserving reset, same jobId re-runs
    else reverted
        Chain-->>Recover: failure receipt
            Recover->>Job: own tx re-recorded as tx_reverted, inherited tx reset
    else pending or inconclusive
        Chain-->>Recover: nothing established
        Recover->>Job: stay held, ask next tick
    end

Loading

Raw-lift send window

Before:

sequenceDiagram
    participant Job
    participant Publisher
    participant Chain
    Publisher->>Chain: sign and send, no durable record
    Note over Job: crash here leaves status validated, no txHash
    Publisher->>Job: the recover loop resets to accepted
    Publisher->>Chain: re-broadcast under a NEW hash
    Note over Chain: possible double publish

Loading

After:

sequenceDiagram
    participant Job
    participant Publisher
    participant Chain
    Publisher->>Job: durable broadcast record with signed txHash
    Publisher->>Chain: send
    Note over Job: crash here leaves a held job carrying the hash
    Publisher->>Chain: dispatcher asks about THAT hash
    Chain-->>Publisher: verdict decides finalize, reset, or hold

Loading

Files changed

File	What
packages/chain/src/chain-adapter.ts	PublishTransactionResolution five-state union; resolvePublishTransaction? on the adapter interface, with the second-entry-point rationale
packages/chain/src/evm-adapter-publish.ts	tri-state receipt resolution; asks for the transaction when there is no receipt; RPC failure throws rather than resolving
packages/chain/src/mock-adapter.ts	transactionStates seam so tests can declare pending vs not-found (the mock previously read every unknown hash as absent); getFinalizedAccountNonce parity
packages/chain/src/evm-adapter-base.ts	typed PreBroadcastSignal write-ahead callback awaited before the send (nonce off the signed transaction, zero extra RPC); readFinalizedChainProofSnapshot is the ONE minted/nonce surface (granular methods deleted; revert classification adapter-private); every mined verdict gated behind receipt-block finality + canonicality
packages/publisher/src/async-lift-publisher-types.ts	config field renamed chainProofResolver (typed lookup contract); the legacy chainRecoveryResolver key is REJECTED at construction with an error naming the replacement — loud break, no shim
packages/cli/src/publisher-chain-proof.ts	the chain-proof policy module (adapter map, createChainProofResolver, nonce + identity proofs, result mapping); legacy-null → inconclusive; publisher-runner.ts stays the composition root
packages/cli/src/daemon/routes/knowledge-assets.ts	503 retryable flipped back to true with the chase wording
packages/publisher/src/async-lift-publisher-types.ts	publisher-owned AsyncLiftChainProofResolution; resolver contract returns verdicts
packages/publisher/src/async-lift-publisher-impl.ts	dispatchFailedJobsOnChainProof (verdict switch, never-guard, paused gate, no-time-escape doc); shared createPreSendBroadcastRecorder now wraps raw-lift sends; finalizeProvenKnowledgeAssetVmPublish shared by both lanes; KA-VM canRetryFailedRecovery = isHeldForChainProof
packages/publisher/src/lift-job-failures.ts	tx_reverted allowed from included (reorg truth); pre-send states stay excluded
packages/publisher/src/async-lift-retry-disposition.ts	isHeldForChainProof doc names its five surfaces
packages/cli/skills/dkg-node/SKILL.md	held jobs now resolve without operator action when the chain can answer; 503 wording
publisher tests (3 files + 1 new, registered) + chain tests (2 + 1 new) + cli tests (3)	verdict × carrier matrix, double-publish falsifier, pause asymmetry pair, cross-package boundary rows, mock discrimination rows, flip row

Test plan

• pnpm build:packages 23/23; clean-room tsc --noEmit (tsbuildinfo purged) on chain, publisher, cli, agent — zero errors
• Publisher unit lane 55 files / 664 tests (base 54/645, +1 file/+19 exactly the new suite); chain unit 51 / 973+1; publisher FULL hardhat lane run ALONE 131 files / 1916+6 green
• CLI: affected suites green (84 rows re-verified by lead); full-lane red byte-identical to the environmental baseline re-established at da3cf07
• 13 mutants across both chunks, serial and reverted, each killed by named rows — including the cross-package pair (a real publisher driven by the real runner resolver: legacy-null must hold, tri-state not-found must release) and two against a real Hardhat chain; one survivor reported as equivalent (no non-recovered member carries recovery), with the meaningful direction killed instead
• Fail-before: the dispatch suite cannot compile at the parent (the contract does not exist there); the faithful stand-in — restoring canRetryFailedRecovery: () => false — turns every release/finalize row red (7/19), stated in-file; the raw-lift write-ahead fail-before is 5/6 rows with the no-regression row named
• Double-publish falsifier: a send counter on the handler is asserted 0 in every dispatcher row, including the full recover-and-finalize cycle
• Post-chain WSL testnet validation (scenarios A1/A2/B/C) before merge go-ahead

🤖 Generated with Claude Code