[RAI-2943] Bound the orchestrator-approve receipt wait and return the tx hash

SPEC.md

@@ -1459,7 +1459,10 @@ against the local SQLite store, and is where future issuer actions (e.g. `mint`,
   token) through the Turnkey signer, after verifying the configured address
   answers as an orchestrator. Idempotent — an already-unlimited allowance sends
   nothing — and success is re-verified by an on-chain allowance read, never
-  inferred from the receipt.
+  inferred from the receipt. It signs the approval before broadcasting it and
+  waits up to 5 minutes for a verified result (no gateway sits in front of it);
+  an approval still unconfirmed then fails with its hash and a non-zero exit, so
+  a script does not proceed as if the allowance were in place.
 - `issuer verify-orchestrator-signing <UNDERLYING>` — signs, WITHOUT
   broadcasting, one transaction per shape the Turnkey signing policy must allow
   before cutover (`orchestrator.mint`, `orchestrator.burn`, `vault.approve`,
@@ -4960,6 +4963,49 @@ Tiers and routes:
 Freezing gates token supply, so freeze/unfreeze are **capital**, not debug: a
 debug identity cannot freeze, burn excess, force-complete, or close.
 
+`orchestrator-approve` runs against one 18-second deadline from when the route
+starts, after IAP verification (which may take up to 10 seconds fetching
+Google's keys). It covers the vault lookup, the signer, the orchestrator checks,
+the wait for the service wallet lock, the broadcast, the receipt, and the
+allowance re-read, so the bot's own verdict arrives before a load balancer's
+default 30-second backend timeout and the client's request timeout. The approval
+is signed before it is broadcast, so its hash is known from then on. It responds
+with `{ outcome, tx_hash }`:
+
+- `already_unlimited` (200, `tx_hash` null): nothing was sent.
+- `approved` (200): the receipt succeeded and the re-read allowance is
+  unlimited.
+- `submitted_unconfirmed` (202): the approval was signed and broadcast (or its
+  broadcast's answer was lost, cut off, "already known", or an error that does
+  not prove a refusal), but the receipt and the allowance re-read did not both
+  come back within the deadline (failed reads are retried until then), so it may
+  still land. The operator looks `tx_hash` up on chain instead of retrying: once
+  it lands a retry sends nothing; if the chain does not know it, nothing was
+  sent and a retry is safe. The wallet lock is released at the deadline, as a
+  live mint's is after broadcast. If the approval then stays pending without
+  mining, it keeps its nonce and later mints and burns on that network queue
+  behind it until it mines or the operator replaces that nonce through Turnkey;
+  re-running the approve only adds another transaction behind it.
+- 404: no vault is listed for that symbol on that network.
+- 422, nothing signed or sent: the network is unknown or the symbol empty, the
+  network has no `[orchestrator.addresses]` entry or no vault service, the
+  signer is not Turnkey, or the configured orchestrator did not verify
+  (`vaultLogicIsExpected()` false). The bot's warning for the request says
+  which; the operator fixes it before retrying.
+- 422, refused: the node answered the broadcast with an error that proves the
+  pool refused it (insufficient funds, underpriced, nonce too low, a malformed
+  request) and a lookup found no such transaction. The bot logs the node's
+  reason with `tx_hash`; nothing is pending and retrying after fixing the cause
+  is safe.
+- 409: an unresolved mint or redemption signer intent holds the network's wallet
+  nonce; nothing was signed or sent, and the approve is re-run once that flow
+  resolves.
+- 503: the deadline passed before the broadcast (for example the wallet lock
+  stayed held by a live mint or burn); nothing was sent and a retry is safe.
+
+The client prints the 202 body and exits 0 like any success, so a script must
+branch on `outcome`, not the exit code.
+
 Both `burn-excess` routes respond with `{ executed, outcome }`: `executed`
 echoes whether a mutation was requested, and `outcome` is a tagged `plan`,
 `terminal`, or `close` view. For a `Start` or `Resume` request with
@@ -5124,16 +5170,20 @@ outage during sign-in. Failures are explained:
   (no `OPS_API_*_AUDIENCE` configured).
 - **503:** the deployment could not serve the request, for example the bot could
   not fetch Google's IAP keys, a burn-excess transfer poller could not be paused
-  in time (it did not park, or another breakglass run held it), or the load
-  balancer had no healthy backend. A read can be retried; before retrying a
-  write, the logs show whether it was applied.
+  in time (it did not park, or another breakglass run held it), an
+  `approve-orchestrator` ran out of time before broadcasting anything (nothing
+  sent), or the load balancer had no healthy backend. A read can be retried;
+  before retrying a write, the logs show whether it was applied.
 - **502 or 504:** a gateway gave up waiting, either the load balancer's backend
-  timeout (an `approve-orchestrator` waiting for its receipt, or a long
-  burn-excess run, can outlast it) or the bot's own bound (its wait on the
-  chain, or a burn-excess run's), so the outcome is unknown. A read can be
-  retried. A burn-excess dry-run changed nothing and an `--execute` is resumed
-  by re-running the same command, which reads the persisted burn stream; for any
-  other write, the logs show whether it was applied before a retry.
+  timeout (a long burn-excess run can outlast it) or the bot's own bound (its
+  wait on the chain, or a burn-excess run's), so the outcome is unknown. An
+  `approve-orchestrator` also answers 502 when an RPC call failed, Turnkey
+  failed to sign it (nothing sent), its approval reverted, or it landed but the
+  re-read allowance is not unlimited; the bot's log for the request says which.
+  A read can be retried. A burn-excess dry-run changed nothing and an
+  `--execute` is resumed by re-running the same command, which reads the
+  persisted burn stream; for any other write, the logs show whether it was
+  applied before a retry.
 - **No response:** a connection that timed out or dropped after the request was
   sent leaves the outcome unknown, and the client says so.
 - **Unwritable output:** the bot answered with success but stdout could not take

crates/ops-client/src/transport.rs

@@ -23,9 +23,9 @@ use url::Url;
 /// bot caps the engine run at 120 s and answers 504. Cutting the request
 /// earlier would report a transport failure for a run whose outcome the bot is
 /// about to state. That holds only while the load balancer's backend timeout
-/// for the breakglass tier exceeds those 150 s. An orchestrator approval waits
-/// for its on-chain receipt and can still outlast it, which surfaces as
-/// `NoResponse`: the outcome is unknown and the logs say what happened.
+/// for the breakglass tier exceeds those 150 s. An orchestrator approval
+/// answers within its own 18 s deadline (202 with the transaction hash when
+/// not yet confirmed), well inside this bound.
 const REQUEST_TIMEOUT: Duration = Duration::from_secs(180);
 const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
 
@@ -134,9 +134,10 @@ pub(crate) enum TransportError {
         "HTTP 503 Service Unavailable: the deployment could not serve the \
          request (for example the bot could not fetch Google's IAP keys, a \
          burn-excess transfer poller could not be paused in time because it \
-         did not park or another breakglass run held it, or the load balancer \
-         had no healthy backend). A read can be retried; before retrying a \
-         write, check the logs for whether it was applied.{}",
+         did not park or another breakglass run held it, an \
+         approve-orchestrator ran out of time before broadcasting anything, \
+         or the load balancer had no healthy backend). A read can be retried; \
+         before retrying a write, check the logs for whether it was applied.{}",
         server_said(.body)
     )]
     Unavailable { body: String },
@@ -147,10 +148,13 @@ pub(crate) enum TransportError {
     #[error(
         "HTTP {status}: a gateway gave up waiting for the bot (the bot's own \
          bound, or the load balancer in front of it), so the outcome is \
-         unknown. A read can be retried. A burn-excess dry-run changed \
-         nothing, and re-running the same command for a burn-excess --execute \
-         reads its persisted state and resumes it; for any other write, check \
-         the logs before retrying.{}",
+         unknown. An approve-orchestrator also answers 502 when an RPC read \
+         or its Turnkey signing failed, or its approval reverted or did not \
+         take effect; its log says which. A read can be \
+         retried. A burn-excess dry-run changed nothing, and re-running the \
+         same command for a burn-excess --execute reads its persisted state \
+         and resumes it; for any other write, check the logs before \
+         retrying.{}",
         server_said(.body)
     )]
     OutcomeUnknown { status: StatusCode, body: String },

docs/runbooks/orchestrator-onboarding.md

@@ -227,6 +227,14 @@ read. When this step actually submits, the transaction is also live proof that
 the policy's `approve` allowance works; the idempotent no-op path proves nothing
 new — step 4's signing proof covers `approve` in that case.
 
+If it fails with "submitted, unconfirmed", the approval was signed but not
+confirmed within 5 minutes. Look the printed hash up on the explorer before
+doing anything else: once it lands, a re-run sends nothing; if the explorer does
+not know it, nothing was sent and a re-run is safe. If it stays pending without
+mining, it holds the wallet's nonce and later mints and burns on that network
+queue behind it until it mines or that nonce is replaced through Turnkey;
+re-running the approval only adds another transaction behind it.
+
 Record each executed approval in the table below.
 
 ## 6. Final gate: preflight must print READY

src/tokenized_asset/cli.rs

@@ -11,6 +11,7 @@ use std::path::{Path, PathBuf};
 use std::str::FromStr;
 use std::sync::Arc;
 use std::time::Duration;
+use tokio::time::Instant;
 use url::Url;
 
 use super::view::{
@@ -795,11 +796,20 @@ async fn run_orchestrator_preflight(
     Ok(())
 }
 
+/// The offline approval's budget, from its allowance read through the fill,
+/// the Turnkey signature, the broadcast, the receipt, and the allowance
+/// re-read; running out before the broadcast fails with nothing sent. No
+/// gateway sits in front of the CLI, so it outwaits a congested block rather
+/// than matching the `/ops` route's request budget.
+const OFFLINE_APPROVAL_DEADLINE: Duration = Duration::from_secs(300);
+
 /// Executes one asset's one-time unlimited approval through the Turnkey
 /// signer, after an explicit confirmation naming the asset, vault,
 /// orchestrator, and wallet. The mutation itself is idempotent
 /// ([`ensure_unlimited_approval`]), so a re-run after a completed approval
-/// reports "already unlimited" instead of sending again.
+/// reports "already unlimited" instead of sending again. An approval still
+/// unconfirmed at [`OFFLINE_APPROVAL_DEADLINE`] fails with its hash, so a
+/// script never proceeds as if the allowance were in place.
 async fn run_approve_orchestrator(
     args: ApproveOrchestratorArgs,
     confirm: impl Fn(&str) -> io::Result<bool>,
@@ -873,7 +883,17 @@ async fn run_approve_orchestrator(
         );
     }
 
-    match ensure_unlimited_approval(&provider, vault, orchestrator, bot).await?
+    let deadline = Instant::now()
+        .checked_add(OFFLINE_APPROVAL_DEADLINE)
+        .ok_or_else(|| anyhow::anyhow!("approval deadline overflows"))?;
+    match ensure_unlimited_approval(
+        &provider,
+        vault,
+        orchestrator,
+        bot,
+        deadline,
+    )
+    .await?
     {
         ApprovalOutcome::AlreadyUnlimited => println!(
             "Already unlimited: {} vault {vault} needs no approval \
@@ -885,6 +905,17 @@ async fn run_approve_orchestrator(
              on {} vault {vault} in {tx_hash}.",
             args.underlying
         ),
+        ApprovalOutcome::SubmittedUnconfirmed { tx_hash } => anyhow::bail!(
+            "submitted, unconfirmed: {tx_hash} was signed for {} vault \
+             {vault} but not confirmed within {}s. Look it up on chain before \
+             re-running: once it lands a re-run sends nothing; if the chain \
+             does not know it, nothing was sent and a re-run is safe; if it \
+             stays pending, later mints and burns on this network queue \
+             behind its nonce until it is replaced through Turnkey, and a \
+             re-run only adds another transaction behind it.",
+            args.underlying,
+            OFFLINE_APPROVAL_DEADLINE.as_secs()
+        ),
     }
     println!("Run orchestrator-preflight to verify overall readiness.");