diff --git a/SPEC.md b/SPEC.md index f86ca02b..58fca942 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1459,7 +1459,10 @@ against the local SQLite store, and is where future issuer actions (e.g. `mint`, token) through the Turnkey signer, after verifying the configured address answers as an orchestrator. Idempotent — an already-unlimited allowance sends nothing — and success is re-verified by an on-chain allowance read, never - inferred from the receipt. + inferred from the receipt. It signs the approval before broadcasting it and + waits up to 5 minutes for a verified result (no gateway sits in front of it); + an approval still unconfirmed then fails with its hash and a non-zero exit, so + a script does not proceed as if the allowance were in place. - `issuer verify-orchestrator-signing ` — signs, WITHOUT broadcasting, one transaction per shape the Turnkey signing policy must allow before cutover (`orchestrator.mint`, `orchestrator.burn`, `vault.approve`, @@ -4960,6 +4963,49 @@ Tiers and routes: Freezing gates token supply, so freeze/unfreeze are **capital**, not debug: a debug identity cannot freeze, burn excess, force-complete, or close. +`orchestrator-approve` runs against one 18-second deadline from when the route +starts, after IAP verification (which may take up to 10 seconds fetching +Google's keys). It covers the vault lookup, the signer, the orchestrator checks, +the wait for the service wallet lock, the broadcast, the receipt, and the +allowance re-read, so the bot's own verdict arrives before a load balancer's +default 30-second backend timeout and the client's request timeout. The approval +is signed before it is broadcast, so its hash is known from then on. It responds +with `{ outcome, tx_hash }`: + +- `already_unlimited` (200, `tx_hash` null): nothing was sent. +- `approved` (200): the receipt succeeded and the re-read allowance is + unlimited. +- `submitted_unconfirmed` (202): the approval was signed and broadcast (or its + broadcast's answer was lost, cut off, "already known", or an error that does + not prove a refusal), but the receipt and the allowance re-read did not both + come back within the deadline (failed reads are retried until then), so it may + still land. The operator looks `tx_hash` up on chain instead of retrying: once + it lands a retry sends nothing; if the chain does not know it, nothing was + sent and a retry is safe. The wallet lock is released at the deadline, as a + live mint's is after broadcast. If the approval then stays pending without + mining, it keeps its nonce and later mints and burns on that network queue + behind it until it mines or the operator replaces that nonce through Turnkey; + re-running the approve only adds another transaction behind it. +- 404: no vault is listed for that symbol on that network. +- 422, nothing signed or sent: the network is unknown or the symbol empty, the + network has no `[orchestrator.addresses]` entry or no vault service, the + signer is not Turnkey, or the configured orchestrator did not verify + (`vaultLogicIsExpected()` false). The bot's warning for the request says + which; the operator fixes it before retrying. +- 422, refused: the node answered the broadcast with an error that proves the + pool refused it (insufficient funds, underpriced, nonce too low, a malformed + request) and a lookup found no such transaction. The bot logs the node's + reason with `tx_hash`; nothing is pending and retrying after fixing the cause + is safe. +- 409: an unresolved mint or redemption signer intent holds the network's wallet + nonce; nothing was signed or sent, and the approve is re-run once that flow + resolves. +- 503: the deadline passed before the broadcast (for example the wallet lock + stayed held by a live mint or burn); nothing was sent and a retry is safe. + +The client prints the 202 body and exits 0 like any success, so a script must +branch on `outcome`, not the exit code. + Both `burn-excess` routes respond with `{ executed, outcome }`: `executed` echoes whether a mutation was requested, and `outcome` is a tagged `plan`, `terminal`, or `close` view. For a `Start` or `Resume` request with @@ -5124,16 +5170,20 @@ outage during sign-in. Failures are explained: (no `OPS_API_*_AUDIENCE` configured). - **503:** the deployment could not serve the request, for example the bot could not fetch Google's IAP keys, a burn-excess transfer poller could not be paused - in time (it did not park, or another breakglass run held it), or the load - balancer had no healthy backend. A read can be retried; before retrying a - write, the logs show whether it was applied. + in time (it did not park, or another breakglass run held it), an + `approve-orchestrator` ran out of time before broadcasting anything (nothing + sent), or the load balancer had no healthy backend. A read can be retried; + before retrying a write, the logs show whether it was applied. - **502 or 504:** a gateway gave up waiting, either the load balancer's backend - timeout (an `approve-orchestrator` waiting for its receipt, or a long - burn-excess run, can outlast it) or the bot's own bound (its wait on the - chain, or a burn-excess run's), so the outcome is unknown. A read can be - retried. A burn-excess dry-run changed nothing and an `--execute` is resumed - by re-running the same command, which reads the persisted burn stream; for any - other write, the logs show whether it was applied before a retry. + timeout (a long burn-excess run can outlast it) or the bot's own bound (its + wait on the chain, or a burn-excess run's), so the outcome is unknown. An + `approve-orchestrator` also answers 502 when an RPC call failed, Turnkey + failed to sign it (nothing sent), its approval reverted, or it landed but the + re-read allowance is not unlimited; the bot's log for the request says which. + A read can be retried. A burn-excess dry-run changed nothing and an + `--execute` is resumed by re-running the same command, which reads the + persisted burn stream; for any other write, the logs show whether it was + applied before a retry. - **No response:** a connection that timed out or dropped after the request was sent leaves the outcome unknown, and the client says so. - **Unwritable output:** the bot answered with success but stdout could not take diff --git a/crates/ops-client/src/transport.rs b/crates/ops-client/src/transport.rs index 56884a90..6c6c1f23 100644 --- a/crates/ops-client/src/transport.rs +++ b/crates/ops-client/src/transport.rs @@ -23,9 +23,9 @@ use url::Url; /// bot caps the engine run at 120 s and answers 504. Cutting the request /// earlier would report a transport failure for a run whose outcome the bot is /// about to state. That holds only while the load balancer's backend timeout -/// for the breakglass tier exceeds those 150 s. An orchestrator approval waits -/// for its on-chain receipt and can still outlast it, which surfaces as -/// `NoResponse`: the outcome is unknown and the logs say what happened. +/// for the breakglass tier exceeds those 150 s. An orchestrator approval +/// answers within its own 18 s deadline (202 with the transaction hash when +/// not yet confirmed), well inside this bound. const REQUEST_TIMEOUT: Duration = Duration::from_secs(180); const CONNECT_TIMEOUT: Duration = Duration::from_secs(10); @@ -134,9 +134,10 @@ pub(crate) enum TransportError { "HTTP 503 Service Unavailable: the deployment could not serve the \ request (for example the bot could not fetch Google's IAP keys, a \ burn-excess transfer poller could not be paused in time because it \ - did not park or another breakglass run held it, or the load balancer \ - had no healthy backend). A read can be retried; before retrying a \ - write, check the logs for whether it was applied.{}", + did not park or another breakglass run held it, an \ + approve-orchestrator ran out of time before broadcasting anything, \ + or the load balancer had no healthy backend). A read can be retried; \ + before retrying a write, check the logs for whether it was applied.{}", server_said(.body) )] Unavailable { body: String }, @@ -147,10 +148,13 @@ pub(crate) enum TransportError { #[error( "HTTP {status}: a gateway gave up waiting for the bot (the bot's own \ bound, or the load balancer in front of it), so the outcome is \ - unknown. A read can be retried. A burn-excess dry-run changed \ - nothing, and re-running the same command for a burn-excess --execute \ - reads its persisted state and resumes it; for any other write, check \ - the logs before retrying.{}", + unknown. An approve-orchestrator also answers 502 when an RPC read \ + or its Turnkey signing failed, or its approval reverted or did not \ + take effect; its log says which. A read can be \ + retried. A burn-excess dry-run changed nothing, and re-running the \ + same command for a burn-excess --execute reads its persisted state \ + and resumes it; for any other write, check the logs before \ + retrying.{}", server_said(.body) )] OutcomeUnknown { status: StatusCode, body: String }, diff --git a/docs/runbooks/orchestrator-onboarding.md b/docs/runbooks/orchestrator-onboarding.md index bde0bbc0..17ade376 100644 --- a/docs/runbooks/orchestrator-onboarding.md +++ b/docs/runbooks/orchestrator-onboarding.md @@ -227,6 +227,14 @@ read. When this step actually submits, the transaction is also live proof that the policy's `approve` allowance works; the idempotent no-op path proves nothing new — step 4's signing proof covers `approve` in that case. +If it fails with "submitted, unconfirmed", the approval was signed but not +confirmed within 5 minutes. Look the printed hash up on the explorer before +doing anything else: once it lands, a re-run sends nothing; if the explorer does +not know it, nothing was sent and a re-run is safe. If it stays pending without +mining, it holds the wallet's nonce and later mints and burns on that network +queue behind it until it mines or that nonce is replaced through Turnkey; +re-running the approval only adds another transaction behind it. + Record each executed approval in the table below. ## 6. Final gate: preflight must print READY diff --git a/src/tokenized_asset/cli.rs b/src/tokenized_asset/cli.rs index a466224a..3b499478 100644 --- a/src/tokenized_asset/cli.rs +++ b/src/tokenized_asset/cli.rs @@ -11,6 +11,7 @@ use std::path::{Path, PathBuf}; use std::str::FromStr; use std::sync::Arc; use std::time::Duration; +use tokio::time::Instant; use url::Url; use super::view::{ @@ -795,11 +796,20 @@ async fn run_orchestrator_preflight( Ok(()) } +/// The offline approval's budget, from its allowance read through the fill, +/// the Turnkey signature, the broadcast, the receipt, and the allowance +/// re-read; running out before the broadcast fails with nothing sent. No +/// gateway sits in front of the CLI, so it outwaits a congested block rather +/// than matching the `/ops` route's request budget. +const OFFLINE_APPROVAL_DEADLINE: Duration = Duration::from_secs(300); + /// Executes one asset's one-time unlimited approval through the Turnkey /// signer, after an explicit confirmation naming the asset, vault, /// orchestrator, and wallet. The mutation itself is idempotent /// ([`ensure_unlimited_approval`]), so a re-run after a completed approval -/// reports "already unlimited" instead of sending again. +/// reports "already unlimited" instead of sending again. An approval still +/// unconfirmed at [`OFFLINE_APPROVAL_DEADLINE`] fails with its hash, so a +/// script never proceeds as if the allowance were in place. async fn run_approve_orchestrator( args: ApproveOrchestratorArgs, confirm: impl Fn(&str) -> io::Result, @@ -873,7 +883,17 @@ async fn run_approve_orchestrator( ); } - match ensure_unlimited_approval(&provider, vault, orchestrator, bot).await? + let deadline = Instant::now() + .checked_add(OFFLINE_APPROVAL_DEADLINE) + .ok_or_else(|| anyhow::anyhow!("approval deadline overflows"))?; + match ensure_unlimited_approval( + &provider, + vault, + orchestrator, + bot, + deadline, + ) + .await? { ApprovalOutcome::AlreadyUnlimited => println!( "Already unlimited: {} vault {vault} needs no approval \ @@ -885,6 +905,17 @@ async fn run_approve_orchestrator( on {} vault {vault} in {tx_hash}.", args.underlying ), + ApprovalOutcome::SubmittedUnconfirmed { tx_hash } => anyhow::bail!( + "submitted, unconfirmed: {tx_hash} was signed for {} vault \ + {vault} but not confirmed within {}s. Look it up on chain before \ + re-running: once it lands a re-run sends nothing; if the chain \ + does not know it, nothing was sent and a re-run is safe; if it \ + stays pending, later mints and burns on this network queue \ + behind its nonce until it is replaced through Turnkey, and a \ + re-run only adds another transaction behind it.", + args.underlying, + OFFLINE_APPROVAL_DEADLINE.as_secs() + ), } println!("Run orchestrator-preflight to verify overall readiness."); diff --git a/src/tokenized_asset/orchestrator_ops.rs b/src/tokenized_asset/orchestrator_ops.rs index 4ed3987c..303f8e6f 100644 --- a/src/tokenized_asset/orchestrator_ops.rs +++ b/src/tokenized_asset/orchestrator_ops.rs @@ -5,7 +5,7 @@ //! wallet from the Turnkey signer, the RPC from the per-network environment. use alloy::primitives::Address; -use alloy::providers::{Provider, ProviderBuilder}; +use alloy::providers::ProviderBuilder; use rocket::http::Status; use rocket::request::FromParam; use rocket::serde::json::Json; @@ -13,6 +13,7 @@ use rocket::{State, get, post}; use serde::Serialize; use sqlx::{Pool, Sqlite}; use std::str::FromStr; +use tokio::time::Instant; use tracing::{error, info, warn}; use super::api::UnderlyingParam; @@ -24,9 +25,9 @@ use crate::chain::{RpcEndpoint, rpc_client}; use crate::config::Config; use crate::mint::has_unresolved_signer_intent; use crate::vault::onboarding::{ - ApprovalOutcome, OnboardingError, OrchestratorReadiness, - check_orchestrator_readiness, ensure_unlimited_approval, - prove_signing_shapes, + APPROVAL_DEADLINE, APPROVAL_RECEIPT_POLL, ApprovalOutcome, OnboardingError, + OrchestratorReadiness, check_orchestrator_readiness, + ensure_unlimited_approval, prove_signing_shapes, }; use crate::vault::{NetworkVaultServices, VaultService}; use crate::wallet::SignerConfig; @@ -166,7 +167,11 @@ pub(crate) async fn orchestrator_verify_signing_ops( /// Capital-tier one-time unlimited approval of an asset's vault shares to the /// orchestrator, signed and broadcast through Turnkey. Idempotent: an already /// unlimited allowance sends nothing. Refuses if the configured address does -/// not verify as a healthy orchestrator. +/// not verify as a healthy orchestrator. The whole request runs against one +/// [`APPROVAL_DEADLINE`] from its start: running out before the broadcast +/// answers 503 with nothing sent, and an approval signed but not confirmed in +/// time answers 202 with its hash, so the operator reconciles it on chain +/// rather than sending a second approval. #[post("/ops/capital/orchestrator-approve//")] #[tracing::instrument( target = "auth", @@ -181,46 +186,65 @@ pub(crate) async fn orchestrator_approve_ops( vault_services: &State, network: &str, underlying: UnderlyingParam, -) -> Result, Status> { +) -> Result<(Status, Json), Status> { + let deadline = Instant::now() + .checked_add(APPROVAL_DEADLINE) + .ok_or(Status::InternalServerError)?; let network = parse_network(network)?; let UnderlyingParam(symbol) = underlying; let OrchestratorContext { orchestrator, bot, rpc, chain_id, turnkey } = resolve_orchestrator_context(config.inner(), network)?; - let vault = find_vault(pool.inner(), &symbol, &network) - .await - .map_err(|error| { - error!(target: "asset", %error, "Failed to look up vault"); - Status::InternalServerError - })? - .ok_or(Status::NotFound)?; - - let resolved = - resolve_turnkey_signer(turnkey, chain_id).await.map_err(|error| { - error!(target: "asset", %error, "Failed to resolve Turnkey signer"); - Status::InternalServerError - })?; + let vault = before_deadline( + deadline, + "vault lookup", + find_vault(pool.inner(), &symbol, &network), + ) + .await? + .map_err(|error| { + error!(target: "asset", %error, "Failed to look up vault"); + Status::InternalServerError + })? + .ok_or(Status::NotFound)?; + + let resolved = before_deadline( + deadline, + "turnkey signer", + resolve_turnkey_signer(turnkey, chain_id), + ) + .await? + .map_err(|error| { + error!(target: "asset", %error, "Failed to resolve Turnkey signer"); + Status::InternalServerError + })?; + // A remote RPC client polls receipts every 7 s by default, which would + // spend most of the approval's deadline between two looks at a receipt + // that landed in the first block or two. + let rpc_client = rpc_client(&rpc).map_err(|error| { + error!(target: "asset", %error, "approve could not connect to RPC"); + Status::BadGateway + })?; let provider = ProviderBuilder::new() .with_chain_id(chain_id) .wallet(resolved.wallet) - .connect_client(rpc_client(&rpc).map_err(|error| { - error!(target: "asset", %error, "approve could not connect to RPC"); - Status::BadGateway - })?); + .connect_client(rpc_client.with_poll_interval(APPROVAL_RECEIPT_POLL)); // The spender is about to receive an unlimited allowance from the // production wallet, so prove the address is a healthy orchestrator first: // a stale or typo'd entry fails these reads or reports vault logic false. - let readiness = - check_orchestrator_readiness(&provider, orchestrator, bot, &[]) - .await - .map_err(|error| { - error!(target: "asset", %orchestrator, %error, - "Refusing approve: orchestrator could not be verified" - ); - map_onboarding_error(&error) - })?; + let readiness = before_deadline( + deadline, + "orchestrator readiness", + check_orchestrator_readiness(&provider, orchestrator, bot, &[]), + ) + .await? + .map_err(|error| { + error!(target: "asset", %orchestrator, %error, + "Refusing approve: orchestrator could not be verified" + ); + map_onboarding_error(&error) + })?; if !readiness.vault_logic_expected { warn!(target: "asset", %orchestrator, "Refusing approve: vaultLogicIsExpected() is false" @@ -236,73 +260,138 @@ pub(crate) async fn orchestrator_approve_ops( vault_service.as_ref(), pool.inner(), network, - &provider, - vault, - orchestrator, - bot, + deadline, + async { + Box::pin(ensure_unlimited_approval( + &provider, + vault, + orchestrator, + bot, + deadline, + )) + .await + .map_err(|error| { + log_approval_failure(&error, orchestrator, vault); + map_onboarding_error(&error) + }) + }, ) .await?; - let response = match outcome { - ApprovalOutcome::AlreadyUnlimited => { - ApproveResponse { outcome: "already_unlimited", tx_hash: None } - } - ApprovalOutcome::Approved { tx_hash } => ApproveResponse { - outcome: "approved", - tx_hash: Some(tx_hash.to_string()), - }, - }; + let (status, response) = approve_reply(outcome); info!(target: "asset", %orchestrator, %vault, outcome = response.outcome, - "Orchestrator approval settled" + tx_hash = ?response.tx_hash, "Orchestrator approval settled" ); - Ok(Json(response)) + Ok((status, Json(response))) } -/// Broadcasts the approval from the production wallet under the network's -/// service wallet lock, the one every live mint and redemption burn holds -/// while it checks intents and signs. The lock is taken before the intent -/// check so no live flow can persist a signed nonce between the check and -/// this broadcast; while it is held no live flow fills a nonce either, so the -/// pending count this route's provider reads is the one to use. The lock is -/// held until the receipt is in, since the service's nonce manager resyncs -/// from the chain's pending count and must not observe this transaction only -/// halfway through. Refuses with 409 while an unresolved mint or redemption -/// signer intent already holds a signed nonce on this network: both would -/// fill from the same pending nonce and one would fail nonce-too-low, leaving -/// the bot's recovery to reconcile a submission it did not make. -async fn approve_under_wallet_lock( +/// Logs why the approval did not settle: running out of time before the +/// broadcast is the same refusal the pre-broadcast steps log (WARN, nothing +/// sent); anything else is a failed approval (ERROR, with the node's or +/// chain's reason). +fn log_approval_failure( + error: &OnboardingError, + orchestrator: Address, + vault: Address, +) { + if matches!(error, OnboardingError::DeadlineBeforeBroadcast) { + warn!(target: "asset", step = "approval signing", %orchestrator, %vault, + "Refusing approve: deadline passed before broadcast; nothing sent" + ); + } else { + error!(target: "asset", %orchestrator, %vault, %error, + "Orchestrator approval failed" + ); + } +} + +/// The approval's HTTP answer: 200 for a verified or already-unlimited +/// allowance, 202 for one signed but not confirmed, which may still land. +fn approve_reply(outcome: ApprovalOutcome) -> (Status, ApproveResponse) { + match outcome { + ApprovalOutcome::AlreadyUnlimited => ( + Status::Ok, + ApproveResponse { outcome: "already_unlimited", tx_hash: None }, + ), + ApprovalOutcome::Approved { tx_hash } => ( + Status::Ok, + ApproveResponse { + outcome: "approved", + tx_hash: Some(tx_hash.to_string()), + }, + ), + ApprovalOutcome::SubmittedUnconfirmed { tx_hash } => ( + Status::Accepted, + ApproveResponse { + outcome: "submitted_unconfirmed", + tx_hash: Some(tx_hash.to_string()), + }, + ), + } +} + +/// Runs a pre-broadcast `step` of the approval against its deadline: one that +/// would overrun it answers 503, since nothing has been sent yet and a retry +/// is safe. `step` names it in the log. +async fn before_deadline( + deadline: Instant, + step: &'static str, + work: impl Future, +) -> Result { + tokio::time::timeout_at(deadline, work).await.map_err(|_| { + warn!(target: "asset", step, + "Refusing approve: deadline passed before broadcast; nothing sent" + ); + Status::ServiceUnavailable + }) +} + +/// Runs `approval` under the network's service wallet lock, the lock every +/// live mint and redemption burn holds while it checks intents and signs. The +/// lock is taken before the intent check so no live flow can persist a signed +/// nonce between the check and the broadcast; while it is held no live flow +/// fills a nonce either, so the pending count the route's provider reads is +/// the one to use. Waiting for the lock and the intent check count against +/// the approval's deadline (503 on elapse, nothing sent). The lock is released +/// when `approval` returns, at the latest at the deadline. An approval still +/// pending then keeps its nonce: the service's nonce manager fills the next +/// live flow after it from the node's pending count, so until it mines (or the +/// operator replaces it) later mints and burns queue behind it. Refuses with +/// 409 while an unresolved mint or redemption signer intent already holds a +/// signed nonce on this network: both would fill from the same pending nonce +/// and one would fail nonce-too-low, leaving the bot's recovery to reconcile a +/// submission it did not make. +async fn approve_under_wallet_lock( vault_service: &dyn VaultService, pool: &Pool, network: Network, - provider: &P, - vault: Address, - orchestrator: Address, - bot: Address, + deadline: Instant, + approval: impl Future>, ) -> Result { - let _wallet_guard = vault_service.lock_wallet().await; - - if has_unresolved_signer_intent(pool, network, None).await.map_err( - |error| { - error!(target: "asset", %network, %error, - "Failed to check signer intents before approve" - ); - Status::InternalServerError - }, - )? { + let _wallet_guard = + before_deadline(deadline, "wallet lock", vault_service.lock_wallet()) + .await?; + + let intent_held = before_deadline( + deadline, + "signer intent check", + has_unresolved_signer_intent(pool, network, None), + ) + .await? + .map_err(|error| { + error!(target: "asset", %network, %error, + "Failed to check signer intents before approve" + ); + Status::InternalServerError + })?; + if intent_held { warn!(target: "asset", %network, "Refusing approve: an unresolved signer intent holds the wallet nonce" ); return Err(Status::Conflict); } - ensure_unlimited_approval(provider, vault, orchestrator, bot).await.map_err( - |error| { - error!(target: "asset", %orchestrator, %vault, %error, - "Orchestrator approval failed" - ); - map_onboarding_error(&error) - }, - ) + approval.await } #[derive(Serialize)] @@ -450,19 +539,26 @@ fn parse_assets(assets: &[String]) -> Result, Status> { .collect() } -/// Maps an onboarding failure to an HTTP status. A Turnkey policy denial is a -/// 422 the operator fixes in the policy; every other failure is an on-chain or -/// RPC fault (502). +/// Maps an onboarding failure to an HTTP status. A Turnkey policy denial found +/// by the sign-only proof and a broadcast the node refused (nothing pending; +/// its reason is in the log) are 422s the operator fixes before retrying; a +/// deadline that passed before the broadcast is a 503 with nothing sent; a +/// filled but unsigned transaction is the bot's own fault (500); every other +/// failure, including the approval's own signing failing inside the fill, is +/// an on-chain, RPC, or signer fault (502). const fn map_onboarding_error(error: &OnboardingError) -> Status { use OnboardingError::{ - ApprovalNotEffective, ApprovalReverted, Contract, PendingTransaction, - SigningRejected, Transport, + ApprovalNotEffective, ApprovalReverted, BroadcastRejected, Contract, + DeadlineBeforeBroadcast, SigningRejected, Transport, Unsigned, }; match error { - SigningRejected { .. } => Status::UnprocessableEntity, + SigningRejected { .. } | BroadcastRejected { .. } => { + Status::UnprocessableEntity + } + DeadlineBeforeBroadcast => Status::ServiceUnavailable, + Unsigned => Status::InternalServerError, Contract(_) - | PendingTransaction(_) | Transport(_) | ApprovalReverted { .. } | ApprovalNotEffective { .. } => Status::BadGateway, @@ -471,10 +567,9 @@ const fn map_onboarding_error(error: &OnboardingError) -> Status { #[cfg(test)] mod tests { - use alloy::network::EthereumWallet; - use alloy::primitives::{Address, U256, address}; - use alloy::providers::ProviderBuilder; - use alloy::signers::local::PrivateKeySigner; + use alloy::primitives::{Address, B256, address}; + use alloy::rpc::json_rpc::ErrorPayload; + use alloy::transports::RpcError; use chrono::Utc; use cqrs_es::DomainEvent; use rocket::http::Status; @@ -482,16 +577,24 @@ mod tests { use sqlx::sqlite::SqlitePoolOptions; use sqlx::{Pool, Sqlite}; use std::sync::Arc; + use std::sync::atomic::{AtomicBool, Ordering}; use std::time::Duration; + use tokio::time::Instant; + use tracing::Level; + use tracing_test::traced_test; - use super::approve_under_wallet_lock; + use super::{ + approve_reply, approve_under_wallet_lock, log_approval_failure, + map_onboarding_error, + }; use crate::account::ClientId; - use crate::bindings::OffchainAssetReceiptVault; use crate::mint::{IssuerMintRequestId, MintEvent, TokenizationRequestId}; - use crate::test_utils::LocalEvm; + use crate::test_utils::logs_contain_at; use crate::tokenized_asset::{Network, TokenSymbol, UnderlyingSymbol}; use crate::vault::PreparedMintTx; + use crate::vault::VaultService; use crate::vault::mock::MockVaultService; + use crate::vault::onboarding::{ApprovalOutcome, OnboardingError}; use crate::{Quantity, VaultMode}; async fn migrated_pool() -> Pool { @@ -571,6 +674,10 @@ mod tests { .await; } + fn deadline_after(budget: Duration) -> Instant { + Instant::now().checked_add(budget).unwrap() + } + /// The approval must take the service wallet lock before it checks for /// signer intents: a mint that signs while the approval is queued on the /// lock persists its intent under that lock, and the approval, once it @@ -578,33 +685,25 @@ mod tests { /// nonce the mint just took. #[tokio::test] async fn approve_checks_intents_only_once_it_holds_the_wallet_lock() { - let evm = LocalEvm::new().await.unwrap(); - let signer = PrivateKeySigner::from_bytes(&evm.private_key).unwrap(); - let provider = ProviderBuilder::new() - .wallet(EthereumWallet::from(signer)) - .connect(&evm.endpoint) - .await - .unwrap(); let pool = migrated_pool().await; let vault_service = Arc::new(MockVaultService::new_wallet_lock_blocked()); - let orchestrator = Address::repeat_byte(0x42); + let broadcast = Arc::new(AtomicBool::new(false)); let approval = tokio::spawn({ let vault_service = Arc::clone(&vault_service); let pool = pool.clone(); - let provider = provider.clone(); - let vault = evm.vault_address; - let bot = evm.wallet_address; + let broadcast = Arc::clone(&broadcast); async move { approve_under_wallet_lock( vault_service.as_ref(), &pool, Network::Base, - &provider, - vault, - orchestrator, - bot, + deadline_after(Duration::from_secs(25)), + async { + broadcast.store(true, Ordering::SeqCst); + Ok(ApprovalOutcome::AlreadyUnlimited) + }, ) .await } @@ -626,58 +725,178 @@ mod tests { seed_mint_holding_the_signer(&pool).await; vault_service.release_wallet_lock(); - let error = approval.await.unwrap().unwrap_err(); assert_eq!( - error, - Status::Conflict, + approval.await.unwrap(), + Err(Status::Conflict), "the intent check must see the mint that signed while the approval waited" ); - let allowance = - OffchainAssetReceiptVault::new(evm.vault_address, &provider) - .allowance(evm.wallet_address, orchestrator) - .call() - .await - .unwrap(); - assert_eq!( - allowance, - U256::ZERO, - "a refused approval must not have broadcast" + assert!( + !broadcast.load(Ordering::SeqCst), + "a refused approval must not have run" ); } - /// With the wallet free and no intent outstanding the approval lands. + /// The approval runs while the wallet lock is held, so no live mint or + /// burn can fill a nonce between its intent check and its broadcast. #[tokio::test] - async fn approve_broadcasts_under_a_free_wallet() { - let evm = LocalEvm::new().await.unwrap(); - let signer = PrivateKeySigner::from_bytes(&evm.private_key).unwrap(); - let provider = ProviderBuilder::new() - .wallet(EthereumWallet::from(signer)) - .connect(&evm.endpoint) - .await - .unwrap(); + async fn approval_runs_while_the_wallet_lock_is_held() { let pool = migrated_pool().await; let vault_service = MockVaultService::new_success(); - let orchestrator = Address::repeat_byte(0x42); - approve_under_wallet_lock( + let outcome = approve_under_wallet_lock( &vault_service, &pool, Network::Base, - &provider, - evm.vault_address, - orchestrator, - evm.wallet_address, + deadline_after(Duration::from_secs(25)), + async { + let competing_got_the_wallet = tokio::time::timeout( + Duration::from_millis(50), + vault_service.lock_wallet(), + ) + .await + .is_ok(); + assert!( + !competing_got_the_wallet, + "a live flow must not get the wallet while the approval runs" + ); + Ok(ApprovalOutcome::AlreadyUnlimited) + }, + ) + .await; + + assert_eq!(outcome, Ok(ApprovalOutcome::AlreadyUnlimited)); + } + + /// Waiting for the wallet counts against the approval's deadline: a lock + /// held past it answers 503, before anything was signed or sent. + #[traced_test] + #[tokio::test] + async fn approve_queued_on_the_wallet_past_its_deadline_is_refused() { + let pool = migrated_pool().await; + let vault_service = MockVaultService::new_wallet_lock_blocked(); + + let refused = tokio::time::timeout( + Duration::from_secs(5), + approve_under_wallet_lock( + &vault_service, + &pool, + Network::Base, + deadline_after(Duration::from_millis(100)), + async { Ok(ApprovalOutcome::AlreadyUnlimited) }, + ), ) .await - .unwrap(); + .expect("the lock wait must end at the approval's deadline"); + + assert_eq!(refused, Err(Status::ServiceUnavailable)); + assert!(logs_contain_at!( + Level::WARN, + &[ + "Refusing approve: deadline passed before broadcast", + "nothing sent", + "wallet lock" + ] + )); + } - assert_eq!(vault_service.get_wallet_lock_call_count(), 1); - let allowance = - OffchainAssetReceiptVault::new(evm.vault_address, &provider) - .allowance(evm.wallet_address, orchestrator) - .call() - .await - .unwrap(); - assert_eq!(allowance, U256::MAX); + /// The intent check's database wait counts against the deadline too: a + /// pool with no free connection answers 503 rather than holding the wallet + /// lock, and the request, past it. + #[traced_test] + #[tokio::test] + async fn approve_with_the_database_stalled_past_its_deadline_is_refused() { + let pool = SqlitePoolOptions::new() + .max_connections(1) + .connect(":memory:") + .await + .unwrap(); + sqlx::migrate!("./migrations").run(&pool).await.unwrap(); + let _held = pool.acquire().await.unwrap(); + let vault_service = MockVaultService::new_success(); + + let refused = tokio::time::timeout( + Duration::from_secs(5), + approve_under_wallet_lock( + &vault_service, + &pool, + Network::Base, + deadline_after(Duration::from_millis(100)), + async { Ok(ApprovalOutcome::AlreadyUnlimited) }, + ), + ) + .await + .expect("the intent check must end at the approval's deadline"); + + assert_eq!(refused, Err(Status::ServiceUnavailable)); + assert!(logs_contain_at!( + Level::WARN, + &[ + "Refusing approve: deadline passed before broadcast", + "nothing sent", + "signer intent check" + ] + )); + } + + /// The status is the script-visible half of the answer: only a verified + /// or already-unlimited allowance is 200; one signed but unconfirmed is + /// 202 with the hash to reconcile. + #[test] + fn approval_outcomes_map_to_their_status_and_body() { + let tx_hash = B256::repeat_byte(0xab); + + let (status, body) = + approve_reply(ApprovalOutcome::SubmittedUnconfirmed { tx_hash }); + assert_eq!(status, Status::Accepted); + assert_eq!(body.outcome, "submitted_unconfirmed"); + assert_eq!(body.tx_hash, Some(tx_hash.to_string())); + + let (status, body) = + approve_reply(ApprovalOutcome::Approved { tx_hash }); + assert_eq!(status, Status::Ok); + assert_eq!(body.outcome, "approved"); + assert_eq!(body.tx_hash, Some(tx_hash.to_string())); + + let (status, body) = approve_reply(ApprovalOutcome::AlreadyUnlimited); + assert_eq!(status, Status::Ok); + assert_eq!(body.outcome, "already_unlimited"); + assert_eq!(body.tx_hash, None); + } + + /// A broadcast the node refused is definitive, nothing is pending, so it + /// is a 422 the client reports as a failure, not a 502 it reads as an + /// unknown outcome. Running out of time while signing is the same 503 the + /// pre-broadcast steps answer, logged as a WARN refusal, not an ERROR. + #[traced_test] + #[test] + fn refusals_map_to_definitive_statuses_and_logs() { + let refused = OnboardingError::BroadcastRejected { + tx_hash: B256::repeat_byte(0xcd), + source: RpcError::ErrorResp(ErrorPayload { + code: -32003, + message: "insufficient funds for gas * price + value".into(), + data: None, + }), + }; + assert_eq!(map_onboarding_error(&refused), Status::UnprocessableEntity); + + let late = OnboardingError::DeadlineBeforeBroadcast; + assert_eq!(map_onboarding_error(&late), Status::ServiceUnavailable); + log_approval_failure( + &late, + Address::repeat_byte(1), + Address::repeat_byte(2), + ); + assert!(logs_contain_at!( + Level::WARN, + &[ + "Refusing approve: deadline passed before broadcast", + "approval signing" + ] + )); + assert!(!logs_contain_at!( + Level::ERROR, + &["Orchestrator approval failed"] + )); } } diff --git a/src/vault/onboarding.rs b/src/vault/onboarding.rs index f2be5b3e..0359d49a 100644 --- a/src/vault/onboarding.rs +++ b/src/vault/onboarding.rs @@ -10,16 +10,27 @@ //! read-only, generic over the provider, consumed by the issuer CLI's //! preflight subcommand. +use alloy::consensus::{Transaction, TxEnvelope}; +use alloy::eips::eip2718::Encodable2718; use alloy::network::{ Ethereum, EthereumWallet, TransactionBuilder, TransactionBuilderError, }; use alloy::primitives::{Address, B256, Bytes, U256, keccak256}; -use alloy::providers::Provider; +use alloy::providers::fillers::{FillProvider, TxFiller}; +use alloy::providers::{Provider, SendableTx}; +use alloy::rpc::json_rpc::ErrorPayload; use alloy::rpc::types::TransactionRequest; use alloy::sol_types::SolCall; +use alloy::transports::RpcError; use std::fmt::{self, Display, Formatter}; -use tracing::{debug, info}; +use std::time::Duration; +use tokio::time::Instant; +use tracing::{debug, info, warn}; +use super::service::{ + is_already_known_message, is_definitive_broadcast_rejection, + is_nonce_too_low_message, +}; use crate::bindings::{ IST0xOrchestratorV1, OffchainAssetReceiptVault, OffchainAssetReceiptVaultAuthorizerV1, Receipt, ST0xOrchestrator, @@ -156,8 +167,28 @@ pub(crate) enum ApprovalOutcome { /// An `approve(orchestrator, U256::MAX)` landed and the re-read /// allowance is unlimited. Approved { tx_hash: B256 }, + /// The approval was signed and broadcast (or its broadcast's answer was + /// lost), but its receipt and the allowance re-read did not both come back + /// before the deadline, so it may still land. The hash is what the + /// operator reconciles on chain instead of sending a second approval. + SubmittedUnconfirmed { tx_hash: B256 }, } +/// The whole approval's budget, from the request's start to its answer. Base +/// and HyperEVM mine within seconds, and the `/ops` route polls the receipt +/// every [`APPROVAL_RECEIPT_POLL`], so a healthy approval fits with room to +/// spare. The route starts it only after the IAP guard, which may spend up to +/// 10 seconds fetching Google's keys, so 18 seconds keeps the whole request +/// under a load balancer's default 30-second backend timeout and the bot's +/// verdict, not a gateway's 504, reaches the operator. Every RPC and signing +/// step before broadcast is held to it too: one that would overrun it refuses +/// with nothing sent. +pub(crate) const APPROVAL_DEADLINE: Duration = Duration::from_secs(18); + +/// How often the `/ops` route's provider polls for the approval's receipt, +/// down from a remote client's default 7 seconds. +pub(crate) const APPROVAL_RECEIPT_POLL: Duration = Duration::from_secs(1); + /// One transaction shape the Turnkey signing policy must allow. Target and /// calldata are fixed at build time, so tests can decode exactly what would /// be signed. @@ -184,9 +215,20 @@ pub(crate) enum OnboardingError { #[error(transparent)] Contract(#[from] alloy::contract::Error), #[error(transparent)] - PendingTransaction(#[from] alloy::providers::PendingTransactionError), - #[error(transparent)] Transport(#[from] alloy::transports::TransportError), + #[error( + "the approval deadline passed before its transaction was broadcast; \ + nothing was sent" + )] + DeadlineBeforeBroadcast, + #[error("the approval transaction was filled but not signed")] + Unsigned, + #[error("the node refused approve transaction {tx_hash}: {source}")] + BroadcastRejected { + tx_hash: B256, + #[source] + source: alloy::transports::TransportError, + }, #[error("approve transaction {tx_hash} reverted on-chain")] ApprovalReverted { tx_hash: B256 }, #[error( @@ -302,20 +344,52 @@ pub(crate) async fn check_orchestrator_readiness( /// Success is never inferred from the receipt alone; the allowance the burns /// will rely on is re-read from the chain. /// +/// Everything runs against `deadline`. The approval is signed before it is +/// broadcast, so its hash is known from then on: running out of time before +/// the broadcast sends nothing, and any later shortfall (no receipt, reads +/// that kept failing, a broadcast whose answer was lost, cut off, or not +/// conclusive) reports the hash as [`ApprovalOutcome::SubmittedUnconfirmed`]. +/// Only a broadcast the pool provably refused, and that the node then does +/// not hold, is an error. +/// /// # Errors /// -/// Returns an error if any read or the send fails, if the approve -/// transaction reverts, or if the re-read allowance is not unlimited. -pub(crate) async fn ensure_unlimited_approval( - signing_provider: &P, +/// Returns an error if a read, the fill, or the signing fails, if the +/// deadline passes before the broadcast, if the node refuses the broadcast, +/// if the approve transaction reverts, or if the re-read allowance is not +/// unlimited. +pub(crate) async fn ensure_unlimited_approval( + signing_provider: &FillProvider, vault: Address, orchestrator: Address, bot: Address, -) -> Result { + deadline: Instant, +) -> Result +where + F: TxFiller, + P: Provider, +{ let vault_contract = OffchainAssetReceiptVault::new(vault, signing_provider); - let current = vault_contract.allowance(bot, orchestrator).call().await?; - if current == U256::MAX { + let signed = tokio::time::timeout_at(deadline, async { + let current = + vault_contract.allowance(bot, orchestrator).call().await?; + if current == U256::MAX { + return Ok(None); + } + + let request = vault_contract + .approve(orchestrator, U256::MAX) + .into_transaction_request(); + match signing_provider.fill(request).await? { + SendableTx::Envelope(envelope) => Ok(Some(envelope)), + SendableTx::Builder(_) => Err(OnboardingError::Unsigned), + } + }) + .await + .map_err(|_| OnboardingError::DeadlineBeforeBroadcast)??; + + let Some(envelope) = signed else { info!( target: "vault", %vault, @@ -324,20 +398,200 @@ pub(crate) async fn ensure_unlimited_approval( "Orchestrator allowance already unlimited; nothing to send" ); return Ok(ApprovalOutcome::AlreadyUnlimited); + }; + let tx_hash = *envelope.tx_hash(); + let nonce = envelope.nonce(); + + // Signing may have used up the budget; a send started after it could be + // cut off before it leaves the socket and still be reported as sent. + if Instant::now() >= deadline { + return Err(OnboardingError::DeadlineBeforeBroadcast); } - let receipt = vault_contract - .approve(orchestrator, U256::MAX) - .send() - .await? - .get_receipt() - .await?; - let tx_hash = receipt.transaction_hash; + broadcast_approval( + signing_provider, + &envelope, + deadline, + vault, + orchestrator, + ) + .await?; + + let mut last_read_error = None; + let confirmed = tokio::time::timeout_at( + deadline, + confirm_approval( + signing_provider, + &vault_contract, + tx_hash, + bot, + orchestrator, + &mut last_read_error, + ), + ) + .await; + + match confirmed { + Ok(Ok(())) => { + info!( + target: "vault", + %vault, + %orchestrator, + bot = %bot, + %tx_hash, + nonce, + "Approved unlimited orchestrator allowance" + ); + Ok(ApprovalOutcome::Approved { tx_hash }) + } + Ok(Err(error)) => Err(error), + Err(_) => { + warn!(target: "vault", %vault, %orchestrator, %tx_hash, nonce, + last_read_error = last_read_error.as_ref().map(display), + "Approval signed but not confirmed within the deadline; \ + reporting it unconfirmed" + ); + Ok(ApprovalOutcome::SubmittedUnconfirmed { tx_hash }) + } + } +} + +/// Broadcasts the signed approval. An error answer that proves the pool +/// refused the bytes (insufficient funds, underpriced, nonce too low, or a +/// malformed request) is [`OnboardingError::BroadcastRejected`] carrying the +/// node's reason, unless a lookup within the deadline shows the node holds +/// the transaction anyway. Every other outcome may have left the approval +/// pending, so it is returned as sent and confirmed by hash: an "already +/// known" answer, any other server error (a load-balanced provider may have +/// forwarded the bytes before failing), a lookup that fails or runs out of +/// time, and a send that fails in transport or runs out of time. +async fn broadcast_approval( + signing_provider: &FillProvider, + envelope: &TxEnvelope, + deadline: Instant, + vault: Address, + orchestrator: Address, +) -> Result<(), OnboardingError> +where + F: TxFiller, + P: Provider, +{ + let tx_hash = *envelope.tx_hash(); + let nonce = envelope.nonce(); + let sent = tokio::time::timeout_at( + deadline, + signing_provider.send_raw_transaction(&envelope.encoded_2718()), + ) + .await; + let error = match sent { + Ok(Ok(_)) => return Ok(()), + Ok(Err(error)) => error, + Err(_) => { + warn!(target: "vault", %vault, %orchestrator, %tx_hash, nonce, + "Approval broadcast cut off at the deadline; it may be pending" + ); + return Ok(()); + } + }; + + let RpcError::ErrorResp(response) = &error else { + warn!(target: "vault", %vault, %orchestrator, %tx_hash, nonce, %error, + "Approval broadcast failed in transport; it may be pending" + ); + return Ok(()); + }; + if is_already_known_message(&response.message) { + info!(target: "vault", %vault, %orchestrator, %tx_hash, nonce, + "The node already holds the approval; confirming it by hash" + ); + return Ok(()); + } + if !is_pool_refusal(response) { + warn!(target: "vault", %vault, %orchestrator, %tx_hash, nonce, %error, + "Approval broadcast answered with an inconclusive error; it may \ + be pending" + ); + return Ok(()); + } + + warn!(target: "vault", %vault, %orchestrator, %tx_hash, nonce, %error, + "Approval broadcast refused by the node; checking whether it holds \ + the transaction anyway" + ); + let lookup = tokio::time::timeout_at( + deadline, + signing_provider.get_transaction_by_hash(tx_hash), + ) + .await; + match lookup { + Ok(Ok(None)) => { + Err(OnboardingError::BroadcastRejected { tx_hash, source: error }) + } + Ok(Ok(Some(_))) => Ok(()), + Ok(Err(lookup_error)) => { + warn!(target: "vault", %tx_hash, %lookup_error, + "Lookup after the refusal failed; it may be pending" + ); + Ok(()) + } + Err(_) => { + warn!(target: "vault", %tx_hash, + "Lookup after the refusal ran out of time; it may be pending" + ); + Ok(()) + } + } +} + +/// An error answer that proves the pool did not accept the bytes: a +/// malformed request, or a pool rule the transaction failed. +fn is_pool_refusal(response: &ErrorPayload) -> bool { + let message = response.message.to_ascii_lowercase(); + is_definitive_broadcast_rejection(response.code) + || is_nonce_too_low_message(&message) + || message.contains("insufficient funds") + || message.contains("underpriced") +} + +/// Waits for the sent approval's receipt and re-reads the allowance. A failed +/// poll or read is retried until the caller's deadline cuts this off, so one +/// transient error (a 429, a dropped connection) cannot end the wait early; +/// the last one is kept for the caller's log. +async fn confirm_approval( + signing_provider: &FillProvider, + vault_contract: &OffchainAssetReceiptVault::OffchainAssetReceiptVaultInstance< + &FillProvider, + >, + tx_hash: B256, + bot: Address, + orchestrator: Address, + last_read_error: &mut Option, +) -> Result<(), OnboardingError> +where + F: TxFiller, + P: Provider, +{ + let poll_interval = signing_provider.client().poll_interval(); + + let receipt = loop { + match signing_provider.get_transaction_receipt(tx_hash).await { + Ok(Some(receipt)) => break receipt, + Ok(None) => {} + Err(error) => *last_read_error = Some(error.into()), + } + tokio::time::sleep(poll_interval).await; + }; if !receipt.status() { return Err(OnboardingError::ApprovalReverted { tx_hash }); } - let allowance = vault_contract.allowance(bot, orchestrator).call().await?; + let allowance = loop { + match vault_contract.allowance(bot, orchestrator).call().await { + Ok(allowance) => break allowance, + Err(error) => *last_read_error = Some(error.into()), + } + tokio::time::sleep(poll_interval).await; + }; if allowance != U256::MAX { return Err(OnboardingError::ApprovalNotEffective { tx_hash, @@ -345,16 +599,7 @@ pub(crate) async fn ensure_unlimited_approval( }); } - info!( - target: "vault", - %vault, - %orchestrator, - bot = %bot, - %tx_hash, - "Approved unlimited orchestrator allowance" - ); - - Ok(ApprovalOutcome::Approved { tx_hash }) + Ok(()) } /// Signs — without ever broadcasting — one transaction per shape the Turnkey @@ -530,8 +775,18 @@ const fn pass_fail(passed: bool) -> &'static str { #[cfg(test)] mod tests { - use alloy::providers::ProviderBuilder; + use alloy::consensus::{SignableTransaction, TxEip1559}; + use alloy::eips::eip2930::AccessList; + use alloy::primitives::TxKind; + use alloy::providers::ext::AnvilApi; + use alloy::providers::fillers::FillerControlFlow; + use alloy::providers::mock::Asserter; + use alloy::providers::{ProviderBuilder, RootProvider}; + use alloy::rpc::json_rpc::ErrorPayload; + use alloy::rpc::types::Transaction as RpcTransaction; + use alloy::signers::SignerSync; use alloy::signers::local::PrivateKeySigner; + use alloy::transports::TransportResult; use httpmock::MockServer; use tracing::Level; use tracing_test::traced_test; @@ -707,7 +962,9 @@ mod tests { assert!(!report.is_ready()); } - async fn signing_provider(evm: &LocalEvm) -> impl Provider + use<> { + async fn signing_provider( + evm: &LocalEvm, + ) -> FillProvider, RootProvider> { let signer = PrivateKeySigner::from_bytes(&evm.private_key).unwrap(); ProviderBuilder::new() .wallet(EthereumWallet::from(signer)) @@ -716,6 +973,10 @@ mod tests { .unwrap() } + fn deadline_after(budget: Duration) -> Instant { + Instant::now().checked_add(budget).unwrap() + } + #[traced_test] #[tokio::test] async fn first_approval_sends_verifies_and_reports_the_tx() { @@ -728,6 +989,7 @@ mod tests { evm.vault_address, orchestrator, evm.wallet_address, + deadline_after(APPROVAL_DEADLINE), ) .await .unwrap(); @@ -753,6 +1015,7 @@ mod tests { evm.vault_address, orchestrator, evm.wallet_address, + deadline_after(APPROVAL_DEADLINE), ) .await .unwrap(); @@ -764,6 +1027,7 @@ mod tests { evm.vault_address, orchestrator, evm.wallet_address, + deadline_after(APPROVAL_DEADLINE), ) .await .unwrap(); @@ -792,6 +1056,7 @@ mod tests { evm.vault_address, orchestrator, evm.wallet_address, + deadline_after(APPROVAL_DEADLINE), ) .await .unwrap(); @@ -801,6 +1066,351 @@ mod tests { assert!(report.assets[0].is_unlimited()); } + /// An approval whose receipt never arrives must come back within the + /// deadline as submitted-unconfirmed with the signed hash, rather than + /// hold the request until a gateway cuts it off with no hash at all. + #[traced_test] + #[tokio::test] + async fn unmined_approval_is_reported_unconfirmed_with_its_hash() { + let evm = LocalEvm::new().await.unwrap(); + let orchestrator = evm.deploy_orchestrator().await.unwrap(); + let provider = signing_provider(&evm).await; + provider.anvil_set_auto_mine(false).await.unwrap(); + + let outcome = tokio::time::timeout( + Duration::from_secs(10), + ensure_unlimited_approval( + &provider, + evm.vault_address, + orchestrator, + evm.wallet_address, + deadline_after(Duration::from_secs(1)), + ), + ) + .await + .expect("the approval must answer once its deadline passes") + .unwrap(); + + let ApprovalOutcome::SubmittedUnconfirmed { tx_hash } = outcome else { + panic!("expected SubmittedUnconfirmed, got {outcome:?}"); + }; + assert!( + provider.get_transaction_by_hash(tx_hash).await.unwrap().is_some(), + "the reported hash must be the approval the node holds as pending" + ); + assert!(logs_contain_at!( + Level::WARN, + &[ + "not confirmed within the deadline", + &tx_hash.to_string(), + "nonce" + ] + )); + } + + /// A deadline that passes before the approval is broadcast refuses with + /// nothing sent, so the operator can retry without a second approval in + /// flight. + #[tokio::test] + async fn deadline_before_broadcast_sends_nothing() { + let evm = LocalEvm::new().await.unwrap(); + let orchestrator = evm.deploy_orchestrator().await.unwrap(); + let provider = signing_provider(&evm).await; + let nonce_before = + provider.get_transaction_count(evm.wallet_address).await.unwrap(); + + let error = ensure_unlimited_approval( + &provider, + evm.vault_address, + orchestrator, + evm.wallet_address, + Instant::now().checked_sub(Duration::from_secs(1)).unwrap(), + ) + .await + .unwrap_err(); + + assert!( + matches!(error, OnboardingError::DeadlineBeforeBroadcast), + "expected DeadlineBeforeBroadcast, got {error:?}" + ); + assert_eq!( + provider + .get_transaction_count(evm.wallet_address) + .pending() + .await + .unwrap(), + nonce_before, + "a refusal before broadcast must not have sent a transaction" + ); + } + + /// Stands in for a remote signer whose answer lands just after the + /// deadline: it runs once the request is otherwise complete, after the + /// wallet has signed, and blocks in the same poll, so the fill future is + /// already ready when its timeout is next checked. + #[derive(Clone, Debug)] + struct SignedLate(Duration); + + impl TxFiller for SignedLate { + type Fillable = (); + + fn status(&self, tx: &TransactionRequest) -> FillerControlFlow { + if tx.nonce.is_some() && tx.gas.is_some() { + FillerControlFlow::Ready + } else { + FillerControlFlow::Finished + } + } + + fn fill_sync(&self, _tx: &mut SendableTx) {} + + async fn prepare( + &self, + _provider: &P, + _tx: &TransactionRequest, + ) -> TransportResult<()> { + Ok(()) + } + + async fn fill( + &self, + _fillable: (), + tx: SendableTx, + ) -> TransportResult> { + std::thread::sleep(self.0); + Ok(tx) + } + } + + /// Signing that finishes after the deadline must not be broadcast: a send + /// started then could be cut off before it leaves and still be reported as + /// submitted. + #[tokio::test] + async fn approval_signed_after_its_deadline_is_not_broadcast() { + let evm = LocalEvm::new().await.unwrap(); + let orchestrator = evm.deploy_orchestrator().await.unwrap(); + let signer = PrivateKeySigner::from_bytes(&evm.private_key).unwrap(); + let provider = ProviderBuilder::new() + .wallet(EthereumWallet::from(signer)) + .filler(SignedLate(Duration::from_millis(600))) + .connect(&evm.endpoint) + .await + .unwrap(); + let nonce_before = + provider.get_transaction_count(evm.wallet_address).await.unwrap(); + + let error = ensure_unlimited_approval( + &provider, + evm.vault_address, + orchestrator, + evm.wallet_address, + deadline_after(Duration::from_millis(300)), + ) + .await + .unwrap_err(); + + assert!( + matches!(error, OnboardingError::DeadlineBeforeBroadcast), + "expected DeadlineBeforeBroadcast, got {error:?}" + ); + assert_eq!( + provider + .get_transaction_count(evm.wallet_address) + .pending() + .await + .unwrap(), + nonce_before, + "an approval signed after its deadline must not have been sent" + ); + } + + /// Signed bytes for a broadcast the mocked node answers; never mined. + fn signed_approval() -> TxEnvelope { + let signer = PrivateKeySigner::random(); + let tx = TxEip1559 { + chain_id: 1, + nonce: 7, + gas_limit: 60_000, + max_fee_per_gas: 1, + max_priority_fee_per_gas: 1, + to: TxKind::Call(Address::repeat_byte(0x22)), + value: U256::ZERO, + access_list: AccessList::default(), + input: Bytes::new(), + }; + let signature = signer.sign_hash_sync(&tx.signature_hash()).unwrap(); + TxEnvelope::from(tx.into_signed(signature)) + } + + /// What the mocked node answers to the hash lookup after an error answer. + enum Lookup { + Missing, + Fails, + NotAsked, + } + + /// Only an error that proves the pool refused the bytes, with the node + /// then not holding them, is a refusal (422 with nothing pending). An + /// "already known" answer, an ambiguous server error, or a lookup that + /// fails leaves the approval possibly pending, so it must go on to be + /// confirmed by hash (202 with the hash) instead. + #[traced_test] + #[tokio::test] + async fn only_a_known_pool_refusal_the_node_does_not_hold_is_refused() { + for (code, message, lookup, refused) in [ + ( + -32000, + "insufficient funds for gas * price + value", + Lookup::Missing, + true, + ), + ( + -32000, + "nonce too low: next nonce 8, tx nonce 7", + Lookup::Missing, + true, + ), + (-32000, "already known", Lookup::NotAsked, false), + (-32603, "internal error", Lookup::NotAsked, false), + (-32000, "transaction underpriced", Lookup::Fails, false), + ] { + let envelope = signed_approval(); + let tx_hash = *envelope.tx_hash(); + let asserter = Asserter::new(); + asserter.push_failure(ErrorPayload { + code, + message: message.into(), + data: None, + }); + match lookup { + Lookup::Missing => { + asserter.push_success(&Option::::None); + } + Lookup::Fails => { + asserter.push_failure_msg("lookup failed (test)"); + } + Lookup::NotAsked => {} + } + let provider = + ProviderBuilder::new().connect_mocked_client(asserter.clone()); + + let result = broadcast_approval( + &provider, + &envelope, + deadline_after(Duration::from_secs(5)), + Address::repeat_byte(0x22), + Address::repeat_byte(0x11), + ) + .await; + + assert_eq!( + matches!( + result, + Err(OnboardingError::BroadcastRejected { .. }) + ), + refused, + "{message}: {result:?}" + ); + assert!(asserter.read_q().is_empty(), "{message}: unread answers"); + if refused { + assert!(logs_contain_at!( + Level::WARN, + &["refused by the node", &tx_hash.to_string()] + )); + } + } + } + + /// One failed receipt poll or allowance read (a 429, a dropped + /// connection) must not end the wait while time is left: the next poll + /// can still confirm the approval. + #[tokio::test] + async fn a_failed_receipt_or_allowance_read_is_retried() { + let evm = LocalEvm::new().await.unwrap(); + let orchestrator = evm.deploy_orchestrator().await.unwrap(); + let receipt = OffchainAssetReceiptVault::new( + evm.vault_address, + signing_provider(&evm).await, + ) + .approve(orchestrator, U256::MAX) + .send() + .await + .unwrap() + .get_receipt() + .await + .unwrap(); + + let asserter = Asserter::new(); + asserter.push_failure_msg("receipt poll failed (test)"); + asserter.push_success(&Some(receipt.clone())); + asserter.push_failure_msg("allowance read failed (test)"); + asserter.push_success(&Bytes::from(U256::MAX.to_be_bytes::<32>())); + let provider = + ProviderBuilder::new().connect_mocked_client(asserter.clone()); + let vault_contract = + OffchainAssetReceiptVault::new(evm.vault_address, &provider); + + let mut last_read_error = None; + let confirmed = tokio::time::timeout( + Duration::from_secs(10), + confirm_approval( + &provider, + &vault_contract, + receipt.transaction_hash, + evm.wallet_address, + orchestrator, + &mut last_read_error, + ), + ) + .await + .expect("the retried reads must confirm well within the guard"); + + assert!(confirmed.is_ok(), "{confirmed:?}"); + } + + /// A broadcast the node refuses (here a wallet with no gas) is an error + /// carrying the hash and the node's reason, answered at once, not a + /// 202 for a transaction that will never exist. + #[traced_test] + #[tokio::test] + async fn refused_broadcast_is_an_error_not_an_unconfirmed_approval() { + let evm = LocalEvm::new().await.unwrap(); + let orchestrator = evm.deploy_orchestrator().await.unwrap(); + let unfunded = PrivateKeySigner::random(); + let bot = unfunded.address(); + let provider = ProviderBuilder::new() + .wallet(EthereumWallet::from(unfunded)) + .connect(&evm.endpoint) + .await + .unwrap(); + + let error = tokio::time::timeout( + Duration::from_secs(10), + ensure_unlimited_approval( + &provider, + evm.vault_address, + orchestrator, + bot, + deadline_after(APPROVAL_DEADLINE), + ), + ) + .await + .expect("a refused broadcast must be answered without waiting out the deadline") + .unwrap_err(); + + let OnboardingError::BroadcastRejected { tx_hash, .. } = error else { + panic!("expected BroadcastRejected, got {error:?}"); + }; + assert!( + provider.get_transaction_by_hash(tx_hash).await.unwrap().is_none(), + "the refused approval must not be held by the node" + ); + assert!(logs_contain_at!( + Level::WARN, + &["refused by the node", &tx_hash.to_string()] + )); + } + #[test] fn signing_shapes_target_the_right_contracts_with_decodable_calldata() { let orchestrator = Address::repeat_byte(0x11); diff --git a/src/vault/service.rs b/src/vault/service.rs index 57514478..9ff1074e 100644 --- a/src/vault/service.rs +++ b/src/vault/service.rs @@ -845,7 +845,7 @@ fn classify_burn_broadcast_error( } } -fn is_nonce_too_low_message(message: &str) -> bool { +pub(super) fn is_nonce_too_low_message(message: &str) -> bool { let message = message.to_ascii_lowercase(); ["nonce too low", "nonce is too low", "oldnonce", "old nonce"] .iter() @@ -854,11 +854,11 @@ fn is_nonce_too_low_message(message: &str) -> bool { /// The node's answer when its pool already holds the exact transaction /// (geth and op-geth: "already known"). -fn is_already_known_message(message: &str) -> bool { +pub(super) fn is_already_known_message(message: &str) -> bool { message.to_ascii_lowercase().contains("already known") } -const fn is_definitive_broadcast_rejection(code: i64) -> bool { +pub(super) const fn is_definitive_broadcast_rejection(code: i64) -> bool { matches!(code, -32700 | -32600 | -32601 | -32602) }