From 7b542b09331fc94df5f0e50d68d646c3fa187abd Mon Sep 17 00:00:00 2001 From: pauljacobb Date: Fri, 26 Jun 2026 07:48:05 +0000 Subject: [PATCH] docs: add troubleshooting guide for failed Stellar transactions (#587) --- docs/STELLAR_TROUBLESHOOTING.md | 346 ++++++++++++++++++++++++++++++++ 1 file changed, 346 insertions(+) create mode 100644 docs/STELLAR_TROUBLESHOOTING.md diff --git a/docs/STELLAR_TROUBLESHOOTING.md b/docs/STELLAR_TROUBLESHOOTING.md new file mode 100644 index 0000000..659bd3b --- /dev/null +++ b/docs/STELLAR_TROUBLESHOOTING.md @@ -0,0 +1,346 @@ +# Stellar Transaction Troubleshooting Guide + +This guide covers common failures when submitting transactions to Stellar/Soroban in the SolarProof stack — meter reading anchors, token mints, and governance votes — along with concrete diagnostic steps and remediation. + +--- + +## Overview: Common Failure Categories + +| Category | Typical Symptom | +|---|---| +| Insufficient balance / base reserve | `INSUFFICIENT_BALANCE`, `op_low_reserve` | +| Sequence number mismatch | `tx_bad_seq` | +| Transaction timeout | `tx_too_late`, XDR submission rejected | +| Contract resource limits | `ExceededLimit`, `InvokeHostFunctionResourceLimitExceeded` | +| Authorization failure | `AuthorizationError`, `NotAuthorized` | +| Mint failure (energy_token) | API returns 500, token not received | +| Network / RPC mismatch | Testnet tx sent to mainnet RPC (or vice versa) | + +--- + +## 1. Insufficient XLM Balance / Base Reserve Errors + +Stellar requires every account to maintain a **minimum base reserve** of 0.5 XLM per account entry plus 0.5 XLM per additional trustline, offer, or data entry. Soroban contract entries add to this. + +**Symptoms** +- `INSUFFICIENT_BALANCE` in Horizon response +- `op_low_reserve` operation result code +- API error: `account cannot afford fee` + +**Diagnose** + +```bash +# Check account balance and subentry count +stellar account show --account --network testnet + +# Or query Horizon directly +curl https://horizon-testnet.stellar.org/accounts/ \ + | jq '{balance: .balances, subentry_count: .subentry_count, min_balance: (.subentry_count * 0.5 + 1)}' +``` + +The minimum spendable balance = `(2 + subentry_count) * 0.5 XLM`. + +**Remediate** + +```bash +# Fund account on testnet via Friendbot +curl "https://friendbot.stellar.org/?addr=" + +# Or transfer XLM from another funded account +stellar tx payment \ + --source \ + --destination \ + --asset XLM \ + --amount 10 \ + --network testnet \ + --sign +``` + +For the SolarProof API service account, ensure `STELLAR_SERVICE_ACCOUNT_SECRET` in `.env.local` corresponds to an account with ≥ 5 XLM available above base reserve before any mint burst. + +--- + +## 2. Transaction Timeout / Sequence Number Errors + +**Symptoms** +- `tx_bad_seq` — the transaction sequence number does not match the account's current sequence +- `tx_too_late` — `max_ledger_version` was exceeded before the transaction was included +- `tx_too_early` — `min_ledger_version` not reached yet + +**Sequence number mismatch** + +This typically happens when multiple concurrent requests share one source account, or when a transaction was built against a stale sequence. + +```bash +# Fetch current sequence number for an account +curl https://horizon-testnet.stellar.org/accounts/ | jq .sequence + +# In code, always reload the account before building a transaction: +# const account = await server.loadAccount(sourcePublicKey); +# const tx = new TransactionBuilder(account, { fee, networkPassphrase }) +``` + +Fix: serialize mint requests through a queue (Redis-backed in the SolarProof stack) so only one transaction is in-flight per source account at a time. See `apps/web/src/lib/stellar/queue.ts`. + +**Transaction timeout** + +Soroban transactions must be submitted within ~30 seconds of construction (before `max_ledger_version` is reached). + +```bash +# Check current ledger +curl https://horizon-testnet.stellar.org/ | jq .core_latest_ledger + +# Soroban RPC: get latest ledger +curl -X POST https://soroban-testnet.stellar.org \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":1,"method":"getLatestLedger","params":{}}' \ + | jq .result +``` + +Fix: rebuild and resubmit the transaction immediately. Do not cache built transactions for later submission. + +--- + +## 3. Contract Invocation Errors (Resource Limits & Auth Failures) + +### Resource limit exceeded + +Soroban enforces per-transaction limits on CPU instructions, memory, ledger reads/writes, and event bytes. + +**Symptoms** +- `ExceededLimit` in the simulation response +- `InvokeHostFunctionResourceLimitExceeded` on submission +- RPC `simulateTransaction` returns an error object instead of `results` + +**Diagnose** + +Always simulate before submitting: + +```bash +# Encode your XDR, then simulate +stellar contract invoke \ + --id \ + --source \ + --network testnet \ + --simulate-only \ + -- mint \ + --to \ + --amount 1000000 +``` + +The simulation response includes `cost` — check `cpuInsns` and `memBytes` against network limits. + +**Remediate** +- Break large batch mints into individual transactions +- Reduce the number of ledger entries touched per invocation +- Bump `soroban_resources` fee in `TransactionBuilder` if simulation succeeds but submission fails + +### Authorization failure + +**Symptoms** +- `AuthorizationError` or `NotAuthorized` in host function result +- `op_not_authorized` in operation result codes + +**Diagnose** + +```bash +# Check who is the current admin/minter on energy_token +stellar contract invoke \ + --id \ + --source \ + --network testnet \ + -- admin +``` + +```bash +# Verify the signing key matches the expected admin +stellar keys show +``` + +**Common causes in SolarProof** +- `STELLAR_ADMIN_SECRET` env var points to a key that is not the contract admin +- Contract was redeployed and the admin was reset — check `docs/deployments.md` for current contract IDs +- `audit_registry` anchor call uses a different key than the one registered for the meter + +--- + +## 4. Mint Failures and tracer-sim Diagnosis + +When the SolarProof API's `POST /api/readings` endpoint fails to mint the `energy_token` certificate, the server logs will include a `tracer-sim` diagnostic report. + +**How tracer-sim works** + +`tracer-sim` replays the failed Soroban transaction locally against the ledger snapshot at the time of failure, producing a step-by-step execution trace with the exact host function that errored and the contract state at that point. + +**Reading a tracer-sim report** + +``` +[tracer-sim] replaying tx +[tracer-sim] ledger sequence: 12345678 +[tracer-sim] invoke energy_token::mint(to=G..., amount=1000000) +[tracer-sim] ERROR at step 42: AuthorizationError +[tracer-sim] contract_id: C... +[tracer-sim] function: mint +[tracer-sim] invoker: G...AAAA <-- mismatch with admin G...BBBB +``` + +**Run tracer-sim manually** + +```bash +# Replay any failed transaction by hash +npx tracer-sim replay \ + --tx-hash \ + --network testnet \ + --contract-id +``` + +**Common mint failure causes** + +| Cause | tracer-sim signal | Fix | +|---|---|---| +| Wrong admin key | `AuthorizationError` at `mint` | Update `STELLAR_ADMIN_SECRET` | +| Contract ID stale | `contract not found` | Re-run `stellar contract deploy`, update `deployments.md` | +| Recipient has no trustline | `TrustlinesNotAuthorized` | Recipient must add trustline before receiving tokens | +| Fee too low | simulation succeeds, submission fails | Increase `fee` in `TransactionBuilder` | +| Ledger entry TTL expired | `MissingLedgerEntry` | Re-upload contract Wasm and restore entry | + +--- + +## 5. Network Connectivity Issues (Testnet vs Mainnet) + +**Symptoms** +- Transactions submitted successfully but never appear on-chain +- `getTransaction` returns `NOT_FOUND` indefinitely +- RPC returns 404 or connection refused + +**Verify active network** + +```bash +# Show the network configured in stellar CLI +stellar network ls + +# Confirm the passphrase your code uses matches the target network +# Testnet: "Test SDF Network ; September 2015" +# Mainnet: "Public Global Stellar Network ; September 2015" +``` + +Check `.env.local`: + +```bash +grep STELLAR_NETWORK apps/web/.env.local +# Should be: STELLAR_NETWORK=testnet (or mainnet for production) +``` + +**Test RPC connectivity** + +```bash +# Soroban RPC health check +curl https://soroban-testnet.stellar.org \ + -X POST -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":1,"method":"getHealth","params":{}}' \ + | jq .result.status +# Expected: "healthy" + +# Horizon health check +curl https://horizon-testnet.stellar.org/health +``` + +**Testnet resets** + +The Stellar testnet resets periodically (roughly quarterly). After a reset: +1. All testnet account balances and contract deployments are wiped +2. Re-fund accounts via Friendbot +3. Redeploy all three contracts and update `docs/deployments.md` +4. Regenerate meter keys via `node scripts/gen-meter-key.mjs` + +Check the reset schedule at https://developers.stellar.org/docs/networks/testnet-and-futurenet. + +--- + +## 6. Inspecting Failed Transactions with Stellar Expert and Horizon + +### Stellar Expert (UI) + +1. Go to https://stellar.expert/explorer/testnet (testnet) or https://stellar.expert/explorer/public (mainnet) +2. Paste the transaction hash or account ID in the search box +3. Open the transaction → **Operations** tab shows each operation result code +4. **Effects** tab shows what state changes were applied (or not) + +### Horizon (API) + +```bash +# Fetch full transaction detail by hash +curl https://horizon-testnet.stellar.org/transactions/ | jq . + +# Fetch operations for a transaction +curl https://horizon-testnet.stellar.org/transactions//operations | jq . + +# Fetch recent transactions for an account +curl "https://horizon-testnet.stellar.org/accounts//transactions?order=desc&limit=5" \ + | jq '._embedded.records[] | {hash, created_at, successful, result_code: .result_codes}' +``` + +### Soroban RPC (for contract invocations) + +```bash +# Get transaction status (Soroban) +curl -X POST https://soroban-testnet.stellar.org \ + -H "Content-Type: application/json" \ + -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"getTransaction\",\"params\":{\"hash\":\"\"}}" \ + | jq '{status: .result.status, resultXdr: .result.resultXdr}' + +# Decode the result XDR to see the host function error +stellar xdr decode --type TransactionResult --xdr +``` + +--- + +## 7. Quick Reference: Error Codes + +### Horizon transaction result codes + +| Code | Meaning | Typical Cause | +|---|---|---| +| `tx_success` | All operations succeeded | — | +| `tx_failed` | One or more operations failed | Check `operations[].result_codes` | +| `tx_bad_seq` | Wrong sequence number | Concurrent submissions; reload account | +| `tx_too_late` | Submitted after max_ledger | Rebuild and resubmit immediately | +| `tx_too_early` | Submitted before min_ledger | Wait for target ledger | +| `tx_insufficient_fee` | Fee below network minimum | Increase base fee | +| `tx_no_account` | Source account does not exist | Create and fund account | +| `tx_bad_auth` | Invalid signature | Check signing key | +| `tx_internal_error` | Unexpected ledger error | Check Stellar status page | + +### Horizon operation result codes + +| Code | Meaning | +|---|---| +| `op_success` | Operation succeeded | +| `op_low_reserve` | Account below minimum balance | +| `op_not_authorized` | Account lacks required auth flag | +| `op_no_destination` | Destination account does not exist | +| `op_no_trust` | Recipient has no trustline for asset | +| `op_line_full` | Recipient trustline is at limit | +| `op_underfunded` | Insufficient balance to send | + +### Soroban host function error types + +| Error | Meaning | +|---|---| +| `AuthorizationError` | Invoker not authorized for function | +| `InvalidInput` | Argument type or value rejected | +| `ExceededLimit` | CPU/memory/ledger resource cap hit | +| `MissingLedgerEntry` | Contract or storage entry not found (TTL expired?) | +| `ContractError(N)` | Contract-defined error — see contract source for code `N` | + +For SolarProof contract-specific error codes, see [`docs/CONTRACT_INTERFACE_DOCS.md`](./CONTRACT_INTERFACE_DOCS.md). + +--- + +## See Also + +- [`docs/DEPLOYMENT.md`](./DEPLOYMENT.md) — deploy and upgrade contracts +- [`docs/deployments.md`](./deployments.md) — current contract IDs per network +- [`docs/METER_INTEGRATION.md`](./METER_INTEGRATION.md) — meter key setup +- [Stellar Developers: Error Codes](https://developers.stellar.org/docs/data/horizon/api-reference/errors/result-codes) +- [Soroban RPC Reference](https://developers.stellar.org/docs/data/rpc/api-reference)