CloudApiClient: no retry for idempotent reads, one transient 503 fails the whole operation

[Nicolas0315]

Summary

Every method on CloudApiClient (packages/arkor/src/core/client.ts) is single-shot (rg -in "retry" src/core/client.ts → no hits). The SSE stream gets a full reconnect state machine in trainer.ts, but plain reads (job detail, deployment list/get, ...) fail on the first transient 503 / 429 / network blip. Studio surfaces these as errors the user has to manually refresh past, and SDK callers see avoidable throws.

Proposal

A small bounded retry for idempotent READ operations only:

• attempts: 3 total; delay: full-jitter exponential (base 300 ms, ×2, cap 2 s);
• retry on network-level errors and 408 / 429 / 5xx; never on other 4xx;
• explicitly NOT retried: createJob, cancelJob, deployment mutations, chat, openEventStream (POSTs have no upstream idempotency keys; the stream already has its own reconnect logic in trainer.ts and must not be double-retried).

Injectable sleep via CloudApiClientOptions (next to the existing fetch seam) keeps tests instant and deterministic.

(A PR implementing this, with vitest coverage incl. "createJob is never retried", is ready to follow this issue.)

[ARYAN0-work]

Hi, I'd like to work on this issue.

I traced through CloudApiClient and noticed that the most GET methods call the typed RPC client directly ( e.g. this.rpc.v1...$get(...) ), and there doesn't seem to be a shared request helper inside CloudApiClient.

I also looked through the existing tests in client.test.ts to understand the current testing setup and patterns and saw that the client is constructed with an injected fetch, which seems to be the existing testing seam.

Before I start implementing, I wanted to clarify the intended insertion point:

Should the retry logic wrap the individual idempotent GET operations inside CloudApiClient, or is the expectation to wrap the injected fetch used by the RPC client (while ensuring POST operations are not retried)?

I wanted to confirm the intended architecture before opening a PR. Thanks!

[Nicolas0315]

Thanks for the careful analysis. The intended boundary is the fetch implementation passed to the typed RPC client: retry idempotent GET requests there, while keeping direct POST operations and the event-stream path single-shot.

I already have a work in progress along that seam, including retry tests, so please hold off on an implementation for this one to avoid duplicate work. I appreciate the offer and the precise questions.

[ARYAN0-work]

Thanks for explaining! I’ll hold off on working on this issue so I don’t duplicate the work you already have in progress. I appreciate you taking the time to clarify the intended approach. Thanks again!