Transient S3 503 on describe_table misroutes existing table into create path, failing scheduled sync

[tenequm]

Symptom

Scheduled pond sync against the Hetzner S3 store intermittently fails with:

Error: failed to create table messages
Caused by:
    0: Namespace error: ... Failed to create table messages: Generic { store: "S3",
       source: RetryError(... method: PUT,
       uri: .../pond/messages.lance/.lance-reserved, retries: 3, max_retries: 3,
       inner: Status { status: 503, body: "<Code>ServiceUnavailable</Code>..." }) }

Observed 8 failures out of 250 runs in 24h (~3.2%) on host beelink-eq13. Every failure self-recovered on the next 5-minute cycle. The messages table obviously exists (2.2M rows) - pond should never be on the create path at all.

Root cause

A transient Hetzner 503 on a LIST gets misclassified as "table does not exist" inside lance-namespace-impls, which routes pond into the create path, where a second 503 then surfaces as the visible error.

Chain:

1. Hetzner object storage returns a transient 503 (ServiceUnavailable, RequestId: N/A). Lance's retry layer (3 retries) usually absorbs these, but occasionally exhausts.
2. open_or_create_via_ns (src/substrate.rs) calls describe_table first. In lance-namespace-impls-8.0.0, that bottoms out in check_table_status (dir.rs:2001), which swallows every object-store error as non-existence:

   match self.object_store.read_dir(table_path).await {
       Ok(entries) => { let exists = !entries.is_empty(); ... }
       Err(_) => TableStatus { exists: false, .. },   // transient 503 == "absent"
   }

   So describe_table returns TableNotFound for a table that exists.
3. pond matches TableNotFound and falls through to create (src/substrate.rs, "fall through to create" -> Dataset::write_into_namespace).
4. The create's PUT of messages.lance/.lance-reserved hits a second 503, exhausts retries, and that error - failed to create table messages - is what the user sees. The reported error names the wrong operation; the trigger was the earlier LIST failure.

Why no data was at risk

Even if the .lance-reserved PUT had succeeded, create_table re-checks check_table_status and errors with TableAlreadyExists when the table has actual manifests, and WriteMode::Create conflicts on the existing _versions/. The 503 only aborted the sync run; the next cron cycle recovered cleanly.

Fix options

1. pond-side (preferred): in open_or_create_via_ns, don't trust a single TableNotFound from describe_table. Retry the describe on ambiguity, or verify absence independently before entering the create path, so a transient store blip cannot route an existing table into create.
2. Upstream: check_table_status in lance-namespace-impls should propagate transient store errors instead of mapping every Err to exists: false. Worth filing against lance-namespace.
3. Cosmetic: Restart=on-failure on pond-sync.service tightens the recovery window, but the misclassification remains.

This is the same Hetzner-503 class documented in CLAUDE.md ("never tune Lance's AIMD limiter") - the correct response is tolerating/retrying transients on the pond side, not widening the rate limiter.

Environment

• pond built against lance-namespace / lance-namespace-impls 8.0.0, lance 7.0.0 pinned
• Store: s3+https://nbg1.your-objectstorage.com/pondarium/pond (Hetzner object storage)
• systemd user timer, every 5m