Define stack payload cache behavior for Python APIs

[drernie]

Context

The original CLI problem has largely been addressed. Commands such as quiltx bucket add now use the shared ensure_stack_payload() helper, which reads an existing cache or discovers and caches CloudFormation metadata on demand. Commands that can operate without full CloudFormation access may use a lightweight in-memory account/region fallback.

The remaining work is to define and consistently implement the cache contract for public Python APIs and incomplete or stale payloads.

Current behavior

• ensure_stack_payload() returns a cache hit or discovers and persists the full stack payload on a cache miss.
• With allow_lightweight=True, failed CloudFormation discovery can return an in-memory payload derived from catalog configuration and ambient AWS identity; this fallback is not cached.
• CLI consumers such as bucket add use the on-demand helper.
• Catalog.payload, load_stack_payload(), and require_stack_payload() remain cache-only.
• The lower-level quiltx.bucket.add_bucket() API still reads Catalog.payload directly and instructs callers to populate the cache manually.
• Existing cache entries are accepted without age, schema-version, or required-field validation.

Scope

• Define which public Python APIs are cache-only and which must discover on demand.
• Make quiltx.bucket.add_bucket() follow the chosen contract rather than having behavior that differs unexpectedly from the CLI.
• Decide whether lightweight payloads should remain ephemeral or be persisted with explicit provenance and completeness metadata.
• Define validation and refresh behavior for incomplete, incompatible, or stale cache entries.
• Document automatic discovery, lightweight fallback, explicit refresh, and cache-only APIs.

Acceptance criteria

• Public API behavior is documented and consistent: cache-only access and on-demand discovery are clearly distinguished.
• quiltx.bucket.add_bucket() either discovers on demand or explicitly accepts a payload/context that removes its hidden cache prerequisite.
• Cache entries are validated for required fields and schema compatibility before use.
• The refresh policy is deterministic and available to callers without deleting cache files manually.
• Lightweight fallback persistence is an explicit, tested design decision.
• Tests cover cache hits, cache misses with successful discovery, forced refresh, invalid/incomplete cache entries, and lightweight fallback.