Add a fal.ai Compute provider

[coygeek]

Summary

Add fal as a built-in Crabbox provider backed by fal.ai Compute. The first implementation should target fal Compute dedicated GPU instances as a direct SSH lease provider, not fal Model APIs or fal Serverless, because Crabbox providers need to run arbitrary repo commands and fal Compute exposes full SSH access to Linux GPU machines.

Why

Crabbox already supports direct cloud and GPU-oriented SSH lease providers such as Lambda Cloud, RunPod, Nebius, and NVIDIA Brev. fal.ai Compute fits the same Crabbox execution model:

• fal Compute provides dedicated GPU instances that stay running under the user's control.
• The documented flow provisions an instance, accepts an SSH public key, waits for the instance to become ready, and connects as ubuntu@<instance-ip>.
• fal exposes Platform APIs for compute instance create, get, list, and delete.
• Instance types documented today are 1xH100-SXM and 8xH100-SXM.

This would let users run:

FAL_KEY=... crabbox run --provider fal -- nvidia-smi
FAL_KEY=... crabbox warmup --provider fal --keep --slug fal-h100
FAL_KEY=... crabbox ssh --provider fal fal-h100
FAL_KEY=... crabbox stop --provider fal fal-h100

Provider shape

Implement fal Compute as an SSH lease provider:

• Provider name: fal
• Optional alias: fal-ai
• Kind: ProviderKindSSHLease
• Family: fal
• Targets: Linux only
• Initial features:
  • ssh
  • crabbox-sync
  • cleanup
• Coordinator: CoordinatorNever

Do not route this through the Worker coordinator initially. Crabbox's coordinator provider registry currently only brokers aws, azure, gcp, and hetzner, and the existing direct GPU providers are CLI-owned.

Non-goals for the first PR

• Do not implement fal Model API inference as the Crabbox provider path. Model APIs run endpoint-specific JSON requests and do not provide an SSH host for arbitrary command execution.
• Do not implement fal Serverless deployment or fal run as a delegated-run backend in the first PR. Serverless owns the app container and endpoint contract, which is a different execution model from a Crabbox SSH lease.
• Do not add coordinator support until there is a separate design for storing fal credentials and lifecycle state in the broker.
• Do not expose FAL_KEY through CLI flags or persisted config output.

fal.ai docs evidence

Source analyzed: fal-docs.ai/2026-06-24_fal_ai_combined.md.

Relevant docs:

• https://fal.ai/docs/documentation/compute
  • fal Compute is dedicated GPU infrastructure with full SSH access.
  • Compute instances are intended for training, fine-tuning, batch jobs, and workloads needing sustained hardware access.
  • Documented instances include 1xH100-SXM and 8xH100-SXM.
• https://fal.ai/docs/documentation/compute/quickstart
  • Users paste an SSH public key at creation time.
  • Ready state usually takes 2 to 3 minutes.
  • SSH example uses ubuntu@<instance-ip>.
• https://fal.ai/docs/api-reference/platform-apis
  • Platform APIs list create, get, list, and delete operations for Compute instances.
• https://fal.ai/docs/documentation/setting-up/authentication
  • fal keys are read from FAL_KEY.
  • Raw API examples use Authorization: Key $FAL_KEY.
  • API keys are account or team scoped.

Additional docs were reviewed for Model APIs, queues, uploads, concurrency, retries, and webhooks. Those are useful context for future fal inference features, but not the best fit for this Crabbox provider because they do not expose a normal SSH lease.

Crabbox codebase analysis

Relevant Crabbox provider architecture:

• internal/cli/provider_backend.go
  • Defines Provider, ProviderSpec, ProviderKindSSHLease, FeatureSSH, FeatureCrabboxSync, FeatureCleanup, and SSHLeaseBackend.
  • Core owns provider selection, sync, command execution, claims, status rendering, and release workflows.
• docs/provider-backends.md and docs/features/provider-authoring.md
  • New SSH providers should return a usable LeaseTarget.SSH and let core handle rsync, command execution, results, and release.
• internal/providers/all/all.go
  • Built-in providers register through side-effect imports.
• internal/providers/lambda
  • Good pattern for a direct GPU cloud SSH provider with an HTTP API client, per-lease SSH key handling, readiness polling, cleanup, and doctor checks.
• internal/providers/runpod
  • Good pattern for direct GPU provider lifecycle with public SSH coordinates and REST API polling.
• docs/providers/provider-metadata.json and docs/providers/README.md
  • Provider metadata and the generated provider matrix must be kept in sync.
• scripts/generate-provider-matrix.mjs and scripts/check-provider-matrix.mjs
  • Docs checks fail when provider registration, metadata, docs paths, or the generated matrix drift.

Proposed implementation

1. Add provider package

Create:

internal/providers/fal/
  provider.go
  backend.go
  client.go
  config.go
  flags.go
  doctor.go
  *_test.go

Provider spec:

func (Provider) Spec() core.ProviderSpec {
	return core.ProviderSpec{
		Name:        "fal",
		Family:      "fal",
		Kind:        core.ProviderKindSSHLease,
		Targets:     []core.TargetSpec{{OS: core.TargetLinux}},
		Features:    core.FeatureSet{core.FeatureSSH, core.FeatureCrabboxSync, core.FeatureCleanup},
		Coordinator: core.CoordinatorNever,
	}
}

Register it in internal/providers/all/all.go.

2. Add fal config and flags

Add config fields consistent with direct providers:

provider: fal
fal:
  instanceType: 1xH100-SXM
  sector: default
  user: ubuntu
  apiURL: ""

Suggested env vars:

• FAL_KEY, primary because fal docs and SDKs use it.
• CRABBOX_FAL_KEY, optional Crabbox-prefixed override for consistency with other providers.
• CRABBOX_FAL_INSTANCE_TYPE
• CRABBOX_FAL_SECTOR
• CRABBOX_FAL_API_URL, for tests and enterprise/private endpoints if needed.

Suggested flags:

• --fal-instance-type
• --fal-sector
• --fal-user
• --fal-api-url

Security requirements:

• Do not accept the API key via CLI flag.
• Do not print the API key in doctor, config show, errors, debug output, or tests.
• Redact fal credentials the same way LAMBDA_API_KEY and RUNPOD_API_KEY are handled.

3. Implement a small fal Compute API client

Use Go's net/http, following the direct provider client style. The client should:

• Read the key from config/env.
• Send Authorization: Key <token>.
• Set Accept: application/json.
• Use Content-Type: application/json for writes.
• Limit response bodies.
• Reject cross-origin redirects unless the configured API URL is a trusted loopback test URL.
• Decode structured error bodies when available and include HTTP status, fal request ID, and safe error type details.

Needed API operations:

• Create compute instance with:
  • instance type,
  • sector,
  • SSH public key,
  • Crabbox-owned name or labels if the API supports them.
• Get compute instance by ID.
• List compute instances.
• Delete compute instance by ID.

Important implementation note: verify exact REST paths and request/response schemas from fal's OpenAPI before coding. The combined docs list the operations, but not all operation payload schemas are present in the local bundle.

4. Implement SSH lease lifecycle

Acquire should:

1. Generate a Crabbox lease ID and direct lease slug.
2. Generate or prepare a Crabbox-managed SSH key for the lease, matching patterns from existing direct SSH providers.
3. Create the fal Compute instance with the public key.
4. Poll the fal instance until it has:
   • terminal non-error lifecycle state,
   • public SSH host or IP,
   • SSH user, defaulting to ubuntu if the API does not return one.
5. Wait for SSH readiness.
6. Return LeaseTarget with Server, SSHTarget, and LeaseID.
7. Claim the lease for the repo.
8. If creation or readiness fails and --keep was not set, delete the partially created instance.

Resolve should:

• Resolve by Crabbox lease ID, slug, or provider instance ID when possible.
• Rebuild LeaseTarget.SSH from fal instance state.
• Preserve existing local claim behavior.

List should:

• List fal Compute instances owned or recognizable by Crabbox.
• Map them to normalized LeaseView records.
• Include provider instance ID, instance type, region/sector, state, public IP, and created time when available.

ReleaseLease should:

• Delete the fal Compute instance.
• Remove the local lease claim after successful delete.
• Treat already-deleted/not-found responses as successful cleanup when safe.

Touch should:

• Reuse Crabbox's direct lease label/touch conventions where possible.

Cleanup should:

• Provide dry-run and mutating cleanup for orphaned Crabbox-owned fal instances.
• Never delete instances that are not clearly owned by Crabbox.
• Include a conservative ownership rule if fal labels/tags are unavailable.

Doctor should:

• Verify FAL_KEY or CRABBOX_FAL_KEY is present.
• Call a read-only endpoint such as list instances or account/billing readiness.
• Report auth, Compute access, inventory readability, and default config.
• Avoid creating any paid resource.

5. Add docs and provider matrix metadata

Add or update:

• docs/providers/fal.md
• docs/providers/provider-metadata.json
• docs/providers/README.md, generated by provider matrix script
• docs/source-map.md, if the provider list there is kept manually
• README.md, if the provider table is generated or expected to include the new provider

Suggested provider metadata:

"fal": {
  "status": "built-in",
  "category": "gpu-cloud",
  "substrate": "fal Compute H100 instance",
  "location": "cloud",
  "ssh": "crabbox-managed",
  "sync": "crabbox-sync",
  "gpu": "yes",
  "lifecycle": "Crabbox",
  "cleanup": "instance delete",
  "bestFit": "Dedicated H100 GPU workloads over SSH",
  "caveat": "Requires fal Compute access, FAL_KEY, quota, and available H100 capacity",
  "docs": "fal.md"
}

6. Add tests

Unit tests should use fake HTTP servers and no live fal credentials.

Suggested tests:

• Provider registration:
  • ProviderFor("fal") resolves.
  • Optional alias fal-ai resolves if added.
  • Spec is Linux-only SSH lease, direct only, with ssh, crabbox-sync, and cleanup.
  • Provider exposes DoctorProvider.
• Config:
  • FAL_KEY and CRABBOX_FAL_KEY loading.
  • config show does not leak token or token env names.
  • Invalid API URL is rejected.
  • HTTPS required except trusted loopback test URL.
• Client:
  • Sends Authorization: Key <token>.
  • Handles create/get/list/delete.
  • Handles 401/403 with clear auth errors.
  • Handles 404 delete idempotently where intended.
  • Limits response bodies.
  • Rejects cross-origin redirects.
• Backend:
  • Acquire creates instance, waits ready, waits SSH, claims lease.
  • Acquire rolls back on readiness failure when not kept.
  • Resolve reconstructs SSH target.
  • List maps fal instances to LeaseView.
  • Release deletes instance and removes claim.
  • Cleanup does not delete unowned instances.
• Docs:
  • Provider matrix check passes.
  • Provider docs path exists.

7. Optional live smoke

Add a guarded live smoke only if credentials and Compute access are available:

CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=fal FAL_KEY=... scripts/live-fal-smoke.sh

The live smoke should:

1. Run doctor --provider fal.
2. Assert starting inventory is safe or isolated by Crabbox-owned names.
3. warmup --provider fal --keep --slug fal-live-....
4. status --provider fal --wait.
5. run --provider fal --id <lease> --no-sync -- nvidia-smi.
6. stop --provider fal <lease>.
7. Run cleanup dry-run and final inventory check.

The script must skip cleanly without CRABBOX_LIVE=1, without fal selected in CRABBOX_LIVE_PROVIDERS, or without FAL_KEY.

Acceptance criteria

• crabbox providers lists fal as a built-in GPU cloud SSH lease provider.
• crabbox doctor --provider fal verifies credentials and Compute access without creating resources.
• crabbox warmup --provider fal --keep provisions a fal Compute instance and returns a reusable lease.
• crabbox run --provider fal -- nvidia-smi runs on the fal Compute instance over Crabbox-managed SSH.
• crabbox ssh --provider fal <id-or-slug> opens an SSH session.
• crabbox list --provider fal --json returns normalized lease data without leaking credentials.
• crabbox stop --provider fal <id-or-slug> deletes the fal Compute instance and removes the local claim.
• Failed acquire rolls back the fal instance unless --keep was requested.
• Cleanup only touches instances Crabbox can prove it owns.
• Docs, provider matrix, and generated provider listings are in sync.
• No fal credential appears in command output, config output, logs, tests, or docs.

Validation checklist

Before opening the PR:

gofmt -w $(git ls-files '*.go')
go vet ./...
go test -race ./...
scripts/check-docs.sh

If Worker files are touched for coordinator support, also run:

npm ci --prefix worker
npm run format:check --prefix worker
npm run lint --prefix worker
npm run check --prefix worker
npm test --prefix worker
npm run build --prefix worker

Suggested PR plan

1. Create a local branch, for example feat/fal-provider.
2. Land the provider package, config, and tests in small commits.
3. Add docs and regenerate the provider matrix.
4. Run the validation checklist.
5. If live credentials are available, run the guarded live smoke and include the result in the PR.
6. Submit this issue and the PR together, linking both by full GitHub URLs.

Open questions

• What are the exact fal Platform API base URL, paths, and request/response schemas for Compute instance lifecycle?
• Does fal Compute expose labels/tags or user metadata that Crabbox can use for safe cleanup ownership?
• Does create support caller-provided names, sectors, and SSH keys directly, or does any of that require prior setup?
• What states should be treated as provisioning, ready, terminal failure, and deleted?
• Are Compute API calls available with normal API scoped keys, or is ADMIN scope required?
• Are there per-team or enterprise constraints that should be surfaced in doctor?
• Should fal-ai be accepted as an alias, or should the provider follow newer Crabbox guidance and use only canonical fal?

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed June 25, 2026, 4:20 PM ET / 20:20 UTC.

Summary
Keep open. Current main and the latest release do not contain a fal provider, and the linked implementation PR is still open with review/proof work remaining, so this issue should continue tracking the request until that PR lands or is closed.

Reproducibility: not applicable. this is a new provider request rather than a report of broken existing behavior. Source inspection verifies that the provider is absent on current main.

Root-cause cluster
Relationship: canonical
Canonical: #694
Summary: This issue is the canonical feature request, and #693 is an open candidate fix using closing syntax.

Members:

• fixed_by_candidate: feat(fal): add SSH lease provider #693 - The PR body says it closes this issue and implements the requested fal SSH lease provider.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.

Next step
No separate ClawSweeper repair lane should race the open implementation PR that already owns this feature request.

Review details

Best possible solution:

Land a reviewed fal provider through #693 after its blockers are resolved, or keep this issue as the canonical request if that PR is closed unmerged.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this is a new provider request rather than a report of broken existing behavior. Source inspection verifies that the provider is absent on current main.

Is this the best way to solve the issue?

Yes, the direct SSH-lease provider direction matches Crabbox's provider contract, but the best path is to review the already-linked implementation PR rather than start a competing fix lane.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The linked implementation PR is open and currently has proof/review blockers, so closing this issue now could hide the remaining merge decision.

Codex review notes: model internal, reasoning high; reviewed against 0ec69d642764.

Label changes

Label changes:

• add P2: A new GPU cloud provider is a normal-priority feature with bounded provider-surface impact.
• add impact:other: The issue affects provider availability for users who want fal.ai Compute as a Crabbox execution backend.
• add issue-rating: 🌊 off-meta tidepool: Current issue advisory state selects this label.
• add clawsweeper:linked-pr-open: Current issue advisory state selects this label.
• add clawsweeper:no-new-fix-pr: Current issue advisory state selects this label.
• add clawsweeper:needs-product-decision: Current issue advisory state selects this label.

Label justifications:

• P2: A new GPU cloud provider is a normal-priority feature with bounded provider-surface impact.
• impact:other: The issue affects provider availability for users who want fal.ai Compute as a Crabbox execution backend.

Evidence reviewed

What I checked:

• Repository policy read: AGENTS.md was read fully; its provider-neutral core boundary and provider-token handling guidance are relevant to a new built-in provider. (AGENTS.md:1, 0ec69d642764)
• Current main registry lacks fal: The built-in provider registry imports existing providers such as Lambda, Nebius, NVIDIA Brev, and RunPod, but no internal/providers/fal package is registered. (internal/providers/all/all.go:3, 0ec69d642764)
• No fal source or docs on main: Targeted search found no fal, fal-ai, FAL_KEY, fal.md, fal Compute, or fal.ai references under the implementation, docs, scripts, README, or changelog paths checked. (0ec69d642764)
• Provider contract supports the requested shape: SSHLeaseBackend is the core interface for providers that acquire, resolve, list, touch, and release SSH-reachable leases. (internal/cli/provider_backend.go:96, 0ec69d642764)
• Existing provider docs show adjacent GPU SSH providers: The generated provider matrix lists Lambda and RunPod as built-in GPU cloud SSH lease providers, but there is no fal row on current main. (docs/providers/README.md:107, 0ec69d642764)
• Latest release also lacks fal: The v0.33.0 provider registry and generated provider matrix include adjacent direct GPU providers but no fal provider. (internal/providers/all/all.go:39, 966e99599db4)

Likely related people:

• coygeek: Current-main history shows substantial adjacent SSH-lease provider work, including the Coder provider addition and provider lifecycle hardening, and blame covers the provider registry and SSH lease interface lines used by this request. (role: recent provider lifecycle contributor; confidence: high; commits: 1b5e52ddaf66, 89019eb7c50f, beb0dabaa97e; files: internal/providers/coder, internal/providers/all/all.go, internal/cli/provider_backend.go)
• Vincent Koc: Recent current-main commits touch Coder live smoke coverage, Coder rollback behavior, and provider SSH defaults adjacent to this issue's lifecycle and live-smoke concerns. (role: recent adjacent provider and smoke-test contributor; confidence: medium; commits: 0ec69d642764, b3dfa1ab3140, c886fc794cef; files: scripts/live-smoke.sh, internal/providers/coder, internal/providers/firecracker)
• Peter Steinberger: History shows adjacent built-in provider work for the AWS Lambda MicroVM provider and provider metadata/registry updates. (role: adjacent built-in provider contributor; confidence: medium; commits: 0301236b2752; files: internal/providers/lambda, internal/providers/all/all.go, docs/providers/provider-metadata.json)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.