feat(installer): node-local storage prototype (RFC-0003 Option C, flag-gated)

[saadqbal]

Draft / prototype — do not merge. Flag-gated behind TB_STORAGE_MODE, default hostpath (today's behavior, byte-for-byte unchanged). Opens the RFC-0003 Option C (node-local) path for testing on a real machine.

Implements the storage half of client#367. Design: backend#1151. RFC: tracebloc/cli#366.

Goal (Asad + Lukas)

Deleting the cluster must destroy dataset data, and data must not sit as a browsable folder in ~/.tracebloc — it should live inside the cluster's sandbox.

What TB_STORAGE_MODE=node-local does

• Stores datasets on k3s's built-in local-path provisioner inside the k3d node → data dies with cluster delete, is not a host folder, needs no chmod 777.
• Drops the -v ${HOST_DATA_DIR}:/tracebloc@all bind-mount and keeps k3s local-storage (the installer disables it in hostpath mode because the chart ships its own manual StorageClass).
• Chart values: storageClass.create=false + name=local-path, hostPath.enabled=false (reuses the chart's existing dynamic-PVC path — same one EKS/AKS use). The mysql privileged chown init-container auto-drops when hostPath.enabled=false.
• C1: forces AGENTS=0 (single-node). local-path is RWO + WaitForFirstConsumer and provisions on one node, but the shared data PVC is mounted by jobs-manager-spawned Jobs that could otherwise land on a second node with no volume.

Not in scope / notes

• Default hostpath path unchanged; existing installs upgrade via --reuse-values and do not silently migrate (they move to node-local only on a clean delete + reinstall — matches "delete means gone", no data-copy tool needed).
• chmod 777 removal is not a separate PR: on the hostpath model it's load-bearing (host-user writes into /data/shared + kubelet doesn't apply fsGroup to hostPath volumes), so it can't be dropped standalone safely. node-local removes the host dirs entirely, so 777 is gone by construction here.
• HOST_DATASET_DIR (network-mount datasets, backend#743) is left on the hostpath path for now; combining it with node-local is a follow-up.

Tested here

• bats scripts/tests/cluster.bats — 29/29 (incl. 2 new: node-local omits mount + keeps local-storage; hostpath keeps both).
• bats scripts/tests/common.bats — pass.
• shellcheck — no new findings.
• Rendered the generated values for both modes + the HOST_DATASET_DIR case → valid YAML.

Needs real-machine verification (can't run k3d in review env) — checklist from client#367

1. helm template with node-local values → no hostPath PV, PVCs bind local-path, no mysql init-container.
2. Fresh TB_STORAGE_MODE=node-local install → ingest a dataset → confirm nothing under ~/.tracebloc, pod mounts fine.
3. Run a training job → the spawned Job mounts the data PVC on the single node (the C1 assertion).
4. k3d cluster stop && start → data survives.
5. tracebloc delete (and a raw k3d cluster delete) → data gone.
6. Existing hostpath install upgrade → still works, no silent migration.

[saadqbal]

Verified on a real Mac (k3d v5.8.3, k3s v1.31.5, docker desktop)

Ran this against a throwaway single-node cluster built with the exact args node-local emits (no -v bind-mount, local-storage kept, --agents 0), isolated kubeconfig, real cluster untouched.

Chart rendering (helm template):

	node-local	hostpath (default)
hostPath PV objects	0	3 (/tracebloc/<rel>/…)
PVC storageClassName	local-path	client-storage-class
mysql chown init-container	absent	present

Live storage mechanics:

• local-path StorageClass present (rancher.io/local-path, WaitForFirstConsumer).
• PVC bound, pod wrote a marker file.
• Data is not under ~/.tracebloc — it lives in the Docker named volume backing the node's /var/lib/rancher/k3s (on macOS that's inside the Docker VM, not the host FS).
• cluster stop && start → marker survived (restart-persistence holds).
• cluster delete → node + its Docker volume removed, data gone (delete-persistence holds).

So the RFC-0003 goal is proven at the substrate level: data sandboxed in the node, survives restart, destroyed on delete, never a browsable host folder.

Still needs a credentialed run (can't do without client creds): a full TB_STORAGE_MODE=node-local install → ingest → run a training job (step 3 — the C1 assumption that jobs-manager-spawned Jobs mount the shared data PVC). Note RWO = ReadWriteOnce per node, so multiple pods on the single node share the volume fine; the only failure mode was cross-node, which AGENTS=0 removes.

[saadqbal]

Full validation on a real credentialed install (TB_STORAGE_MODE=node-local, dev)

Installed a fresh node-local client from this branch (isolated: CLUSTER_NAME=tb-nodelocal, own HOST_DATA_DIR), untouching the existing hostpath install.

Install came up clean: single-node (1 server, 0 agents), all three PVCs (mysql-pvc, client-pvc, client-logs-pvc) Bound RWO on local-path, zero hostPath PVs, full stack Running (mysql, jobs-manager 2/2, proxies).

C1 under a real workload (not synthetic): ingested a tabular dataset (train=120, test=40). Each ingest ran as a real jobs-manager-triggered ingest-job pod that scheduled on the single node and mounted the shared client-pvc at /data/shared alongside jobs-manager — completing with all validators passing and the datasets registered to the dev backend. That's the exact spawned-pod-shares-RWO-local-path-volume-on-one-node pattern a training job uses.

Delete-lifecycle (Problem A):

• Before: both ingested CSVs + the MySQL InnoDB files (ibdata1, ib_logfile*, dataset tables) present inside the node's Docker volume; ~/.tracebloc-nodelocal held no dataset data.
• After k3d cluster delete: node container gone, the k3s Docker volume destroyed with it, data gone. Clean slate by construction — no host paths for offboard to chase.

Bug found + fixed here (5e4ea45): node-local still created empty 777 ~/.tracebloc/{data,logs,mysql} dirs because _ensure_tracebloc_dirs is called from a second site (install_client_helm) the first gate missed. No data leaked (dirs were empty), but now node-local only ensures the base dir (for values.yaml + log) and skips the world-writable subdirs.

Not exercised: a full backend-driven training experiment. It was blocked on a dev-account auth issue (/dataset/admin/ → 401) that is orthogonal to storage — the spawned ingest-job already exercises the same node-local storage path a training pod would. Worth completing separately once the dev-account/permissions are sorted, but it adds no new storage-validation coverage.

Bottom line: node-local (Option C, single-node) works end-to-end on a real install and delivers the RFC-0003 goal — data sandboxed in the node, shared correctly across pods on one node, destroyed on cluster delete, never a browsable host folder.