fix(infra): training image ENTRYPOINT; document build-branch and pref…

[Enskc05]

fix(infra): training image ENTRYPOINT; document build-branch and prefix traps

Follow-up to #30. The first smoke test on the server hit three blockers; all
three are closed here.

1. CMD → ENTRYPOINT

Kubernetes args replace a container's CMD, but are appended to its
ENTRYPOINT. With the training command in CMD, a Job that passes only Hydra
overrides as args made the runtime try to execute the override itself:

StartError: exec: "training.epochs=1": executable file not found in $PATH

Moving the command to ENTRYPOINT makes args behave as intended. The example
manifest now carries an explicit "do not set command" note, since setting it
would override the entrypoint and silently drop every Hydra override.

2. Build can silently use the wrong branch

The training code lives on ml/feature while the Dockerfile lives on main,
so the build overlays three files from main onto an ml/feature checkout.
If the file list is dropped from that second checkout, git checkout origin/main switches the entire tree to main — the build then succeeds
against main's services/ml, and the failure only surfaces at run time:

No module named services.ml.training.train

This is easy to miss because requirements/ml.txt looks correct either way
(#30 added the missing packages to main). The runbook now includes a
pre-build verification step that checks for files which only exist on
ml/feature.

3. MinIO prefix mismatch — confirmed

services/ml/conf/data/default.yaml sets bucket_name: datasets and
minio_prefix: datasets/training-512/v1. The actual layout on the server:

$ mc ls dh/datasets/training-512/v1/
clean/ degraded/

So the prefix should be training-512/v1; as configured, boto3 looks for
datasets/datasets/training-512/v1 and finds nothing. The permanent fix
belongs to the ML squad — until then, runs need
data.minio_prefix=training-512/v1 as an override, which the runbook now
documents.

Testing

Image builds clean and the dependency verification step passes
(torch 2.6.0+cu124 | cuda 12.4 | numpy 2.2.2). The container starts and
imports resolve inside the cluster. End-to-end training smoke test is running
against the rebuilt image; results to follow.