import: the registry token is fetched once and never refreshed, so long multi-layer pulls fail with 401

[100milliongold]

Summary

enroot import docker://... authenticates once, before the layer downloads start, and reuses that
token for every blob request. On a registry whose bearer tokens are short-lived, a pull that runs
longer than the token's remaining lifetime fails with 401 Unauthorized partway through, and the
import aborts without producing an image. Neither ENROOT_TRANSFER_RETRIES nor GNU parallel's own
retries recover from it, because both reuse the same expired token.

Code path

Line numbers are from src/docker.sh at master (ead3a25e, identical to v4.2.1).

docker::_download authenticates once and turns the result into curl arguments:

# 145-149
docker::_authenticate "${user}" "${registry}" "${url_manifest}"
if [ -f "${token_dir}/${registry}.$$" ]; then
    req_params+=("-K" "${token_dir}/${registry}.$$")
fi

docker::_authenticate writes the token into that file (line 89) and is called from only two places
in the file: line 146 here and line 425 in docker::_configure. There is no call inside the layer
loop. The loop passes the same req_params to every worker:

# 188-189
BASH_ENV="${BASH_SOURCE[0]}" parallel --plain ${TTY_ON+--bar} --xapply --shuf --retries 2 -j "${ENROOT_MAX_CONNECTIONS}" -q \
  docker::_download_extract "{1}" "{2}" "${curl_opts[@]}" -f "${req_params[@]}" -- "${url_digest}sha256:{1}" ::: "${missing_digests[@]}" ::: "${missing_media_types[@]}"

v3.2.0 has the same shape (single call at line 124, loop at line 167), so this is not a regression
in the 4.x line.

Why the existing retries do not help

There are two retry layers, and neither one recovers from an expired token, so this is not
something a user can configure around.

• curl_opts is readonly (lines 22 and 25) and carries --retry "${ENROOT_TRANSFER_RETRIES}".
  curl --retry does not retry a 401, and if it did it would resend the same token.
• parallel --retries 2 is hard-coded on line 188, independent of ENROOT_TRANSFER_RETRIES. It
  does re-run docker::_download_extract, but req_params was fixed in the caller, so the rerun
  reads the same unrefreshed -K file and can only repeat the same 401.

What we measured on nvcr.io

Anonymous bearer tokens from nvcr.io:

• The JWT's own lifetime is fixed: exp - iat = 600 seconds.
• The expires_in field of the response is not that lifetime. It is the time left on the token
  the server hands out, because the server serves cached tokens. Ten consecutive requests returned:
  600 379 600 378 98 598 600 596 96 595. A separately inspected token had expires_in = 95 and
  exp - now = 95, with exp - iat = 600.
• The registry grants a short grace period past expiry: a blob request 30 s after expiry still
  succeeded, 150 s after expiry returned 401.

One detail decides when the failure happens: a blob request answers 307, redirecting to a signed
CDN URL, and the token is only needed for that 307. The transfer that follows is not authenticated
against the registry. We downloaded an 85 MB layer with --limit-rate 200k over 7 min 24 s and it
succeeded.

So the failure condition is not "the download is slow". It is "a late layer's 307 request happens
after the token's expiry plus grace". Two consequences follow from the code, and both are easy to
check:

• A single-layer image never hits this, however large, because there is only one 307.
• Re-running the import eventually succeeds: cached layers shrink missing_digests, so the later
  307 requests move earlier in the run.

The token lifetime is not an nvcr.io peculiarity

Anonymous token requests to four registries, with the lifetime read from the JWT (exp - iat):

Registry	expires_in	JWT lifetime	Time already elapsed when issued
nvcr.io	600	600 s	varies; see above
docker.io	300	300 s	0
quay.io	absent	3600 s	0
ghcr.io	anonymous request rejected with 403	not measured	not measured

Docker Hub's tokens live half as long as nvcr.io's. They are handed out fresh, so the window is a
fixed 300 s rather than a variable one, which means every layer's 307 has to happen within five
minutes of the single authentication.

To be clear about what this is: a lifetime measurement, not a failure we have reproduced. We
have only seen the 401 on nvcr.io. We are including the other registries because the token
lifetime is a property of the registry and import does not account for any of them.

Reproduction

enroot import docker://nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.1 on Enroot 4.2.1:

[+   2s] [INFO] Authentication succeeded      <- the only authentication in the run
[+   3s] [INFO] Downloading 35 missing layers...
[+ 718s] curl: (22) The requested URL returned error: 401
[+ 918s] command exits, rc=1

The import leaves no .sqsh file. The 200 s between the 401 and the exit is the already-started
layers finishing. The single authentication is 716 s before the failure, and the token issued there
had at most 600 s of life.

We cannot say which layer failed: enroot prints one Downloading N missing layers... line and no
per-layer progress, so the log only shows that it happened partway through a 35-layer pull.

Suggested fix

Refresh the token when a blob request comes back 401 and retry that request.

The token already lives in a shared file, ${token_dir}/${registry}.$$, referenced by curl -K, so
one refresh can serve every worker: a worker that hits 401 takes a lock on that file, re-runs
docker::_authenticate, and releases it, and the other workers pick up the new token on their next
request. Without a lock, ENROOT_MAX_CONNECTIONS workers would each request a token.

parallel --retries 2 already re-runs a failed layer, and the retry re-reads the -K file, so a
refresh that lands before the job exits makes the existing retry succeed on its own. That keeps the
patch small, though the budget of two attempts is thin enough that retrying inline after the refresh
is probably safer.

Two details in the current structure that a patch has to deal with:

• docker::_download_extract receives only curl arguments, so it has no user, registry, or
  manifest URL to authenticate with. Those would have to be passed in or exported.
• It runs curl directly with -f rather than common::curl, so the 401 arrives as curl exit
  status 22 and the HTTP status is not otherwise visible.

Would you take a PR along these lines? We have the environment to test it against a registry with
short-lived tokens.

Environment

• Enroot 4.2.1 (packaged), library at /usr/lib/enroot/docker.sh
• Ubuntu, GNU parallel, curl 8.x
• Registry: nvcr.io, anonymous pull