Add nomad trigger: safely update HashiCorp Nomad-managed containers - #1123
Open
therustyrobot wants to merge 2 commits into
Open
Add nomad trigger: safely update HashiCorp Nomad-managed containers#1123therustyrobot wants to merge 2 commits into
therustyrobot wants to merge 2 commits into
Conversation
The docker/dockercompose triggers stop, remove, and recreate a container directly via the Docker API. When that container is actually managed by Nomad, Nomad notices its container disappeared and reconciles independently -- fighting whatever the trigger just created, since the replacement isn't part of any allocation Nomad knows about (no service registration, no template-rendered secrets, no restart-policy supervision). This trigger instead calls Nomad's own allocation restart API (POST /v1/client/allocation/:alloc_id/restart), reading the com.hashicorp.nomad.alloc_id / task_name labels Nomad already stamps onto every container its Docker driver creates. Nomad's own driver then handles the pull (given force_pull = true on the task) and recreation, so the allocation stays under normal Nomad supervision throughout.
com.hashicorp.nomad.task_name is missing on at least Nomad v2.0.4 (alloc_id was present, task_name was not), despite existing in Nomad's own driver source. Since Nomad's Docker driver always names containers "<task_name>-<alloc_id>" (drivers/docker/driver.go), we can recover the task name reliably without that label. If neither the label nor the container-name parse works, refuse to restart rather than silently falling back to AllTasks -- that could mean restarting a database sidecar nobody asked to touch.
Tom-V
reviewed
Aug 5, 2026
| const config = configuration || this.configuration; | ||
| return { | ||
| ...config, | ||
| token: config.token ? '_****_' : undefined, |
Contributor
There was a problem hiding this comment.
There is a mask function in WUD available to use
Suggested change
| token: config.token ? '_****_' : undefined, | |
| token: Nomad.mask(config.token) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
nomadtrigger provider for containers whose lifecycle is managed by HashiCorp Nomad rather than plain Docker/Compose.Why not the existing
docker/dockercomposetriggers?Those triggers update a container by stopping it, removing it, and creating a replacement directly through the Docker API. That's the right approach when WUD (or the operator) is the sole owner of the container's lifecycle.
It breaks down when something else is supervising the container -- Nomad included. Nomad's client continuously reconciles the containers backing its allocations against what it expects to be running. If a
docker/dockercomposetrigger deletes and recreates a Nomad-managed container out from under it, Nomad sees its tracked container disappear, treats it as a task failure, and reconciles independently -- fighting whatever the trigger just created. The replacement container also isn't part of any allocation Nomad knows about, so it loses Nomad's service registration, health checks, and template-rendered secrets/config.What this trigger does instead
It calls Nomad's own allocation restart API (
POST /v1/client/allocation/:alloc_id/restart) and lets Nomad's own Docker driver handle the pull and recreation, the same way it would for any other restart. The allocation stays under Nomad's normal supervision throughout -- no fighting, no orphaned containers.Nomad stamps every container it creates with a
com.hashicorp.nomad.alloc_idlabel (and, on at least some versions,com.hashicorp.nomad.task_name) -- seedrivers/docker/driver.goin the Nomad source. This trigger reads those to target the restart, with a fallback to parsing the task name out of the container's own name (Nomad always names containers<task_name>-<alloc_id>) for the Nomad versions/setups where thetask_namelabel isn't present -- I observed it missing entirely on Nomad v2.0.4 whilealloc_idwas still set. If neither source yields a task name, the trigger logs a warning and does not restart anything, rather than silently falling back to restarting every task in the allocation (which could mean restarting an unrelated database sidecar in the same task group).For the restart to actually pick up a new image, the target Nomad task needs
force_pull = trueset in its job spec -- otherwise Nomad's driver reuses whatever's already cached locally, same as any other restart would.Testing done
Nomad.test.ts), 11 passing, 100% line coverage on the new filenpm run lint,npm run build(tsc --noEmit), and the full existing suite (63 suites / 612 tests) all pass with no regressionsAUTO=falsefor manual-only execution via the existing "run trigger" UI/API)wud.trigger.include, invoked it through WUD's/api/triggers/nomad/:nameendpoint, and confirmed vianomad alloc statusthat the same allocation ID restarted cleanly (0 → 1 restarts) with no duplicate/orphaned containers, and the service was healthy again afterwardDocs
Added
docs/configuration/triggers/nomad/README.mdfollowing the existing per-trigger doc convention, and registered it indocs/configuration/triggers/sidebar.md.Happy to adjust naming/config shape if it doesn't match where you'd want this to land -- this is my first contribution here, so let me know if there's anything I should do differently.