Skip to content

Add nomad trigger: safely update HashiCorp Nomad-managed containers - #1123

Open
therustyrobot wants to merge 2 commits into
getwud:mainfrom
therustyrobot:feature/nomad-trigger
Open

Add nomad trigger: safely update HashiCorp Nomad-managed containers#1123
therustyrobot wants to merge 2 commits into
getwud:mainfrom
therustyrobot:feature/nomad-trigger

Conversation

@therustyrobot

Copy link
Copy Markdown

Summary

Adds a nomad trigger provider for containers whose lifecycle is managed by HashiCorp Nomad rather than plain Docker/Compose.

Why not the existing docker/dockercompose triggers?

Those triggers update a container by stopping it, removing it, and creating a replacement directly through the Docker API. That's the right approach when WUD (or the operator) is the sole owner of the container's lifecycle.

It breaks down when something else is supervising the container -- Nomad included. Nomad's client continuously reconciles the containers backing its allocations against what it expects to be running. If a docker/dockercompose trigger deletes and recreates a Nomad-managed container out from under it, Nomad sees its tracked container disappear, treats it as a task failure, and reconciles independently -- fighting whatever the trigger just created. The replacement container also isn't part of any allocation Nomad knows about, so it loses Nomad's service registration, health checks, and template-rendered secrets/config.

What this trigger does instead

It calls Nomad's own allocation restart API (POST /v1/client/allocation/:alloc_id/restart) and lets Nomad's own Docker driver handle the pull and recreation, the same way it would for any other restart. The allocation stays under Nomad's normal supervision throughout -- no fighting, no orphaned containers.

Nomad stamps every container it creates with a com.hashicorp.nomad.alloc_id label (and, on at least some versions, com.hashicorp.nomad.task_name) -- see drivers/docker/driver.go in the Nomad source. This trigger reads those to target the restart, with a fallback to parsing the task name out of the container's own name (Nomad always names containers <task_name>-<alloc_id>) for the Nomad versions/setups where the task_name label isn't present -- I observed it missing entirely on Nomad v2.0.4 while alloc_id was still set. If neither source yields a task name, the trigger logs a warning and does not restart anything, rather than silently falling back to restarting every task in the allocation (which could mean restarting an unrelated database sidecar in the same task group).

For the restart to actually pick up a new image, the target Nomad task needs force_pull = true set in its job spec -- otherwise Nomad's driver reuses whatever's already cached locally, same as any other restart would.

Testing done

  • New unit tests (Nomad.test.ts), 11 passing, 100% line coverage on the new file
  • npm run lint, npm run build (tsc --noEmit), and the full existing suite (63 suites / 612 tests) all pass with no regressions
  • Built the image from this branch and smoke-tested it standalone (registers correctly, respects AUTO=false for manual-only execution via the existing "run trigger" UI/API)
  • Live end-to-end test against a real 3-node Nomad v2.0.4 cluster: opted a running container into the trigger via wud.trigger.include, invoked it through WUD's /api/triggers/nomad/:name endpoint, and confirmed via nomad alloc status that the same allocation ID restarted cleanly (0 → 1 restarts) with no duplicate/orphaned containers, and the service was healthy again afterward

Docs

Added docs/configuration/triggers/nomad/README.md following the existing per-trigger doc convention, and registered it in docs/configuration/triggers/sidebar.md.

Happy to adjust naming/config shape if it doesn't match where you'd want this to land -- this is my first contribution here, so let me know if there's anything I should do differently.

The docker/dockercompose triggers stop, remove, and recreate a
container directly via the Docker API. When that container is
actually managed by Nomad, Nomad notices its container disappeared
and reconciles independently -- fighting whatever the trigger just
created, since the replacement isn't part of any allocation Nomad
knows about (no service registration, no template-rendered secrets,
no restart-policy supervision).

This trigger instead calls Nomad's own allocation restart API
(POST /v1/client/allocation/:alloc_id/restart), reading the
com.hashicorp.nomad.alloc_id / task_name labels Nomad already stamps
onto every container its Docker driver creates. Nomad's own driver
then handles the pull (given force_pull = true on the task) and
recreation, so the allocation stays under normal Nomad supervision
throughout.
com.hashicorp.nomad.task_name is missing on at least Nomad v2.0.4
(alloc_id was present, task_name was not), despite existing in
Nomad's own driver source. Since Nomad's Docker driver always names
containers "<task_name>-<alloc_id>" (drivers/docker/driver.go), we
can recover the task name reliably without that label.

If neither the label nor the container-name parse works, refuse to
restart rather than silently falling back to AllTasks -- that could
mean restarting a database sidecar nobody asked to touch.
const config = configuration || this.configuration;
return {
...config,
token: config.token ? '_****_' : undefined,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a mask function in WUD available to use

Suggested change
token: config.token ? '_****_' : undefined,
token: Nomad.mask(config.token)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants