Skip to content

feat(inventory): report agent health to Anchore Enterprise - #480

Draft
bradleyjones wants to merge 1 commit into
mainfrom
maam/integration-impl-giddy-quail
Draft

feat(inventory): report agent health to Anchore Enterprise#480
bradleyjones wants to merge 1 commit into
mainfrom
maam/integration-impl-giddy-quail

Conversation

@bradleyjones

@bradleyjones bradleyjones commented Aug 17, 2026

Copy link
Copy Markdown
Member

⛔️ Do not merge before Anchore Enterprise branch maam/enterprise-impl-30a48222
(commit a11b88453) lands.
See Cross-repo coordination.

What this does

anchore-ecs-inventory now registers itself with Anchore Enterprise as an integration of
type ecs_inventory_agent and reports its health on an interval, following the prior art
in anchore/k8s-inventory.

Until now the agent reported inventory and nothing else. There was no way to see from
Enterprise whether an agent was alive, and a failure that stopped it collecting anything —
expired AWS credentials, a revoked ecs:ListClusters, a bad region — was only ever a line
in the agent's own log. Those failures now mark the integration UNHEALTHY in Enterprise,
per ECS cluster.

New packages

Package Purpose
internal/time Datetime/Duration with the JSON encoding Enterprise expects (RFC3339 strings, durations as bare seconds). Ported from k8s-inventory.
internal/anchore Client for the v2 registration and health-report endpoints, with classifiers for offline / unsupported / forbidden / unauthenticated. pkg/reporter keeps its own client — its v1/v2 fallback logic is unrelated.
pkg/integration Registration, version gating, and derivation of a stable registration identity from the ECS task metadata endpoint v4.
pkg/healthreporter The periodic health report and the per-cluster result store shared with the inventory loop.

GetInventoryReportsForRegion now returns what happened per cluster, including gather
failures that were previously only a log line. Two new direct dependencies:
github.com/google/uuid and github.com/hashicorp/go-version (both already used by
k8s-inventory).

Degradation is deliberate and never fatal

Against an Enterprise older than 6.2.0, one that rejects the new integration type, an
offline Enterprise, or a user missing the registerIntegration / reportHealth RBAC
actions, the agent logs clearly and carries on reporting inventory. Registration runs
alongside the inventory loop rather than gating it, so an unbounded offline retry can never
starve inventory reporting. --dry-run neither registers nor health-reports.

API / contract

Two Anchore Enterprise v2 endpoints, in addition to the existing inventory endpoint:

Purpose Endpoint RBAC action
Register POST /v2/system/integrations/registration registerIntegration
Health report POST /v2/system/integrations/{integration_uuid}/health-report reportHealth

One substantive shape difference from the k8s agent: account_ecs_inventory_reports
maps each account to a list of per-cluster results (each with a required cluster_arn
and an optional region), not to the single flat object the k8s agent sends. The ECS agent
sweeps many clusters within a single Anchore account, so a k8s-shaped map would hide the
failures of all but one cluster. Several fields deliberately carry no omitempty and
batches is initialised with make(...) — a nil slice marshals to null and Enterprise
400s. A golden-JSON unit test pins this, and QA read the persisted shape back out of the
Enterprise database.

Registration identity. Enterprise keys an integration on the pair
(registration_id, registration_instance_id) and creates a new record for any pair it
has not seen, marking superseded ones INACTIVE. ECS has no Deployment UID equivalent, so
identity is derived from the task metadata endpoint: the task family and the ECS
service name
(falling back to the family for tasks in no service), each qualified with
the AWS account (read from the task ARN), the cluster the agent's own task runs in,
and the region it scans — e.g.
anchore-ecs-inventory/111122223333/anchore-qa/us-east-1.

The task ARN is deliberately not used: ECS mints a new task id on every deployment, scale
event, health-check replacement and spot interruption, which would leave a new integration
record behind each time. Family and service name give stability; account, cluster and
region give uniqueness. Values set explicitly in anchore-registration.* are used verbatim
and are not qualified.

Nothing has shipped, so there is no migration — but any Enterprise instance used while
developing this branch holds stale integration records under earlier id shapes that will
not be reused.

Config

No new CLI flags. New config keys, env-overridable as usual with the
ANCHORE_ECS_INVENTORY_ prefix:

# frequency of which to send health reports to anchore enterprise (30-600)
health-report-interval-seconds: 60

# how this agent identifies itself when registering as an integration with anchore
# enterprise. anything left empty is derived at runtime from the ECS task metadata endpoint.
anchore-registration:
  registration-id: ""
  registration-instance-id: ""
  integration-name: ""
  integration-description: ""

health-report-interval-seconds is validated to 30–600 at startup. The
anchore-registration keys are registered as viper defaults — without that AutomaticEnv
cannot resolve them at all, so the whole block would have been unreachable by environment
variable.

README.md gains a Health reporting section (endpoints, RBAC actions, degradation
behaviour, identity rules and when to set the ids explicitly), and
docker-compose/anchore-ecs-inventory.yaml gains the same keys with commentary. There is
no Helm chart or webhook in this repo, so neither was touched.

QA

Exercised end-to-end against a live Enterprise docker-compose stack (all 11 containers
healthy) with a binary confirmed built from the change (anchore-ecs-inventory version
GitCommit: 1f1584681a1f7b9c2a307483e44ef993d03e1e1f). The stack was probed by contract
rather than by version string, found not to carry the Enterprise change, and a11b88453's
production files were synced into the api and catalog containers first — every result below
is against a stack carrying a11b88453. 24 checks, 24 PASS, 0 FAIL.

  • Happy path: registration → integration_status.state = active,
    reported_status.state = healthy, with uptime, last_seen and cluster_name
    populated; ECS inventory still landing in Enterprise. The persisted health_data read
    back out of the DB is the map-of-arrays shape, batches a real list (never null), and
    a cluster with zero containers correctly absent — neither claimed successful nor
    failed.
  • Identity is unique and stable: two agents whose /task reports a bare cluster name
    with identical family/service/region and differing only in AWS account register as two
    distinct integrations. With one agent's region completely broken, both states were
    sampled every 10s for 3.5 minutes — one healthy throughout, the other unhealthy
    throughout, zero oscillation. An earlier revision flapped and hid a totally broken
    agent; that is what the account qualifier fixes, and it is pinned by a regression test
    QA independently confirmed fails without the fix.
  • Restart stability holds: the account is read from TaskARN, so an agent was
    restarted after its task ARN was rewritten with a new task id — it re-registered onto the
    same integration UUID with the count unchanged.
  • Multi-region distinctness, and clean degradation when the account is underivable
    (unparseable ARN, and no metadata endpoint at all).
  • Five auth/API failure modes: a real 401, a genuine 403 from a real Enterprise
    account/user missing registerIntegration, a reportHealth 403, the verbatim
    pre-change 400, and offline exponential backoff. Inventory reporting continued in every
    case and the process never died.
  • Dry run: 0 registrations, 0 health reports, no integration record created.
  • Explicit registration config used verbatim and unscoped; config bounds checked at
    their exact edges (29 / 30 / 600 / 601); README prose verified against observed
    behaviour rather than read in isolation.
  • go build, go vet, go test ./..., go test -race -count=1 and make lint
    (0 issues, gofmt clean) all pass. pkg/integration coverage 89.7%.

Follow-ups (not in this PR)

  • Remove the pre-existing "validating-creds" dummy report POSTed to /v2/ecs-inventory
    in cmd/root.go — its own comment says health reporting should replace it. It predates
    this change and is retained here so credential validation still works against Enterprises
    without the branch. Note it still fires under --dry-run.
  • The integration display name is unscoped by design, so several agents from one task
    definition register correctly as separate integrations but all display as
    anchore-ecs-inventory, distinguishable by cluster_name/account_name. A README
    sentence recommending a per-deployment anchore-registration.integration-name would
    close it.
  • With no task metadata and no integration-name, the integration registers with an empty
    display name (Enterprise accepts it).

@bradleyjones
bradleyjones requested a review from a team as a code owner August 17, 2026 09:33
@bradleyjones bradleyjones changed the title PR title feat(inventory): report agent health to Anchore Enterprise Aug 17, 2026
@bradleyjones
bradleyjones marked this pull request as draft August 17, 2026 12:50
@bradleyjones
bradleyjones force-pushed the maam/integration-impl-giddy-quail branch from 3e99f25 to 3bd3d99 Compare August 17, 2026 12:54
Register the agent as an `ecs_inventory_agent` integration and report
per-cluster health on an interval, following the k8s-inventory prior art.

- add internal/anchore client for the v2 registration and health-report
  endpoints
- add pkg/integration (registration + identity from ECS task metadata) and
  pkg/healthreporter (periodic report + per-cluster result store)
- add internal/time helpers for Enterprise's JSON encoding
- return per-cluster results (including gather failures) from
  GetInventoryReportsForRegion
- add health-report-interval-seconds and anchore-registration.* config keys

Registration and health reporting degrade gracefully and never gate
inventory reporting; --dry-run does neither.

Signed-off-by: bradleyjones <jones.bradley@me.com>
@bradleyjones
bradleyjones force-pushed the maam/integration-impl-giddy-quail branch from 3bd3d99 to 8b4e40c Compare August 17, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant