Skip to content

ci: collect EVE coverage per matrix job in Eden CI#1173

Open
eriknordmark wants to merge 4 commits into
lf-edge:masterfrom
eriknordmark:ci-coverage-collection
Open

ci: collect EVE coverage per matrix job in Eden CI#1173
eriknordmark wants to merge 4 commits into
lf-edge:masterfrom
eriknordmark:ci-coverage-collection

Conversation

@eriknordmark

@eriknordmark eriknordmark commented May 12, 2026

Copy link
Copy Markdown
Contributor

Description

Per-job coverage collection for the Eden test matrix plus a fan-in
merge job, intended to be consumed by a coverage-instrumented EVE
build on the lf-edge/eve side.

Each suite in the Eden test matrix expands into ~12 independent
runners with their own ./eden setup -> start -> stop -> make clean
lifecycle, so per-job collection is the only point at which to
capture covdata. After make clean the runner is wiped — there is no
later opportunity.

Two commits:

collect-coverage: preserve raw covdata in output

CollectEveCoverage SCP'd the raw covmeta.* / covcounters.*
files into a temp dir, ran go tool covdata textfmt on them, and
let the deferred cleanup wipe the binary files on return. Only the
text profile at <output-dir>/eden_e2e_coverage.txt survived.

Callers that want to merge coverage across multiple runs (e.g.
across the Eden test matrix) need the raw covdata so they can use
go tool covdata merge; the text profile alone only supports
concatenation, which is fine for a per-line summary but doesn't
aggregate the way merge does.

Copy the SCP target into <output-dir>/covdata/ instead of an
auto-deleted temp dir. The text-profile output path is unchanged.

ci: collect eden e2e coverage per matrix job

  • run-eden-test composite action gains a Collect/Upload coverage
    pair, gated as best-effort (if: always() and || true), that
    invokes ./eden eve collect-coverage and uploads the resulting
    eden_coverage/ directory (text profile + raw covdata subtree) as
    artifact covdata-<suite>-fs-<fs>-tpm-<tpm>-attempt-<n>.
  • test.yml gains a coverage-merge job that downloads every
    covdata-* artifact, runs go tool covdata merge on the raw
    covmeta.* / covcounters.* files to produce a single merged covdata
    directory, converts it to a text profile via covdata textfmt,
    and runs go tool cover -func for a per-package summary.

Companion PR

This change is the prerequisite for
lf-edge/eve#5950,
which adds an opt-in COVER=y EVE build (gated on a coverage PR
label) and wires eden-trusted.yml to consume the resulting
artifact. The EVE PR calls
lf-edge/eden/.github/workflows/test.yml@master, so this PR has to
land first; otherwise the tests-master-cover job on the EVE side
will run tests but emit no covdata-merged artifact.

How to test and validate this PR

Without a coverage-instrumented EVE artifact, the new steps run as
no-ops:

  1. Confirm eden.yml (the eden-only CI path that consumes
    lfedge/eve:16.6.0) still passes after this change. The
    Collect coverage step warns ("no covcounters appeared")
    because zedbox isn't built with -cover, and the
    Upload coverage artifact step skips because the
    eden_e2e_coverage.txt hashFiles check is false. No new artifact
    is uploaded.
  2. Confirm the coverage-merge fan-in job runs to completion and
    emits the "No raw covdata found across jobs" warning instead of
    failing.
  3. Locally: build EVE with COVER=y and run a single suite via
    make eden-cover (or by passing
    --coverage-dir <dir> to eden test); confirm
    <dir>/eden_e2e_coverage.txt AND <dir>/covdata/coverage/
    (containing covmeta.* and covcounters.*) both exist after
    the run.
  4. Once the companion EVE PR lands on master, exercise the full
    path end-to-end by labelling a coverage-instrumented EVE PR with
    coverage and confirming the covdata-merged artifact is
    produced at the end of the workflow run.

Notes

No user-facing changes. CI / dev-tooling only.

go tool cover -func=combined_coverage.txt | tee coverage-summary.txt
- name: Upload merged covdata
if: always() && hashFiles('combined_coverage.txt') != ''
uses: actions/upload-artifact@v4

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's now mandatory to have a codecov TOKEN in order to upload coverage data. See https://github.com/lf-edge/eve/blob/dc7f587911a2a3d39a459e1150f5758c51429e10/.github/workflows/publish.yml#L294 for reference.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — added an "Upload coverage to Codecov" step to the coverage-merge job using codecov/codecov-action@v6.0.0 with token: ${{ secrets.CODECOV_TOKEN }}, matching the EVE publish.yml reference. Since test.yml is a reusable workflow, eden.yml now passes secrets: inherit so the token reaches it. (This assumes CODECOV_TOKEN is configured as a secret on lf-edge/eden — let me know if it still needs to be added on the repo side.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed CODECOV_TOKEN is already configured on lf-edge/eden, so this works on push-to-master. Fork PRs get no secrets, so I set fail_ci_if_error: false to skip cleanly there.

@rene rene left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eriknordmark , please, add the codecov token support to the workflow....

eriknordmark and others added 2 commits July 9, 2026 15:14
CollectEveCoverage previously SCP'd the raw covmeta.* / covcounters.*
files into a temp dir, ran `go tool covdata textfmt` on them, and let
the deferred cleanup wipe the binary files on return. Only the text
profile at <output-dir>/eden_e2e_coverage.txt survived.

Callers that want to merge coverage across multiple runs (e.g. across
the Eden test matrix) need the raw covdata so they can use
`go tool covdata merge`; the text profile alone only supports
concatenation, which is fine for a per-line summary but doesn't
aggregate the way merge does.

Copy the SCP target into <output-dir>/covdata/ instead of an
auto-deleted temp dir. The text-profile output path is unchanged.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each suite in the Eden test matrix runs on its own runner with its own
./eden setup -> start -> stop -> make clean lifecycle, so per-job
collection is the only point at which to capture covdata from a
coverage-instrumented EVE build. After make clean the runner is wiped
and there is no second chance.

Add a Collect/Upload coverage pair to the run-eden-test composite
action: invoke `eden eve collect-coverage` with `if: always()` and
best-effort error handling so a failed test still surfaces whatever
covdata was written, and upload the eden_coverage/ output directory
(both the eden_e2e_coverage.txt text profile and the raw covdata
subtree).

Add a coverage-merge job to test.yml that downloads every covdata-*
artifact, runs `go tool covdata merge` on the raw covmeta.* /
covcounters.* files to produce a single merged covdata directory,
converts it to a text profile via `covdata textfmt`, and runs
`go tool cover -func` for a per-package summary.

Producing useful coverage data requires the EVE artifact consumed by
these jobs to be built with COVER=y so zedbox registers the SIGUSR2
handler and sets GOCOVERDIR=/persist/coverage. That is a separate
change on the EVE side.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@eriknordmark
eriknordmark force-pushed the ci-coverage-collection branch from c900dfb to 2f7cdb3 Compare July 9, 2026 22:17
eriknordmark and others added 2 commits July 20, 2026 16:13
The coverage-merge job produced combined_coverage.txt only as a
downloadable artifact. Upload it to Codecov as well so coverage is
tracked over time. Codecov now requires an upload token, so the
reusable test workflow references secrets.CODECOV_TOKEN and eden.yml
passes secrets through with `secrets: inherit`.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set fail_ci_if_error: false explicitly on the Codecov upload step so a
failed or skipped upload (e.g. fork PRs, where secrets are withheld and
CODECOV_TOKEN is empty) does not fail the job.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants