ci: collect EVE coverage per matrix job in Eden CI#1173
Conversation
2744e25 to
c900dfb
Compare
| go tool cover -func=combined_coverage.txt | tee coverage-summary.txt | ||
| - name: Upload merged covdata | ||
| if: always() && hashFiles('combined_coverage.txt') != '' | ||
| uses: actions/upload-artifact@v4 |
There was a problem hiding this comment.
It's now mandatory to have a codecov TOKEN in order to upload coverage data. See https://github.com/lf-edge/eve/blob/dc7f587911a2a3d39a459e1150f5758c51429e10/.github/workflows/publish.yml#L294 for reference.
There was a problem hiding this comment.
Thanks — added an "Upload coverage to Codecov" step to the coverage-merge job using codecov/codecov-action@v6.0.0 with token: ${{ secrets.CODECOV_TOKEN }}, matching the EVE publish.yml reference. Since test.yml is a reusable workflow, eden.yml now passes secrets: inherit so the token reaches it. (This assumes CODECOV_TOKEN is configured as a secret on lf-edge/eden — let me know if it still needs to be added on the repo side.)
There was a problem hiding this comment.
Confirmed CODECOV_TOKEN is already configured on lf-edge/eden, so this works on push-to-master. Fork PRs get no secrets, so I set fail_ci_if_error: false to skip cleanly there.
rene
left a comment
There was a problem hiding this comment.
@eriknordmark , please, add the codecov token support to the workflow....
CollectEveCoverage previously SCP'd the raw covmeta.* / covcounters.* files into a temp dir, ran `go tool covdata textfmt` on them, and let the deferred cleanup wipe the binary files on return. Only the text profile at <output-dir>/eden_e2e_coverage.txt survived. Callers that want to merge coverage across multiple runs (e.g. across the Eden test matrix) need the raw covdata so they can use `go tool covdata merge`; the text profile alone only supports concatenation, which is fine for a per-line summary but doesn't aggregate the way merge does. Copy the SCP target into <output-dir>/covdata/ instead of an auto-deleted temp dir. The text-profile output path is unchanged. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each suite in the Eden test matrix runs on its own runner with its own ./eden setup -> start -> stop -> make clean lifecycle, so per-job collection is the only point at which to capture covdata from a coverage-instrumented EVE build. After make clean the runner is wiped and there is no second chance. Add a Collect/Upload coverage pair to the run-eden-test composite action: invoke `eden eve collect-coverage` with `if: always()` and best-effort error handling so a failed test still surfaces whatever covdata was written, and upload the eden_coverage/ output directory (both the eden_e2e_coverage.txt text profile and the raw covdata subtree). Add a coverage-merge job to test.yml that downloads every covdata-* artifact, runs `go tool covdata merge` on the raw covmeta.* / covcounters.* files to produce a single merged covdata directory, converts it to a text profile via `covdata textfmt`, and runs `go tool cover -func` for a per-package summary. Producing useful coverage data requires the EVE artifact consumed by these jobs to be built with COVER=y so zedbox registers the SIGUSR2 handler and sets GOCOVERDIR=/persist/coverage. That is a separate change on the EVE side. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
c900dfb to
2f7cdb3
Compare
The coverage-merge job produced combined_coverage.txt only as a downloadable artifact. Upload it to Codecov as well so coverage is tracked over time. Codecov now requires an upload token, so the reusable test workflow references secrets.CODECOV_TOKEN and eden.yml passes secrets through with `secrets: inherit`. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set fail_ci_if_error: false explicitly on the Codecov upload step so a failed or skipped upload (e.g. fork PRs, where secrets are withheld and CODECOV_TOKEN is empty) does not fail the job. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Description
Per-job coverage collection for the Eden test matrix plus a fan-in
merge job, intended to be consumed by a coverage-instrumented EVE
build on the lf-edge/eve side.
Each suite in the Eden test matrix expands into ~12 independent
runners with their own
./eden setup -> start -> stop -> make cleanlifecycle, so per-job collection is the only point at which to
capture covdata. After
make cleanthe runner is wiped — there is nolater opportunity.
Two commits:
collect-coverage: preserve raw covdata in outputCollectEveCoverageSCP'd the rawcovmeta.*/covcounters.*files into a temp dir, ran
go tool covdata textfmton them, andlet the deferred cleanup wipe the binary files on return. Only the
text profile at
<output-dir>/eden_e2e_coverage.txtsurvived.Callers that want to merge coverage across multiple runs (e.g.
across the Eden test matrix) need the raw covdata so they can use
go tool covdata merge; the text profile alone only supportsconcatenation, which is fine for a per-line summary but doesn't
aggregate the way merge does.
Copy the SCP target into
<output-dir>/covdata/instead of anauto-deleted temp dir. The text-profile output path is unchanged.
ci: collect eden e2e coverage per matrix jobrun-eden-testcomposite action gains a Collect/Upload coveragepair, gated as best-effort (
if: always()and|| true), thatinvokes
./eden eve collect-coverageand uploads the resultingeden_coverage/directory (text profile + raw covdata subtree) asartifact
covdata-<suite>-fs-<fs>-tpm-<tpm>-attempt-<n>.test.ymlgains acoverage-mergejob that downloads everycovdata-*artifact, runsgo tool covdata mergeon the rawcovmeta.* / covcounters.* files to produce a single merged covdata
directory, converts it to a text profile via
covdata textfmt,and runs
go tool cover -funcfor a per-package summary.Companion PR
This change is the prerequisite for
lf-edge/eve#5950,
which adds an opt-in
COVER=yEVE build (gated on acoveragePRlabel) and wires
eden-trusted.ymlto consume the resultingartifact. The EVE PR calls
lf-edge/eden/.github/workflows/test.yml@master, so this PR has toland first; otherwise the
tests-master-coverjob on the EVE sidewill run tests but emit no covdata-merged artifact.
How to test and validate this PR
Without a coverage-instrumented EVE artifact, the new steps run as
no-ops:
eden.yml(the eden-only CI path that consumeslfedge/eve:16.6.0) still passes after this change. TheCollect coveragestep warns ("no covcounters appeared")because zedbox isn't built with
-cover, and theUpload coverage artifactstep skips because theeden_e2e_coverage.txthashFiles check is false. No new artifactis uploaded.
coverage-mergefan-in job runs to completion andemits the "No raw covdata found across jobs" warning instead of
failing.
COVER=yand run a single suite viamake eden-cover(or by passing--coverage-dir <dir>toeden test); confirm<dir>/eden_e2e_coverage.txtAND<dir>/covdata/coverage/(containing
covmeta.*andcovcounters.*) both exist afterthe run.
master, exercise the fullpath end-to-end by labelling a coverage-instrumented EVE PR with
coverageand confirming thecovdata-mergedartifact isproduced at the end of the workflow run.
Notes
No user-facing changes. CI / dev-tooling only.