Add provider→scanner→output integration tests and per-package coverage floors#26
Merged
Conversation
…e floors The final piece of the uplift: end-to-end tests across the layers, a per-package coverage gate so no package can silently regress, and a latent bug the new tests surfaced. ──────────────────────── integration tests ──────────────────────── internal/integration runs the provider → scanner → output path with a fixture provider registered through the real provider registry (providers.NewRegistry / Capable), resolved by capability, run through each domain scanner, and rendered to both JSON and table. Six domains (orphans, storage, network, certs, tags, secrets) assert multiple rendered fields each, plus a case that proves the scanner's MinSeverity actually drops a below-threshold finding, and the no-provider error path. A command's RunE resolves providers via providers.Resolve (→ Default()), which is AWS-backed and intentionally has no test-injection seam, so the cobra shell (flag→ScanOptions, the output-format switch, gate/exit-code) is covered by package cmd's unit tests rather than here. ──────────────────────── remediate bugfix ───────────────────────── Adding cmd helper tests (cmd/remediate_test.go) exposed that remediate's bare-array fallback was unreachable: a bare JSON array fails the envelope unmarshal and returned an error before the fallback ran, so `cloudgov remediate --from <hand-crafted-array.json>` was silently broken. The three unmarshalers now try the envelope, then the bare array, then report a useful error — with tests for envelope, bare-array, invalid-JSON, and the empty-envelope path. ──────────────────────── per-package floors ─────────────────────── .coverage-floors sets a floor per package (a few points below current, to ratchet up as coverage lands). scripts/coverage.sh enforces them in ci.yml and fails on three conditions: a package below its floor, a floored package that produced no coverage line (stale/typo'd name), and a package that reports coverage but has no floor (so new tested code can't land ungated). This replaces the single 50% total floor. golangci-lint and coverage profiling were already wired into CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The final target of the uplift: end-to-end tests across the layers, a per-package coverage gate, and a latent bug the new tests surfaced.
Integration tests
internal/integrationruns the provider → scanner → output path with a fixture provider registered through the real registry (providers.NewRegistry/Capable), resolved by capability, run through each domain scanner, and rendered to both JSON and table. Six domains (orphans, storage, network, certs, tags, secrets) assert multiple rendered fields each, plus a case proving the scanner'sMinSeverityactually drops a below-threshold finding, plus the no-provider error path.A command's
RunEresolves viaproviders.Resolve(→Default()), which is AWS-backed and intentionally has no test-injection seam — so the cobra shell (flag→ScanOptions, output-format switch, gate/exit-code) is covered by packagecmd's unit tests, not here. (Honest framing, per review.)Remediate bugfix (found by the new tests)
The
cmdhelper tests exposed that remediate's bare-array fallback was unreachable: a bare JSON array fails the envelope unmarshal and errored before the fallback ran, socloudgov remediate --from <hand-crafted-array.json>was silently broken. The three unmarshalers now try envelope → bare array → useful error, with tests for envelope, bare-array, invalid-JSON, and empty-envelope.Per-package coverage floors
.coverage-floors(one floor per package, a few points below current, ratcheting) enforced byscripts/coverage.shinci.yml. Fails on three conditions: a package below its floor, a floored package with no coverage line (stale/typo'd name), and a tested package with no floor (ungated new code). Replaces the single 50% total floor. golangci-lint + coverage profiling were already in CI.Verification
Local: full suite green, all 25 floors met (total ~61%), shellcheck + lint clean; both new floor-guard paths verified to fail correctly. An adversarial review workflow (3 dimensions → verify) raised 12 findings — bugfix + floor logic verified correct; the rest (seam labeling honesty, multi-field assertions, filter discrimination, the two floor guards, bugfix edge-case tests) are all addressed in this PR.
Completes backlog section 7 — the cloudgov uplift.