Repository navigation
feat(ci): modernize ADO integration test pipeline - #912
Draft
Hangyi (HangyiWang) wants to merge 26 commits into
Draft
Hangyi (HangyiWang) wants to merge 26 commits into
Hangyi (HangyiWang) wants to merge 26 commits into
Conversation
…retries Scope pipeline 147 to supported services, add strict immutable attempt qualification and offline retry diagnostics, preserve owned fixture cleanup, and refresh ADO WIF credentials for long-running jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the selectable retry self-test while retaining offline regression coverage and service-level manual retry targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scope ADR/ADU cleanup inventory to ARM roots observed by each test process and its workers so parallel services do not falsely report one another's resources as leaks. Preserve strict cleanup, retry evidence and per-service matrix serialization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collect the full unit inventory in each serial pytest shard, assign files by measured duration, and require complete consistent results before merging coverage. Cache only pip downloads for lint and unit jobs and expose cache and tox timing measurements. Keep every integration service behind Build, Lint and Unit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove cross-job pip caching after cold/warm runs showed no net gain. Preserve real Digital Twins LRO success/failure coverage with immediate mocked Retry-After and bounded completion, cutting the two unit cases from141 seconds to0.1 seconds. Rebalance the four unit shards using the corrected timing weight. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep cleanup, integration, unknown definitions and changed revisions fail-closed; identify blocking builds in diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Carry extension dependency paths into standalone controllers and children after initial Azure login, and verify main/DPS interpreter startup during setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the merge-definition revision exception along with the ADO build scan. Preserve GitHub live-run admission, immutable candidates, and manual failed-case retry history. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove HTTP interception and the coupled inventory gate. Retain existing service cleanup, owned Hub/DPS phase checks, immutable candidates, and manual retry evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Separate immutable execution integrity from test recovery eligibility. Preserve all sibling phase evidence, validate worker completion and combined DPS stages, and publish actionable rejected-attempt reports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep exact-case recovery for complete call failures; rerun every affected service phase after fixture, startup or execution failures. Record accepted, pending and failed cleanup separately without polling for deletion completion, retaining dependent leftovers for operational cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract reusable duration-balanced serial unit partitions and strict completeness/coverage aggregation from the ADO migration. Run four GitHub integration precheck shards alongside independent lint, with attempt-scoped evidence and CI-neutral run identity. Preserve the existing tox platform matrix and remove mocked Digital Twins polling sleeps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stack the migration on the standalone unit-test performance branch. Keep the reusable runner, GitHub wiring, profile and Digital Twins speedup identical to its new base, adapt ADO run identity to the shared contract, and keep ADO-specific assertions with the migration tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hangyi (HangyiWang)
changed the base branch from
release/1.0.0-preview
to
users/hangyiwang/unit-test-parallelism
October 6, 2026 17:23
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hosted runners resolved Python 3.13 to both 3.13.15 and 3.13.16 in the same workflow. Validate the requested major/minor series while retaining exact versions in receipts and summaries; preserve all collection, execution and artifact checks. Add mixed-patch and incompatible-version regressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hangyi (HangyiWang)
force-pushed
the
users/hangyiwang/unit-test-parallelism
branch
from
October 6, 2026 21:39
85eafd3 to
51ef7cc
Compare
Hangyi (HangyiWang)
force-pushed
the
users/hangyiwang/ado147-integration-parity-retries
branch
from
October 6, 2026 21:39
4f76bb7 to
da9d5d7
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep fresh-attempt override rejection intact while isolating offline tests from legacy ADO sentinels. Use a high-resolution counter to order equal wall-clock cleanup events, retaining old receipt compatibility and append-only evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since we're reusing ADO pipeline https://dev.azure.com/azureiotdevxp/aziotcli/_build?definitionId=147&_a=summary for FIC migration, this PR modernizes pipeline 147 / Azure IoT CLI - Integration Tests for DPS, HubControl, HubData, ADU and ADR, using the existing ADO service connection with renewable WIF and a simple Dry run / Integration tests form with Python, region and ARM-routing selection. The form defaults to Integration tests with the named DPS/Hub/ADR/ADU preset; clear the preset to select individual suites, or explicitly choose Dry run for planning only. Shipped CLI behavior, pipeline default branch, GitHub release/schedule callers and FICs are unchanged.
Build, Lint and Unit run in parallel, followed by parallel selected integration services against one immutable candidate wheel, with manual same-run recovery that preserves original results. Completed call failures rerun only the failed cases, including service/transport exceptions. When exact selection is unsafe or unavailable, the next manual attempt reruns the entire affected service/Python/region combination with fresh fixtures, including passing authentication phases. This covers fixture failures, incomplete execution, worker crashes, timeouts and installation/login failures. Full retries replace effective service results rather than hiding new failures behind old passes. Commit, candidate wheel, dependency fingerprint, target, ancestry and raw-artifact integrity remain enforced; changed immutable evidence requires a new run. Rejected attempts publish explicit failure reports, completed sibling receipts remain available, and startup gaps cannot reuse an older pass. Runs pinned to the older attempt format need a new run to use this policy.
ADO cleanup uses submission handoff rather than waiting for resource absence where supported. Accepted deletion is recorded separately from confirmed absence. Pending dependencies and failed submissions retain exact IDs in
cleanup-status.jsonand ownership receipts for later operational cleanup, as requested; no extra cleanup job or new cleanup polling deadline is added. CSR targets stay retained while their namespace/device cleanup is unresolved. Pending resources may continue consuming quota. Ownership checks, worker termination and actual deletion assertions inside tests remain intact, and no generic ADR/ADU HTTP or inventory guard is introduced.ADO runs are no longer blocked by other project builds or a fixed cleanup-time window; the separate GitHub live-run check remains. Standalone controllers and their subprocesses inherit candidate dependencies, with main/DPS interpreter startup checks before Azure login. ADU has no added per-test timeout, and DPS regular/service-SAS attempts retain seven-worker execution with worker-safe result collection; existing job/phase and process-shutdown limits, serial local-auth toggling and non-ADO cleanup requirements remain intact.
Depends on #917. The reusable unit shard planner, pytest plugin, duration profile, completeness/coverage gate, GitHub precheck wiring, and mocked Digital Twins polling speedup now belong to that standalone PR and are excluded from this diff. This PR retains the ADO-specific four-shard Unit/UnitGate adapter, pipeline artifact wiring, and adapter regression checks. ADO cross-job pip caching remains disabled after measurement showed no net benefit.
Merge order: merge #917 into
release/1.1.0-previewfirst, then retarget this PR fromusers/hangyiwang/unit-test-parallelismtorelease/1.1.0-preview. If the unit PR is squash/rebase merged, also merge the updated release base into this migration branch to preserve the clean separation. Both branches have been rebased onto the release 1.1 stack; this PR remains based on the unit branch to keep its ADO-only diff.