[CI] Use ACES in CI builds by dalexsoto · Pull Request #25572 · dotnet/macios

dalexsoto · 2026-05-29T01:20:07Z

Using ACES only for CI builds, PR will be enabled in the future.

✅ CI Test run: https://devdiv.visualstudio.com/DevDiv/DevDiv%20Team/_build/results?buildId=14226458&view=results
✅ CI Api Diff: https://devdiv.visualstudio.com/DevDiv/DevDiv%20Team/_build/results?buildId=14226006&view=results
✅ CI Build: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=14225709&view=results

…M when using ACES

Copilot

Pull request overview

This PR adds ACES pool support for CI Azure DevOps builds while keeping PR pipelines on the existing Mac pools. It also adjusts CI/test behavior for VM-backed agents and adds a few test resiliency updates.

Changes:

Adds and propagates a useACES template parameter through build, API diff, and simulator-test pipeline templates.
Switches CI entry pipelines to ACES while leaving PR entry pipelines on legacy pools.
Updates selected tests/utilities to account for ACES VM environments and CI-specific network/folder behavior.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`tools/devops/automation/templates/variables/common.yml`	Defines ACES pool variables and documents the per-pipeline switch.
`tools/devops/automation/templates/tests/stage.yml`	Adds ACES pool selection and `VM_VENDOR` for simulator tests.
`tools/devops/automation/templates/tests/build.yml`	Adds retry handling for Provisionator dependencies.
`tools/devops/automation/templates/tests-stage.yml`	Propagates `useACES` to simulator tests and macOS-test build jobs.
`tools/devops/automation/templates/pipelines/run-tests-pipeline.yml`	Exposes and forwards the `useACES` parameter.
`tools/devops/automation/templates/pipelines/api-diff-pipeline.yml`	Adds `useACES` forwarding for API diff pipelines.
`tools/devops/automation/templates/main-stage.yml`	Adds `useACES` forwarding to the main build stage.
`tools/devops/automation/templates/build/build-stage.yml`	Switches build job pool selection based on `useACES`.
`tools/devops/automation/templates/build/build-mac-tests-stage.yml`	Switches macOS test-build pool selection based on `useACES`.
`tools/devops/automation/templates/build/api-diff-stage.yml`	Switches API diff job pool selection based on `useACES`.
`tools/devops/automation/templates/api-diff-stage.yml`	Propagates `useACES` to API diff job template.
`tools/devops/automation/run-pr-api-diff.yml`	Keeps PR API diff off ACES.
`tools/devops/automation/run-post-pr-build-tests.yml`	Keeps PR post-build simulator tests off ACES.
`tools/devops/automation/run-post-ci-build-tests.yml`	Enables ACES for CI post-build simulator tests.
`tools/devops/automation/run-ci-api-diff.yml`	Enables ACES for CI API diff.
`tools/devops/automation/build-pull-request.yml`	Keeps PR builds off ACES.
`tools/devops/automation/build-pipeline.yml`	Enables ACES for CI builds.
`tests/xharness/AppRunner.cs`	Forwards `VM_VENDOR` into launched test apps in CI.
`tests/monotouch-test/VideoToolbox/VTMotionEstimationSessionTest.cs`	Skips selected tests on VM-backed macOS/Mac Catalyst runs.
`tests/monotouch-test/VideoToolbox/VTCompressionSessionTests.cs`	Skips multi-image compression test on VM-backed macOS/Mac Catalyst runs.
`tests/monotouch-test/System.Net.Http/NetworkResources.cs`	Adds an HTTP Microsoft URI helper.
`tests/monotouch-test/Network/NWParametersTest.cs`	Uses the HTTP Microsoft URI host for port-80 Network tests.
`tests/monotouch-test/Network/NWConnectionTest.cs`	Uses the HTTP Microsoft URI host for Network connection tests.
`tests/monotouch-test/Metal/MTLDeviceTests.cs`	Skips one Metal test on VM-backed runs.
`tests/monotouch-test/AudioToolbox/AudioQueueTest.cs`	Skips channel assignment test on VM-backed runs.
`tests/linker/link sdk/LinkSdkRegressionTest.cs`	Tolerates missing standard user folders on macOS CI images.
`tests/common/TestRuntime.cs`	Makes VM detection active on all platforms.

dalexsoto · 2026-05-29T01:36:32Z

/azp run xamarin-macios-pr,xamarin-macios-pr-apidiff

azure-pipelines · 2026-05-29T01:36:51Z

Azure Pipelines successfully started running 2 pipeline(s).

rolfbjarne · 2026-05-29T07:44:46Z


 	class ConnectionManager : IDisposable {
-		string host = NetworkResources.MicrosoftUri.Host;
+		string host = NetworkResources.MicrosoftHttpUri.Host;


This looks weird; https links don't work, but http does?

yeah, I am not sure why I wonder if there is a rule somewhere to just disallow any 80 (unsecured) connection which would not surprise me...

This the other way around: unsecure (http) is allowed, while secure (https) is not allowed... which looks suspicious. What were the test failures you were seeing?

…kRegressionTest

dalexsoto · 2026-05-29T12:22:55Z

/azp run xamarin-macios-pr,xamarin-macios-pr-apidiff

azure-pipelines · 2026-05-29T12:23:17Z

Azure Pipelines successfully started running 2 pipeline(s).

vs-mobiletools-engineering-service2 · 2026-05-29T12:56:02Z

🔥 [CI Build #1adb358] Build failed (Detect API changes) 🔥

Build failed for the job 'Detect API changes' (with job status 'Failed')

Pipeline on Agent
Hash: 1adb3588e5dbd0cd05700f7eac294c295fb91dc8 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-29T12:59:10Z

🔥 Unable to find the contents for the comment: D:\a\1\s\change-detection\results\gh-comment.md does not exist :fire

Pipeline on Agent
Hash: 1adb3588e5dbd0cd05700f7eac294c295fb91dc8 [PR build]

dalexsoto · 2026-05-29T15:19:10Z

/azp run xamarin-macios-sim-pr-tests

azure-pipelines · 2026-05-29T15:19:28Z

Azure Pipelines successfully started running 1 pipeline(s).

vs-mobiletools-engineering-service2 · 2026-05-29T16:12:15Z

✅ [CI Build #1adb358] Prepare .NET Release succeeded ✅

📦 Published NuGet packages (32 packages)

iOS

Microsoft.iOS.Ref.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.iOS.Runtime.ios-arm64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.iOS.Runtime.ios.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.iOS.Runtime.iossimulator-arm64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.iOS.Runtime.iossimulator-x64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.iOS.Sdk.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.iOS.Templates.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.iOS.Windows.Sdk.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.NET.Sdk.iOS.Manifest-10.0.400-preview.0.26.5.10264-ci.release-test-use-aces.nupkg

MacCatalyst

Microsoft.MacCatalyst.Ref.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.MacCatalyst.Runtime.maccatalyst-arm64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.MacCatalyst.Runtime.maccatalyst-x64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.MacCatalyst.Runtime.maccatalyst.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.MacCatalyst.Sdk.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.MacCatalyst.Templates.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.NET.Sdk.MacCatalyst.Manifest-10.0.400-preview.0.26.5.10264-ci.release-test-use-aces.nupkg

macOS

Microsoft.macOS.Ref.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.macOS.Runtime.osx-arm64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.macOS.Runtime.osx-x64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.macOS.Runtime.osx.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.macOS.Sdk.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.macOS.Templates.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.NET.Sdk.macOS.Manifest-10.0.400-preview.0.26.5.10264-ci.release-test-use-aces.nupkg

tvOS

Microsoft.NET.Sdk.tvOS.Manifest-10.0.400-preview.0.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.tvOS.Ref.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.tvOS.Runtime.tvos-arm64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.tvOS.Runtime.tvos.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.tvOS.Runtime.tvossimulator-arm64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.tvOS.Runtime.tvossimulator-x64.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.tvOS.Sdk.net10.0_26.5.26.5.10264-ci.release-test-use-aces.nupkg
Microsoft.tvOS.Templates.26.5.10264-ci.release-test-use-aces.nupkg

Other

Sharpie.Bind.Tool.26.5.0.264-ci.release-test-use-aces.nupkg

Pipeline on Agent
Hash: 1adb3588e5dbd0cd05700f7eac294c295fb91dc8 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-29T18:16:04Z

✅ [PR Build #1adb358] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 1adb3588e5dbd0cd05700f7eac294c295fb91dc8 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-29T19:16:55Z

✅ [PR Build #1adb358] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 1adb3588e5dbd0cd05700f7eac294c295fb91dc8 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-31T15:01:43Z

🔥 [CI Build #1adb358] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

1 tests crashed, 21 tests failed, 149 tests passed.

Failures

❌ monotouch tests (iOS) [attempt 3]

1 tests failed, 19 tests passed.

Failed tests

monotouch-test/iOS - simulator/Release (trimmable static registrar, NativeAOT, x64): Failed

Html Report (VSDrops) Download

❌ monotouch tests (MacCatalyst)

🔥 Failed catastrophically on VSTS: test results - monotouch_maccatalyst (no summary found).

Html Report (VSDrops) Download

❌ monotouch tests (tvOS) [attempt 3]

20 tests failed, 0 tests passed.

Failed tests

monotouch-test/tvOS - simulator/Debug: LaunchTimedOut
monotouch-test/tvOS - simulator/Release (link sdk): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (link all): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (LinkSdk): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (static registrar): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (all optimizations): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (NativeAOT, ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (trimmable static registrar, NativeAOT, ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (managed static registrar): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (managed static registrar, all optimizations): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (NativeAOT, x64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (trimmable static registrar, NativeAOT, x64): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (interpreter): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (interpreter): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (compat inline Class.GetHandle): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (strict inline Class.GetHandle): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (compat inline dlfcn): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (strict inline dlfcn, link sdk): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (NativeAOT, .NET 11 defaults): LaunchTimedOut

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. (⚠️ Html Report Publish failed ⚠️) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. (⚠️ Html Report Publish failed ⚠️) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. (⚠️ Html Report Publish failed ⚠️) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 6 tests passed. Html Report (VSDrops) Download
✅ linker (iOS): All 11 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ linker (MacCatalyst): All 11 tests passed. Html Report (VSDrops) Download
✅ linker (macOS): All 11 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ linker (tvOS): All 11 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ monotouch (macOS): All 23 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. (⚠️ Html Report Publish failed ⚠️) Download

macOS tests

✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Ventura (13): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 1adb3588e5dbd0cd05700f7eac294c295fb91dc8 [PR build]

dalexsoto added 16 commits May 27, 2026 12:51

[CI] Use ACES for CI builds

2cfff87

[CI] Use ACES for CI API Diff

99e2604

[CI] Run CI simulator tests on ACES

a98508a

[CI] Prepare PR chain pipelines for future ACES use

cf6bde1

[CI] Create a global ACES switch for PR and CI builds

9446c3f

Merge remote-tracking branch 'origin/main' into release-test/use-aces

436e9bb

[Tests] Fix SpecialFolder tests on macOS CI VMs

18ef1a8

[Tests] remove retry in order to get results from CI monotouch-tests run

60feeed

[CI] Fix global usage of UseACES_XX

c58dfa2

[CI] Unfortunately no global switch so going back to docs and switches

ae929aa

[CI] onboard Build macOS tests into ACES too

cb53e40

[CI] Fix monotouch-tests by propagating that we are running under a V…

8489ece

…M when using ACES

Merge remote-tracking branch 'origin/main' into release-test/use-aces

ae01ef1

[tests] A couple of more VM checks on VideoToolbox

c0475b0

[CI] Retry the provisioning if it fails

541d5bf

[Tests] bring back retry on tests

c9f4987

Copilot AI review requested due to automatic review settings May 29, 2026 01:20

dalexsoto requested a review from rolfbjarne as a code owner May 29, 2026 01:20

Copilot started reviewing on behalf of dalexsoto May 29, 2026 01:20 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

Comment thread tests/xharness/AppRunner.cs

This comment has been minimized.

Sign in to view

rolfbjarne requested changes May 29, 2026

View reviewed changes

Copilot started work on behalf of dalexsoto May 29, 2026 12:21 View session

[tests] refine CI-only tolerance check for user directories in LinkSd…

1adb358

…kRegressionTest

dalexsoto requested a review from rolfbjarne May 29, 2026 12:22

Copilot finished work on behalf of dalexsoto May 29, 2026 12:24

This comment has been minimized.

Sign in to view

This was referenced Jun 1, 2026

[CI] Infrastructure: 'Download secrets: azdev-secrets' EADDRNOTAVAIL on multiple bots #25596

Open

[CI Postmortem] Widespread LaunchTimedOut failures on iOS/tvOS simulator tests #25299

Open

Conversation

dalexsoto commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

dalexsoto commented May 29, 2026

Uh oh!

azure-pipelines Bot commented May 29, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

rolfbjarne May 29, 2026

Choose a reason for hiding this comment

Uh oh!

dalexsoto May 29, 2026

Choose a reason for hiding this comment

Uh oh!

rolfbjarne Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dalexsoto commented May 29, 2026

Uh oh!

azure-pipelines Bot commented May 29, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 29, 2026

🔥 [CI Build #1adb358] Build failed (Detect API changes) 🔥

Uh oh!

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 29, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

dalexsoto commented May 29, 2026

Uh oh!

azure-pipelines Bot commented May 29, 2026

Uh oh!

vs-mobiletools-engineering-service2 commented May 29, 2026

✅ [CI Build #1adb358] Prepare .NET Release succeeded ✅

iOS

MacCatalyst

macOS

tvOS

Other

Uh oh!

vs-mobiletools-engineering-service2 commented May 29, 2026

✅ [PR Build #1adb358] Build passed (Build packages) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 29, 2026

✅ [PR Build #1adb358] Build passed (Build macOS tests) ✅

Uh oh!

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 31, 2026

🔥 [CI Build #1adb358] Test results 🔥

Test results

Failures

❌ monotouch tests (iOS) [attempt 3]

Failed tests

❌ monotouch tests (MacCatalyst)

❌ monotouch tests (tvOS) [attempt 3]

Failed tests

Successes

macOS tests

Linux Build Verification

dalexsoto commented May 29, 2026 •

edited

Loading