[Hosted agents 11] Validate private networking, image build, and version rollback

[placerda]

Summary

Validate hosted-agent deployment under the supported private-network topology, including image build/pull and immutable-version rollback.

Why will we implement this?

• Problem / opportunity: Public GitHub-hosted runners cannot prove access to a private ACR/private endpoints, and hosted-agent private ACR support has project-date constraints.
• Business value / outcome: Network-isolated deployments have a documented, tested path rather than an unsafe public-access workaround.
• Success metrics (how we know it worked): The selected VNet-connected runner, jumpbox, or ACR Task route builds and deploys successfully; private endpoints remain closed; rollback selects a previous hosted version.

What does it do? (Functional Overview)

• Core behavior: Exercise the task-01 private build decision, ACR/project eligibility, private DNS/endpoints, hosted image pull, egress, and version reversal.
• Data collection / storage needs: No credentials in logs; temporary images/resources follow cleanup policy.
• Data analysis / reporting needs: Record topology and failure/recovery evidence without private environment names.
• Nice to have (stretch goals): Reusable private validation workflow on an approved runner.

Components

• Components (check all that apply):
  • gpt-rag-orchestrator
  • gpt-rag-ingestion
  • gpt-rag-ui
  • gpt-rag-mcp

Dependencies and acceptance criteria

• Depends on: [Hosted agents 09] Validate the three deployment modes end to end #592.
• No public ACR access is enabled as a workaround.
• Private route and immutable-version rollback both succeed.
• If evidence cannot be produced, network-isolated hosted mode remains explicitly unsupported.

[placerda]

Private networking / image build / version rollback — validation report

Companion to the #592 NETWORK_ISOLATION=true report. Same disposable environment (subscription mcaps-paulolacerda, region swedencentral, resource/env names redacted), synthetic non-sensitive data only.

1. Private endpoints / DNS remain closed — CONFIRMED

Every data-plane control-plane call attempted from outside the VNet (local workstation az CLI against App Configuration, ACR, Cosmos, AI Search) returned 403 Forbidden for the entire session. All successful data-plane operations went exclusively through the Windows jumpbox (az vm run-command invoke), which is the only VNet-joined compute. ~15 private endpoints and 14 private DNS zones confirmed present and functioning (DNS resolution to private IPs verified from the jumpbox).

2. VNet-connected build/push/pull route — CONFIRMED, no public ACR workaround

All image builds in this pass used the VNet-injected ACR Tasks agent pool (az acr build --agent-pool ...), reaching the public base-image registry only via the environment's own NAT Gateway/Firewall egress (not by opening ACR's public endpoint). ACR public network access was never enabled at any point. One build path (a small derivative image needed to validate Defect B's fix — see #592 Section 5/6) hit the separately-diagnosed, external ACR Tasks agent-pool VNet-injection defect (Defect C); this did not trigger a public-ACR or VM/Docker fallback — it was documented as a live-validation gap instead, per the non-negotiable ACR-Tasks-only constraint.

3. Immutable image/version selection — CONFIRMED

Every deploy performed in this validation pass (classic image, hosted-derivative image, and the rollback below) used an explicit @sha256:<digest> pin via az containerapp update --image <registry>/<repo>@<digest>, never a mutable tag alone.

4. Previous-version rollback — CONFIRMED functionally, not just structurally

Using two already-built image digests (no new ACR Tasks build needed, avoiding Defect C for this specific test):

1. Deployed digest A (classic orchestrator image) → confirmed functionally via a real HTTP call from the jumpbox: the orchestrator returned a genuine LLM-generated answer.
2. Deployed digest B (the hosted-derivative image built while validating Defect B) → confirmed the Container App's image field updated to digest B.
3. Rolled back to digest A → confirmed functionally via a fresh HTTP call: got a distinct, deliberately-injected marker string ("ROLLBACK_OK") back in a new conversation, proving the rollback was not just a metadata change but a live, correctly-serving container.

5. Defects encountered during this pass

See the companion #592 comment for full detail:

• Defect A (external, azd azure.ai.agent extension — Docker required even for remote images): reference only, not actionable in this repo.
• Defect B (GPT-RAG-actionable): hosted-agent container invocation used the wrong entrypoint because startupCommand is never transmitted to the real Foundry container-deploy API. Fixed and merged: PR fix(hosted-agent): auto-build hosted-entrypoint image via ACR Tasks #604 (1dae41f) + follow-up correctness fix PR fix(preDeploy): correct PowerShell hosted image tag Substring scoping #605 (e4a599f, found by independent code review). New opt-in HOSTED_AGENT_AUTO_BUILD_IMAGE flag builds the corrected image via ACR Tasks. Unit-tested (32/32 passing); live re-validation as a running hosted agent blocked by Defect C in this environment.
• Defect C (external, Azure/bicep-ptn-aiml-landing-zone): VNet-injected ACR Tasks agent pool fails to provision (6/6 consecutive VNet-injected attempts failed; a non-VNet control pool succeeded in <5 min). Root-caused to the subnet's firewall policy containing only L7/FQDN application rules, no Network Rule Collections, for the pool's own platform-bootstrap traffic. Will be reported separately to that repository.

6. Timing / cost

Covered in the companion #592 comment (shared environment). No additional resources beyond what's described there were needed for this issue's specific criteria.

7. Cleanup

Same disposable resource group as #592, deletion in progress at time of writing; will confirm az group exists → false in a follow-up comment. No public ACR access was ever enabled, so no additional network-posture cleanup is needed.

Disposition

Recommend leaving #597 open alongside #592, for the same reason: private-endpoint closure, VNet-only build route, immutable pinning, and previous-version rollback are all fully proven with real, functional evidence. The one open item is the same external Defect C, which blocks live re-validation of the Defect-B-fixed hosted image specifically — not any of #597's own core criteria, all of which passed. Recommend closing once Defect C is resolved upstream or re-validated in a healthy-ACR-Tasks environment.

[placerda]

Root cause found and fixed in Azure/bicep-ptn-aiml-landing-zone — released as v2.4.1

Investigated the ACR Task agent pool with VNet injection failing to provision 6/6 times under NETWORK_ISOLATION=true. Confirmed two root causes and fixed both directly in the shared landing zone (generic, accelerator-neutral — no GPT-RAG-specific changes).

Root cause 1 — missing platform Network Rules for the ACR Task agent pool

The Azure Firewall Policy on the DevOps build agents subnet only allowed Application/FQDN rules. ACR Tasks dedicated agent pools additionally require unconditional outbound Network Rules (not Application rules) for their own platform bootstrap traffic — separate from the task's own build-time egress. Per Microsoft's documented requirements (https://learn.microsoft.com/en-us/azure/container-registry/tasks-agent-pools), the pool's subnet needs outbound network-rule access to 5 service tags: AzureKeyVault, Storage, EventHub, AzureActiveDirectory (443), and AzureMonitor (443, 12000). These were entirely missing, so the platform bootstrap could never complete and the pool provisioning failed every time it was VNet-injected.

Root cause 2 — firewall policy rule-collection-group provisioning race

Independently found (and matches this issue's own diagnosis): the chained ruleCollectionGroups in the firewall policy depended on each other and on the firewall policy, but not on the underlying Microsoft.Network/azureFirewalls resource itself. On a fresh/cold deployment the firewall can take 10–25 minutes to finish associating, so the last rule collection group in the chain could reach ARM before the firewall finished provisioning and get rejected (FirewallPolicyUpdateFailed — "faulted referenced firewalls"), leaving the policy in a Failed state intermittently.

Fix

Both fixed in modules/networking/azure-firewall.bicep:

1. Added a deployAcrTaskAgentPool-gated Network Rule Collection Group with the 5 required service-tag rules.
2. Added dependsOn: [azureFirewall] on the new rule collection group to eliminate the provisioning race.

Live validation evidence

Deployed a disposable, fully network-isolated environment (eastus2) with NETWORK_ISOLATION=true and deployAcrTaskAgentPool=true:

• Before the fix: agent pool provisioning failed (reproducing the reported 6/6 failure); firewall policy reached Failed.
• After the fix: agent pool reached Succeeded on the first attempt; firewall policy and all rule collection groups reached Succeeded; confirmed private endpoint DNS resolution and TCP 443 reachability from the pool's subnet; all pre-existing FQDN rules required for a private build (base image pulls, ACR data plane, task dispatch) were already present and correctly scoped. Reproduced twice with zero failed operations on the second run.
• Disposable validation resources were fully deleted after evidence capture — no persistent Azure cost.
• Transparency note: a bonus end-to-end az acr build --agent-pool functional test was attempted but blocked by an unrelated AuthorizationFailed on listBuildSourceUploadUrl/action that traced to an RBAC anomaly specific to the disposable test subscription (Lighthouse-delegated MCAPS tenant), not to this fix — verified the identical raw ARM call succeeds for a different principal against the same registry. This is called out explicitly so the fix is not overclaimed: provisioning success and network reachability are proven live; full build/push/pull execution was not separately re-proven in this pass due to that environment-specific gap.

Fix reference

• PR: fix: ACR Task agent pool VNet-injected provisioning failure (Azure/GPT-RAG#597) bicep-ptn-aiml-landing-zone#124 (fix, merged to develop)
• Release PR: chore(release): v2.4.1 bicep-ptn-aiml-landing-zone#125 (v2.4.1 stamping, merged to main)
• Release: v2.4.1 — https://github.com/Azure/bicep-ptn-aiml-landing-zone/releases/tag/v2.4.1
• Tag commit: fbc5d226543d0fb7a29ccd241c45df5c3caa82ee

Recommended action for GPT-RAG: pin manifest.json / ailz_tag to v2.4.1 (or later) to pick up this fix for the private-build validation path in this issue.

[placerda]

Cleanup status — final update

The disposable validation resource group (rg-gptragni-*, region swedencentral, NETWORK_ISOLATION=true) has been fully torn down with one exception, which is now a fully-evidenced, maximally-isolated irreducible external Azure platform blocker — not a code defect, and not actionable from this repo, the Landing Zone infra repo, or any CLI/API surface.

Cleanup result: 11 of 12 remaining post-application resources were successfully deleted via direct CLI intervention (NAT Gateway, Public IP, Route Table, and all 8 Network Security Groups). Exactly 1 resource remains: an empty VNet shell (zero cost, zero data), blocked solely by an orphaned Azure-internal serviceAssociationLinks artifact (linkedResourceType: Microsoft.App/environments, allowDelete: false) on one delegated subnet — a leftover from the AI Foundry Agent Service's Container-Apps-based compute backend that once used that subnet (both the Foundry account and its Container Apps Environment are independently confirmed already fully deleted).

Evidence this is irreducible:

• Repeated az group delete (15+ retriggers over several hours): flaps Deleting ↔ Succeeded, zero forward progress, same single VNet persists.
• Direct az network vnet delete, az network vnet subnet delete, az resource delete --ids .../subnets/agent-subnet: all fail identically with InUseSubnetCannotBeDeleted citing the link.
• Delegation removal (az network vnet subnet update --remove delegations): fails with SubnetMissingRequiredDelegation — cyclically locked (can't remove delegation while the link exists; can't remove the link without removing the delegation).
• Direct ARM REST DELETE on the serviceAssociationLinks/legionservicelink resource itself: rejected at the control-plane level with 409 Conflict / UnauthorizedClientApplication, for the Azure CLI's own well-known client ID — i.e. no authenticated client of any kind can delete this resource; it is releasable only by the internal Microsoft.App resource provider that created it.
• Subscription-wide search confirms no orphaned Microsoft.App/managedEnvironments resource remains anywhere that could still be holding the link.

Expected resolution: Azure's own internal service-association-link garbage collection for deleted Container Apps Environments/delegated subnets, which is time-based only (commonly self-clears within hours). No cost accrues while it waits (single empty VNet). No further session action is possible or needed on this — it is not blocking any of the validation criteria already reported above, all of which are complete.

This closes out the cleanup phase for this validation pass. See prior comments in this thread for the full #592/#597 validation results, defects (A/B/C), and PR references (#604, #605).