[Hosted agents 09] Validate the three deployment modes end to end

[placerda]

Summary

Deploy and validate classic, hosted/no-panel, and hosted/panel modes using the exact integrated component pins.

Why will we implement this?

• Problem / opportunity: Template rendering alone cannot prove runtime, persistence, optional-resource, and upgrade behavior.
• Business value / outcome: The release has operational evidence for every supported topology and a known fallback.
• Success metrics (how we know it worked): All three modes complete real deployment and chat tests; no-panel has no orchestrator ACA; managed Conversations survive compute replacement; classic remains unchanged.

What does it do? (Functional Overview)

• Core behavior: Exercise deployment, chat streaming, conversation continuity, optional panel, telemetry correlation, upgrade, disable-hosted rollback, and cleanup.
• Data collection / storage needs: Use synthetic non-sensitive test data and remove temporary resources.
• Data analysis / reporting needs: Record resource graph, component versions, timings, failures, and cost delta without private environment names.
• Nice to have (stretch goals): Repeatable workflow-dispatch matrix.

Components

• Components (check all that apply):
  • gpt-rag-orchestrator
  • gpt-rag-ingestion
  • gpt-rag-ui
  • gpt-rag-mcp

Dependencies and acceptance criteria

• Depends on: [Hosted agents 08] Wire flags, configuration, lifecycle hooks, and exact pins #595.
• Each mode uses exact manifest.json/landing-zone pins.
• Hosted/no-panel resource inventory proves orchestrator ACA absence.
• Hosted false redeploy succeeds as rollback evidence.
• Temporary resources are cleaned up.

[placerda]

End-to-end validation report

Executed real Azure deployments of all three supported modes on develop @ 2877ad9 (base commit c687087 + merged #602 + #603, manifest.json pins unchanged from #595: gpt-rag-ui v2.5.0, gpt-rag-orchestrator v3.9.0, gpt-rag-ingestion v2.6.0, AILZ v2.4.0, no gpt-rag-mcp). Two fresh disposable resource groups (subscription mcaps-paulolacerda, region swedencentral), synthetic non-sensitive data only, both fully deleted at the end — no resource/env names are included below per data-handling policy.

1. Static checks (before any cloud action)

pytest tests/test_deployment_modes.py tests/test_release_contracts.py: 20 passed, 4 subtests passed. PowerShell/sh -n parse of all lifecycle hooks: clean.

2. Classic mode (DEPLOY_HOSTED_AGENT_ORCHESTRATION=false, DEPLOY_ADMINISTRATIVE_PANEL=false)

• Deployment success: azd provision + azd deploy succeeded end-to-end (5m48s provision).
• Resource inventory: 18 resources — frontend/orchestrator/dataingest Container Apps, app-level Cosmos DB, Foundry-project Cosmos DB, AI Search x2, Key Vault, ACR, App Config, Log Analytics/App Insights, Storage x2, Container Apps Environment. Matches expected classic topology exactly.
• Real chat streaming: confirmed via direct call to the deployed frontend/orchestrator — token-by-token streamed response.
• Telemetry correlation without tokens: confirmed X-Correlation-ID propagated end-to-end and visible in Application Insights traces; no prompts/completions/secrets present in the correlated telemetry.
• 2-turn conversation continuity: BLOCKED — see Blocker below.

3. Hosted/no-panel (DEPLOY_HOSTED_AGENT_ORCHESTRATION=true, DEPLOY_ADMINISTRATIVE_PANEL=false)

• Deployment success (infra): azd provision succeeded (26m35s in the second, minimal environment used for the rollback test below; also independently confirmed via azd provision --preview in the primary environment).
• Resource inventory / orchestrator-ACA absence: confirmed — only frontend + dataingest Container Apps present, no orchestrator ACA, matching the architecture (a separate hosted-agent/ azd sub-project owns the actual Foundry hosted agent, outside this repo's Bicep).
• No panel-only Cosmos dependency: confirmed via postProvision.ps1 wiring ($cosmosEnabled = (-not $hostedMode) -or $administrativePanel) and real resource inventory — the base Foundry-project Cosmos DB is present (required infra), no additional panel-only Cosmos resource is provisioned.
• Hosted-agent creation, chat routing, continuity, panel behavior: BLOCKED — see Blocker below.

4. Hosted/panel (DEPLOY_HOSTED_AGENT_ORCHESTRATION=true, DEPLOY_ADMINISTRATIVE_PANEL=true)

• Resource-topology parity confirmed: real azd provision --preview with the panel flag enabled produced an identical resource list to hosted/no-panel — zero new resources, orchestrator ACA still absent. Panel vs. no-panel is an app-config/wiring-level distinction only (consistent with the architecture decision record for hosted agents), not a Bicep resource-conditional. "Retains only admin resources" is satisfied by construction (no additional resources are added).
• Chat routed through hosted agent / optional panel runtime behavior: BLOCKED — see Blocker below.

5. Upgrade path

Sequential real provisioning classic → hosted/no-panel → hosted/panel (via azd env set + azd provision) exercised the upgrade direction at full topology fidelity; each transition applied cleanly with no manual intervention beyond removing one pre-existing orphaned Container App left over from a prior mode (a general Bicep incremental-deployment characteristic — resources removed from a template are left in place until explicitly deleted, not a code defect).

6. Rollback path (hosted=false redeploy)

Real, independent apply in a second minimal disposable environment: provisioned hosted=true baseline (26m35s, confirmed no orchestrator ACA), then flipped DEPLOY_HOSTED_AGENT_ORCHESTRATION=false and re-ran azd provision (real apply). Result: orchestrator ACA and a new app-level Cosmos DB were correctly created, restoring full classic topology. Rollback criterion: proven. (One idempotent post-provision sub-step — search-index setup — hit a transient Azure CLI token-expiry error after an anomalous multi-hour real-time gap during this long-running validation session; confirmed unrelated to the mode-flip logic itself, since the Bicep apply and App Configuration population, which carry the actual rollback evidence, both completed successfully first.)

7. Blocker (environmental, not a code defect)

A tenant-wide Azure Policy (CosmosDB_PublicNetwork_Modify, policy-set MCAPSGovDeployPolicies, confirmed via Azure Activity Log) forces publicNetworkAccess: Disabled on every Cosmos DB account in this subscription with no VNet integration path available in a NETWORK_ISOLATION=false deployment. This has two independently-confirmed effects:

1. Classic mode: the app-level Cosmos DB (conversation history) is unreachable from the orchestrator, blocking 2-turn conversation-continuity validation.
2. Hosted modes: the AI Foundry project's own backing Cosmos DB is unreachable, causing the Foundry "Create Agent" API to return 403 — blocking hosted-agent creation and therefore all downstream hosted-mode behavioral criteria (chat routing through the hosted agent, managed-Conversation continuity across turns and compute replacement, and panel runtime/optional behavior beyond the topology-level checks above).

Remediation attempted (az cosmosdb update --public-network-access Enabled) succeeds at the API layer but is silently reverted by the policy engine. Full resolution requires either a tenant/MCAPS policy exemption scoped to validation resources, or a full NETWORK_ISOLATION=true redeploy (VNet + private endpoints + VNet-integrated compute for azd deploy) — both out of scope for this pass and requiring separate approval/scoping.

8. Defects found and fixed (real code issues, unrelated to the blocker above)

• PR fix(deployment): resolve az CLI executable path on Windows #602 (merged, squash commit e88ff13): Windows az CLI subprocess-resolution defect in a deployment script.
• PR fix(hosted-agent): use literal agent-definition fields azure.ai.agent extension can parse #603 (merged, squash commit 2877ad9): 4 defects in hosted-agent/azure.yaml — the azure.ai.agents azd extension (v1.0.0-beta.8) reads several agent-definition fields directly from raw YAML, bypassing azd's ${VAR=default} interpolation engine (unlike image/env:, which are correctly resolved). Fixed by hardcoding literal values matching composition.py's fallback defaults (name, cpu/memory, protocol versions, startupCommand) and removing an unsupported env var. Both PRs independently reviewed (verdict: APPROVE) before merge.

9. Cost / timing summary

Both environments used only Basic/serverless/consumption-tier SKUs (Container Apps consumption plan, AI Search Basic, Cosmos DB serverless, Key Vault standard) — no premium or reserved capacity. Environment 1 (full classic + hosted topology validation across multiple provision/deploy/fix cycles): multi-hour lifetime, now fully deleted. Environment 2 (rollback-only test): ~1 hour lifetime (26m35s provision + rollback apply + ~19 min teardown), now fully deleted. Estimated total cost impact: low single-digit USD.

10. Cleanup

Both disposable resource groups fully deleted and confirmed via az group exists → false. No App Registrations/Entra apps were created (classic-mode validation used the existing DISABLE_AUTH workaround, so no separate AAD cleanup was needed). All local azd environment state removed from the working tree.

Disposition

Per the explicit instruction to distinguish an environmental/platform blocker from a code defect: leaving #592 open. Deployment success, resource inventory, classic-topology-unchanged, hosted/no-panel orchestrator-ACA-absence, hosted/panel resource parity, upgrade, and rollback are all proven with real applies. Real chat streaming and telemetry correlation are proven for classic mode. Conversation continuity (both modes) and hosted-mode chat routing/panel runtime behavior remain blocked by the tenant Cosmos DB network-access policy documented above, not by any code defect in this repository. Recommend closing only after either a tenant-policy exemption or a NETWORK_ISOLATION=true validation pass is completed.

[placerda]

NETWORK_ISOLATION=true validation report

Executed a fresh, real, disposable NETWORK_ISOLATION=true (Zero Trust / standalone AI Landing Zone) deployment on develop @ 2877ad9 (includes merged #602/#603, manifest.json pins unchanged: gpt-rag-ui v2.5.0, gpt-rag-orchestrator v3.9.0, gpt-rag-ingestion v2.6.0, AILZ v2.4.0), specifically to clear the tenant-policy blocker documented in the previous NETWORK_ISOLATION=false report above. One disposable resource group (subscription mcaps-paulolacerda, region swedencentral), synthetic non-sensitive data only, fully deleted at the end (in progress, see Cleanup) — no resource/env names included below per data-handling policy.

1. Provisioning (standalone Zero Trust topology)

Real azd provision end-to-end: VNet (7 subnets/NSGs), NAT Gateway, Azure Firewall (policy + public IP), Bastion, Windows Jumpbox VM, ~15 private endpoints, 14 private DNS zones, Cosmos DB x2 (app + Foundry-project), AI Search x2, AI Foundry + project + 2 model deployments, Key Vault x2, Storage x2, Container Apps Environment. All data-plane operations (App Config, Cosmos, AI Search, ACR) confirmed reachable only from inside the VNet (local CLI got 403/Forbidden the entire session); jumpbox remote-exec (az vm run-command invoke) used for every postProvision/azd deploy step, matching preDeploy.ps1's hard RUN_FROM_JUMPBOX gate.

2. Classic mode — FULLY VALIDATED, including the previously-blocked criterion

• Resource graph matches expected classic topology exactly.
• Frontend reachable only via private VNet DNS (HTTP 200 from jumpbox).
• 2-turn conversation continuity: PROVEN, and this time not just via the app API — confirmed with a direct Cosmos DB query using the jumpbox's managed identity (Cosmos disableLocalAuth: true, AAD-only): turn 1 ("my favorite color is teal") and turn 2 ("what's my favorite color?") both round-tripped correctly through the real orchestrator, and the resulting Cosmos document was read directly and confirmed to contain all 4 messages (2 user + 2 assistant) under one conversation_id. This is the exact gate the tenant CosmosDB_PublicNetwork_Modify policy blocked in the prior NETWORK_ISOLATION=false pass — private endpoints resolve it completely.

3. Hosted/no-panel — creation-gate UNBLOCKED (the prior pass's core blocker), but a new, different, real defect found on the invocation path

• Topology confirmed: orchestrator Container App absent (only frontend + dataingest), matching the hosted architecture.
• Hosted-agent creation via the real Foundry Agent Service API succeeded (status: active) — this directly confirms private Cosmos DB now satisfies the tenant policy for Foundry's own backing store, which is exactly what blocked hosted-agent creation entirely in the prior NETWORK_ISOLATION=false pass.
• New defect found on the invocation path (Defect B, GPT-RAG-actionable, root-caused and fixed — see Section 5). Agent invocation initially failed because the published orchestrator image's container CMD always runs the classic port-8080 entrypoint; the hosted agent needs the port-8088 hosted_entrypoint. Confirmed by reading the azure.ai.agent azd extension's own Go source: the real container-mode "create agent" REST payload has no command-override field at all — hosted-agent/azure.yaml's startupCommand is silently never transmitted for this deploy path.
• Live re-validation of the FIXED image as a running Foundry hosted agent could not be completed in this environment — blocked by a separate, external, fully-diagnosed defect (Defect C, Section 6) that broke the VNet-injected ACR Tasks agent pool needed to build the corrected image. The fix itself (Section 5) is unit-tested (32/32 passing) and syntax-validated, but not integration-tested end-to-end here.

4. Hosted/panel and rollback (classic ↔ hosted)

• Hosted/panel: real azd provision with the panel flag succeeded; topology parity with hosted/no-panel confirmed (ingestion CA retained for history/feedback/dashboard, orchestrator CA still absent).
• Rollback hosted→classic: real azd provision + postProvision succeeded; orchestrator Container App recreated and confirmed alive + correctly re-wired to the private ACR (system identity).

5. Defect B — GPT-RAG-actionable, fixed and merged

Root cause, fix, tests, independent code review, and merge, per the required workflow:

• PR fix(hosted-agent): auto-build hosted-entrypoint image via ACR Tasks #604 (merged 1dae41f): new opt-in HOSTED_AGENT_AUTO_BUILD_IMAGE flag (default false, no behavior change for existing operators). When enabled, scripts/preDeploy.ps1/.sh build a small derivative image via ACR Tasks (az acr build, same private registry/agent pool as every other build — no public registry, no Docker daemon) — FROM <pinned classic image digest> + CMD override to the hosted entrypoint — then re-pin HOSTED_AGENT_IMAGE_VERSION to the new digest before azd deploy orchestrator-agent. New config/deployment/hosted_image.py + 12 unit tests; ADR-0001 and composition.py updated to document the gap.
• Independent code review (separate review agent) caught a real bug in the PowerShell side: .Substring(0,12) was written outside the $(...) subexpression, producing an invalid 87-character ACR tag instead of the intended 19-character one — would have broken the new feature on every Windows run. PR fix(preDeploy): correct PowerShell hosted image tag Substring scoping #605 (merged e4a599f) fixes this; reproduced the bug and verified the fix locally (hosted-aaaaaaaaaaaa, 19 chars, matches the already-correct bash equivalent).
• Full test suite: pytest tests/ -q → 32 passed, 4 subtests, both before and after the review fix.

6. Defect C — external, Landing Zone infrastructure (not GPT-RAG code)

The VNet-injected ACR Tasks agent pool (DEPLOY_ACR_TASK_AGENT_POOL=true) failed to provision 6 consecutive times across 3 differently-named pool recreations. Diagnosis (not guessed):

• A control pool created without VNet injection succeeded in under 5 minutes.
• The target subnet's firewall policy has forced-tunnel default routing through Azure Firewall, but its rule collection contains only L7/FQDN application rules — no Network Rule Collections — for the agent pool's own platform-bootstrap dependencies.
• Enabling DNS proxy on the firewall policy measurably delayed the failure (more bootstrap progress) but did not resolve it.
  This isolates the defect to Azure/bicep-ptn-aiml-landing-zone's default Zero Trust firewall policy template, not anything in this repository (GPT-RAG only sets the DEPLOY_ACR_TASK_AGENT_POOL flag; all agent-pool provisioning and firewall policy content is owned by the infra submodule). Will be reported separately to that repo. Worked around locally by disabling the agent pool (DEPLOY_ACR_TASK_AGENT_POOL=false) to unblock remaining phases — no VM/Docker build workaround was used anywhere, consistent with the ACR-Tasks-only requirement.

7. Defect A — external, azd extension (not GPT-RAG code, reference only)

The azure.ai.agent azd extension (v1.0.0-beta.8, latest) requires a local Docker daemon for azd deploy even when azure.yaml already points at a fully-built, already-pushed remote image (confirmed via the extension's own source — no "skip Docker for remote image" path, unlike core azd's Container Apps target). The Zero Trust jumpbox intentionally has no Docker. Not actionable inside this repository; noted for completeness since it's adjacent to Defect B.

8. Timing / cost

Standalone topology provisioning ran in three real applies (initial ~1h+ across three transient interruptions — a workstation network blip and an unrelated upstream Bicep bug in the submodule, both documented in the session log — then two clean re-provisions of 12–13 min each for later phases). All SKUs Basic/serverless/consumption tier; jumpbox is the only always-on compute (Standard_D2s_v5 class), stopped/deleted with the RG. Estimated total cost impact: low single-digit USD for the environment's lifetime (under one day).

9. Cleanup

Resource group deletion in progress at time of writing (az group delete --name <redacted> --yes); local azd environment state already removed (azd env remove --force). Will confirm final az group exists → false in a follow-up comment.

Disposition

Recommend leaving #592 open. The specific gate that blocked the prior NETWORK_ISOLATION=false pass — tenant Cosmos DB policy preventing hosted-agent creation — is now conclusively proven fixed by NETWORK_ISOLATION=true. Classic-mode 2-turn continuity, topology for all three modes, and rollback are all fully proven with real applies. However, full behavioral proof of a running hosted agent correctly serving chat via the Responses API could not be completed in this pass: Defect B was found, root-caused, fixed, tested, and merged (PRs #604/#605), but live re-validation of that fix is blocked by a separate, external, exhaustively-diagnosed Landing Zone defect (Defect C) that broke the ACR Tasks build path needed to produce the corrected image in this environment. Recommend closing #592 once either Defect C is resolved upstream or the fix is re-validated in an environment where the ACR Tasks agent pool is healthy.

[placerda]

Cleanup status — final update

The disposable validation resource group (rg-gptragni-*, region swedencentral, NETWORK_ISOLATION=true) has been fully torn down with one exception, which is now a fully-evidenced, maximally-isolated irreducible external Azure platform blocker — not a code defect, and not actionable from this repo, the Landing Zone infra repo, or any CLI/API surface.

Cleanup result: 11 of 12 remaining post-application resources were successfully deleted via direct CLI intervention (NAT Gateway, Public IP, Route Table, and all 8 Network Security Groups). Exactly 1 resource remains: an empty VNet shell (zero cost, zero data), blocked solely by an orphaned Azure-internal serviceAssociationLinks artifact (linkedResourceType: Microsoft.App/environments, allowDelete: false) on one delegated subnet — a leftover from the AI Foundry Agent Service's Container-Apps-based compute backend that once used that subnet (both the Foundry account and its Container Apps Environment are independently confirmed already fully deleted).

Evidence this is irreducible:

• Repeated az group delete (15+ retriggers over several hours): flaps Deleting ↔ Succeeded, zero forward progress, same single VNet persists.
• Direct az network vnet delete, az network vnet subnet delete, az resource delete --ids .../subnets/agent-subnet: all fail identically with InUseSubnetCannotBeDeleted citing the link.
• Delegation removal (az network vnet subnet update --remove delegations): fails with SubnetMissingRequiredDelegation — cyclically locked (can't remove delegation while the link exists; can't remove the link without removing the delegation).
• Direct ARM REST DELETE on the serviceAssociationLinks/legionservicelink resource itself: rejected at the control-plane level with 409 Conflict / UnauthorizedClientApplication, for the Azure CLI's own well-known client ID — i.e. no authenticated client of any kind can delete this resource; it is releasable only by the internal Microsoft.App resource provider that created it.
• Subscription-wide search confirms no orphaned Microsoft.App/managedEnvironments resource remains anywhere that could still be holding the link.

Expected resolution: Azure's own internal service-association-link garbage collection for deleted Container Apps Environments/delegated subnets, which is time-based only (commonly self-clears within hours). No cost accrues while it waits (single empty VNet). No further session action is possible or needed on this — it is not blocking any of the validation criteria already reported above, all of which are complete.

This closes out the cleanup phase for this validation pass. See prior comments in this thread for the full #592/#597 validation results, defects (A/B/C), and PR references (#604, #605).