feat: Add inboundInterception=transparent (drops port stealing)

[huang195]

Summary

Adds an opt-in inbound shape for proxy-sidecar / lite that stops stealing the agent's port. Companion to rossoctl/cortex#776 (merged), which implements the listener and iptables rules; this is the switch that turns them on. Until this lands, that code is inert — nothing sets inbound_interception or INBOUND_TRANSPARENT_PORT.

With inboundInterception: transparent, proxy-init installs a PREROUTING REDIRECT and AuthBridge recovers each connection's real destination via SO_ORIGINAL_DST, so the agent keeps the port it already binds.

Why

Port stealing has three problems, all of which the existing code acknowledges:

1. The relocated port is an unvalidated entrance. originalPort+1 is declared in the pod spec and directly reachable, so any pod can reach the agent without JWT validation. Pods are the granularity NetworkPolicy and ztunnel enforce at, so this was the boundary that mattered.
2. It depends on the agent honoring PORT. pod_mutator.go concedes that agents which hardcode their listen port "won't be affected" — in practice they collide with AuthBridge on the stolen port and the pod never starts. No config can fix that.
3. Only Ports[0] of the first container with ports is relocated, so a second declared port was never proxied. Undeclared ports aren't even in usedPorts, so findFreePort can hand the agent a port it already uses.

Transparent interception removes all three: no relocation, no PORT env var, no second port to discover, and every port the agent listens on is covered.

Default is unchanged

Default stays reverse-proxy. Transparent costs a privileged proxy-init container, so it is chosen deliberately; with the field unset every existing path is byte-identical. Two fallbacks are deliberately biased toward the unprivileged shape:

• An unrecognized value falls back to reverse-proxy, so a typo cannot silently grant a NET_ADMIN init container.
• transparent + egressEnforcement: none falls back to reverse-proxy. The two features share one proxy-init container; without it nothing would REDIRECT to the inbound listener and inbound would be silently unenforced — worse than port stealing, which at least validates Service-routed traffic.

proxy.allowedInboundInterception lets a platform admin forbid it (["reverse-proxy"]) or mandate it (["transparent"]).

Deliberately NOT adding an AgentRuntime spec field

The obvious move was to mirror spec.egressEnforcement — but that field is dead surface. The pod mutator has no access to the CR and resolves everything from the namespace ConfigMap, so nothing reads spec.egressEnforcement (spec.mtlsMode reaches the pod only as a rollout-triggering annotation, not as mutator input). An enum-validated spec field that silently does nothing is worse than no field, and plumbing CR → mutator properly is a separate change that would have to fix mtlsMode and egressEnforcement too.

Worth deciding separately: spec.egressEnforcement should probably be either wired up or removed.

Details

• The per-agent ConfigMap gets inbound_interception + transparent_inbound_addr and must not get reverse_proxy_addr/reverse_proxy_backend — authbridge's config validation rejects that combination, so emitting both would crash-loop the pod.
• SIDECAR_PORTS_EXCLUDE carries the resolved forward-proxy port, not the script's 8081 default: findFreePort may have moved it, and an unexempted forward-proxy port would be swallowed by the inbound REDIRECT. Gating 9091 would put kubelet probes behind JWT validation.
• The sidecar declares its inbound port as transparent-in, so which shape is running is visible in the pod spec.
• Config validation rejects transparentInboundPort == transparentPort: both are listeners in one container, so a shared value fails the second bind at pod start, long after admission succeeded.

Upgrade-safe: the config loader overlays YAML onto compiled defaults, so an existing ConfigMap without the new keys keeps 8083 / both-allowed. Verified by running this operator against an unmodified 22-day-old cluster ConfigMap.

Verification

• make test — 21 packages green, including 13 new injector tests and 3 new config-validation tests (table-driven, 9 further cases).
• No CRD changes (none needed); make manifests generate produces no drift.
• Verified live on Kind: agent keeps :8000 with no PORT override, sidecar declares transparent-in=8083 + forward-proxy=8081, proxy-init receives INBOUND_TRANSPARENT_PORT=8083 / POD_IP (downward API) / SIDECAR_PORTS_EXCLUDE=8081,9091,9093,9094, and the per-agent ConfigMap contains inbound_interception: transparent + transparent_inbound_addr: :8083 with no reverse_proxy_* keys.

Port safety

usedPorts previously recorded the inbound port as reserved but never checked whether the agent had declared it. Under port stealing such a collision was accidentally survivable (Ports[0] got relocated); transparent mode deliberately removes that, so an agent declaring 8083 would have left two processes binding one port in a shared netns — admission succeeds, then the sidecar dies on address already in use in a container the user never wrote.

All sidecar-owned ports (8082, 8083, 9091, 9093, 9094) are now reserved unconditionally, and a declared collision on a port the sidecar actually binds falls back to port stealing with a warning naming the port and the owning listener. Reservation is separate from collision because containerPort is informational: under reverse-proxy nothing binds 8083, so declaring it is harmless.

This also closed a pre-existing hole — the transparent egress port (8082) was never reserved on main either, so an agent declaring 8081 could push the forward proxy onto a listener that is always on in this mode.

Sequencing note

Default image tags are :latest (values.yaml:250-252, defaults.go:38-45). If a workload selects transparent while running an authbridge image predating cortex#776, the pod crash-loops rather than silently bypassing: the old binary ignores the unknown inbound_interception key, finds no reverse_proxy_addr/reverse_proxy_backend (deliberately omitted for transparent), and fails config validation. Loud rather than dangerous, and unreachable by accident since the default is reverse-proxy — but on a cluster pinned to an older tag, opting in needs the image bumped first.

Verified end-to-end, with one gap

E2E suite 10 passed, 0 skipped on Kind (rossoctl/rossoctl#2393). The operator-side injection was confirmed live: the agent keeps its port with no PORT override, the sidecar declares transparent-in=8083, proxy-init receives INBOUND_TRANSPARENT_PORT / POD_IP / POD_IPS / SIDECAR_PORTS_EXCLUDE, and the per-agent ConfigMap correctly omits reverse_proxy_*. The A/B that matters: the transparent agent's real port returns 401 while the reverse-proxy control's relocated port returns 200 — the bypass this closes.

Still unverified: the ambient (HBONE) path's runtime behavior. The cluster used runs only istiod — no ztunnel DaemonSet, no istio-cni — so there is no ambient data plane, and counters confirm all traffic took plain PREROUTING (AB_INBOUND REDIRECT 4 packets, ambient DNAT 0). The rules are verified installed and correctly ordered; whether ztunnel's re-originated connection matches mark 0x539 + dst-type LOCAL is still reasoned from redirect-mode precedent rather than observed. Needs a cluster with ztunnel deployed.

[huang195]

Adds POD_IPS (status.podIPs) alongside POD_IP.

Review found that proxy-init's ambient DNAT target was keyed off POD_IP — the pod's primary address — so on a dual-stack pod the other family's HBONE delivery passed unvalidated while that family's PREROUTING rules were installed. proxy-init now selects a target per family; details and the companion fix in rossoctl/cortex#776.

make test still 21 packages green, with a new assertion that POD_IPS comes from the Downward API.

Assisted-By: Claude Code

[huang195]

Companion rossoctl/cortex#776 is now merged, so the authbridge listener and proxy-init rules this PR switches on are in place.

CI green (Unit, Integration, Lint, Shellcheck, Trivy, action pinning). E2E verified 10/10 on Kind — see rossoctl/rossoctl#2393; the operator-side injection specifically was confirmed live (agent keeps its port with no PORT override, sidecar declares transparent-in=8083, proxy-init receives INBOUND_TRANSPARENT_PORT/POD_IP/POD_IPS/SIDECAR_PORTS_EXCLUDE, and the per-agent ConfigMap correctly omits reverse_proxy_*).

One sequencing note before this merges

The default image tags are :latest (values.yaml:250-252, defaults.go:38-45), so behavior depends on which authbridge a cluster actually pulls. If a workload selects inboundInterception: transparent while running an authbridge image predating #776, the pod crash-loops rather than silently bypassing: the old binary ignores the unknown inbound_interception key, finds no reverse_proxy_addr/reverse_proxy_backend (this PR deliberately omits them for transparent), and fails config validation.

Loud rather than dangerous, and it cannot happen by accident since the feature is opt-in and defaults to reverse-proxy. But worth knowing: on a cluster pinned to an older authbridge tag, opting in needs the image bumped first.

Assisted-By: Claude Code

[huang195]

All four addressed (eee000e)

Verified each against the code first — all valid.

#1 port collision (must-fix)

Confirmed, and it extends one step further than reported. usedPorts[transparentInboundPort] only stopped findFreePort handing out 8083; it never checked what the agent declared. Your framing of why that matters here specifically — port stealing made a collision accidentally survivable, transparent mode removes that — is exactly right.

While fixing it I found the transparent egress port (8082) was never reserved either, on main or in this PR. So an agent declaring 8081 could push the forward proxy onto 8082 and collide with a listener that is always on in proxy-sidecar mode. Pre-existing, same family, closed by the same change.

Two design points worth flagging:

• Reservation is now separate from collision. All five sidecar ports are reserved unconditionally (so assignment stays stable if a namespace later flips mechanism), but only ports the sidecar actually binds in the resolved mechanism trigger the guard. containerPort is informational, so under reverse-proxy nothing binds 8083 and an agent declaring it is harmless.
• Fell back rather than rejected. Rejecting would block pods that declare a colliding port informationally without ever binding it — which works today. And your point that a first-port collision is resolvable by relocation is what makes fallback a real fix rather than a dodge. The warning names the port, the owning listener, and the effective mechanism.

A pre-existing test expected the forward proxy on 8082 when 8081 was taken. That expectation encoded the bug — now 8084, with the reason in a comment plus an assertion that it is none of the sidecar's ports.

You were right that no test covered this: all ten used ContainerPort: 8000. Added coverage for each of the five ports colliding, a non-colliding port staying transparent, and the forward proxy never landing on a sidecar port in either mechanism.

#2 rebase (must-fix)

Rebased onto a12afd0. One conflict: main added an agentRuntime argument to ensurePerAgentConfigMap while this PR replaced the inline map with listenerOverrides — took both.

#3 dead _ = agentBackendPort (nit)

Correct, and correct that it predates this PR. Dropped the parameter rather than annotating it; two call sites, internal package.

#4 resolver ordering (nit)

Valid, with one correction to the rationale. The resolver ran before the mode was even resolved, so I moved the whole block after mode resolution; envoy-sidecar/waypoint namespaces now log that the value is inapplicable instead of a resolved mechanism.

But authbridge would not have refused: the operator writes listener.inbound_interception only in the proxy-sidecar branch, so those modes never see it. The stray top-level inboundInterception that survives ConfigMap rendering is an unknown key to authbridge and ignored — same as egressEnforcement already is today. The misleading log was the real problem.

Fixing this also surfaced an observability bug of my own: the fallback set a source label nothing read, so ineffassign caught that the "resolved" line still said transparent after the code had overridden it.

Not included

make generate also rewrites zz_generated.deepcopy.go — main's committed copy is stale relative to its own types (a12afd0 added AgentRuntimeSpec.Auth without regenerating). Reverted it here rather than ship an unrelated API-looking diff in a feature PR. Worth a separate commit on main.

make test: 21 packages green. Lint clean apart from three pre-existing findings in files this PR does not touch.

Assisted-By: Claude Code

[cwiklik]

Summary

Well-engineered, well-tested, opt-in feature that replaces port-stealing with transparent inbound interception (iptables PREROUTING REDIRECT + SO_ORIGINAL_DST), closing a real JWT-bypass hole — the relocated originalPort+1 was a directly-reachable, unvalidated entrance. Default stays reverse-proxy, so no blast radius on merge. The PR body is exemplary: it enumerates the port-stealing failure modes it fixes and honestly flags what's still unverified.

Verified correct:

• Dual-mode ConfigMap parity — strict if transparentInbound / else in listenerOverrides: transparent emits inbound_interception + transparent_inbound_addr and never the reverse_proxy_* keys (which authbridge would reject together); reverse-proxy emits them. No path emits both or neither, and the ConfigMap, sidecar container, and proxy-init env all read one transparentInbound local, so the three artifacts can't disagree.
• Port reservation — all five sidecar ports (8082, 8083, 9091, 9093, 9094) reserved unconditionally in usedPorts; a declared agent collision on a port the sidecar actually binds falls back to port-stealing with a named warning.
• Config validation (types.go:213-220) — rejects transparentInboundPort == transparentPort and out-of-range.
• Enum defaults fail-safe — empty/unknown → reverse-proxy; allowedInbound[0] index is length-guarded; no unguarded map/slice access.
• Feature-gated / off by default — complies with the feature-flag policy.

Non-blocking findings:

1. SIDECAR_PORTS_EXCLUDE omits 8082 (pod_mutator.go:753 builds forwardProxyPort,9091,9093,9094). 8083 is correctly omitted (it's the REDIRECT target), but 8082 — the transparent egress listener you note is "always on in this mode" — is left out while 8081 (forward proxy) is excluded. Likely fine if the init script applies exclusions only to PREROUTING (8082 takes OUTPUT-originated traffic, never inbound), but the asymmetry with 8081 is worth confirming — it lands squarely in the ambient/HBONE path you flagged as unverified.
2. Re-injection idempotency — the transparent inbound env is added only when a proxy-init container is absent, so on re-admission the sidecar still binds the transparent listener and the ConfigMap still emits transparent_inbound_addr, but no PREROUTING rules get (re)installed → inbound silently unenforced. Mirrors the existing egress pattern, but the consequence (a bound listener nothing redirects to) is sharper here.
3. Your self-disclosed follow-ups — ambient/HBONE runtime still reasoned-not-observed (single-istiod cluster, ambient DNAT counter 0), egressEnforcement wire-or-remove, and the :latest sequencing (crash-loops loudly rather than bypassing on a pre-cortex#776 image) — are all legitimate follow-ups, not merge blockers, given default-off and the confirmed 401-on-real-port result.

Approving. The security posture strictly improves over port-stealing, the closed path is verified live, and the residual items are either opt-in-only or already flagged.

Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go webhook injector/config, Helm values, dual-mode parity, port/iptables safety, tests
CI: 15 green; E2E Tests pending.

[huang195]

Thanks @cwiklik — both findings checked. One was a non-issue with an undocumented reason; the other was real and is now fixed (1cabcaa).

#1 SIDECAR_PORTS_EXCLUDE omits 8082 — not a bug, but you shouldn't have had to guess

8082 is exempted; just not through this variable. init-iptables.sh's emit_inbound_exemptions exempts TRANSPARENT_PORT and INBOUND_TRANSPARENT_PORT unconditionally in its own loop, because it already knows both from env. cortex#776's harness asserts it directly (PASS: egress transparent port exempted).

So the asymmetry with 8081 is intentional: this variable carries only what the script cannot know — the forward-proxy port, which findFreePort may have moved off 8081, plus the health/stats/session constants. Listing 8082/8083 again would only duplicate RETURN rules.

One correction to the reasoning though: your hypothesis was "likely fine if the init script applies exclusions only to PREROUTING." It doesn't — after a fix in cortex#776 they apply to both hooks, including the ambient path. That fix exists because the ambient DNAT originally carried no exemptions, which made every one of them a silent no-op for mesh traffic (a JWT-gated :9091 crash-loops the pod). So your instinct to look here was right, just one round late. Added a comment at the construction site so the next reader doesn't have to reconstruct any of this.

#2 Re-injection idempotency — real, fixed

You're right, and it's worth more than a note because the failure is silent: sidecar binds :8083, ConfigMap says transparent, nothing installed PREROUTING → inbound unenforced with no error anywhere. That's the exact outcome this mode exists to prevent.

It's narrow — the webhook is CREATE-only on pods and a fresh pod comes from an unmutated Deployment template, so it needs a hand-authored proxy-init — but I'd rather guard it than rely on that reasoning holding.

The guard sits in the pre-flight block next to the port-collision check, before any artifact is built, which is the structural point: the ConfigMap is rendered ~60 lines earlier than the proxy-init injection, so flipping at injection time would have been too late and left exactly the divergence you described. An existing proxy-init without INBOUND_TRANSPARENT_PORT now falls back to reverse-proxy with a warning naming the effective mechanism and the fix; one that is armed is consistent and stays transparent. Both cases tested.

#3 Your triage of the follow-ups

Agreed on all three, and thanks for stating them as follow-ups explicitly. The ambient/HBONE gap is the one I'd most like closed — it's now merged-but-unexercised in cortex#776, and this PR is what activates it. It needs a cluster with ztunnel; the rules are verified installed and correctly ordered, but the mark 0x539 + dst-type LOCAL matching is still reasoned from redirect-mode precedent rather than observed.

make test: 21 packages green. Lint clean apart from three pre-existing findings in untouched files.

Assisted-By: Claude Code