Skip to content

feat: upstream traffic distribution and cluster DNS forward options - #5

Merged
rubencosta merged 1 commit into
mainfrom
ruben/upstream-same-node-failfast
Oct 19, 2026
Merged

rubencosta merged 1 commit into
mainfrom
ruben/upstream-same-node-failfast

Conversation

@rubencosta

@rubencosta rubencosta commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

For MOB-768. Needed by holoplot/sw__holokube#99, which gives node-local-dns one CoreDNS per controller as its upstream.

Change (chart 0.3.0)

  • config.upstreamServiceTrafficDistribution: sets trafficDistribution on the upstream service. With PreferSameNode, node-local-dns asks the cluster DNS pod on its own node first, and pods on other nodes only when its own node has none ready. PreferSameNode needs Kubernetes 1.35, or 1.34 with the PreferSameTrafficDistribution feature gate.
  • config.clusterDnsForwardConfig: extra forward options for the cluster domain and the reverse zones (in-addr.arpa, ip6.arpa), like upstreamForwardConfig already does for the default block. For example failfast_all_unhealthy_upstreams makes node-local-dns answer SERVFAIL at once when no upstream is healthy, instead of waiting for one.
  • upstreamForwardConfig now defaults to []. The template iterates over it as a list, and the old {} default made helm warn whenever a list was set.

The default render is unchanged.

Why

On an HA controller ring, a controller holding both k3s CoreDNS replicas lost power, and the cluster had no CoreDNS for 4 min. node-local-dns kept answering the cached names, thanks to serve_stale. But a name looked up without a trailing dot is first tried with the search domains. Those queries weren't in the cache, and node-local-dns waited on the dead upstream for each of them, so lookups timed out before reaching the cached name. Details in MOB-768.

Tested

  • helm unittest: 9 passed. The new tests/upstream_test.yaml covers both values and their defaults.
  • helm lint: clean.
  • The holokube values render without warnings. node-cache 1.26.7 parses and runs the rendered Corefile, and rejects an unknown forward option, so the parse check is real.
  • failfast_all_unhealthy_upstreams with node-cache 1.26.7, against an upstream that drops packets:
    • without it, every query hung until the client gave up (10 s);
    • with it, queries got SERVFAIL in 0.02 s once the forward health check had marked the upstream down (after about 40 s in that test).
  • trafficDistribution: PreferSameNode is accepted by the API server on k3s v1.35.7 (server-side dry run). The PreferSameTrafficDistribution feature is enabled there.

🤖 Generated with Claude Code

Two values for a node-local-dns whose upstream can go away with a node
(MOB-768):

- config.upstreamServiceTrafficDistribution sets trafficDistribution on
  the upstream service, e.g. PreferSameNode to ask the cluster DNS pod
  on the same node first.
- config.clusterDnsForwardConfig adds forward options to the cluster
  domain and the reverse zones, like upstreamForwardConfig does for the
  default block, e.g. failfast_all_unhealthy_upstreams.

upstreamForwardConfig now defaults to [] (the template ranges over a
list; the map default made helm warn when a list was set).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rubencosta
rubencosta merged commit bce0742 into main Oct 19, 2026
1 check passed
@rubencosta
rubencosta deleted the ruben/upstream-same-node-failfast branch October 19, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant