fix: detect silent k8s-dqlite datastore failures in inspect - #5532
Open
bugra-gokcek wants to merge 1 commit into
Open
fix: detect silent k8s-dqlite datastore failures in inspect#5532bugra-gokcek wants to merge 1 commit into
bugra-gokcek wants to merge 1 commit into
Conversation
`microk8s inspect` previously gave no hint when the k8s-dqlite datastore could not start: the daemon wrapper exits before the binary can log a fatal error, so neither `microk8s status` nor the inspection report pointed at the cause. Two common failure modes were invisible: - a missing TLS keypair in the backend directory (for example after the backend dir was wiped), which aborts dqlite on start with "failed to load keypair from cluster.crt and cluster.key" - a stale raft node address still pointing at the old hostname after a rename, which loops with "no available dqlite leader server found" Add a `check_dqlite` step to the inspect report (control-plane nodes only) that FAILs on a missing cert/key and WARNs when the stored node address no longer matches the current hostname, so these failures are no longer silent. Fixes canonical#5524
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #5524.
microk8s inspectgave no actionable hint when thek8s-dqlite datastore could not start, because the daemon wrapper
(
run-k8s-dqlite-with-args) exits before the binary can log its fatalerror. The reporter hit two variants of this silent failure:
(
var/kubernetes/backend/) also removescluster.crt/cluster.key,so dqlite aborts on start with
failed to load keypair from cluster.crt and cluster.keywhilemicrok8s status/microk8s inspectshow nothing wrong.address still pointed at the old hostname, so dqlite looped with
no available dqlite leader server found. The existing uppercase-hostnamewarning disappears once the host is renamed to lowercase, leaving nothing
pointing at the stale address.
As the reporter notes, the real gap is the silent
inspect, so this PR makesboth conditions visible.
Changes
scripts/inspect.sh: newcheck_dqlitestep, run on control-plane (non-worker)HA nodes alongside the existing dqlite inspection:
cluster.crtorcluster.keyis missing from the backenddirectory, with a short recovery hint (sets a non-zero return code, like the
other
FAILchecks).Addressininfo.yamlis a hostname that no longermatches the current hostname.
The address check intentionally ignores IPv4/IPv6 literals, so the default
single-node (
127.0.0.1) deployment and IP-based clusters never trigger afalse positive —
info.yamlalways stores this node's own address, so amismatch genuinely indicates a stale name.
Testing
scripts/inspect.shhas no unit-test harness (it is a pure bash diagnosticscript, consistent with how previous inspect checks were added). I verified the
new logic in isolation across these cases:
127.0.0.1:19001)10.0.0.5:19001)[fd00::5]:19001)oldnamevsnewname)MyHostvsmyhost)info.yamlbash -n scripts/inspect.shandcodespell(the lint config used in CI) bothpass.