kube-control: pre-existing ca-certificate secret is not granted to new relations and ca-certificate-secret-id is not published

[mostafaCamel]

Bug Description

Notes:

• I am creating the issue here because the repo https://github.com/charmed-kubernetes/interface-kube-control does not have an Issues section
• The bulk of the issue was created via Copilot

kube-control provider does not publish or grant pre-existing CA secret to new relations

Repository

https://github.com/charmed-kubernetes/interface-kube-control

Related consumer path

https://github.com/canonical/k8s-operator/blob/main/charms/worker/k8s/src/kube_control.py

Summary

When a CA secret labeled ca-certificate already exists before new kube-control relations are added, the provider does not publish ca-certificate-secret-id and does not grant relation access to that CA secret.

Affected code

• https://github.com/charmed-kubernetes/interface-kube-control/blob/main/ops/interface_kube_control/provides.py

Expected behavior

For each kube-control relation, set_ca_certificate should:

• Grant the CA secret to the relation
• Publish ca-certificate-secret-id in provider relation data

This should happen even when the CA secret already exists from earlier runs.

Actual behavior

• Creds secrets are created and granted per requirer, and creds secret-id values are published
• CA secret exists and contains valid certificate content
• CA secret has no relation access entries for newly added requirers
• Relation data does not include ca-certificate-secret-id

Root cause

In set_ca_certificate, grant and publication are gated on secret.id.

If the secret is obtained via label lookup, secret.id may be empty on that object instance.
When that happens, grant and relation-data publication are skipped for existing CA secrets.

Why this is a bug

New kube-control requirers cannot read or reference the CA secret, even though token secrets are correctly granted and advertised.

Evidence from live deployment

• Installed package path on unit:
  /var/lib/juju/agents/unit-k8s-1/charm/venv/lib/python3.10/site-packages/ops_interface_kube_control-0.3.3.dist-info
• Provider code on unit confirms set_ca_certificate checks secret.id before grant/publish
• CA secret was created earlier (June 8), while affected relations were added later (June 10)
• Creds secrets added on June 10 include relation access entries; CA secret does not

Proposed fix

In set_ca_certificate:

• Resolve secret locator using either secret.id or secret.get_info().id
• If needed, re-fetch by resolved id before grant
• Use resolved id to:
  • grant secret to each relation
  • publish ca-certificate-secret-id in relation data

Optional hardening

Apply the same id-resolution pattern in sign_auth_request after _refresh_secret_content, to avoid similar edge cases for pre-existing credential secrets.

To Reproduce

1. Ensure provider has a pre-existing ca-certificate secret (created before target requirer relation is added)
2. Add a new kube-control requirer relation
3. Trigger provider reconcile
4. Observe:
   • creds contains secret-id entries for requirers
   • ca-certificate-secret-id is missing from relation data
   • CA secret has no relation-scoped access entries

Environment

• Juju: 3.6.2
• k8s charm revision: 1845 (channel 1.35/stable )
• ops.interface_kube_control: 0.3.3 (verified on unit filesystem)

Relevant log output

"Missing required certificates"

`juju secrets` shows a secret with a k8s CA but it does not have anya ccess grant to the kube-control relation requirer

Additional context

No response

[mostafaCamel]

Script that you can run on the juju client to patch the existing units of the provider juju app (k8s)

MODEL=k8s
PROVIDING_APP=k8s

UNITS=$(juju status -m "$MODEL" --format json | jq -r --arg app "$PROVIDING_APP" '.applications[$app].units | keys[]')

for UNIT in $UNITS; do
  echo "===> Patching $UNIT"
  UNIT_DASH=${UNIT//\//-}

  juju ssh -m "$MODEL" "$UNIT" "UNIT_DASH=$UNIT_DASH bash -s" <<'EOSSH'
set -euo pipefail

FILE=$(sudo find "/var/lib/juju/agents/unit-$UNIT_DASH/charm/venv/lib" -path "*/site-packages/ops/interface_kube_control/provides.py" 2>/dev/null | head -n1)

if [ -z "${FILE:-}" ]; then
  echo "ERROR: provides.py not found on this unit"
  exit 1
fi

echo "Target: $FILE"
sudo cp "$FILE" "${FILE}.bak.$(date +%s)"

sudo patch -u "$FILE" <<'PATCH'
@@ -1,7 +1,7 @@
 import ipaddress
 import json
 import logging
-from typing import Generator, List, Tuple
+from typing import Generator, List, Optional, Tuple
 
 import ops
 from pydantic import TypeAdapter
@@ -61,6 +61,16 @@
         """List of relations on this endpoint."""
         return self.charm.model.relations[self.endpoint]
 
+    @staticmethod
+    def _secret_locator_id(secret: ops.Secret) -> Optional[str]:
+        """Resolve a secret locator ID even when fetched by label only."""
+        if secret.id:
+            return secret.id
+        try:
+            return secret.get_info().id
+        except ops.SecretNotFoundError:
+            return None
+
     def set_api_endpoints(self, endpoints) -> None:
         """Send the list of API endpoint URLs to which workers should connect."""
         endpoints = json.dumps(endpoints)
@@ -77,10 +87,14 @@
         secret = self._refresh_secret_content(
             "ca-certificate", content, "Kubernetes API endpoint CA certificate"
         )
+        secret_id = self._secret_locator_id(secret)
+        if secret_id and not secret.id:
+            secret = self.charm.model.get_secret(id=secret_id, label="ca-certificate")
+
         for relation in self.relations:
-            if secret.id:
+            if secret_id:
                 secret.grant(relation)
-                relation.data[self.unit]["ca-certificate-secret-id"] = secret.id
+                relation.data[self.unit]["ca-certificate-secret-id"] = secret_id
 
     def set_cluster_name(self, cluster_name) -> None:
         """Send the cluster name to the remote units."""
@@ -241,8 +255,11 @@
             label = f"{request.user}-creds"
             description = f"Credentials for {request.user}"
             secret = self._refresh_secret_content(label, content, description)
-            if secret.id:
-                log.info(f"Granting secret {secret.id} to {request_relation.name}")
+            secret_id = self._secret_locator_id(secret)
+            if secret_id and not secret.id:
+                secret = self.charm.model.get_secret(id=secret_id, label=label)
+            if secret_id:
+                log.info(f"Granting secret {secret_id} to {request_relation.name}")
                 secret.grant(request_relation, unit=request_unit)
                 # Intentionally set the tokens to empty strings in order to
                 # be valid credentials for units still receiving these on schema 0
@@ -251,7 +268,7 @@
                 tokens.client_token = ""
                 tokens.kubelet_token = ""
                 tokens.proxy_token = ""
-                tokens.secret_id = secret.id
+                tokens.secret_id = secret_id
                 creds[request.user] = tokens.model_dump(by_alias=True, exclude_none=True)
         else:
             creds[request.user] = tokens.model_dump(by_alias=True, exclude_none=True)
PATCH

sudo grep -n "_secret_locator_id\|ca-certificate-secret-id\|Granting secret" "$FILE"
EOSSH
done

echo
echo "Patched units:"
echo "$UNITS"

• then remove the kube-control relation
• then wait until the relation is removed (check with juju status --relations)
• then readd the kube-control relation

[mostafaCamel]

No changes needed in requirers as the change is only in provides.py

No changes needed in the k8s-operator charm as the dependency is not pinned:

k8s-operator/charms/worker/k8s/pyproject.toml

Line 30 in c36a555

"ops-interface-kube-control",

However, we will need a k8s and k8s-worker charm revision that are buit after the new interface has been published

[mostafaCamel]

Let's use this issue for the k8s-operator changes as well. The uv.lock file in the k8s-operator repo (for each of the k8s and k8s-worker charm) should be updated with the new version of ops-interface-kube-control once the new version of kube-control is released with the changes

[mohamedgalal99]

I faced the same problem with new deployed k8s

$ juju status --relations
Model  Controller  Cloud/Region  Version  SLA          Timestamp
k8s    manual      manual        3.6.25   unsupported  14:22:07Z

App                         Version  Status   Scale  Charm                       Channel         Rev  Exposed  Message
cinder-csi                           waiting      1  cinder-csi                  latest/stable    47  no       Waiting for kube-control relation
easyrsa                     v3.0.9   active       1  easyrsa                     latest/stable    79  no       Certificate Authority connected.
k8s                         1.35.0   error        1  k8s                         1.35/stable    1845  no       hook failed: "kube-control-relation-departed"
k8s-worker                  1.35.0   active       2  k8s-worker                  1.35/stable    1842  no       Ready
openstack-cloud-controller           waiting      1  openstack-cloud-controller  latest/stable    48  no       Waiting for kube-control relation
openstack-integrator        yoga     active       1  openstack-integrator        latest/stable   123  no       Ready

Unit                             Workload  Agent  Machine  Public address  Ports     Message
easyrsa/0*                       active    idle   0        10.10.10.6                Certificate Authority connected.
k8s-worker/0*                    active    idle   1        10.10.10.129              Ready
k8s-worker/1                     active    idle   2        10.10.10.88               Ready
k8s/0*                           error     idle   0        10.10.10.6      6443/tcp  hook failed: "kube-control-relation-departed"
openstack-integrator/1*          active    idle   0        10.10.10.6                Ready
  cinder-csi/0*                  waiting   idle            10.10.10.6                Waiting for kube-control relation
  openstack-cloud-controller/0*  waiting   idle            10.10.10.6                Waiting for kube-control relation

Machine  State    Address       Inst id                  Base          AZ  Message
0        started  10.10.10.6    manual:k8s-controler-01  ubuntu@22.04      Manually provisioned machine
1        started  10.10.10.129  manual:k8s-worker-01     ubuntu@22.04      Manually provisioned machine
2        started  10.10.10.88   manual:k8s-worker-02     ubuntu@22.04      Manually provisioned machine

Integration provider                                Requirer                                 Interface                Type         Message
easyrsa:client                                      cinder-csi:certificates                  tls-certificates         regular
easyrsa:client                                      openstack-cloud-controller:certificates  tls-certificates         regular
k8s-worker:upgrade                                  k8s-worker:upgrade                       upgrade                  peer
k8s:cluster                                         k8s:cluster                              k8s-cluster              peer
k8s:containerd                                      k8s-worker:containerd                    containerd               regular
k8s:cos-tokens                                      k8s:cos-tokens                           cos-k8s-tokens           peer
k8s:cos-worker-tokens                               k8s-worker:cos-tokens                    cos-k8s-tokens           regular
k8s:k8s-cluster                                     k8s-worker:cluster                       k8s-cluster              regular
k8s:kube-control                                    cinder-csi:kube-control                  kube-control             regular
k8s:kube-control                                    openstack-cloud-controller:kube-control  kube-control             regular
k8s:upgrade                                         k8s:upgrade                              upgrade                  peer
openstack-cloud-controller:external-cloud-provider  k8s:external-cloud-provider              external_cloud_provider  regular
openstack-integrator:clients                        cinder-csi:openstack                     openstack-integration    subordinate
openstack-integrator:clients                        openstack-cloud-controller:openstack     openstack-integration    subordinate

problem was kube-api self-signed cert, not trusted by cinder-csi and openstack-cloud-controller

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/ops/manifests/manifest.py", line 112, in client
    load_in_cluster_generic_resources(client)
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/lightkube/generic_resource.py", line 246, in load_in_cluster_generic_resources
    for crd in crds:
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/lightkube/core/generic_client.py", line 109, in __iter__
    for rv, chunk in self._inner_iter:
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/lightkube/core/generic_client.py", line 390, in list_chunks
    resp = self.send(req)
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/lightkube/core/generic_client.py", line 344, in send
    return self._client.send(req, stream=stream)
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/var/lib/juju/agents/unit-cinder-csi-0/charm/venv/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)
2026-07-19 20:41:15 WARNING unit.cinder-csi/0.juju-log server.go:405 Encountered retryable installation error: ('Failed to load in cluster CRDs', ConnectError('[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)'))
2026-07-19 20:41:15 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
2026-07-19 20:43:21 INFO unit.cinder-csi/0.juju-log server.go:405 certificates:15: Auth Req for cinder-csi/0 with group system:masters on schema [0, 1]
2026-07-19 20:43:21 ERROR unit.cinder-csi/0.juju-log server.go:405 certificates:15: certificates relation data not yet available.

---

SOLUTION

juju ssh -m k8s k8s/0 -- bash -s <<'EOF'
set -euo pipefail

FILE=$(
  sudo find /var/lib/juju/agents/unit-k8s-0/charm/venv/lib \
    -path '*/site-packages/ops/interface_kube_control/provides.py' \
    -type f |
  head -n1
)

echo "Editing: $FILE"

sudo cp --preserve=all \
  "$FILE" \
  "${FILE}.before-departed-fix.$(date +%Y%m%d%H%M%S)"

sudo python3 - "$FILE" <<'PY'
from pathlib import Path
import sys

path = Path(sys.argv[1])
text = path.read_text()

old = '''        for relation in self.relations:
            if secret_id:
                secret.grant(relation)
                relation.data[self.unit]["ca-certificate-secret-id"] = secret_id
'''

new = '''        for relation in self.relations:
            if not secret_id:
                continue

            try:
                secret.grant(relation)
                relation.data[self.unit]["ca-certificate-secret-id"] = secret_id
            except (ops.ModelError, ops.SecretNotFoundError) as exc:
                log.warning(
                    "Unable to publish CA secret to relation %s: %s",
                    relation.id,
                    exc,
                )
'''

if new in text:
    print("Departed-relation protection is already installed")
    sys.exit(0)

count = text.count(old)

if count != 1:
    print(
        "ERROR: expected patched CA relation block was not found exactly once; "
        f"found {count}"
    )
    sys.exit(1)

path.write_text(text.replace(old, new, 1))
print("Departed-relation protection installed")
PY

sudo python3 -m py_compile "$FILE"

sudo sed -n '75,115p' "$FILE"
EOF

juju remove-relation --force \
  k8s:kube-control \
  cinder-csi:kube-control

juju remove-relation --force  \
  k8s:kube-control \
  openstack-cloud-controller:kube-control

wait till relation rremoved

juju integrate  \
  k8s:kube-control \
  cinder-csi:kube-control

juju integrate \
  k8s:kube-control \
  openstack-cloud-controller:kube-control


juju resolved -m k8s k8s/0 --no-retry

[mostafaCamel]

Clarified with Galal: for his usecase, the existing codebase did not work then my patch alone did not work. His patch on top of my patch worked