How One Expired Certificate Quietly Stopped an Entire Kubernetes Control Plane

Every Kubernetes engineer eventually runs into certificates.

Most of the time they're invisible. They sit quietly in /etc/kubernetes/pki, doing their job while the cluster keeps serving workloads.

Until one day they don't.

A while ago I hit a production incident that perfectly demonstrated how much the Kubernetes control plane depends on secure communication between its own components and how fixing the obvious problem isn't always the same as fixing the system.

The outage wasn't caused by etcd. It wasn't networking. It wasn't even a bug in Kubernetes itself.

It was a single expired API server certificate.

The setup

This wasn't one of our customer-facing application clusters. It was a kubeadm-managed cluster three control-plane nodes behind a load balancer running operational workloads:

• scheduled etcd backups
• automation jobs
• maintenance tasks
• some legacy applications

Because nothing customer-facing lived on it, the cluster had been running quietly for more than a year without a Kubernetes upgrade.

That detail turned out to be the whole story. kubeadm issues certificates with a one-year lifetime, and the normal thing that rotates them is a cluster upgrade. No upgrades means no rotation — a countdown timer nobody was watching.

The first symptom

The first thing we noticed was that kubectl stopped working. Every request returned:

Unable to connect to the server:
x509: certificate has expired or is not yet valid

Like many engineers, my first thought wasn't Kubernetes. It was authentication maybe an expired OIDC token.

So I refreshed the token.

Nothing changed.

Verify before you assume

Instead of continuing down the authentication rabbit hole, I checked the API server's serving certificate directly:

openssl x509 \
  -in /etc/kubernetes/pki/apiserver.crt \
  -noout \
  -dates

The problem was immediately obvious: the certificate had expired months earlier.

You can get the same answer (plus the state of every other certificate in the cluster PKI) with:

kubeadm certs check-expiration

Since this was a kubeadm-managed cluster, renewal was straightforward:

kubeadm certs renew all

Within minutes, the API server was accepting connections again. kubectl worked.

Problem solved?

Not quite.

Something still felt wrong

Access was restored, but the system didn't feel healthy. Checking the operational workloads, several things stood out:

• CronJobs hadn't executed for weeks.
• Our scheduled etcd backups had silently stopped.
• New Pods were being created but stayed Pending forever.
• Deployments weren't progressing.

The cluster looked alive. But it wasn't doing anything.

That observation completely changed the direction of the investigation.

Kubernetes wasn't dead it had stopped reconciling

This is one of the most important mental models in Kubernetes.

Kubernetes doesn't "execute" your YAML. Almost everything happens through reconciliation loops: controllers continuously watch the API server, and whenever desired state differs from actual state, they act to close the gap.

User applies manifest
        ↓
   API server (desired state stored in etcd)
        ↓
Controllers watch for changes
        ↓
Desired state ≠ actual state?
        ↓
      Reconcile
        ↓
  Cluster converges

If reconciliation stops, nothing happens. Not "things break loudly" literally nothing happens. Existing workloads keep running on the kubelets, but no new decisions get made. The cluster freezes in place.

And the symptom pattern we were seeing Pending Pods, dead CronJobs, stalled Deployments is exactly what "nothing happens" looks like.

How the control plane actually communicates

A common misconception is that control plane components all talk directly to each other, or directly to etcd.

They don't. The API server is the hub. Everything flows through it.

                 kube-apiserver
               /       |        \
              /        |         \
      Scheduler   Controller    kubectl,
                   Manager      kubelets
                       
                 (only the API
                  server talks
                    to etcd)

The controller manager never talks directly to etcd. Neither does the scheduler. They authenticate to the API server over TLS using certificates.

You can probably see where this is going.

So why didn't renewing the certificates fix everything?

This was the interesting part.

kubeadm certs renew all updated the certificate files on disk in /etc/kubernetes/pki/ and the kubeconfigs the control plane components use.

But the controller manager and scheduler were already running. Like any long-running Linux process, they had loaded their credentials into memory at startup months ago, when the old certificates were still valid.

Updating files on disk doesn't change what a running process has loaded in memory.

So both components kept presenting their old, expired certificates to the freshly-fixed API server which correctly rejected them.

This is where the diagnosis came together. Since the API server was reachable again, I could pull the control plane component logs directly:

kubectl -n kube-system logs kube-controller-manager-<control-plane-node> --tail=50
kubectl -n kube-system logs kube-scheduler-<control-plane-node> --tail=50

Both were full of the same steady stream of authentication failures:

E ... leaderelection.go: error retrieving resource lock
  kube-system/kube-controller-manager: Unauthorized
E ... x509: certificate has expired or is not yet valid

That's the smoking gun: the API server was healthy, but its own control plane components couldn't authenticate to it.

(Worth knowing for the harsher version of this incident: if the API server itself is still down, kubectl logs won't work either. In that case, go to the control-plane node directly and use crictl logs <container-id> or journalctl -u kubelet, since the control plane components are just containers managed by the kubelet.)

No communication meant no watching. No watching meant no reconciliation.

Because the controller manager couldn't reach the API server:

• CronJobs never created Jobs (that's the CronJob controller's job)
• Deployments never reconciled ReplicaSets
• Garbage collection stopped
• Our backup automation- which is just a CronJob - silently died

Because the scheduler couldn't reach the API server:

• No scheduling decisions were made
• Every new Pod sat in Pending, forever

The cluster looked operational. But the control plane had effectively stopped thinking.

The fix

Anticlimactically simple: restart the control plane components so they reload the renewed certificates from disk.

On a kubeadm cluster, the scheduler and controller manager run as static pods, so you can't just kubectl rollout restart them. Two options:

# Option 1: delete the mirror pods (kubelet recreates them from the static manifests)
kubectl -n kube-system delete pod kube-controller-manager-<node> kube-scheduler-<node>
 
# Option 2: move the static pod manifests out and back
mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/ && sleep 20 && \
mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/

We did this across all three control-plane nodes. Fresh processes, fresh TLS connections, valid certificates.

Within minutes:

• CronJobs began executing again
• etcd backups resumed
• Pending Pods were scheduled
• Deployments reconciled

No rebuilding. No restoring from backup. No reinstalling Kubernetes.

Just understanding how the control plane works.

The biggest lesson

The technical root cause was an expired certificate.

The operational root cause was the absence of certificate lifecycle management. And that's the fix that actually matters, because "remember to renew certificates" is not a control — human memory fails, and this cluster proved it.

The goal is to make certificate expiry impossible to miss:

• Upgrade kubeadm clusters on a regular cadence. kubeadm upgrade renews certificates as part of the process staying current on upgrades makes expiry a non-event.
• Run a scheduled expiry check. A simple cron job running kubeadm certs check-expiration on the control-plane nodes, alerting when anything is inside 30 days, costs nothing and catches everything.
• Export expiry as metrics. Surface certificate expiry timestamps into Prometheus and let Alertmanager page the team weeks before anything expires — the same way you'd alert on disk space, not after it runs out.
• Audit your other clusters. We treated this cluster as the symptom, not the disease — the same latent condition existed anywhere upgrades had drifted.

One more painful detail worth internalizing: the outage revealed our etcd backup job had been silently dead for weeks before anyone noticed. Monitoring the certificates also means you'll never again discover a dead backup pipeline by accident.

Final thoughts

What I took away from this incident wasn't how to renew Kubernetes certificates. That's one command.

It was something broader: in distributed systems, restoring access is not the same as restoring the system.

When kubectl started working again, it would have been easy to declare the incident resolved. Instead, the real fix came from asking behavioral questions:

• Are controllers reconciling?
• Are CronJobs actually creating Jobs?
• Are Pods getting scheduled?

Sometimes the most valuable troubleshooting step isn't fixing the first symptom you see it's asking whether the system is actually behaving the way it's designed to.