Pod Restarted 15 Times: How a Podman Bridge Triggered a microk8s Certificate Rotation

A small app of mine had been running quietly for months. Then one day, it showed: 15 restarts.

The only log it produced was two lines:

1
2
3
4

Error: timed out

Caused by:
    timed out

The strange part: every other pod in the same cluster had restarted exactly 6 times. Same node, same moment — why did only this one crash 9 extra times?

The answer turned out to be two problems stacked on top of each other. One was cluster-wide: all pods were being killed together. The other was app-level: only my app turned a brief dependency hiccup into a crash loop.

1. First, don’t stare at your own pod

When a pod restarts N times, the first step is not reading its logs. It is looking at its neighbours:

1

kubectl -n my-app get pods -o wide

The result: one service had restarted 4 times, and the rest — the database, the cache, the small side services — had all restarted exactly 6 times. Only my app was at 15.

That single command split the problem in two:

• Every pod restarted 6 times → there were 6 node-level events that killed all containers. That is not my app’s fault.
• Only my app reached 15 → the extra 9 restarts were self-inflicted.

Then I checked how the container had exited:

1
2
3

kubectl -n my-app get pod app-6d9f7c8b5-x2k4p \
  -o jsonpath='{.status.containerStatuses[0].lastState}'
# {"terminated":{"exitCode":1,"reason":"Error","startedAt":"...","finishedAt":"..."}}

Exit code 1, after about 36 seconds, and the log never showed a successful startup. It was dying during boot.

2. Who is killing the containers?

The node runs containerd (bundled with microk8s). I searched the system journal:

1

journalctl --since "6 hours ago" | grep -E 'containerd|kubelite' | grep -i kill

And found this:

1
2
3
4

systemd[1]: snap.microk8s.daemon-containerd.service:
  Sent signal SIGKILL to main process 2230966 (containerd) on client request.
systemd[1]: snap.microk8s.daemon-containerd.service:
  Main process exited, code=killed, status=9/KILL

The process IDs of containerd and kubelite kept changing, so those daemons were being restarted again and again. And on client request told me the killer was not the kernel — it was something on the host, asking systemd to do it. That something was microk8s’s own apiserver-kicker.

The kicker is a small shell script (/snap/microk8s/<rev>/apiservice-kicker), and its core logic is easy to read:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

if snapctl services microk8s.daemon-kubelite | grep active &> /dev/null
then
  certs_modified="$(manage_certs)"
  if [[ "$certs_modified" -eq "1" ]] && ...
  then
    echo "cert change detected. Restarting the cluster-agent"
    snapctl restart microk8s.daemon-cluster-agent

    echo "cert change detected. Reconfiguring the kube-apiserver"
    rm -rf .srl
    snapctl stop microk8s.daemon-kubelite
    remove_all_containers          # ← kill every container
    kill_all_container_shims
    snapctl restart microk8s.daemon-containerd
    snapctl start microk8s.daemon-kubelite
    start_all_containers
    ...

In other words: whenever the apiserver certificate needs to be re-issued, microk8s stops kubelite, kills every container on the node, restarts containerd, and brings everything back up. That is a very blunt instrument, and the price is a cluster-wide restart of all containers.

So the real question became: why did the certificate keep “changing”?

3. Why did the certificate keep changing?

The kicker calls manage_certs, which ends up in produce_certs. That function does something simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

render_csr_conf
if ! cmp -s certs/csr.conf.rendered certs/csr.conf; then
    force=true
    cp certs/csr.conf.rendered certs/csr.conf
fi

if $force; then
    gen_server_cert        # re-issue the apiserver certificate
    gen_proxy_client_cert
    echo "1"               # ← triggers the "kill everything" path
fi

And render_csr_conf does this:

1
2
3

local IP_ADDRESSES="$(get_ips)"     # get_ips = hostname -I, minus CNI interfaces
...
"$SNAP/bin/sed" -i "s/#MOREIPS/${ips}/g" certs/csr.conf.rendered

It renders the host’s hostname -I address list into the certificate’s SAN list, then compares the result with the previous version. So any change to the host’s IP list — an interface appearing, an interface disappearing, an address being added or removed — makes microk8s believe the node’s IPs have changed and the certificate must be re-issued. And re-issuing means killing every container.

So who kept changing the host’s IP list? I lined up the logs by time, and the answer was remarkably tidy:

Six events, six hits. Every time a podman container started or stopped, the whole Kubernetes cluster was restarted.

The reason is in how podman creates networks:

1
2

$ podman network inspect podman
[{ "subnet": "10.88.0.0/16", "gateway": "10.88.0.1" }]  podman0

The bridge of the default podman network (podman0, gateway 10.88.0.1) is created on demand: while a container is attached to that network, the bridge exists; when the last container stops, the bridge and its IP disappear. And a bridge is a host-side interface, so its IP shows up in hostname -I:

1
2
3

$ hostname -I
203.0.113.10 192.0.2.10 10.88.0.1 2001:db8:1::1 2001:db8:2::1
#              ↑ public IP  ↑ podman bridge (sometimes there, sometimes not)

Now the chain closes:

1
2
3
4
5
6
7

podman container starts / stops
  → default network bridge is created / destroyed
  → hostname -I changes
  → the rendered csr.conf changes
  → microk8s re-issues the apiserver certificate
  → remove_all_containers / kill_all_container_shims
  → every container in the cluster is SIGKILLed and restarts

One small piece of supporting evidence: the certificate’s SAN list did contain the gateway of a permanent podman bridge (from another network that always has a container attached), but not the temporary one — because at the moment the certificate was issued, the temporary bridge did not exist.

4. Why did only my app crash 9 extra times?

That explains the 6 cluster-wide events. Now for 15 − 6 = 9.

My app starts like this:

1
2

let redis_pool = cache::init_redis().await?;   // ← if it can't connect, exit
println!("Redis connected");

And init_redis connects eagerly: creating the connection manager actually opens the connection. If that fails, it returns Err, the ? propagates it, main returns Err, and the process exits with code 1. That is where the two-line log comes from:

1
2
3
4

Error: timed out

Caused by:
    timed out

Interestingly, the same message appears twice. In the redis-rs source, each connection attempt is wrapped in a 1-second timeout, and a timeout converts tokio’s Elapsed into an io::Error:

1
2
3
4
5

impl From<Elapsed> for RedisError {
    fn from(_: Elapsed) -> Self {
        io::Error::from(io::ErrorKind::TimedOut).into()
    }
}

The Display of io::ErrorKind::TimedOut is literally timed out. RedisError wraps it, prints it as its own Display, and also returns it from source(). So when anyhow prints the error chain, the same string appears on the first line and again under “Caused by” — it looks like an echo, but it is one error wearing two hats.

So this is what each cluster-wide event looked like from my app’s point of view:

1. It was killed by remove_all_containers (restart #1);
2. When it came back, Redis, the Service, and the network were not ready yet, so the eager connection timed out and the process exited (restart);
3. kubelet backed off for 10 seconds, then 20, and tried again — if the dependency was still not ready, it exited once more (restart).

The container creation times show the pattern perfectly — they always come in bursts:

1
2
3
4
5
6

03:07:16, 03:07:54, 03:08:29   ← 1 kill + 2 self-crashes
03:33:58, 03:34:36, 03:35:13   ← 1 kill + 2 self-crashes
04:34:24, 04:35:03             ← 1 kill + 1 self-crash
04:36:49, 04:37:27, 04:38:01   ← 1 kill + 2 self-crashes
07:19:31, 07:20:09             ← 1 kill + 1 self-crash
07:21:03, 07:21:40             ← 1 kill + 1 self-crash (finally stable)

6 kills + 9 self-crashes = 15 restarts. The truth was that simple.

5. The fix

5.1 Node level: stop re-issuing certificates on IP changes

microk8s provides a lock file for exactly this. The script checks for ${SNAP_DATA}/var/lock/no-cert-reissue:

1

touch /var/snap/microk8s/current/var/lock/no-cert-reissue

With the lock in place, the kicker takes the ensure_server_ca branch instead — it only re-issues when the certificate is genuinely broken (for example, a wrong issuer), not when the IP list changes. On a single-node machine with fairly stable IPs, this is the easiest fix. To revert, just rm the file.

5.2 App level: don’t make a startup dependency a hard dependency

A cache is best-effort by nature: on a miss you fetch the data again, and a failed write is not a disaster. So the app should not die for it at startup. Two ways to fix it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// Option A: lazy connection — startup never blocks, reconnects in the background
let client = redis::Client::open(url)?;
let pool = client.new_lazy_connection_manager()?;   // no I/O until the first command

// Option B: retry with backoff during startup
for attempt in 0..N {
    match init_redis().await {
        Ok(p) => return Ok(p),
        Err(e) => { warn!(...); sleep(backoff).await }
    }
}

After the change, the next node-level event would cost my app exactly one kill and one restart — the same as every other pod.

5.3 Alternative: keep the bridge alive

If you would rather not touch the certificate lock, you can keep one long-running container attached to the default podman network, so the bridge never disappears and hostname -I stays stable. But that only removes one trigger; a VPN or a virtual interface could do the same thing tomorrow. That is why I prefer 5.1 + 5.2.

6. What I learned

• When a pod restarts N times, look at its neighbours first. All pods restarting means a platform problem; only yours restarting means an app problem; both means two problems — fix them separately.
• Anything on the host that appears and disappears can change hostname -I: container runtime bridges, VPNs, virtual interfaces. If some component watches the host’s IP list, watch out.
• microk8s certificate rotation is expensive. When it detects an IP change, it kills every container and rebuilds, instead of hot-reloading. Knowing this saves a lot of confusion when the whole cluster shudders at once.
• A hard dependency at startup amplifies hiccups. Thirty seconds of dependency downtime becomes a string of CrashLoopBackOff restarts if the process chooses to exit; it stays a single cache miss if the process chooses to degrade.

And finally, back to that two-line log. It was not lying — it just said too little. The neighbours’ restart counts, the system journal, and a few seconds of timing correlation told the rest of the story.