Only 54 GB Free: Reclaiming 74 GB from containerd, Podman, and Build Caches

A 200 GB disk, and df -h said only 54 GB was left — 72% used.

Most days that is survivable. But this machine also runs a small Kubernetes cluster, and the moment an image pull or a build needs a few GB of scratch space, “three-quarters full” stops being a comfortable place to be.

This is the write-up of two things: where the space actually went, and — the more important half — how I convinced myself that deleting it did not break anything that was running.

1. Start with df, then du

The first step is always to look at the numbers, not to guess:

1
2
3

df -h /
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sda1       197G  136G   54G  72% /

Then walk the top-level directories to find the heavy hitters:

1
2
3
4
5

du -xsh /* | sort -rh | head
# 92G  /var
# 41G  /root
# 3.2G /usr
# 2.0G /tmp

/var and /root are the whole story. And that is where the first lesson of the whole exercise lives:

The biggest directory is not the directory that is most worth reclaiming.

2. Trap #1: /tmp is tmpfs, not disk

/tmp looked like it held 2 GB and was tempting to clear. But first check which filesystem it lives on:

1
2
3

df -h /tmp
# Filesystem      Size  Used Avail Use% Mounted on
# tmpfs           7.9G  2.0G  5.9G  25% /tmp

tmpfs is memory, not disk. Emptying /tmp gives RAM back and does not hand a single byte to that 200 GB disk.

Before you clean anything, confirm that the thing you are about to clean is actually on the filesystem you are trying to free.

3. The bulk in /var: containerd blobs and snapshots

Drill further into /var:

1
2
3
4

du -xsh /var/* | sort -rh | head
# 67G  /var/snap
# 24G  /var/lib
# 785M /var/cache

/var/snap belongs to the cluster, /var/lib/containers belongs to Podman. Split the cluster half open:

1
2
3

du -xsh /var/snap/microk8s/common/*
# 55G  .../common/var             <- containerd's runtime directory
# 12G  .../common/default-storage <- persistent volume (PVC) data

One level deeper into containerd:

1
2
3
4

du -xsh .../var/lib/containerd/io.containerd.content.v1.content/blobs/sha256
# 15G
du -xsh .../var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
# 40G

Two numbers, and they tell a story:

• content/blobs (15 GB) is the compressed image-layer store — a shared, content-addressed pool deduplicated by sha256.
• overlayfs snapshots (40 GB) is those layers unpacked and stacked into writable snapshots. It is far larger than the blobs because a layer’s unpacked size usually exceeds its compressed one, and every layer change adds another diff.

That 12 GB default-storage directory is persistent volume data — databases, user uploads, model weights. That is live data, a different category from cache. It does not get touched.

By now the reclaimable bulk is obvious. One aside: logs were not even on the list — the journal was tens of megabytes. The reflex “full disk means big logs” was simply wrong this time.

4. The safe reclaim: ctr images prune needs --all

containerd has no household-name equivalent of docker system prune, but its debug client ctr carries an images prune. And that is the first trap:

1
2
3

ctr images prune --help
# --all  Remove all unused images, not just dangling ones
#        (if all is not specified no images will be pruned)

Without --all it does nothing — and it does not error. You will believe you reclaimed space while having quietly run a no-op. So:

1
2

# microk8s ships its own ctr wrapper that sets the k8s.io namespace for you
microk8s ctr images prune --all

The output is a long list of deleted image: sha256:.... The image-record count dropped from 237 to 22.

Two details worth understanding:

1. It is safe because it deletes by reference. If an image is still referenced by any container, or by a kubelet lease, it will not be removed. Which means every image belonging to a running pod is protected by construction, and what disappears is old versions and throwaway build artifacts.
2. The space comes back asynchronously. prune removes image references; the blob and snapshot garbage collection runs afterward. So df does not move immediately — give it a moment.

5. Podman: prune images and volumes separately

On the Podman side, read the ledger before acting:

1
2
3
4
5

podman system df
# TYPE           TOTAL  ACTIVE  SIZE      RECLAIMABLE
# Images         143    2       21.01GB   20.33GB (97%)
# Containers     3      1       135.8kB   123.7kB (91%)
# Local Volumes  4      2       2.633GB   114.9MB (4%)

97% reclaimable is basically a statement that the whole thing is historical debris.

1
2

podman image prune -a -f    # remove every image with no container referencing it
podman volume prune -f      # remove only unmounted volumes

The semantics of image prune -a are worth stating exactly: it keeps the image backing the running container, and it keeps the images referenced by the two stopped database containers; it removes everything else — a couple of gigabytes of an old runtime, several <none> intermediate layers, and a pile of forgot-they-existed test tags. Three images left, clean and neat.

One precondition: -a also removes base images (compilers, language runtimes). They are all re-pullable, but only if the registry happens to be reachable right now. It is worth ten seconds to check — otherwise you may have deleted something you cannot easily get back for a while. I ran that check.

6. Build caches: costly to rebuild, but always rebuildable

Most of that 41 GB under /root was the kind of thing that grows back on its own:

These are safe precisely because they are reproducible by nature. But the cost has to be stated plainly: after this, the next build is a full cold start. That is a trade-off to weigh — if you are about to do heavy code edits, the 8 GB target directory may be worth keeping.

7. The small stuff worth sweeping anyway

Individually minor, but together it adds up to an image:

1
2
3

apt-get clean                                              # 772 MB of package cache
snap remove <name> --revision=<old-rev>                    # ~500 MB of disabled old revisions
rm -rf /var/tmp/<scattered-scratch-dirs>                   # 360 MB from assorted verification runs

Note that snap keeps two versions by default so you can roll back; what gets removed is the revision that is already in a disabled state, not the one currently in use.

8. Verification: do not just look at df

Everything above was easy. The only step that needed real care was the containerd one, because it ran against a live cluster. So I set myself three checks, and the order matters:

Step 1: before deleting, record the images in use.

1
2
3

kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' \
  | sort -u
# 20 images

Step 2: after deleting, check all 20 are still there.

1

ctr images ls | awk '{print $1}' | sort -u

There is a trap here: kubectl returns short names, e.g. team/app:1.2.3, while containerd stores fully-qualified references, i.e. docker.io/team/app:1.2.3. The more common variant adds a library/ prefix: write myapp:1.2.3 in a pod, and containerd keeps docker.io/library/myapp:1.2.3. Grepping the short name directly yields a pile of false MISSING lines and a small fright. The lesson: normalize before comparing (add the docker.io/ and library/ prefixes). After normalization, all 20 were present.

Step 3: confirm the cluster itself is unaffected.

1
2
3

kubectl get pods -A        # all Running, no increase in restart counts
curl -sS -o /dev/null -w '%{http_code}' https://<endpoint>/
# 200

Only when all three pass is it true that you deleted the right things.

9. Result

Roughly 74 GB reclaimed. By source, approximately:

(The individual numbers are estimates and overlap slightly; the df before/after delta of 74 GB is the number to trust.)

10. Lessons

• The biggest directory is not the most reclaimable one. The 12 GB of persistent volumes are right there, and they are live data, not cache. Classify first, delete second.
• Check the filesystem type before cleaning. /tmp is tmpfs — memory, not disk.
• A prune default can be a silent no-op. ctr images prune does nothing without --all, and it does not error. “The command succeeded” is not “the space came back.”
• To delete what is unused, you must be able to name what is in use. Capture the in-use list before, verify it item by item after; containerd’s prune is safe precisely because it deletes by reference, and a running pod’s image cannot be touched.
• Build caches have a price. They always rebuild — but “cold build” is the invoice.
• Be honest about the boundary. The image registry was unreachable for the entire session, so the old image versions I removed cannot be re-pulled in the near term — a rollback to an old tag would have to wait for the registry to come back. I could not verify that path, so I wrote it down explicitly instead of waving it away.