Layer 5 — Encrypted Storage

Everything your cluster writes — objects, database volumes, temp files — lives inside a locked vault on the USB drive. No key, no data.

Why encrypt the USB drive?

Your USB SSD is the most physically vulnerable component in this build. It is small, pluggable, and easy to pocket. Without encryption, a thief who pulls it out gets everything: every database row, every secret that was ever written to disk, every container image layer. With encryption, they get a block of random-looking bytes that cannot be read without the key — and the key never leaves the host.

Think of the USB drive as a safe-deposit box. The data inside is your valuables. LUKS2 is the lock. The passphrase — or the TPM chip on your motherboard — is the key. Without the key, the box is just a sealed lump of metal.

This layer also covers a second problem: Kubernetes writes a lot of temporary data (container image layers, scratch space for running pods) to disk without asking. By pointing k3s at the encrypted USB mount, all of that lands inside the lock too — automatically.

1 — LUKS2: the right encryption for this job

LUKS2 (“Linux Unified Key Setup, version 2”) is the standard full-disk encryption system on Linux. Version 2 added stronger key-derivation and larger metadata. The combination we use:

Argon2id is deliberately slow and RAM-hungry. When you type a passphrase, your machine needs about 2 seconds and 1 GB of RAM to check it. For you, that is fine. For an attacker trying billions of guesses per second, it makes their GPU essentially useless.

Always use the by-id path for the device — /dev/disk/by-id/usb-… — never /dev/sdb or similar. The short names change every time you reboot; the by-id path is stable because it is derived from the drive’s serial number.

1.1 Format the drive

Run scripts/host/15-luks-usb.sh (described in the Scripts section). It will:

1. List all USB block devices and ask you to confirm the one to format.
2. Refuse to proceed if you point it at the system disk.
3. Wipe the existing partition table.
4. Run cryptsetup luksFormat with the parameters above.
5. Add a recovery key slot (store that passphrase offline — printed, in a safe).
6. Optionally enrol the TPM2 chip so the drive unlocks automatically on a trusted boot.
7. Open the container, format ext4, and mount it.
8. Print the libvirt XML snippet if you are running k3s inside a VM.

sudo bash scripts/host/15-luks-usb.sh

1.2 Persistent mount via crypttab + fstab

After the script runs, two lines persist the setup across reboots:

# name       device (by-id — never /dev/sdX)               keyfile                    options
usb-data  /dev/disk/by-id/usb-YOUR_DRIVE_ID-0:0  /etc/luks/usb-data.key  luks,discard,nofail

/dev/mapper/usb-data  /mnt/usb-data  ext4  defaults,noatime,nofail  0  2

nofail means the system boots even if the USB is absent — important because a missing device should not hard-lock the machine. discard passes TRIM commands to the SSD for longevity.

1.3 TPM2 auto-unlock

If the host has a TPM 2.0 chip (most Intel/AMD machines from 2017+), enrol it with:

systemd-cryptenroll \
  --tpm2-device=auto \
  --tpm2-pcrs=0+2+4+7 \
  --tpm2-with-pin=yes \
  /dev/disk/by-id/usb-YOUR_DRIVE_ID-0:0

This uses the same PCR set as the OS disk (PCRs 0+2+4+7): UEFI firmware code (0), option ROMs (2), bootloader (4), and Secure Boot state (7). This set is reliable on standard Ubuntu/Debian installations. PCR 11 (kernel image) was previously listed here but requires a Unified Kernel Image (UKI) to be populated; on standard GRUB-based installs PCR 11 is not measured and the binding would be silently weak. If your system uses UKIs (e.g. Talos, systemd-boot with UKI), you may add PCR 11 for stronger binding.

After any UEFI firmware update or bootloader change, re-enrol the TPM slot or the drive will not unlock automatically. Plan for this maintenance window.

Raspberry Pi 5: a stock Pi 5 has no standard TPM2 chip exposed to Linux — systemd-cryptenroll --tpm2-device=list returns nothing on an unmodified board. Use a strong passphrase or a root-disk keyfile for LUKS auto-unlock instead. A TPM is only reachable on a Compute Module 4/5 carrier board that routes the RP1 security block as a TPM2 device, or via a third-party add-on module; both configurations are out of scope for this guide.

If there is no TPM, the script generates a 512-byte random keyfile at /etc/luks/usb-data.key (readable only by root, mode 0400). This file lives on the root disk — which must itself be on an encrypted OS partition, or you have no defence against physical theft of the whole machine.

1.4 Back up the LUKS header — now

The LUKS2 header is a small block at the start of the drive that stores all key material. If it is corrupted (a bad write, a failed firmware update, stray magnets), the drive is permanently unreadable — even with the correct passphrase. Back it up immediately:

cryptsetup luksHeaderBackup /dev/disk/by-id/usb-YOUR_DRIVE_ID-0:0 \
  --header-backup-file /root/luks-header-usb-$(date +%Y%m%d).bin

Store that .bin file somewhere encrypted and physically separate. A USB thumb drive in a different location is fine. Without this backup, losing the header means losing all data permanently.

2 — Passing the drive to the guest VM

Skip this section if you are running k3s directly on the host (bare-metal). Bare-metal is the simpler and slightly more secure path — no hypervisor in between.

If k3s runs inside a VM (Talos or Flatcar on KVM/QEMU), the host opens the LUKS2 container and passes the unlocked block device — /dev/mapper/usb-data — to the VM as a virtio-blk disk. The VM never sees the USB controller; it sees a plain block device.

The USB controller is a complex piece of software. Handing the whole USB stack to a VM means the VM can exploit bugs in that stack (CVE-2024-8354 was one such bug). Instead, the host decrypts the drive and hands the VM a simple “virtual disk” — just a queue of read/write requests, no USB complexity.

Do not use USB controller passthrough (<hostdev type='usb'>). The virtio-blk path avoids the QEMU USB emulation layer entirely. CVE-2024-8354 is a concrete example of why.

The libvirt XML snippet is printed by the script and also lives at scripts/config/storage/virtio-blk-passthrough.xml. Add it inside the <devices> block of your domain XML:

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
  <source dev='/dev/mapper/usb-data'/>
  <target dev='vdb' bus='virtio'/>
  <serial>usb-data-luks</serial>
</disk>

cache='none' bypasses the host page cache so data is not held in unencrypted RAM. io='native' uses direct I/O for lower latency. discard='unmap' passes TRIM through to the drive.

3 — Three kinds of storage, all encrypted

Once the USB drive is mounted at /mnt/usb-data, three subdirectories serve three purposes:

/mnt/usb-data/
├── object-store/   ← Garage (S3-compatible object storage)
├── longhorn/       ← Longhorn persistent volumes  (or TopoLVM on Pi 4)
└── k8s-volumes/    ← k3s data-dir (container layers + emptyDir)

All three are inside the LUKS2 container. Let’s look at each.

4 — Object storage: Garage v1.1.0

Object storage is the “S3-like” bucket system — store a file by key, retrieve it by key, no filesystem hierarchy needed. It is what Velero (backup) and many applications use for large blobs.

Why Garage, not MinIO?

MinIO was the default homelab choice for years. It is no longer:

• June 2025: MinIO stripped the admin UI out of the free version, restricting it to a paid enterprise tier starting at $96,000/year.
• December 2025: The community repository was placed in maintenance mode.

MinIO community edition is a dead end. Do not build on it.

MinIO did the classic open-core bait-and-switch: build a community, then move the useful parts behind a paywall. Garage is funded by NLnet (a European non-profit), has a stable AGPLv3 licence, and is designed from the start for exactly this use case — small, outside-a-datacenter hardware.

Garage v1.1.0 (released March 2025) is the right choice:

• Single Rust binary, ~200 MB idle RAM — runs on a Pi 4.
• Passes AWS SDK S3 compatibility tests.
• Designed for “outside datacenter” single-node or small-cluster deployments.
• TLS is handled by the Cilium Gateway API (Layer 4) — an HTTPRoute points at the Garage service, and cert-manager issues the certificate.

4.1 Install Garage

scripts/cluster/24-storage.sh installs Garage via its Helm chart. It also pre-creates the two local PersistentVolumes that pin Garage’s data to the USB mount.

sudo bash scripts/cluster/24-storage.sh

4.2 Bootstrap: layout, key, and bucket

After the pod is running, bootstrap the cluster once:

GPOD=$(kubectl -n storage get pod -l app.kubernetes.io/name=garage -o name | head -1)

# Show the node ID
kubectl -n storage exec "$GPOD" -- /garage status

# Apply layout — set zone and capacity (adjust 200G to your drive size)
NODE_ID=$(kubectl -n storage exec "$GPOD" -- /garage node id 2>/dev/null | awk '{print $1}')
kubectl -n storage exec "$GPOD" -- /garage layout assign -z homelab -c 200G "$NODE_ID"
kubectl -n storage exec "$GPOD" -- /garage layout apply --version 1

# Create an access key for Velero
kubectl -n storage exec "$GPOD" -- /garage key create velero-key

# Create the backup bucket and grant access
kubectl -n storage exec "$GPOD" -- /garage bucket create velero-backups
kubectl -n storage exec "$GPOD" -- /garage bucket allow velero-backups \
  --read --write --owner --key velero-key

The Garage admin API runs on port 3902. A NetworkPolicy in 24-storage.sh restricts that port to kube-system only. The S3 API on port 3900 is reachable cluster-wide through the Cilium Gateway API (an HTTPRoute to the Garage service, TLS terminated by cert-manager).

5 — Persistent volumes

A PersistentVolumeClaim (PVC) is a request for durable storage — “give me 10 GB that survives pod restarts.” This section explains how to create one and how it is encrypted.

Think of a PVC like reserving a locker at the gym. You say “I need a 10 GB locker, encrypted.” The storage provisioner finds space, formats a volume, and hands you a key (a mount path). Your pod opens the locker every time it starts; the data is still there when it comes back.

5.1 Which provisioner?

Longhorn is not feasible on a Pi 4 (4 GB RAM). Its per-node daemon alone uses ~50% of that RAM. Use TopoLVM on a Pi 4.

The install script detects the hardware and installs the correct provisioner automatically. On a Pi 4 (IS_PI=1, RAM ≤ 4 GiB), it installs TopoLVM. On all other hardware, it installs Longhorn.

If you are using local-path-provisioner (k3s’s built-in default), upgrade it to v0.0.34 or later immediately. CVE-2025-62878 (CVSS 10.0, disclosed February 2026) allows path traversal via a StorageClass pathPattern field, enabling arbitrary read/write on the host filesystem. There is no workaround — only the upgrade patches it.

5.2 The encrypted StorageClass

scripts/config/storage/storageclass-encrypted.yaml defines a StorageClass that tells Longhorn to wrap every volume in LUKS2 using aes-xts-plain64 + Argon2id. The encryption key comes from a Kubernetes Secret (longhorn-crypto-key in longhorn-system).

# Create the key secret first (replace the placeholder with a real random passphrase)
kubectl -n longhorn-system create secret generic longhorn-crypto-key \
  --from-literal=CRYPTO_KEY_VALUE="$(openssl rand -base64 32)" \
  --from-literal=CRYPTO_KEY_CIPHER="aes-xts-plain64" \
  --from-literal=CRYPTO_KEY_HASH="sha512" \
  --from-literal=CRYPTO_KEY_SIZE="512" \
  --from-literal=CRYPTO_PBKDF="argon2id"

# Apply the StorageClass
kubectl apply -f scripts/config/storage/storageclass-encrypted.yaml

Do not store the plaintext passphrase in git. Manage the Secret via Sealed Secrets or External Secrets Operator. The 24-storage.sh script creates a random passphrase at install time and writes it only to the cluster.

5.3 Provisioning a PersistentVolume — the copy-paste PVC

scripts/config/storage/example-pvc.yaml is a heavily commented template. Copy it, change the name and size, and apply it:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data       # change this to match your application
  namespace: default       # change to the namespace where your app runs
spec:
  accessModes:
    - ReadWriteOnce         # one pod at a time (standard for databases, etc.)
  storageClassName: longhorn-encrypted   # uses per-volume LUKS2 encryption
  resources:
    requests:
      storage: 10Gi        # request 10 GB; Longhorn thin-provisions so it
                            # doesn't actually consume 10 GB until data is written

kubectl apply -f scripts/config/storage/example-pvc.yaml
kubectl get pvc my-app-data   # should reach Bound within ~30 seconds

Attach it to a pod:

spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: my-app-data
  containers:
    - name: app
      image: your-image
      volumeMounts:
        - name: data
          mountPath: /data
      resources:
        limits:
          ephemeral-storage: "1Gi"   # always set this — see §6

ReadWriteOnce means the volume can be mounted by one node at a time. For a single-node cluster that is always correct. If you ever need two pods to share a volume simultaneously, that requires ReadWriteMany and a different provisioner — outside the scope of this layer.

6 — Ephemeral storage: the invisible bytes

Every time a container runs, Kubernetes writes three kinds of data you may not think about:

1. Container writable layer — any file the container creates or modifies on top of its image.
2. emptyDir volumes — scratch space declared in the pod spec; lives only as long as the pod.
3. Container logs — written by the kubelet to the node filesystem.

By default, k3s places all of this under /var/lib/rancher/k3s/ — on the root disk, which may be unencrypted. One line in k3s’s config redirects everything to the USB mount:

data-dir: /mnt/usb-data/k8s-volumes/k3s-data

This single line makes every ephemeral byte land inside the LUKS2 container. The 24-storage.sh script writes this line.

Without this setting, a pod could write a secret to a temp file, the pod terminates, and the secret is sitting unencrypted on the root disk. With this setting, all temp files go onto the locked USB. When the drive is unplugged, those files are unreachable.

6.1 Always set ephemeral storage limits

An unconstrained pod can fill the USB drive and crash the cluster. Set limits:

resources:
  requests:
    ephemeral-storage: "256Mi"
  limits:
    ephemeral-storage: "1Gi"

6.2 Use memory-backed volumes for secrets

Any volume that holds tokens, TLS certificates, or credentials should never touch disk at all. Use medium: Memory:

volumes:
  - name: secret-scratch
    emptyDir:
      medium: Memory
      sizeLimit: "64Mi"

This uses tmpfs (RAM). It never writes to the USB drive. It counts against the container’s memory limit, not its ephemeral-storage quota.

7 — Backup with Velero + Kopia

Encryption at rest protects against physical theft. Backup protects against drive failure. They are different problems; you need both.

7.1 What Velero does

Velero backs up both Kubernetes resource definitions (Deployments, ConfigMaps, Secrets) and the actual data inside PersistentVolumes (via Kopia, its file-copy engine). It stores everything in an S3 bucket — in this case, your Garage bucket.

Velero 1.15 deprecated the old restic backend. From 1.15 onward, the correct uploader is kopia. The commands below use Kopia.

7.2 Install Velero pointing at Garage

# Garage credentials file (same format as AWS)
cat > /tmp/garage-creds <<EOF
[default]
aws_access_key_id=REPLACE_WITH_GARAGE_ACCESS_KEY
aws_secret_access_key=REPLACE_WITH_GARAGE_SECRET_KEY
EOF

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0 \
  --bucket velero-backups \
  --backup-location-config \
    region=garage,s3ForcePathStyle=true,s3Url=https://s3.internal.example.com \
  --use-node-agent \
  --uploader-type=kopia \
  --secret-file /tmp/garage-creds

rm /tmp/garage-creds

7.3 Schedule a daily backup

velero schedule create daily-full \
  --schedule="0 2 * * *" \
  --ttl 168h   # keep 7 days

The Garage bucket on your USB drive is not an off-site backup. If the USB drive fails, you lose both the primary data and the backups simultaneously. Add a second backup target — rclone sync to a remote encrypted location, a second USB drive, or a cloud bucket — to close this gap.

8 — Security summary

Checklist

• Identified USB device by /dev/disk/by-id/ path — never /dev/sdX
• LUKS2 formatted: aes-xts-plain64, 512-bit, Argon2id
• LUKS2 header backed up and stored encrypted offline
• At least 2 key slots (passphrase + recovery); slot 2 = TPM2 if available
• TPM2 enrolled with --tpm2-pcrs=0+2+4+7 (same PCR set as OS disk; add PCR 11 only if using UKIs)
• VM passthrough: virtio-blk (/dev/mapper/usb-data), not USB controller passthrough
• local-path-provisioner at v0.0.34+ if in use (CVE-2025-62878 patched)
• Longhorn encrypted StorageClass applied; crypto Secret managed via Sealed Secrets or ESO
• data-dir: /mnt/usb-data/k8s-volumes/k3s-data in k3s config on every agent
• Ephemeral storage limits set on all pods
• Garage S3 endpoint behind TLS via Cilium Gateway API HTTPRoute (Traefik is disabled in Layer 2)
• Garage admin port 3902 restricted by NetworkPolicy to kube-system only
• Velero (1.15+, --uploader-type=kopia) installed; daily schedule configured
• LUKS header backup tested by dry-run restore on a spare machine

What this layer bought you

Every byte your cluster writes is now locked. A stolen USB drive yields random noise. A decommissioned drive needs no special disposal — the data was already inaccessible without the key. Container scratch files, database rows, object-store blobs, and Velero backup archives all live inside the same cryptographic boundary. The TPM2 binding adds a second guarantee: the drive only unlocks on this specific machine when this specific boot chain runs — firmware tampering or kernel replacement trips the lock. The per-volume LUKS2 on Longhorn adds a third layer: even if an attacker gets a raw volume file, each volume has its own key, so one compromised secret does not decrypt everything.

Object storage, persistent volumes, and ephemeral storage are now all provisioned, encrypted, and backed up — a complete, self-contained storage plane for a hardened single-node cluster.