The Architecture
One picture of the whole machine: what runs where, what talks to what, and which open-source tool does each job.
The previous chapter explained the philosophy. This one makes it concrete. By the end you’ll be able to point at any box in the system and say what it is, why it’s there, and which layer protects it. Keep this page open as a map while you build.
The layered model
Everything in this guide is organized as seven layers wrapped around your workload, plus one cross-cutting layer (supply chain, policy, and secrets) that touches them all. Each layer is independent: it does its job whether or not the others hold.
Read the stack from the outside in. An attacker arrives at the top (L0, the physical machine) and has to chew through every layer to reach your data at the bottom. Each band is a different kind of wall, so the skills and luck needed to beat one don’t help with the next.
The physical topology
Here’s how the layers map onto actual hardware on a powerful (Class A) machine — one box, one USB drive, and your laptop:
The flow of control is deliberately narrow:
- Your admin laptop holds the keys. It never talks to the cluster directly over the internet.
- It dials a WireGuard tunnel to the host — the single open door in the firewall.
- Inside the tunnel,
kubectlreaches the k3s API, which lives inside the immutable VM, not on the host itself. - The encrypted USB drive is plugged into the host, unlocked at boot, and passed straight through to the VM as a virtual disk — so the cluster’s storage is physically separable and encrypted at rest.
On a Raspberry Pi, remove the VM box from that picture: k3s runs directly on the hardened, immutable OS on bare metal. Everything else — WireGuard door, encrypted USB, kubectl path — is identical.
A command’s journey (and why it’s safe)
When you type kubectl get pods, here is everywhere that command is checked:
| Step | What happens | Guarding layer |
|---|---|---|
| 1 | Your laptop encrypts the request into the WireGuard tunnel | L4 |
| 2 | The host firewall accepts it only on the WireGuard port | L0 |
| 3 | It reaches the k3s API inside the VM — isolated from the host | L1 |
| 4 | k3s checks your identity (client cert) and RBAC permissions | L2 |
| 5 | The action is written to the immutable audit log | L2 + L6 |
| 6 | If you create a pod, admission control vets the image & security context | ✦ + L3 |
| 7 | The eBPF runtime sensor watches the resulting process for anything odd | L6 |
No single step is trusted on its own. Identity, authorization, policy, encryption, and observation are all enforced independently.
The component inventory
This is the complete bill of materials — every tool, why it’s chosen, and the layer it serves. All are open-source. Versions are the current stable releases as of this writing; the runbook pins them exactly.
| Layer | Job | Tool (chosen) | Why this one |
|---|---|---|---|
| L0 | Host OS | Ubuntu 24.04 LTS (Debian 13 / Pi OS supported) | Best CIS tooling (USG); broad hardware support |
| L0 | Disk encryption | LUKS2 + TPM2 | Standard, audited, hardware-sealed keys |
| L0 | Host firewall | nftables | Modern, default-deny, in-kernel |
| L0 | Virtualization | KVM / libvirt | Native Linux hypervisor with sVirt isolation |
| L1 | Immutable OS | Flatcar (k3s) · Talos (max) | Read-only root, atomic updates, tiny attack surface |
| L2 | Kubernetes | k3s v1.36 | Lightweight, single-binary, CIS-hardenable |
| L2 | Datastore | embedded etcd (encrypted) | Encrypted snapshots, HA upgrade path |
| L2 | Benchmark | kube-bench (auto-detects CIS profile) | Audits against the CIS benchmark for the running k3s version |
| L3 | Pod sandbox | gVisor (systrap) · Kata (opt.) | User-space kernel; contains container escapes |
| L3 | Profiles | Security Profiles Operator | Auto-generates tight seccomp profiles |
| L4 | CNI / network | Cilium (eBPF) | kube-proxy replacement, L7 policy, encryption |
| L4 | Encryption | WireGuard (via Cilium) | Fast pod-to-pod encryption, great on ARM |
| L4 | Admin access | WireGuard VPN | The only open port; no exposed API |
| L4 | Ingress | Cilium Gateway API + cert-manager | TLS, modern, ingress-nginx is EOL |
| L5 | Object store | Garage v1.1 | S3-compatible, tiny, Pi-friendly (MinIO is dead) |
| L5 | Persistent vols | Longhorn (TopoLVM on Pi) | Encrypted replicated volumes |
| L6 | Metrics | Prometheus (VictoriaMetrics on Pi) | The standard; light variant for small nodes |
| L6 | Logs | Loki + Alloy / Fluent Bit | Ships container, audit, and host logs |
| L6 | Runtime security | Tetragon (Falco alt.) | In-kernel eBPF detect and enforce |
| L6 | Posture / CVEs | Trivy Operator + Kubescape | Continuous scanning & compliance |
| ✦ | Policy engine | Kyverno | Blocks bad pods & unsigned images at admission |
| ✦ | Image signing | cosign (keyless) + syft SBOM | Proves images are trusted; SLSA L2 |
| ✦ | Secrets | SOPS + age · Sealed Secrets | Secrets safe to store in Git |
| ✦ | GitOps | Flux v2 | Reproducible, auditable cluster state |
Every one of these is default-deny in spirit: the firewall denies all ports but one, the network denies all traffic until allowed, admission denies all images but signed ones, and RBAC denies all actions but those granted. You open things deliberately, one at a time — never the other way around.
How the layers are built (the order matters)
You build from the outside in, because each layer assumes the one outside it already exists. The runbook follows exactly this order:
L0 host → L1 VM → L2 k3s → L4 network (so pods can talk) → L3 runtime + ✦ policy (so workloads are sandboxed and vetted) → L5 storage → L6 observability (so you can finally see it all). The Build Runbook is this sequence as copy-paste commands.
What this chapter bought you
You have the whole map: the seven layers, how they sit on real hardware, the path a command travels, and the exact open-source tool doing each job. Every following chapter zooms into one layer of this picture. Start with the host in Layer 0.