The Build Runbook

The whole fortress, start to finish, as an ordered checklist. Follow it top to bottom and don’t skip.

This is the page you actually do. Every other chapter explains the why; this one is the what, in order. Each phase points at one script in the scripts/ folder. You can run the whole thing with one orchestrator, or phase by phase to understand each step. Phase order matters — each layer assumes the one before it exists.

Run these on a machine you are willing to completely wipe. Layer 0 reconfigures the firewall, kernel, and SSH. Do the first run on a screen-and-keyboard you can physically reach, not over a remote SSH session you might lock yourself out of.

Before you start

Complete the Hardware & Prerequisites chapter first. You need: a freshly installed, disk-encrypted Ubuntu/Debian/Pi OS host; a USB SSD plugged in; and kubectl + helm + wireguard on your admin laptop.

Get the code onto the host:

git clone https://github.com/n1healthcare/fortress-k3s.git
cd hardened-k3s/scripts
# Read what you're about to run. Never run hardening scripts blind.
less setup-all.sh

Every script is idempotent — safe to run again. If one fails partway, fix the cause and re-run it; completed steps are skipped automatically. Set ASSUME_YES=1 to run unattended, or leave it unset to be prompted before each major change.

The one-command path

If you’ve read the chapters and trust the defaults, the orchestrator runs every phase in order, pausing between layers:

sudo ./setup-all.sh            # interactive, pauses between layers
# or fully unattended:
sudo ASSUME_YES=1 ./setup-all.sh

Prefer to go phase-by-phase the first time. The rest of this page is exactly what the orchestrator does, broken out so you can run and verify each layer yourself.

Phase 0 — Harden the host (L0)

sudo ./host/10-host-harden.sh        # updates, sysctls, lockdown, auditd, SSH, nftables, CIS audit
sudo ./host/12-tpm-luks-enroll.sh    # seal the disk key to the TPM (x86 with TPM only)
sudo ./host/11-kvm-libvirt.sh        # install + harden KVM/libvirt (x86 only; skipped on Pi)

Verify before continuing:

sudo nft list ruleset | head            # firewall is default-deny
sudo aa-status | head                    # AppArmor enforcing
sudo systemctl is-active auditd          # auditing is on

After this phase the machine has no open ports except (soon) WireGuard, a hardened kernel, encrypted disk, and a CIS audit report at /var/log/fortress-k3s/.

Phase 1 — Boot the immutable VM (L1)

Raspberry Pi users skip this phase — you run k3s directly on the bare-metal hardened OS. Continue at Phase 2.

sudo ./host/13-vm-provision.sh       # downloads Flatcar, renders Ignition, starts the VM

This provisions the immutable VM and, on first boot, it pulls the k3s configuration from Phase 2. Confirm the VM is up:

sudo virsh list --all                 # the k3s VM should be 'running'

Phase 2 — Install hardened k3s (L2)

Run inside the VM (or directly on the Pi):

sudo ./cluster/20-k3s-install.sh     # lays down hardened config.yaml, audit policy, PSA, then installs k3s v1.36

Verify:

sudo k3s kubectl get nodes            # node is Ready
sudo k3s kubectl get --raw='/readyz?verbose'

secrets-encryption and the audit log can only be set at first install. If you ever see them missing, you must rebuild the cluster — do not try to bolt them on later.

Phase 3 — Set up the network & your access (L4)

We do networking before runtime/storage so pods can actually communicate.

sudo ./host/14-wireguard.sh          # host WireGuard VPN — this is how you'll reach kubectl

sudo ./cluster/21-network-cilium.sh  # Cilium eBPF CNI + WireGuard pod encryption + Hubble + default-deny

Get your admin access working now — copy the printed WireGuard client config to your laptop, bring up the tunnel, then fetch the kubeconfig:

sudo wg-quick up wg0                                  # bring up the tunnel
scp host:/etc/rancher/k3s/k3s.yaml ~/.kube/fortress   # (over the tunnel)
# edit the server: address to the WireGuard IP, then:
export KUBECONFIG=~/.kube/fortress
kubectl get nodes                                     # you're in — over an encrypted tunnel

From here on, every kubectl command travels through WireGuard. Port 6443 is never exposed. If you can reach the API without the tunnel up, stop and fix it.

Phase 4 — Seal the runtime & enforce policy (L3 + ✦)

sudo ./cluster/22-runtime.sh         # gVisor sandbox + RuntimeClasses + Security Profiles Operator
sudo ./cluster/23-policy-kyverno.sh  # Kyverno + restricted policies + image-signature enforcement

Verify the sandbox and policy both work:

# A pod that breaks the rules (privileged) must be REJECTED:
kubectl run bad --image=nginx --privileged --restart=Never
# expected: error from the Kyverno/PSA admission webhook — that's success.

Phase 5 — Encrypted storage (L5)

sudo ./host/15-luks-usb.sh           # LUKS2-encrypt the USB SSD, mount it, print the VM passthrough snippet

sudo ./cluster/24-storage.sh         # Garage S3 object store + Longhorn/TopoLVM persistent volumes

Verify:

kubectl get storageclass              # an 'encrypted' class exists
kubectl get pods -n storage           # Garage + Longhorn running

Phase 6 — Turn on the eyes (L6)

sudo ./cluster/25-observability.sh   # Prometheus/Grafana/Loki + Tetragon eBPF + Trivy (auto light-tier on Pi)
sudo ./cluster/26-supplychain.sh     # cosign verify + SOPS/age secrets (+ optional Flux GitOps)

Reach Grafana safely (over the tunnel only):

kubectl -n monitoring port-forward svc/grafana 3000:80
# open http://localhost:3000  — through WireGuard, never exposed publicly

Final acceptance check

Run the full validation suite from the Verify & Validate chapter — kube-bench, Trivy, Kubescape, and the network/policy probes:

# Apply the kube-bench job and tail its output (see Chapter 12 for benchmark details)
sudo k3s kubectl apply -f config/k3s/kube-bench-job.yaml
sudo k3s kubectl wait --for=condition=complete job/kube-bench-k3s -n kube-system --timeout=120s
sudo k3s kubectl logs -n kube-system job/kube-bench-k3s

When kube-bench reports 0 FAIL on the scored controls, Tetragon is logging events, Trivy shows your images scanned, and kubectl only works over WireGuard — the fortress is built. Continue to Day-2 Operations.

What this runbook bought you

A repeatable, ordered path from a blank machine to a seven-layer hardened cluster — and because every script is idempotent and the config is in Git, you can rebuild the entire thing from scratch any time, in minutes, with confidence.