Verify & Validate

Don’t trust that it’s secure — prove it. The same tests a professional auditor would run, in plain steps.

A fortress you haven’t tested is just a hope with good intentions. This chapter is a checklist of concrete tests that prove each layer is doing its job. Run them right after building, and again after any big change. If a test fails, the chapter it maps to tells you how to fix it.

Most of these are read-only and safe to run anytime. The handful that deliberately try to break in (marked 🔴 attack test) are safe too — they’re designed to be blocked, and a block is a pass.

Tip

The fast acceptance test

If you only do one thing, run the cluster benchmark and the deliberate-bad-pod test:

On the server (root)

# 1. CIS benchmark — the authoritative cluster hardening score
sudo k3s kubectl apply -f scripts/config/k3s/kube-bench-job.yaml
sudo k3s kubectl logs -f job/kube-bench | tail -40

On your admin laptop — 🔴 attack test

# 2. A rule-breaking pod MUST be rejected by admission control:
kubectl run evil --image=nginx --privileged --restart=Never
# PASS = "Error from server ... admission webhook ... denied the request"

Pass criteria: kube-bench shows 0 FAIL on scored controls (a few WARN/INFO are normal and explained in its output), and the privileged pod is denied. If both hold, your core hardening is real, not theoretical.

Hardened

Layer-by-layer verification

L0 — Host

On the host (root)

sudo nft list ruleset | grep -E 'policy drop|dport'   # firewall default-deny + only allowed ports
sudo aa-status | head -3                               # AppArmor: profiles in enforce mode
sudo systemctl is-active auditd                        # auditing on
sudo sysctl kernel.kptr_restrict kernel.unprivileged_bpf_disabled  # hardening applied
lsblk -o NAME,FSTYPE,MOUNTPOINT | grep crypt           # disk is LUKS-encrypted
sudo lynis audit system --quick                        # CIS-style audit score

From a second machine, scan the host: nmap -p- <host-ip>. You should see only the WireGuard UDP port (and nothing if the scanner isn’t on the VPN). An open SSH or 6443 to the world is a failure — fix the firewall before going further.

Important

L1 — Immutable VM

On the host (root)

sudo virsh list --all                 # the k3s VM is running
# Confirm the guest root is read-only (Flatcar/Talos): a write to /usr must fail.

L2 — k3s hardening

On your admin laptop

kubectl get --raw='/readyz?verbose'                    # all checks pass
kubectl get apiservices | grep -i False && echo "DEGRADED" || echo "OK"
# Secrets must be encrypted at rest — read the raw etcd value, it should be ciphertext:
sudo k3s secrets-encrypt status                        # 'Encryption Status: Enabled'
# Audit log is being written:
sudo tail -n 2 /var/lib/rancher/k3s/server/logs/audit.log

L3 — Sealed runtime 🔴 attack test

On your admin laptop

# A sandboxed pod should report the gVisor kernel, not the host kernel:
kubectl run gv --image=alpine --restart=Never --overrides='{"spec":{"runtimeClassName":"gvisor"}}' -- uname -a
kubectl logs gv          # gVisor reports a distinctive synthetic kernel string
# A container trying to gain privileges must fail (read-only rootfs / dropped caps):
kubectl exec gv -- sh -c 'touch /test' 2>&1 | grep -i 'read-only' && echo "rootfs locked ✓"

L4 — Network zero-trust 🔴 attack test

On your admin laptop

# Two pods in different namespaces should NOT reach each other by default:
kubectl run a -n ns1 --image=nicolaka/netshoot --restart=Never -- sleep 1d
kubectl run b -n ns2 --image=nicolaka/netshoot --restart=Never -- sleep 1d
kubectl exec -n ns1 a -- curl -m 3 http://<b-pod-ip>   # PASS = times out (blocked)
# Confirm pod traffic is WireGuard-encrypted:
kubectl -n kube-system exec ds/cilium -- cilium status | grep -i encryption  # 'Wireguard'
# See the flows live:
hubble observe --verdict DROPPED | head              # default-deny drops are visible

L5 — Encrypted storage

On your admin laptop

kubectl get storageclass | grep encrypted            # the encrypted class exists
# Provision a test volume, confirm Bound, then confirm it's LUKS on the node:
kubectl apply -f scripts/config/storage/example-pvc.yaml
kubectl get pvc -A | grep Bound

On the host (root)

sudo dmsetup ls --target crypt | head                # the volume shows as a crypt device

L6 — Observability

On your admin laptop

kubectl -n monitoring get pods                        # Prometheus/Grafana/Loki/Tetragon Running
# Generate a benign 'suspicious' event and confirm Tetragon/Falco catches it:
kubectl run shell --image=alpine --restart=Never -- sh -c 'cat /etc/shadow'
# then look for the detection:
kubectl -n kube-system logs ds/tetragon | grep -i shadow   # the read was observed

✦ — Supply chain & policy 🔴 attack test

On your admin laptop

# An image from a registry NOT on the allow-list must be rejected:
kubectl run rogue --image=docker.io/library/nginx --restart=Never
# PASS = denied by Kyverno restrict-registries (if docker.io isn't allow-listed)
# An UNSIGNED image must be rejected when verifyImages is enforced:
kubectl run unsigned --image=<your-registry>/test:unsigned --restart=Never   # PASS = denied

Continuous validation (set and forget)

You don’t have to remember to run these — the cluster scans itself:

Tool	What it checks	Where results show up
Trivy Operator	Image CVEs, misconfigs, exposed secrets, RBAC	`kubectl get vulnerabilityreports -A`; Grafana
Kubescape	NSA/CISA + MITRE ATT&CK posture	Kubescape dashboard / reports
kube-bench (weekly CronJob)	CIS benchmark drift	Job logs; alert on new FAIL
Tetragon / Falco	Live runtime attacks	Loki + your alert channel

Glance at your security posture anytime

kubectl get vulnerabilityreports -A | sort -k7 -r | head   # worst CVEs first
kubectl get configauditreports -A | head

How this guide itself was validated

The scripts and manifests in this repository were checked before publishing — not just claimed to work:

shellcheck + bash -n on every shell script → 0 errors.
kubeconform schema-validation on every Kubernetes manifest → 0 invalid (custom-resource manifests such as Cilium and Kyverno policies are schema-skipped, which is expected offline).
A live boot of the hardened k3s (v1.36.1+k3s1) using the exact config.yaml, audit-policy.yaml, and psa.yaml from this repo. The control plane came up with zero configuration errors, and the logs confirmed the hardening is active:

Check	Result
Hardened `config.yaml` accepted by k3s v1.36.1	Pass no fatal/parse/validation errors
Secrets encryption-at-rest provider loaded	Pass `encryption-provider-config` controller started
Admission chain loaded (incl. PodSecurity)	Pass 19 validating + 16 mutating controllers
PSA + EventRateLimit admission config accepted	Pass `admission-control-config-file` loaded
Audit logging active	Pass 1.4 MB of valid JSON audit events written
`protect-kernel-defaults` satisfied by sysctls	Pass kubelet did not reject

The full seven-layer stack — host firmware, nested VM, TPM-sealed LUKS, USB passthrough — can only be fully exercised on your real hardware, by design (those layers are the hardware). Request-level checks (the privileged-pod denial, node Ready, kube-bench’s full score) likewise need a long-running cluster with registry access. This chapter is exactly how you run those on your machine; the repository’s automated checks cover everything that can be validated without it.

Note

What this chapter bought you

Proof. You can now demonstrate — not just claim — that the firewall is closed, the sandbox holds, the network denies by default, the disk is encrypted, bad pods are rejected, and the sensors are watching. Re-run this checklist after every change and you’ll never wonder whether a tweak quietly opened a hole.