The goal of this project is to leverage a proxmox cluster (named ranch) to build out a Kubernetes Cluster (eventually highly available). This kubernetes cluster should have all of the standard services required:
(Future me:) I spent a lot of time looking into what was necessary to run this cluster using an external Postgresql cluster. The primary reason was I thought I could run 2 controller nodes and save money on a 3rd machine for the cluster. In the end, I could not really find good instructions for running a 2-node postgresql cluster. Everyone felt like 3 was the proper number of nodes and I had to learn so much about postgresql failover that it was overwhelming. The entire point of k3s is that it is batteries included.
I have decided that I will start with a 1 node control-plane using the SQLITE storage backend. Then I will jump to a 3-node control plane (even if I have to run 2 VMs on the same host) converting the cluster to use etcd when that day happens. I will go ahead and set up kube-vip so that my cluster API has a well-known IP address: 192.168.200.210
The code lives in the Git repo at http://gitea.rachelski.internal/mark/k3s. Its README.md has the full deployment instructions. This page was extracted from that README to hold the IP reservations and the service overview.
We will go ahead and plan out a block of IP addresses to reserve for the home lab. This could get rebuilt in the future once I have better networking equipment that allows me to manage multiple subnets. But for now, this is based upon the network created using the Asus ZenWiFi mesh routers at 192.168.200.1/24. This router is currently using 192.168.200.5-192.168.200.180 for its dynamic DHCP pool.
| IP Address | Name | Purpose |
|---|---|---|
| 192.168.200.201 | ranch-1 | MinisForum MS-01 workstation running Proxmox. NOTE: this device has 4 ports and we may wish to reserve more IP addresses for those ports. |
| 192.168.200.202 | ranch-2 | MinisForum MS-01 workstation running Proxmox. NOTE: this device has 4 ports and we may wish to reserve more |
| 192.168.200.203 | reserved for ranch-3 when it is purchased. | |
| 192.168.200.204 | reserved for other lab hardware | |
| 192.168.200.205 | net-1 | firewall host running 4 VMs: DNS, the Omada network controller, OpnSense and Caddy |
| 192.168.200.206 | omada-controller | Runs the network controller that manages the SDN |
| 192.168.200.207 | Caddy VM | Runs docker and within that a Caddy reverse proxy that has been port forwarded (ports 80, 443) from the Asus router |
| 192.168.200.208-209 | reserved for other lab hardware | |
| 192.168.220.201 | a 10Gb port on ranch-1 that forms part of the cluster high speed communication backbone | |
| 192.168.220.202 | a 10Gb port on ranch-2 that forms part of the cluster high speed communication backbone |
There will be a number of IP addresses and virtual machines deployed:
| IP Address | Name | Purpose |
|---|---|---|
| 192.168.200.210 | k8s-control | Virtual IP for k8s cluster control plane API |
| 192.168.200.211 | k8s-control-0 | Primary cluster controller |
| 192.168.200.212 | k8s-control-1 | Reserved for replica controller for HA |
| 192.168.200.213 | k8s-control-2 | Reserved for 3rd replica controller for HA |
| 192.168.200.214-192.168.200.216 | spare space for VMs | |
| 192.168.200.217-192.168.200.218 | k8s-pv-storage-0, k8s-pv-storage-1 | A group of 2 VMs that form the Longhorn Storage Array (using local storage) - 1 still to be moved to ranch-3 when it is procured |
| 192.168.200.219 | reserved for final longhorn node running on ranch-3 | |
| 192.168.200.220 | k8s-node-0 | Worker 1 that hols k8s workloads |
| 192.168.200.221 | k8s-node-1 | Old Worker 2 - - Removed from the cluster. The VM has not been deleted yet. |
| 192.168.200.222 | k8s-node-21 | Worker 21 |
| 192.168.200.223 - 228 | reserved for additional workers | |
| 192.168.200.229 | k8s-pv-storage-20 | VM for Longhorn Storage on larger SSD on ranch-2 (replacing k8s-pv-storage-2 on ranch-1) |
| 192.168.200.230-192.168.200.239 | Reserved for Load Balancer IPs used by MetalLB (see below table) |
| VIP | Purpose |
|---|---|
| 192.168.200.230 | ingress loadbalancer |
| 192.168.200.231 | Bittorrent VIP |
The longhorn components are dependant upon a NAS on the network at 192.168.200.155 that is used as a target for storing backup images
This is a quick summary of the services and controllers running inside the cluster that make it easy for a developer to run apps
The cluster is running traefik as the primary ingress controller that services will use to expose connections to outside of the cluster. A very common ingress pattern that can be used for cut-and-paste looks like this:
className: traefik
annotations:
cert-manager.io/cluster-issuer: ranch-cert-issuer
cert-manager.io/common-name: bookstack.rachelski.internal
external-dns.alpha.kubernetes.io/hostname: "bookstack.rachelski.internal"
hosts:
- host: "bookstack.rachelski.internal"
paths:
- path: /
pathType: Prefix
tls:
- secretName: bookstack-tls
hosts:
- bookstack.rachelski.internal
This example will cover configuring cert-manager, external-dns and traefik so that the developer can expose a service to the home network by name with HTTPS. Note that the hostname is repeated 3 times within the block for different purposes.
A record in Technitium, pointing at the Traefik ingress address.This is the Helm values layout. For a plain Kubernetes manifest and the rules every Ingress must follow, see Deploying Apps on the Ranch Cluster.
Some requests reach Traefik for a hostname that no Ingress claims: stale DNS names, requests to the bare IP 192.168.200.230, or network scanners. Out of the box Traefik would answer with its self-signed "TRAEFIK DEFAULT CERT" and Go's plain-text 404 page not found, which tells anyone probing exactly what proxy is running (and when it last restarted). Instead:
*.rachelski.internal wildcard issued by ranch-cert-issuer and stored as the Secret traefik-default-tls in traefik-system. cert-manager renews it, and Traefik picks up renewals without a restart.catchall-404 in traefik-system) that returns a plain Not found with no Server header.This lives in the k3s repo: the certificate in terraform/traefik-default-cert.tf, the TLS store setting in config/traefik-values.yaml, and the catch-all in bootstrap/network/traefik.tf. On a fresh rebuild the bootstrap network stage runs before the main terraform/ root, so Traefik serves its built-in certificate (and logs an error about the missing Secret) until the main root has issued the wildcard. It then switches over by itself.
This wildcard is only a fallback. Apps must still get their own certificate through their Ingress; see rachelski.internal PKI.
trust-manager takes care of keeping a current trust bundle updated including our root certificate.
See Using Trust Manager for more information.
external-secrets can replicate secrets stored in Hashicorp Vault into a k8s secret. This safely delivers secret configuration information to the pod at startup time for configuration purposes.
It uses a custom resource definition (CRD) to declare the mapping between the Hashicorp Vault path and the K8s secret name and key.
See Using external-secrets for more information.
cloud-native-pg can provision postgresql databases. Very useful for developers as almost all solutions need a backing database as a dependency.
See Using CNPG (Cloud Native Postgres) for more information.
mariadb-operator can provision mariadb databases. Very useful for developers where an application requires a mariadb database server.
See Using MariaDB Operator for more information.
We deploy kube-vip on the cluster controllers. This allows us to have multiple controller nodes that can operate as a single control-plane because they share a virtual IP address. In most cases, an application developer does not use kube-vip directly, but their kubectl tool will contain the VIP inside its configuration by leveraging the name k8s-control.rachelski.internal.
MetalLB is used in the unlikely situation where an application needs a virtual IP address to handle fail-over for services. Since most applications are leveraging the cluster services through ingresses (or gateway API), they should not require this. But MetalLB is used to create Ingress controller load balancers.
Longhorn is an open-source, lightweight distributed block storage system built specifically for Kubernetes to provide highly available, persistent storage for stateful applications. It orchestrates storage by turning local disks on Kubernetes nodes into a resilient, distributed storage pool, allowing users to easily create and manage Persistent Volumes (PVs) without relying on external enterprise storage arrays. By automatically replicating data across multiple nodes, offering built-in incremental snapshots and backups, and supporting live volume migration, Longhorn ensures enterprise-grade data resilience and disaster recovery while remaining native and easy to manage within the Kubernetes ecosystem.
For the most part, a developer does not need to think about CoreDNS. The deployment of CoreDNS running in this cluster will recognize any hostname.rachelski.internal and forward resolver requests to the local network resolver. CoreDNS is also the primary DNS server for IP addresses inside the cluster handling the [service-name].[namespace].svc.cluster.local DNS zone.
external-dns will allow a developer to register their hostname.rachelski.internal name in the DNS resolver so that others can find their service from outside of the cluster.
See Ingress Controller for a useful example of how to use it.
cert-manager helps a developer issue and rotate certificates so services can be exposed over HTTPS.
See Ingress Controller for a useful example on how to use it.
This cluster has a few Taints and Tolerations used to help a developer position the applications on the nodes in the cluster.
In general, unless you are deploying a database, don't bother declaring any tolerations. But for completeness, here are the details.
The cluster uses taints to keep each type of node for its own job. General workers have no taints and run normal app workloads. The control plane and the storage nodes are tainted to keep app pods off.
| Node(s) | Taint | Effect | Purpose |
|---|---|---|---|
k8s-control-0 |
CriticalAddonsOnly=true |
NoExecute |
The controller has little memory (~4 GiB). It runs only the k3s system add-ons (CoreDNS, metrics-server, kube-vip). NoExecute also evicts any pod already running there that doesn't tolerate the taint. |
k8s-pv-storage-0, -1, -21 |
dedicated-purpose=storage |
NoSchedule |
Longhorn storage nodes. Only Longhorn and the databases on longhorn-database volumes are scheduled here. These nodes also carry the labels tier=pv-storage and node.longhorn.io/create-default-disk=true. |
k8s-node-0, k8s-node-21 |
(none) | — | General workers. App pods land here by default. |
Where the taints come from
node-taint in config/k3s-agent-storage-config.yaml. A storage node never exists without its taint, so DaemonSets can't land on it before it's tainted. Older nodes were tainted afterwards with scripts/k3s-setup-longhorn-nodes.sh, which is only for retrofitting.kubectl taint nodes k8s-control-0 CriticalAddonsOnly=true:NoExecute. It can't go in the k3s server config, because CoreDNS and metrics-server need somewhere to run while the cluster is being built. After a rebuild, re-apply it (README → "Tainting the controllers").| Workload | Toleration | Why |
|---|---|---|
| Longhorn (manager, CSI driver/plugins, instance managers, engine images, UI) | dedicated-purpose=storage:NoSchedule |
Set in two places in config/longhorn-values.yaml: global.tolerations for the chart's pods and defaultSettings.taintToleration for the pods Longhorn creates itself (instance managers, engine images). Longhorn runs on every node, both storage and worker, so any node can attach volumes. Only the UI is pinned to storage nodes. |
Database pods (CNPG clusters: paperless-db, wikijs-db, immich-db, nextcloud-db) |
dedicated-purpose=storage:NoSchedule plus nodeSelector: tier: pv-storage |
The longhorn-database StorageClass uses strict-local with 1 replica, so the volume must be on the same node as the pod. The toleration allows scheduling on storage nodes. The nodeSelector requires it. |
| CoreDNS, metrics-server (k3s built-ins) | CriticalAddonsOnly (Exists) |
k3s defaults. This is why they are the only Deployments on the controller. |
| kube-vip DaemonSet | All NoSchedule and NoExecute taints (Exists) |
Holds the control-plane VIP (192.168.200.210), so it has to run on control-plane nodes whatever their taints. Generated by scripts/config-generate-kube-vip.sh (--taint). |
| MetalLB speaker / FRR | node-role.kubernetes.io/control-plane, .../master (chart defaults) |
These match no taint in this cluster, so they have no effect. Node affinity keeps MetalLB off storage and control-plane nodes, not tolerations. |
Traefik and MetalLB speakers have no tolerations for our custom taints. Node affinity keeps them on general workers (they exclude tier=pv-storage and the control-plane role). Their placement therefore doesn't depend on when the taints were applied. It is important that these 2 workloads land on the same nodes.
Normal app (stateless, or using a regular Longhorn PVC): add no tolerations. Pods schedule on the general workers. Longhorn's CSI driver runs on every node, so the volume attaches wherever the pod lands.
Database on longhorn-database: you must pin it to storage nodes. This applies to CNPG Cluster, MariaDB CRs and standalone DB Deployments:
nodeSelector:
tier: pv-storage
tolerations:
- key: dedicated-purpose
operator: Equal
value: storage
effect: NoSchedule
Use both parts. Without the toleration the pod can't schedule on a storage node, so it stays Pending or its strict-local volume can't attach. Without the nodeSelector it can also land on a general worker.
Never add a CriticalAddonsOnly toleration to app workloads. The controller doesn't have the memory for them.