The primary secret store for the homelab is a deployment of Hashicorp Vault. This deployment is responsible for the following functions:
secrets-k8s-ranchuserpass access method for login to the Vault web consoleThere may be other functions added in the future and I hope to remember to add them here.
At the beginning, this Vault was set up to run on the Rachelski NAS. It does not need to remain there as there is a DNS record named hcl-vault.rachelski.internal. But the NAS is a reasonably foundational component of the network, so Vault remains there for now.
The project used to initialize the instance of vault can be found in the Gitea project named hcl-vault. This project simply contains the instructions to execute a rebuild. However, given the fact that this is a stateful data store, it is preferred to use backups and restore to recover. A rebuild from scratch would lose the encrypted content.
Everything uses https://hcl-vault.rachelski.internal. The NAS reverse proxy terminates TLS on port 443 and forwards to Vault. The name is never combined with a port.
Vault's own port, 8200 (plain HTTP), is only published on the NAS's loopback interface, so it can't be reached from the LAN. It is the way in when the normal route is broken: DNS is failing, or the NAS is serving an expired *.rachelski.internal certificate (Vault issues that certificate, so this can happen). Tunnel to it over SSH by IP address:
ssh -L 8200:127.0.0.1:8200 admin@192.168.200.155
# in another terminal
VAULT_ADDR=http://127.0.0.1:8200 vault status
The content of this vault is being backed up using the standard mechanisms for the NAS.
Kopia's copy of /volume3/docker/hcl-vault is taken while Vault is running, so it may not be consistent. A nightly Raft snapshot, together with the unseal keys, is also kept encrypted in Google Drive. To restore Vault after losing the NAS, follow NAS Disaster Recovery.
It is preferred that all basic capabilities for Vault are managed in this initialization project. This project enables capabilities, but it does not actually manage any content 'in' the vault. This project is responsible for:
Other projects are allowed to add content to Vault using these structural primitives.
Note: it is appropriate for a new AppRole role representing an application to also set up the access rights for that role.