We run a server connected to the public internet in the Hetzner cloud for very highly available services designed to be homed on the internet as their primary function. The server is simply named host-1.rachelski.net.
The server is a relatively modest AMD vm. It has 4 CPUs, 8GB of ram, an 80GB hard drive ad a public IPv4 address. This deployment also leverages a Hetzner firewall that only opens ports 80,443, and 22.
Lastly, the Hetzner service has automatic cloud backups turned on that will perform a full VM backup once a day keeping the last 7 backups.
The gitea project named: debian-host-1 holds an ansible playbook for ensuring this host has been properly set up from the base debian OS. This includes a bunch of security hardening along with the automated backups that use kopia to keep an off-site copy of the applications' data. This ansible playbook only handles native applications and does not manage the docker services.
I now have the Hetzner firewall blocking port 22. To compensate, the host is running headscale which is the orchestration server for a tailscale network (tailnet). And the host itself is participating in that tailnet (using the name host-1.rachelski.tailnet). If I must access the SSH port or even key services such as the Keycloak admin app or the traefik dashboard, then the calling machine needs to have the tailnet turned on. Current tailnet rules allow the Caladan laptop to connect to host-1 but not the other way around.
This creates a problem if the headscale service ever stops. So, in this case, the fallback is to go to the Hetzner console and open up port 22 in the firewall. If something ultra bad goes back, the Hetzner console also suports getting access to the host console.
Everything on this host is manged via git repos using infrastructure-as-code practices that are homed within gitea.rachelski.internal. The following projects are in use:
ansible project for managing natively installed apps and servicesThe server has a docker engine installed on it and it runs a few key services that must be reachable from the internet at all times. These applications are as follows:
The first application running in docker is a proxy named Traefik and managed in a similarly named GITEA repo using IaC practices. This traefik proxy is the ONLY service exposed to the internet and listening on ports on the machine. It is handling port 80 (only for Let's Encrypt) and 443 (for proxied services).
To make things even safer, we have a Hetzner firewall installed that is only allowing traffic to ports 80, 443 and 22.
To further protect that SSH port (22), I also installed CrowdSec (see 15-Feb-2025) to reduce the number of attempts being made against that SSH port. Since 23-Oct-2026 CrowdSec also reads traefik's access log and bans IPs that attack the web services (scanners, exploit attempts, password guessing), and the firewall bouncer blocks banned IPs from reaching traefik on 80/443 as well as SSH. Before that, it only watched and protected SSH.
Keycloak is the IAM (Identity and access management) service that performs authentication services for the various applications in the homelab. Running it in a highly available location on the internet improves the service levels. In addition, running against the public name rachelski.net allows me to use this for public services. It would be impractical for a public service to authenticate against an internal IAM.
Architecturally, this is the only service I HAVE to run on the internet. Services that use Keycloak can stay private, because the user's browser carries the login exchange and Keycloak never needs to reach them. Each service, however, MUST be able to reach Keycloak.
Normal logins at https://auth.rachelski.net work from anywhere. The admin console is not reachable from the internet: https://auth.rachelski.net/admin/... and anything under /realms/master return 403 Forbidden (since 23-Oct-2026). To administer Keycloak, you connect through an SSH tunnel to host-1 instead.
What you need: an SSH key that can log in to host-1 as mark, and the master realm admin username, password and OTP app (in BitWarden). If ssh mark@host-1.rachelski.net doesn't log you in, the tunnel won't work either. Sort that out first.
On your own computer, open a terminal and run:
ssh -N -L 8081:$(ssh mark@host-1.rachelski.net "docker inspect -f '{{.NetworkSettings.Networks.proxy.IPAddress}}' keycloak"):8080 mark@host-1.rachelski.net
The terminal will look as if it's hanging. That means the tunnel is open; leave it running.
In your browser, go to http://localhost:8081/admin/ and sign in with the master realm admin account and its OTP code.
When you're done, go back to the terminal and press Ctrl-C to close the tunnel.
What the command does: the part inside $( ) asks host-1 for the Keycloak container's current address on the Docker network. That address changes every time the container is recreated, which is why it's looked up each time. The outer ssh then forwards port 8081 on your computer, through your encrypted SSH connection, to Keycloak's port 8080 on host-1. The connection never goes through traefik, so the 403 block doesn't apply. The address bar shows http://, but everything between your computer and host-1 is encrypted by SSH.
If something goes wrong:
| What you see | What it means |
|---|---|
channel 2: open failed: connect failed: Name or service not known |
The address lookup came back empty. Check that ssh mark@host-1.rachelski.net "docker ps" shows a keycloak container, then try again. Don't split the command into two steps in different terminals. |
bind [127.0.0.1]:8081: Address already in use |
Something else on your computer uses port 8081, maybe an old tunnel you forgot to close. Close it, or change both 8081s to another number such as 8082. |
502 Bad Gateway (or the page won't load) for about 30 seconds |
Keycloak is still starting after a restart or redeploy. Wait and reload. |
| "Account is temporarily disabled" / repeated login failures | Brute-force protection has temporarily locked the admin account after too many wrong passwords. It unlocks by itself within 15 minutes. |
Locked out completely (lost the password or the OTP device): a second master realm admin can unlock or reset the account under Users. As a last resort, Keycloak 26's bootstrap-admin command creates a temporary admin account straight in the database. It needs SSH and Docker access to host-1, but no working Keycloak login. See the Keycloak documentation on recovering admin access.
Headscale is a coordination server for a tailscale network.
We are using it primarily as a safer way to protect the host-1 VM from attack via SSH. It is the primary way to open up a tunnel to host-1 so that we can run ssh and get to the headplane.rachelski.net and auth.rachelski.net/admin hidden services. These are both considered high value targets and should not be exposed directly to the internet for threat actors to try to pick apart.
The Certificate Revocation List server allows applications to trust the root certificate chain managing the rachelski.internal domains. It provides a revocation list of certificates that should not be trusted and the name of this service is embedded into the root certificate.
This is a very simple docker host that runs an nginx container to serve up the revocation list file that was produced by BounCA.
As mentioned above, there are a few things added to this debian server to help to harden it. This includes setting the server up for unattended-security patching and reboot. This is all managed using the ansible configuration management system that organizes settings into idempotent playbooks. This playbook also maintains the following native linux applications
This application is not accessible from the internet, nor is it running as a docker container. I am listing it here for completeness.
This is used to back up the docker environment (at /srv/docker_volumes) remotely to a separate hetzner storagebox named data-barn-1. This is in addition to the native hetzner backups that this server is configured to take. I am doing this in case the server gets cracked and the Hetzner backups become corrupted.
This is an intrusion prevention system that watches verious security logs on the machine and will step in and bounce specific source IP addresses that are considered threats. It does its own monitoring of brute force (similar to Fail2Ban), but it also has a denylist of known bad actors that have been detected by its crowd-sourced users base.
This section is a set of notes representing what was done at the OS level of my cloud server named rachelski-home-1 with DNS name host-1.rachelski.net. Unfortunately, I did not think of keeping this log until after I started building the machine.
The primary runtime for this environment will be the docker runtime. No applications will be installed directly on the host unless they are related to operating or protecting the host itself. This host is currently a pet (maintained by hand rather than rebuilt from automation), so we keep it as close to a stock install as possible. However, certain low-level maintenance is critical.
The host has Hetzner backup services turned on (at extra cost).
This log is focused on changes made directly to the host operating system and does not bother to log changes done to the applications running inside the docker runtime. However, it should capture all changes made to the docker runtime environment itself.
I installed docker on this machine. I have no idea at this point how it was done. I assume this was from following instructions on the docker.com website.
The system has dedicated a section of the host filesystem to hosting persistent storage for docker applications at /srv/docker_volumes/. It will be critical to have good backup mechanisms for the content of this directory. All other parts of the system should be considered disposable.
I also installed traefik on this machine as specified in the traefik Gitea repo. This is my proxy that also sets up let's encrypt certificates so that any web service can be set up behind it with a unique domain name (e.g. headscale.rachelski.net)
Consider installing fail2ban or some other solution on this server. I am getting messages from the traefik log docker logs traefik that say:
2024-07-21T03:28:10Z ERR Error while Peeking first byte error="read tcp 172.18.0.2:80->167.86.108.114:58696: i/o timeout"
I am pretty sure that this is an attacker looking to compromise my machine. They seem to have taken a higher interest on 07-21 with 4 attempts in the same minute.
No action taken at this time, but starting the research.
Watched a video on CrowdSec. This appears to be a distributed intrusion detection system that has agents running on your machines. The agents decide what to block. You then apply bouncers to various parts of your ecosystem (such as fastly or a node's iptables) to stop those attackers. Open source + community contributions of bad actors. Interesting DoS protection using a consensus approach.
TechnoTim has a video on YouTube on how to combine this with traefik at: https://www.youtube.com/watch?v=-GxUP6bNxF0
I have found that the /var/log directory has quite a large log file named btmp along with an even larger btmp.1 rotated file. Further research shows that this represents attackers on the internet attempting to brute force my passwords via SSHD.
mark@rachelski-home-1:/srv/docker_volumes$ sudo ls -l /var/log/
[sudo] password for mark:
Sorry, try again.
[sudo] password for mark:
total 207044
-rw-r--r-- 1 root root 0 Apr 24 2024 alternatives.log
drwxr-xr-x 2 root root 4096 Aug 1 2024 apt
-rw-rw---- 1 root utmp 61578240 Feb 15 09:39 btmp
-rw-rw---- 1 root utmp 150218496 Jan 31 23:59 btmp.1
-rw-r--r-- 1 root adm 98027 Jul 7 2024 cloud-init.log
-rw-r----- 1 root adm 4846 Jul 7 2024 cloud-init-output.log
-rw-r--r-- 1 root root 0 Aug 1 2024 dpkg.log
-rw-r--r-- 1 root root 7938 Jul 7 2024 dpkg.log.1
-rw-r--r-- 1 root root 0 Apr 24 2024 faillog
drwxr-sr-x+ 4 root systemd-journal 4096 Jul 7 2024 journal
-rw-rw-r-- 1 root utmp 292292 Feb 15 08:55 lastlog
drwx------ 2 root root 4096 Jul 7 2024 private
lrwxrwxrwx 1 root root 39 Apr 24 2024 README -> ../../usr/share/doc/systemd/README.logs
-rw-rw-r-- 1 root utmp 54528 Feb 15 08:55 wtmp
mark@rachelski-home-1:/srv/docker_volumes$
You can also review the details using the sudo lastb command, which confirms a number of actors probing the SSHD service
I have decided to deploy the crowdsec application set to defend against this.
This was accomplished with the following commands that were confirmed via the hetzner site.
curl -s https://install.crowdsec.net | sudo sh
mark@rachelski-home-1:/srv/docker_volumes$ curl -s https://install.crowdsec.net | sudo sh
Detected operating system as debian/12.
Detected apt version as 2.6.1
Checking for gpg...
Detected gpg...
Checking for curl...
Detected curl...
Importing packagecloud gpg key...
Packagecloud gpg key imported to /etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg
Installing /etc/apt/sources.list.d/crowdsec_crowdsec.list...
sudo apt update && sudo apt install crowdsec
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
crowdsec
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 72.2 MB of archives.
After this operation, 266 MB of additional disk space will be used.
Get:1 https://packagecloud.io/crowdsec/crowdsec/any any/main amd64 crowdsec amd64 1.6.5 [72.2 MB]
Fetched 72.2 MB in 2s (44.0 MB/s)
Preconfiguring packages ...
Selecting previously unselected package crowdsec.
(Reading database ... 39348 files and directories currently installed.)
Preparing to unpack .../crowdsec_1.6.5_amd64.deb ...
Unpacking crowdsec (1.6.5) ...
Setting up crowdsec (1.6.5) ...
Creating /etc/crowdsec/acquis.yaml
INFO[2025-02-15 09:59:09] crowdsec_wizard: using journald for 'ssh'
Machine 'b9e9e53e9d31402badaf0b03ad8c5ab7vKh2oy13IqtF6hV1' successfully added to the local API.
API credentials written to '/etc/crowdsec/local_api_credentials.yaml'.
Updating hub
Downloading /etc/crowdsec/hub/.index.json
INFO[2025-02-15 09:59:11] crowdsec_wizard: Installing collection 'crowdsecurity/linux'
downloading parsers:crowdsecurity/syslog-logs
downloading parsers:crowdsecurity/geoip-enrich
downloading https://hub-data.crowdsec.net/mmdb_update/GeoLite2-City.mmdb
downloading https://hub-data.crowdsec.net/mmdb_update/GeoLite2-ASN.mmdb
downloading parsers:crowdsecurity/dateparse-enrich
downloading parsers:crowdsecurity/sshd-logs
downloading scenarios:crowdsecurity/ssh-bf
downloading scenarios:crowdsecurity/ssh-slow-bf
downloading scenarios:crowdsecurity/ssh-cve-2024-6387
downloading contexts:crowdsecurity/bf_base
downloading collections:crowdsecurity/sshd
downloading collections:crowdsecurity/linux
enabling parsers:crowdsecurity/syslog-logs
enabling parsers:crowdsecurity/geoip-enrich
enabling parsers:crowdsecurity/dateparse-enrich
enabling parsers:crowdsecurity/sshd-logs
enabling scenarios:crowdsecurity/ssh-bf
enabling scenarios:crowdsecurity/ssh-slow-bf
enabling scenarios:crowdsecurity/ssh-cve-2024-6387
enabling contexts:crowdsecurity/bf_base
enabling collections:crowdsecurity/sshd
enabling collections:crowdsecurity/linux
Run 'sudo systemctl reload crowdsec' for the new configuration to be effective.
downloading parsers:crowdsecurity/whitelists
enabling parsers:crowdsecurity/whitelists
Run 'sudo systemctl reload crowdsec' for the new configuration to be effective.
Created symlink /etc/systemd/system/multi-user.target.wants/crowdsec.service → /lib/systemd/system/crowdsec.service.
Get started with CrowdSec:
* Go further by following our post installation steps : https://docs.crowdsec.net/u/getting_started/next_steps
====================================================================================================================
* Install a remediation component to block attackers: https://docs.crowdsec.net/u/bouncers/intro
====================================================================================================================
* Find more collections, parsers and scenarios created by the community with the Hub: https://hub.crowdsec.net
====================================================================================================================
* Subscribe to additional blocklists, visualize your alerts and more with the console: https://app.crowdsec.net
You can always run the configuration again interactively by using '/usr/share/crowdsec/wizard.sh -c'
sudo cscli collections list
mark@rachelski-home-1:/srv/docker_volumes$ sudo cscli collections list
─────────────────────────────────────────────────────────────────────────────────
COLLECTIONS
─────────────────────────────────────────────────────────────────────────────────
Name ? Status Version Local Path
─────────────────────────────────────────────────────────────────────────────────
crowdsecurity/linux ✔️ enabled 0.2 /etc/crowdsec/collections/linux.yaml
crowdsecurity/sshd ✔️ enabled 0.5 /etc/crowdsec/collections/sshd.yaml
─────────────────────────────────────────────────────────────────────────────────
sudo apt install crowdsec-firewall-bouncer-iptables
sudo cscli bouncers list
mark@rachelski-home-1:/srv/docker_volumes$ sudo cscli bouncers list
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Name IP Address Valid Last API pull Type Version Auth Type
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
cs-firewall-bouncer-1739613769 127.0.0.1 ✔️ 2025-02-15T10:18:26Z crowdsec-firewall-bouncer v0.0.31-debian-pragmatic-amd64-4b99c161b2c1837d76c5fa89e1df83803dfbcc87 api-key
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
To see currently banned IP addresses:
sudo ipset list crowdsec-blacklists-0
At this moment, I have not integrated crowdsec with traefik.
I installed the rachelski.internal root CA into the trust store for this host using the commands:
sudo mv rachelski-internal-root-ca.pem /usr/local/share/ca-certificates/rachelski-internal-root-ca.crt
sudo update-ca-certificates
Added Kopia in order to perform backups.
curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /etc/apt/keyrings/kopia-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kopia-keyring.gpg] http://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list
sudo apt update
sudo apt install kopia
After this, I created a local SSH key for the mark@host-1.rachelski.net user and registered it in the sub2 subaccount on the Hetzner storagebox.
ssh-keygen -t rsa -b 4096
cat ~/.ssh/id_rsa.pub | ssh -p 23 u465088-sub2@u465088.your-storagebox.de install-ssh-key
Then I set up an SFTP repository on that storagebox from host-1.
mkdir .cache/kopia/cache
kopia repository create sftp \
--host=u465088.your-storagebox.de \
--port=23 \
--username=u465088-sub2 \
--path=host-1.rachelski.net \
--keyfile=/home/mark/.ssh/id_rsa \
--known-hosts=/home/mark/.ssh/known_hosts \
--external \
--ssh-args="-i /home/mark/.ssh/id_rsa" \
--cache-directory="/home/mark/.cache/kopia/cache" \
--content-cache-size-limit-mb=5242880000 \
--description="hetzner-data-barn-1_host-1.rachelski.net" \
--max-list-cache-duration=3000s \
--metadata-cache-size-limit-mb=5242880000
The resulting repository password was stored in BitWarden under the title kopia - host-1.rachelski.net repository
After this, I wanted to set up snapshots against /srv/docker_volumes but I fully expected them to fail as many of the docker volumes are owned by 'root'. I established the snapshot using the mark@host-1.rachelski.net account first and then attempted to set up subsequent snapshots using root via the crontab
kopia snapshot create /srv/docker_volumes
As I expected, kopia had issues iterating through the volumes:
mark@rachelski-home-1:~$ kopia snapshot create /srv/docker_volumes
Snapshotting mark@rachelski-home-1:/srv/docker_volumes ...
WARN Unable to estimate
/ 1 hashing, 3 hashed (7.5 KB), 0 cached (0 B), uploaded 200 B, estimating...
! Error when processing "identity/data": cannot create iterator: unable to read directory: open /srv/docker_volumes/identity/data: permission denied
- 1 hashing, 4 hashed (7.5 KB), 0 cached (0 B), uploaded 200 B (1 fatal errors), estimating...
! Error when processing "headscale/data/noise_private.key": unable to open file: unable to open local file: open /srv/docker_volumes/headscale/data/noise_private.key: permission denied
\ 1 hashing, 21 hashed (9.9 MB), 0 cached (0 B), uploaded 200 B (2 fatal errors), estimating...
! Error when processing "headscale/data/private.key": unable to open file: unable to open local file: open /srv/docker_volumes/headscale/data/private.key: permission denied
* 0 hashing, 27 hashed (10 MB), 0 cached (0 B), uploaded 200 B (3 fatal errors), estimating...
Created snapshot with root k338ff37f13eb06a8a1c51bc4c77f7a35 and ID 54bb8b0043f834fe7acb9d58dbd8e71c in 0s
ERROR Found 3 fatal error(s) while snapshotting mark@rachelski-home-1:/srv/docker_volumes.
But it established the snapshot location:
mark@rachelski-home-1:~$ kopia snapshot list
mark@rachelski-home-1:/srv/docker_volumes
2025-08-17 04:55:30 UTC k338ff37f13eb06a8a1c51bc4c77f7a35 10 MB drwxrwxrwx files:25 dirs:17 errors:3 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)
My real intent was to run the nightly backups as root so it had access to the entire filesystem. Unfortunately, it was at this point when I ran into a nasty problem. I did not have the root password for this server. I went back into my email but was unable to easily find the email notifying me of setup of this server size CX32. I think this means that the server does not have a root password because I set it up initially with an SSH key. So, to get to the root account, I will need to 'reset the root password'. Not having the root password is actually preferable, as it is one less thing a hacker can locate. My mark@host-1.rachelski.net account appears to be able to do 99% of everything I need. BUT it appears to be a problem in my backup strategy. I don't think doing a sudo will replicate the automated backup I need.
I did a general security walkthrough of the host (with Claude's help). Most of the defences were already in good shape: SSH password logins are off, the Hetzner firewall only lets in 22/80/443, unattended-upgrades is running, and CrowdSec is banning SSH attackers. The walkthrough found some gaps, and this entry records the fixes made today: the OS-level changes, locking down the Keycloak admin console, and connecting CrowdSec to traefik. The remaining clean-up items are in my remediation plan (~/projects/cloud/debian-host-1/REMEDIATION.md on my workstation) and will get their own entries here when they are done.
One finding is deliberately deferred: traefik has the Docker socket mounted, so anyone who compromised traefik could take over the whole host. The likelihood is low, but I'd rather traefik didn't control Docker at all. I plan a separate project to move traefik's routes from Docker labels to its own config files, possibly restructuring the traefik-related repos at the same time.
The Hetzner backups remain the fallback for this server. Kopia (16-Aug-2025) never got past the first snapshot, so it should not be counted as a backup.
The server had been up for 838 days. Unattended-upgrades was installing new kernels, but nothing ever rebooted into them, so the running kernel (6.1.0-20) was about two years behind the installed one (6.1.0-54). I took a Hetzner snapshot, then ran:
sudo apt full-upgrade
sudo reboot
Note that reboot needs sudo; without it, nothing happens. The server came back in about a minute, and all containers restarted on their own (they all use restart: unless-stopped). This upgrade also took Docker from 28 to 29 and CrowdSec from 1.6.11 to 1.8.1.
To stop this happening again, unattended-upgrades now reboots by itself when an update needs it. The setting is in its own file, so Debian's 50unattended-upgrades stays untouched and package upgrades don't stop to ask about it:
sudo tee /etc/apt/apt.conf.d/52unattended-upgrades-local >/dev/null <<'EOF'
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
Unattended-Upgrade::Automatic-Reboot-Time "20:00";
EOF
The server's clock is UTC, so 20:00 is 03:00 Bangkok time. Expect a short outage of all host-1 services (including Keycloak logins) around then, but only on nights after an update that needs a reboot. To check the settings: apt-config dump | grep -i Automatic-Reboot.
The CrowdSec firewall bouncer only hooked the INPUT chain. Docker forwards traffic for 80/443 to the traefik container through the FORWARD chain, so it never passed through INPUT, and IPs banned by CrowdSec could still reach traefik. I added Docker's DOCKER-USER chain to the list in /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml:
iptables_chains:
- INPUT
- DOCKER-USER
and restarted the bouncer with sudo systemctl restart crowdsec-firewall-bouncer. To confirm it is working, sudo iptables -S DOCKER-USER should include -A DOCKER-USER -j CROWDSEC_CHAIN. (Docker 29 no longer adds a -j RETURN line at the end of that chain; that's expected.)
The package upgrade in "Quick Notes" replaced this file with the maintainer's version, so if a future upgrade asks about it again, check that DOCKER-USER is still in the list afterwards.
The root account accepted SSH logins with the same key as mark, which meant that key gave root directly and skipped the sudo password. SSH settings now live in a separate file:
sudo tee /etc/ssh/sshd_config.d/10-hardening.conf >/dev/null <<'EOF'
PermitRootLogin no
X11Forwarding no
MaxAuthTries 3
EOF
sudo sshd -t && sudo systemctl reload ssh
sudo rm /root/.ssh/authorized_keys
sshd_config includes sshd_config.d/*.conf at the top, and sshd uses the first value it sees, so these settings win over anything later in the main file.
I also replaced my old RSA-2048 login key with an ed25519 key (~/.ssh/id_ed25519_host1 on my workstation, with a Host host-1 entry in ~/.ssh/config), and removed the RSA key from mark's authorized_keys. Logging in now looks like:
ssh host-1
If I am ever locked out of SSH, these all still work, because none of them use SSH:
qemu-guest-agent running on the host;Correction to 16-Aug-2025: the walkthrough showed root does have a password (set on 07-Jul-2024 when the server was created). If I don't know it, I should reset it from the Hetzner panel and store it in BitWarden before I need it.
SSH sessions to host-1 already use a post-quantum key exchange (sntrup761x25519-sha512), so recorded traffic can't be decrypted later by a quantum computer. The login keys themselves are not post-quantum yet. OpenSSH 9.2 on Debian 12 doesn't support post-quantum login keys, so I'll revisit this after the Debian 13 upgrade.
This is an application change rather than an OS one, but it changes how the server is administered, so it's logged here.
The traefik route for auth.rachelski.net sent every path to Keycloak, so the admin console (/admin/master/console/) and the master realm's login pages were open to the whole internet. That let anyone try passwords against the most powerful account on the server, and would expose us immediately to any future Keycloak admin vulnerability.
The fix is in the identity Gitea repo (docker-compose/identity/docker-compose.yaml). A second, higher-priority traefik router matches /admin and /realms/master and passes them through an IP allowlist containing only 127.0.0.1. No internet client has that address, so they all get 403. Everything else, including the rachelski-net realm that the apps log in through, still goes through the normal router. Admin access is now only through an SSH tunnel; see "Getting to the Keycloak admin console" in the Keycloak section above.
I also tightened the master realm itself:
I left brute-force lockout off for the rachelski-net realm. Its login page has to be public, and Keycloak locks accounts, not attackers' IPs, so anyone could lock household members out on purpose. Password guessing there is handled by CrowdSec reading traefik's logs instead (see below), which bans the attacker's IP.
After any redeploy of the identity stack, expect 502 from auth.rachelski.net for about 30 seconds while Keycloak starts.
This finishes what I started on 15-Feb-2025 ("I have not integrated crowdsec with traefik"). Until today, CrowdSec only read the SSH logs, so web attacks against traefik went unnoticed.
Two changes were needed. First, traefik now writes an access log: one line per request, to its standard output. This is in the traefik Gitea repo, data/traefik.yml:
accessLog: {}
Second, CrowdSec on the host now reads that log straight from Docker. I installed the traefik collection, which also pulls in the generic HTTP attack scenarios and the HTTP CVE scenarios:
sudo cscli collections install crowdsecurity/traefik
sudo tee /etc/crowdsec/acquis.d/traefik.yaml >/dev/null <<'EOF'
source: docker
container_name:
- traefik
labels:
type: traefik
EOF
sudo systemctl reload crowdsec
CrowdSec follows the container by name and listens for Docker events, so it reconnects automatically when traefik is recreated on a redeploy. Banned IPs are blocked by the same firewall bouncer as before. Because that now also covers Docker's DOCKER-USER chain (see above), a ban keeps the attacker away from both SSH and the websites.
I deliberately did not install the CrowdSec traefik plugin (from the TechnoTim video in 22-Jul-2024). The firewall bouncer already blocks banned IPs, and the plugin would run third-party code inside traefik, which is the most sensitive container on this host.
Checking it's working: sudo cscli metrics show acquisition should list a docker:traefik source with lines read and parsed. It appears only after traefik has logged at least one request since CrowdSec was last reloaded, so load any https://auth.rachelski.net page first.
If someone in the household gets banned (CrowdSec bans by IP, so this blocks everyone behind that IP, including from SSH):
sudo cscli decisions list # what is banned, and why
sudo cscli decisions delete --ip <their IP> # lift a ban
If it's my IP that's banned, SSH to host-1 is blocked too, so use the Hetzner Cloud console to run these.