The 2:14 AM Wake-Up Call
Last Tuesday, my phone started screaming at 2:14 AM. I dragged myself to the desk, squinting at the terminal as logs from a production jump host flooded the screen. A botnet had sniffed out a temporary SSH port I’d opened for a contractor. Within minutes, /var/log/auth.log was bloating at 50 login attempts per second. Even with public key authentication enforced, watching a production system endure that kind of aggressive probing is enough to ruin any sysadmin’s night.
Changing the default port from 22 to something high like 49221 is a common first step. But let’s be realistic: tools like Masscan can scan the entire IPv4 space at 25 million packets per second. Security through obscurity is a brittle strategy that snaps at the first sign of a determined scanner. I didn’t just want to move the port. I wanted it to be invisible—completely non-existent to the public internet, yet instantly accessible when I needed it.
The Core Issue: Digital Beacons
Standard network services require a port to be ‘Open’ or ‘Listening’ to accept connections. This state is a beacon. Even if you use Fail2Ban to block IPs after three failed attempts, the port still responds to a SYN packet. This tells an attacker exactly which service you’re running. If a zero-day vulnerability like ‘regreSSHion’ (CVE-2024-6387) drops, a visible port becomes a liability before you can even finish your first cup of coffee and run apt upgrade.
Comparing the Walls: Port Knocking vs. SPA
I started by evaluating my team’s existing baseline. Key Auth and Fail2Ban are great, but they don’t hide the service. I looked at Port Knocking first. It’s a classic trick where the firewall stays closed until it detects a specific sequence of connection attempts—hitting port 7000, then 8000, then 9000. Only then does it open port 22 for your IP.
The catch? Port knocking is vulnerable to packet sniffing. If an attacker is on your network path, they can see the ‘secret knock’ and replay it to gain access. It’s also famously finicky on unstable mobile networks where packets often arrive out of order, locking you out of your own server.
I eventually landed on Single Packet Authorization (SPA). Unlike knocking, SPA uses a single, encrypted, non-replayable packet. It doesn’t rely on a sequence of connection attempts. Instead, a background daemon monitors the wire via libpcap, looking for a specifically formatted UDP packet. Once detected, the daemon decrypts the packet, verifies the signature, and dynamically updates the firewall (iptables/nftables) for your specific IP for a few seconds.
The Solution: Implementing fwknop
I chose fwknop (FireWall Knock Operator) because it is the industry standard for SPA. It utilizes Rijndael (AES) or GnuPG encryption, making it virtually impossible for an attacker to spoof or replay the authorization trigger.
1. Server-Side Configuration
I started by installing the server component on an Ubuntu host. My first priority was ensuring the default policy for the INPUT chain was set to DROP for port 22.
sudo apt update
sudo apt install fwknop-server iptables-persistentThe configuration requires two keys: a strong encryption key and a Hash-based Message Authentication Code (HMAC) key. For these, I used the generator at toolcraft.app/en/tools/security/password-generator. It runs locally in the browser, so no sensitive data ever hits the network. I generated two 64-character strings for the setup.
Next, I defined the access rules in /etc/fwknop/access.conf:
SOURCE ANY
OPEN_PORTS tcp/22
KEY YOUR_LONG_ENCRYPTION_KEY
HMAC_KEY YOUR_LONG_HMAC_KEY
FW_ACCESS_TIMEOUT 30Finally, I configured /etc/fwknop/fwknopd.conf to listen on the correct interface, typically eth0 or ens3:
PCAP_INTF eth02. Client-Side Setup
On my local machine, I installed the fwknop client and saved my credentials to avoid repetitive typing.
# On your local machine
fwknop -A tcp/22 -D [Server_IP] --key-gen --use-hmac --save-rc-stanzaThis generates a ~/.fwknoprc entry. I manually updated it with the 64-character keys created earlier.
3. Testing the Cloak
With fwknopd running and the firewall dropping port 22 traffic, I attempted a standard SSH connection. It hung indefinitely. Perfect. To a scanner, the server appeared dead.
Then, I sent the SPA packet:
fwknop -n my-production-serverThe client confirmed transmission. Within milliseconds, fwknopd intercepted the UDP packet (usually on port 62201), verified the keys, and injected a temporary iptables rule for my current public IP.
ssh user@server_ipI was in. The /var/log/auth.log stayed silent. No brute force noise, no scanners, just a clean connection.
Operational Security in Practice
Switching to SPA made a night-and-day difference in my on-call stress levels. Two lessons stood out. First, always maintain a secondary ’emergency’ access path, such as a cloud provider console, in case the fwknopd process fails. Second, remember that SPA is an extra layer of armor, not a replacement for SSH keys. I still use Ed25519 keys with a strong passphrase. The difference is that now my SSH gateway is a ‘black hole’ to the internet until I decide to let myself in.
