Posted inSecurity

SSH Two-Factor Authentication with Google Authenticator PAM on Linux

Security tutorial - IT technology blog
Security tutorial - IT technology blog

Real-World Problem: When SSH Keys Are No Longer Enough

At 2:14 AM on a Tuesday last October, our on-call PagerDuty alarm woke me up. A junior developer’s laptop had caught an infostealer malware payload through a typo-squatted npm dependency in a weekend side project. Within 90 seconds, the malware harvested everything on the system: active browser cookies, AWS session tokens, and an unpassphrased 256-bit Ed25519 private key sitting unprotected in ~/.ssh/id_ed25519.

Eighteen minutes later, automated bot scripts connected to three staging nodes and our primary production jump box (10.0.1.15). Our infrastructure ran standard baseline hardening: password logins were off, non-standard SSH ports were monitored, and root logins were disabled. Yet not a single alert tripped. The incoming connection presented a valid, cryptographically correct key signature. To the SSH daemon, the attacker looked like a trusted engineer.

This failure pattern repeats across engineering organizations every week. Teams treat SSH public-key authentication like an unbreachable fortress. But when a private key leaks from a lost laptop, an accidental public GitHub push, or a compromised endpoint, that fortress turns into an open door.

Root Cause Analysis: Single-Factor Vulnerability in Asymmetric Keys

Most engineering teams classify SSH keys as multi-factor auth by default. They assume keys satisfy “something you have” (the private key file) and “something you know” (the passphrase). In production environments, this assumption regularly breaks down:

  • Unencrypted keys on disk: Engineers frequently skip setting a passphrase. They do this to avoid typing passwords across dozens of daily git operations, automated deployment scripts, and new tmux panes.
  • Long-lived SSH agents: Even with strong passphrases, background utilities like ssh-agent or GNOME Keyring keep decrypted keys in RAM for hours or days. An attacker who gains low-privilege execution on your laptop can simply point to the active SSH_AUTH_SOCK environment variable and hijack your keys without ever knowing your passphrase.
  • Zero client context verification: When sshd validates a cryptographic signature, it checks mathematical alignment against authorized_keys and nothing else. The daemon cannot tell whether that signature originated from a senior sysadmin or a bot executing inside a stolen session socket.

Relying on a single static file stored on a client filesystem creates an architectural single point of failure. If an endpoint gets compromised, every server trusting that key is compromised too.

Evaluating Remediation Strategies: Hardware Tokens vs. TOTP PAM

Closing this security gap requires a server-side second factor before opening an interactive shell. Two realistic patterns solve this issue:

Approach A: Hardware Security Keys (FIDO2 / U2F)

Modern OpenSSH (version 8.2 and newer) natively supports FIDO2 security keys via key types like ed25519-sk. Each signing operation demands a physical capacitive tap on a USB hardware device like a YubiKey 5 series.

  • Pros: Phishing-resistant architecture, zero private key exportability from silicon, and zero server-side PAM module dependencies.
  • Cons: Costly physical rollout ($45–$85 per key across 50+ developers), shipping bottlenecks for remote hires, complex account recovery when keys get lost, and broken compatibility on legacy enterprise systems running OpenSSH < 8.2 (such as CentOS 7).

Approach B: Time-Based One-Time Passwords (TOTP) via PAM

Linux relies on Pluggable Authentication Modules (PAM) to coordinate authentication handshakes. By plugging the open-source Google Authenticator PAM module into your SSH pipeline, the server enforces a dynamic, time-limited 6-digit TOTP code. Users generate these codes through mobile authenticator apps like Aegis, 1Password, Bitwarden, or Google Authenticator.

  • Pros: Zero hardware procurement costs, universal compatibility across every major Linux distribution, and deployment across a 100-person team in under an hour using standard configuration management.
  • Cons: Vulnerable to real-time man-in-the-middle reverse proxies when used as a lone factor (a risk mitigated entirely when paired with SSH key authentication).

The Best Approach: Enforcing Multi-Factor Authentication (SSH Key + TOTP)

For standard cloud VMs and bare-metal nodes, pairing SSH keys with Google Authenticator PAM delivers the best defense-in-depth setup. An intruder holding a stolen private key hits a concrete wall because they lack your phone’s physical TOTP generator.

Strong fallback credentials remain essential for out-of-band console access like AWS EC2 Serial Console or Hetzner KVM. For generating unpredictable emergency root credentials, browser-based offline generators like toolcraft.app/en/tools/security/password-generator keep passwords off the wire. For day-to-day SSH access, we will require both the cryptographic key and the rotating 6-digit TOTP code.

Step 1: Install the PAM Module

Install the Google Authenticator PAM module package through your package manager. On Ubuntu (22.04 / 24.04 LTS) and Debian systems:

sudo apt update
sudo apt install -y libpam-google-authenticator

For RHEL 9, Rocky Linux, or AlmaLinux, pull the binary from EPEL:

sudo dnf install -y epel-release
sudo dnf install -y google-authenticator

Step 2: Generate the User Secret and Recovery Codes

Switch to the non-root developer user account requiring SSH access. Run the setup wizard directly in your shell:

google-authenticator

Walk through the interactive prompts using these hardened configuration options:

  • Do you want authentication tokens to be time-based (y/n)? Enter y to enforce standard RFC 6238 time-based tokens.
  • The tool will render an ASCII QR code in your terminal alongside a base32 secret key, an initial verification code, and five 8-digit emergency scratch codes. Scan the QR code into your authenticator app. Store the 5 emergency codes in an offline corporate password manager immediately.
  • Do you want me to update your “~/.google_authenticator” file (y/n)? Enter y to write the state to disk with 0400 permissions.
  • Do you want to disallow multiple uses of the same authentication token (y/n)? Enter y. This rejects token replays within the 30-second window.
  • By default, tokens are good for 30 seconds… Increase window (y/n)? Enter n. Keep the clock skew check strictly bounded to 3 possible codes (±1 window, 90 seconds total) to prevent wide replay gaps.
  • Do you want to enable rate-limiting (y/n)? Enter y. This caps failed attempts to 3 logins every 30 seconds, stopping automated brute-force attacks in their tracks.

Step 3: Configure PAM for SSH

Instruct Linux PAM to challenge interactive logins for TOTP codes. Open /etc/pam.d/sshd in your editor:

sudo nano /etc/pam.d/sshd

Locate the standard password authentication directive. On Debian/Ubuntu, look for @include common-auth. On RHEL derivatives, look for auth substack password-auth. Comment that directive out with a # so users bypass traditional Unix passwords. Then add the authenticator directive to the bottom of the file:

# @include common-auth
auth required pam_google_authenticator.so nullok

The nullok argument acts as a temporary safety net during company rollouts. Users who have not yet run google-authenticator can still log in using keys alone. Once all team accounts complete Step 2, delete nullok to enforce MFA across every account on the server.

Step 4: Update OpenSSH Daemon Settings

Next, configure OpenSSH to enforce both validation stages sequentially. Open your daemon configuration file at /etc/ssh/sshd_config (or create a dedicated file under /etc/ssh/sshd_config.d/mfa.conf):

sudo nano /etc/ssh/sshd_config

Review and apply these parameters:

# Enable PAM and interactive challenge mechanics
UsePAM yes
KbdInteractiveAuthentication yes

# Keep plaintext passwords disabled
PasswordAuthentication no

# Enforce multi-factor chain: SSH Key FIRST, then TOTP prompt
AuthenticationMethods publickey,keyboard-interactive

Pay close attention to AuthenticationMethods publickey,keyboard-interactive. This comma-separated syntax is critical: it requires both checks in strict order. OpenSSH must validate the client’s public key first. Only after the key check succeeds will the server trigger the PAM keyboard-interactive TOTP prompt.

Step 5: Test Without Locking Yourself Out

Do not close your current terminal session. First, run an OpenSSH configuration syntax check to spot typo regressions:

sudo sshd -t

If that command returns exit code 0 without output, safely reload or restart the service:

sudo systemctl restart sshd

Keep your existing terminal window active as an emergency lifeline. Open an entirely new terminal window on your local machine and initiate a verbose connection test:

ssh -v username@your-server-ip

Watch the verbose debug output. You will see public key exchange complete, followed immediately by OpenSSH’s partial authentication response:

Authenticated with partial success.
Verification code:

Punch in the current 6-digit TOTP number from your phone. You will drop straight into your bash shell. If an attacker exfiltrates your SSH private key tomorrow, their automated connection halts abruptly at the verification prompt—dead in the water without your phone.

Share:
Tags: