Mastering ssh-agent and SSH Key Forwarding on Linux: Manage Multiple Keys and Chain Servers Securely

The Problem: Too Many Keys, Too Many Prompts

If you’ve managed more than two Linux servers, you know the pain. One SSH key for GitHub, another for production, one for staging, maybe one for a client’s VPS. Every connection either asks for a passphrase — or worse, you’re reusing the same key everywhere and hoping nothing gets compromised.

Engineers deal with this differently. The approaches vary enough that picking blindly will usually cost you later.

Four Ways to Handle Multiple SSH Keys

Approach 1: One Key for Everything

The simplest setup — one key pair, used for all servers and services. Most people start here because it requires zero configuration.

Approach 2: Per-Host Keys with SSH Config

Create separate key pairs per server or service, then specify which key to use per host inside ~/.ssh/config:

Host github.com
    IdentityFile ~/.ssh/github_ed25519

Host production
    HostName 203.0.113.10
    User ubuntu
    IdentityFile ~/.ssh/prod_ed25519

Approach 3: ssh-agent with Key Loading

Start an ssh-agent process, load your keys into it once (entering passphrases only that one time), and SSH reuses them automatically for the rest of your session.

Approach 4: ssh-agent + SSH Key Forwarding

The most capable setup. Your agent runs locally. When you SSH into a bastion or jump host, the connection forwards your local agent socket to that machine — so you can authenticate all the way into internal servers without a private key ever touching the remote disk.

Pros and Cons

Approach 1: One Key for Everything

Pro: Zero setup, works immediately
Con: One compromised key exposes access to every server
Con: If you skip the passphrase for convenience, your private key on disk is unprotected

Approach 2: Per-Host Keys with SSH Config

Pro: Better isolation — compromise one key and the others remain safe
Pro: Clean, readable config that self-documents which key goes where
Con: You still re-enter passphrases every session, per key
Con: Useless when you need to jump from server A to server B without copying private keys onto A

Approach 3: ssh-agent Only

Pro: Type each passphrase once per session instead of once per connection
Pro: Private keys stay encrypted on disk — the agent holds them in memory only
Con: Agent dies when the terminal closes unless you persist it
Con: Still can’t chain server hops without putting keys on intermediate machines

Approach 4: ssh-agent + SSH Key Forwarding

Pro: Private keys never leave your local machine, even in multi-hop scenarios
Pro: One terminal, the whole chain: laptop → bastion → internal server → database
Con: The bastion host must be trusted — a compromised root user on it could momentarily hijack the forwarded socket
Con: A bit more initial setup than the other approaches

Recommended Setup

For most engineers managing a handful of servers, Approach 4 is worth the setup time. Per-host keys loaded into ssh-agent, with forwarding enabled only on specific bastion hosts you control.

Keep Approach 2 (SSH config with IdentityFile) as your foundation. Layer ssh-agent on top for passphrase convenience. Enable agent forwarding only where you explicitly need it — never globally.

On my production Ubuntu 22.04 server (4GB RAM), the difference with Ansible was immediate. Playbooks connecting to 10+ internal nodes through a single bastion used to trigger passphrase prompts per host — or required unencrypted keys sitting on the bastion itself. With agent forwarding, the whole playbook runs unattended. The private key never touches the remote filesystem.

Implementation Guide

Step 1: Generate Separate Keys per Purpose

Use Ed25519 — it’s faster and cryptographically stronger than RSA for most use cases:

# Key for GitHub
ssh-keygen -t ed25519 -C "github" -f ~/.ssh/github_ed25519

# Key for production servers
ssh-keygen -t ed25519 -C "prod-servers" -f ~/.ssh/prod_ed25519

# Key for a client VPS
ssh-keygen -t ed25519 -C "client-vps" -f ~/.ssh/client_ed25519

Always set a strong passphrase when prompted. Without it, the key file on disk is just a text file waiting to be stolen.

Step 2: Start ssh-agent

GNOME and KDE desktops usually start an SSH agent automatically with your session. Check first:

echo $SSH_AUTH_SOCK
# Should print something like: /run/user/1000/keyring/ssh

If the variable is empty, start it manually:

eval "$(ssh-agent -s)"
# Agent pid 12345

To auto-start in every shell session, add this to ~/.bashrc or ~/.zshrc:

if [ -z "$SSH_AUTH_SOCK" ]; then
    eval "$(ssh-agent -s)"
fi

Step 3: Add Keys to the Agent

ssh-add ~/.ssh/github_ed25519
# Enter passphrase: (type once — done for the session)

ssh-add ~/.ssh/prod_ed25519
ssh-add ~/.ssh/client_ed25519

# Confirm what's loaded
ssh-add -l
# 256 SHA256:abc123... github (ED25519)
# 256 SHA256:def456... prod-servers (ED25519)

On shared or less-trusted machines, add a time limit. The key auto-removes after the duration expires:

# Auto-remove after 4 hours (14400 seconds)
ssh-add -t 14400 ~/.ssh/prod_ed25519

Step 4: Configure ~/.ssh/config

Wire keys to hosts and enable forwarding only for trusted bastion hosts:

Host github.com
    IdentityFile ~/.ssh/github_ed25519
    AddKeysToAgent yes

Host bastion
    HostName 203.0.113.5
    User ubuntu
    IdentityFile ~/.ssh/prod_ed25519
    ForwardAgent yes        # Enable ONLY for this trusted host
    AddKeysToAgent yes

Host internal-db
    HostName 10.0.1.20
    User ubuntu
    ProxyJump bastion       # Reach via bastion — no IdentityFile needed here

Host client-vps
    HostName 198.51.100.10
    User root
    IdentityFile ~/.ssh/client_ed25519
    AddKeysToAgent yes
    # ForwardAgent intentionally omitted — don't forward to external hosts

Step 5: Test the Chain

With this config, reaching your internal database from your laptop is a single command:

ssh internal-db

Here’s what SSH actually does:

Connects to the bastion using your local prod key via the forwarded agent
Opens a second connection from bastion to 10.0.1.20
The internal server authenticates against your forwarded local agent
Your private key file never touched the bastion disk

To verify forwarding is active on the remote side:

# After connecting to bastion:
echo $SSH_AUTH_SOCK
# /tmp/ssh-XXXXXX/agent.NNNN  ← forwarded socket is present

ssh-add -l
# Lists your keys from your local machine

Step 6: Persist the Agent Across Terminal Sessions

One remaining annoyance: close your terminal and the agent dies, taking your loaded keys with it. The keychain utility fixes this by keeping a single agent alive across sessions:

sudo apt install keychain

# Add to ~/.bashrc:
eval "$(keychain --eval --quiet ~/.ssh/prod_ed25519 ~/.ssh/github_ed25519)"

Your first login after a reboot asks for passphrases once. Every terminal you open after that — new tabs, reconnections, tmux panes — reuses the same running agent. No more prompts.

Security Checklist Before You Go Live

Set ForwardAgent yes only on bastion hosts you fully control
Never add ForwardAgent yes to a wildcard Host * block
Use ssh-add -t <seconds> for keys loaded on shared machines
Audit what’s currently loaded with ssh-add -l before sensitive operations
Restrict bastion /tmp permissions and disable root SSH login on it

The actual risk with agent forwarding isn’t key copying — your private key bytes genuinely stay local. The exposure is the socket itself. A root user on the bastion can use that socket temporarily while your SSH session is open. Treat your bastion host with the same discipline you’d apply to your own laptop.

Set this up once and authentication stops being something you think about. Type ssh internal-db and you’re in. Your keys stay on your local machine where they belong.