Self-Hosted Mail Server with Mailcow on Docker: SMTP, IMAP, Anti-Spam, and Webmail for Your Own Domain

HomeLab tutorial - IT technology blog
HomeLab tutorial - IT technology blog

Why I Stopped Paying for Google Workspace

Running email for a personal domain used to mean one thing: Google Workspace, $6/user/month, forever. For a couple of personal domains and a HomeLab I maintain on the side, that adds up fast — and the data-ownership story never sat right with me.

After spending a weekend with Mailcow, I haven’t looked back. It handles SMTP, IMAP, anti-spam (rspamd), DKIM, DMARC, SPF, and ships a full Webmail interface (SOGo). Everything runs as Docker containers, so it slots cleanly into a HomeLab already running Portainer and a stack of other self-hosted services.

Owning your mail stack sounds intimidating. Email has a well-earned reputation for being painful to self-host. But Mailcow changes the equation — it wraps Postfix, Dovecot, rspamd, and a half-dozen supporting services into a single Compose file and handles the hard parts for you.

What you need going in:

  • A VPS or bare-metal server with at least 2 vCPU / 4 GB RAM (Mailcow’s stated minimum; 2 GB technically works but swap will get hammered)
  • A domain you control (you’ll need to set DNS records)
  • Docker + Docker Compose installed
  • Ports 25, 80, 110, 143, 443, 465, 587, 993, 995 open in your firewall

Port 25 is the painful one. Most residential ISPs and many VPS providers — including Hetzner by default — block outbound port 25 to cut down on spam. Check this before you start. If your VPS blocks it, open a support ticket or switch providers. There’s no workaround that keeps you off blocklists.

Installation

DNS Setup First — Always

Touch DNS before you touch Docker. Mailcow needs a hostname with a valid PTR record (reverse DNS). Set your VPS’s PTR to match your mail hostname, e.g. mail.yourdomain.com. Then add these records in your DNS panel:

# Replace yourdomain.com and your server IP
mail.yourdomain.com.   A     203.0.113.10
yourdomain.com.        MX    10  mail.yourdomain.com.
yourdomain.com.        TXT   "v=spf1 mx ~all"
_dmarc.yourdomain.com. TXT   "v=DMARC1; p=quarantine; rua=mailto:[email protected]"

DKIM keys get generated after Mailcow starts — you’ll add that TXT record in a later step.

Clone and Configure Mailcow

git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized
./generate_config.sh

The script asks for your mail hostname (mail.yourdomain.com) and timezone. It writes a mailcow.conf file. Open it and tweak a few defaults before first boot:

# mailcow.conf — key settings to review
MAILCOW_HOSTNAME=mail.yourdomain.com
TZ=Asia/Tokyo

# Skip ClamAV if RAM is tight (saves ~700 MB)
SKIP_CLAMD=y

# Optional: external Acme/Let's Encrypt skip if you handle certs yourself
# SKIP_LETS_ENCRYPT=y

ClamAV is nice but heavy. On a 4 GB server I keep it disabled and lean on rspamd alone — rspamd’s ClamD integration can always be switched back on later if you add more RAM.

Start the Stack

docker compose pull
docker compose up -d

The initial pull takes 5–10 minutes on a decent connection. Mailcow is a substantial stack — Postfix, Dovecot, rspamd, Redis, MySQL, SOGo, nginx, and more. Watch the logs as it initializes:

docker compose logs -f --tail=50

Once nginx is up, hit https://mail.yourdomain.com. Default admin credentials are admin / moohoo. Change them immediately — before anything else.

Configuration

Add Your Domain and First Mailbox

Three steps in the Mailcow admin UI:

  1. Configuration → Mail Setup → Domains → Add domain — enter yourdomain.com
  2. Mailboxes → Add mailbox — create [email protected] with a strong password
  3. Optionally add aliases: postmaster@, abuse@, and info@ pointed at your main mailbox

DKIM Key — The One Step People Forget

After adding your domain, Mailcow generates a DKIM keypair automatically. Grab the public key:

Configuration → Mail Setup → Domains → DKIM — click your domain and copy the TXT record value.

Add it in DNS:

dkim._domainkey.yourdomain.com. TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqh..."

DNS propagation takes up to an hour. Once it settles, every outbound message gets signed — and that signature is what convinces Gmail, Outlook, and others to trust your server instead of binning your mail.

Tuning rspamd Anti-Spam

Mailcow’s rspamd UI lives at https://mail.yourdomain.com/rspamd. The defaults are reasonable, but a fresh server benefits from tightening the thresholds slightly:

# Connect into the rspamd container
docker exec -it mailcowdockerized-rspamd-mailcow-1 sh

# Check current action scores
cat /etc/rspamd/local.d/actions.conf

On a new server, I drop the reject score from 15 to 12 and set greylist at 6. That rejects obvious junk more aggressively while holding borderline messages for review. Drop a local override file rather than editing the defaults directly:

# /data/conf/rspamd/local.d/actions.conf
reject = 12;
greylist = 6;
add_header = 4;

Restart rspamd to pick up the change:

docker compose restart rspamd-mailcow

Connecting a Mail Client

IMAP and SMTP settings for Thunderbird, Apple Mail, or any client:

Incoming (IMAP):
  Server:    mail.yourdomain.com
  Port:      993
  Security:  SSL/TLS
  Username:  [email protected]

Outgoing (SMTP):
  Server:    mail.yourdomain.com
  Port:      587
  Security:  STARTTLS
  Username:  [email protected]

SOGo Webmail is at https://mail.yourdomain.com/SOGo — calendar and contacts are baked in, which makes it a fuller replacement for Gmail’s interface than you might expect.

Verification and Monitoring

Checking Your Mail Reputation

Once DNS propagates, run a deliverability check. Send a test message to mail-tester.com — it scores SPF, DKIM, DMARC, blacklists, and HTML hygiene in one shot. Hit 10/10 before sending real mail.

Confirm your PTR record resolves correctly:

dig -x 203.0.113.10 +short
# Should return: mail.yourdomain.com.

Also verify that DKIM is actually signing outbound mail. Send something to Gmail, open the original message headers, and look for dkim=pass in the Authentication-Results line.

Watching Logs and Queue

The Mailcow UI has a built-in log viewer under Configuration → Logs. For the CLI:

# Postfix mail queue
docker exec -it mailcowdockerized-postfix-mailcow-1 mailq

# Flush stuck queue manually
docker exec -it mailcowdockerized-postfix-mailcow-1 postqueue -f

# Live Dovecot auth log
docker compose logs -f dovecot-mailcow | grep -i auth

# rspamd scan results
docker compose logs -f rspamd-mailcow | tail -20

Keeping Mailcow Updated

Mailcow ships an update script. Run it monthly:

cd mailcow-dockerized
git fetch origin
git checkout origin/master
./update.sh

The script pulls new images, runs any migrations, and restarts containers. Mailboxes, settings, and certificates survive intact.

Backup Strategy

Mail data lives in Docker volumes. Back these up daily at minimum:

# Volumes to back up
/var/lib/docker/volumes/mailcowdockerized_vmail-vol-1    # Mail storage
/var/lib/docker/volumes/mailcowdockerized_mysql-vol-1    # Database
./mailcow.conf                                           # Config
./data/conf/                                             # Custom configs

Rsync to a remote server or S3-compatible bucket handles this well. For a consistent MySQL dump, source the config to get the DB password, then dump before rsyncing the volume:

source mailcow.conf
docker exec mailcowdockerized-mysql-mailcow-1 \
  mysqldump -u root -p"${DBROOT}" mailcow > /backup/mailcow-$(date +%F).sql

Keep at least 7 days of dumps. Recovering from a corrupted database without one is a bad afternoon.

After You’re Up and Running

  • Enable DNSSEC on your domain registrar — strengthens the DKIM validation chain end-to-end
  • Set up 2FA in Mailcow admin and SOGo — both support TOTP natively
  • Blacklist / whitelist persistent senders in the rspamd UI rather than training per-message
  • Set mailbox quotas — mail accumulates faster than you’d think; 10 GB per mailbox is a sensible starting point

Running your own mail server takes more upfront effort than a SaaS subscription. No argument there. But after the first few weeks, the overhead fades into background noise. What you keep is the visibility — full log access, queue control, spam tuning, zero per-seat cost — that no hosted provider will ever hand you.

Share: