Posted inDatabase

Deploying Milvus on Docker: A Production Guide for Enterprise AI and RAG Applications

Database tutorial - IT technology blog
Database tutorial - IT technology blog

Six Months with Milvus in Production — Here’s What I Learned

I’ve run production systems on MySQL, PostgreSQL, and MongoDB. Each earned its place. But building a retrieval-augmented generation (RAG) system that needed to handle tens of millions of embeddings at low latency — that’s where all three hit their ceiling.

I evaluated the usual suspects. ChromaDB felt too lightweight for anything beyond a prototype. Pinecone would have worked, but vendor lock-in and egress costs were non-starters. Weaviate’s operational overhead was more than I wanted to carry. Milvus was different: built from the ground up for billion-scale vector search, with a design that treats enterprise deployment as a first-class concern, not an afterthought.

This guide reflects six months of running Milvus in Docker for a production RAG pipeline. Not a demo — an actual deployment handling ~50 million vectors with multiple services querying it concurrently.

Why Milvus for Enterprise-Scale AI

The architecture is the answer. Milvus separates storage, indexing, and query layers, which means each component scales independently. When query load doubled on our pipeline, I scaled the query nodes without touching storage. That kind of flexibility doesn’t exist in PostgreSQL or MongoDB for vector workloads — you’re stuck scaling the whole thing.

Key reasons Milvus earned its place in my stack:

  • Multiple index types: HNSW, IVF_FLAT, IVF_SQ8, DISKANN — choose based on your memory/accuracy tradeoff
  • Hybrid search: combine dense vector search with scalar filtering (filter by user_id or created_at while running ANN search)
  • Multi-tenancy: partition-level isolation makes it practical for SaaS architectures
  • Active maintenance: Zilliz backs it commercially, production issues get resolved quickly

For smaller projects under a few million vectors, lighter options are perfectly fine. Past that threshold, the slightly higher setup complexity of Milvus pays back quickly.

Installation: Milvus on Docker

Milvus ships in three deployment modes: Lite (embedded), Standalone, and Cluster. For teams running Docker without Kubernetes, Standalone is the right call. It bundles etcd and MinIO internally and runs as a manageable set of containers — no orchestration platform needed.

Prerequisites

  • Docker Engine 20.10+ and Docker Compose v2
  • At least 4 CPU cores and 8 GB RAM (16 GB recommended for production)
  • Linux host preferred — macOS works for development but is not recommended for production

Download the Official Docker Compose File

mkdir -p /opt/milvus && cd /opt/milvus

# Download the official standalone compose file
wget https://github.com/milvus-io/milvus/releases/download/v2.4.9/milvus-standalone-docker-compose.yml \
  -O docker-compose.yml

Always start from the official file. Here’s what the key sections look like, trimmed for clarity:

version: '3.5'
services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
    volumes:
      - ./volumes/etcd:/etcd

  minio:
    image: minio/minio:RELEASE.2023-03-13T19-46-17Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    volumes:
      - ./volumes/minio:/minio_data
    command: minio server /minio_data

  standalone:
    image: milvusdb/milvus:v2.4.9
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    ports:
      - "19530:19530"   # gRPC
      - "9091:9091"     # HTTP / metrics
    volumes:
      - ./volumes/milvus:/var/lib/milvus
    depends_on:
      - etcd
      - minio

Start the Stack

docker compose up -d

# Verify all containers are running
docker compose ps

Expect three containers — etcd, minio, and milvus-standalone — all showing Up status. First startup takes 30–60 seconds while Milvus initializes its metadata structures.

Configuration: Tuning for Production

Default settings will get you through a demo. They won’t survive real traffic. These are the changes I made after month one, once actual usage patterns became clear.

Custom milvus.yaml

Mount a custom config file to override defaults without rebuilding the image:

mkdir -p /opt/milvus/config
cat > /opt/milvus/config/milvus.yaml << 'EOF'
log:
  level: warn          # reduce log noise; use "info" for debugging

dataCoord:
  segment:
    maxSize: 512        # MB; smaller segments = faster indexing
    sealProportion: 0.8

queryNode:
  gracefulTime: 5000    # ms to wait before stopping a query

common:
  retentionDuration: 86400  # seconds; 0 = keep data forever

cache:
  cacheSize: 4          # GB; set to ~30-40% of available RAM
EOF

Add the config mount to your docker-compose.yml under the standalone service:

volumes:
  - ./volumes/milvus:/var/lib/milvus
  - ./config/milvus.yaml:/milvus/configs/milvus.yaml

Resource Limits

Skip this step and Milvus will consume all available memory during heavy indexing. Add resource limits to the standalone service definition:

deploy:
  resources:
    limits:
      memory: 12G
    reservations:
      memory: 6G

Choosing the Right Index Type

Index selection matters more than almost any other single configuration decision. On my workload — 768-dimensional embeddings, ~50 million vectors — HNSW delivered the best recall. Here’s how to set it up:

from pymilvus import Collection, FieldSchema, CollectionSchema, DataType, connections

connections.connect(host="localhost", port="19530")

# HNSW: best recall, higher memory usage
index_params = {
    "metric_type": "IP",       # Inner Product for normalized embeddings
    "index_type": "HNSW",
    "params": {
        "M": 16,               # higher M = better recall, more memory
        "efConstruction": 200  # higher = better index quality, slower build
    }
}

collection = Collection("my_embeddings")
collection.create_index(field_name="embedding", index_params=index_params)
collection.load()  # must load into memory before querying

Running tight on memory? Switch to IVF_SQ8. Quantization cuts the memory footprint by roughly 75% at the cost of only a 2–3% recall drop — a reasonable trade for most use cases.

Verification and Monitoring

Health Check

# HTTP health endpoint
curl -f http://localhost:9091/healthz
# Expected: {"status":"healthy"}

Connect and Run a Quick Sanity Test

pip install pymilvus
from pymilvus import connections, utility

connections.connect(host="localhost", port="19530")

print("Connected:", utility.get_server_version())
print("Collections:", utility.list_collections())

Attu: The Milvus Management UI

Attu is the official GUI for Milvus — think pgAdmin, but for vector databases. Add it to your compose file:

attu:
  image: zilliz/attu:v2.4.9
  environment:
    MILVUS_URL: standalone:19530
  ports:
    - "3000:3000"
  depends_on:
    - standalone

Restart and open http://localhost:3000. You get collection browsing, index stats, interactive search queries, and segment status — all without writing a line of code. I use Attu every day for operational visibility. It’s not optional in my setup.

Prometheus Metrics

Milvus exposes Prometheus-compatible metrics at http://localhost:9091/metrics. The three I track most closely in production:

  • milvus_querynode_search_latency_bucket — p50/p99 search latency
  • milvus_datanode_flush_segment_size_bytes — segment flush behavior
  • milvus_rootcoord_collection_num — collection count over time
# prometheus.yml scrape config
scrape_configs:
  - job_name: 'milvus'
    static_configs:
      - targets: ['milvus-host:9091']

Wire this up to Grafana using Milvus’s official dashboard (ID 17777) and you have solid observability without building anything custom.

Log Monitoring

# Watch live logs, filter for errors only
docker compose logs -f standalone | grep -E "ERROR|WARN"

# Check etcd health — Milvus depends on it heavily
docker compose exec etcd etcdctl endpoint health

What I’d Do Differently Starting Over

Four things I wish I’d known on day one:

  • Always normalize embeddings before inserting when using Inner Product metric. I forgot this once and spent two hours confused by nonsensical search results.
  • Call collection.load() explicitly after every service restart. Milvus does not auto-load collections into memory — this trips up everyone at least once.
  • Set up Attu immediately, not as an afterthought. Debugging segment compaction issues without any UI visibility is genuinely painful.
  • Plan your partition strategy early. Adding partitions after tens of millions of inserts is possible but disruptive to query patterns.

Six months in, Milvus is the component I touch least in my entire stack. That’s exactly what you want from infrastructure. Get the initial configuration right, and it stays out of your way.

Share:
Tags: