Posted inHomeLab

Deploy a Local AI Server with Ollama and Open WebUI on Docker: Self-Hosted RAG and Web Search

HomeLab tutorial - IT technology blog
HomeLab tutorial - IT technology blog

Comparing Approaches: Bare-Metal vs. Dockerized Local AI

Running Large Language Models (LLMs) on homelab hardware comes down to one question: do you want to spend your weekend debugging CUDA dependencies, or actually build workflows with your AI assistant? For self-hosted inference, three main architectures stand out:

  • Raw llama.cpp / CLI Binaries: Compiling llama.cpp straight on the host yields maximum throughput and minimal memory overhead. But managing GGUF weights by hand, writing bespoke API shims, and stitching together frontends turns basic maintenance into a tedious chore.
  • Production Engines (vLLM / TGI): High-throughput frameworks shine in multi-tenant production setups thanks to PagedAttention and continuous batching. In a single-user homelab, though, vLLM aggressively pre-allocates up to 90% of your VRAM by default and lacks a built-in user interface.
  • Ollama + Open WebUI on Docker: Ollama handles quantization, GPU offloading, and model pulls behind a clean REST API. Packaging both Ollama and Open WebUI in Docker containers keeps your host system clean. You get multi-user auth, chat history, document RAG, and live search right out of the box.

For most homelab setups, containerizing Ollama alongside Open WebUI hits the sweet spot between raw speed and low maintenance.

Tradeoffs of Running a Private AI Stack

Self-hosting delivers total privacy, but hardware reality sets hard limits on what you can run.

Advantages

  • Complete Data Privacy: Prompts, company documents, proprietary code, and chat logs stay entirely inside your local LAN.
  • Zero Token Billing: Once you buy the hardware, running 500,000 tokens a day costs only the wall power your machine draws—typically 30W to 250W under load.
  • Offline Operation: Your coding assistant, summarizer, and internal knowledge base keep working even during an ISP outage.
  • Granular Control: You choose the exact context window size, sampling temperature, chunk overlap, and vector embedding model.

Disadvantages

  • VRAM Dictates Model Size: A quantized 8B model needs roughly 5.5 GB to 6 GB of VRAM for the weights plus context. Stepping up to 70B models requires at least 40 GB to 48 GB of VRAM across multiple GPUs.
  • Sluggish CPU Fallback: An RTX 3060 12GB delivers roughly 35–45 tokens per second on Llama 3.1 8B (Q4_K_M). On an 8-core CPU without GPU acceleration, that speed tumbles down to 3–6 tokens per second.

Self-hosting won’t match cloud frontier models on complex multi-step reasoning, but it gives you zero subscription fees, predictable latency, and complete control over your data.

Recommended Hardware Baseline

To keep inference responsive while running embedding pipelines in the background, aim for these hardware targets:

  • Host OS: Ubuntu Server 24.04 LTS or Debian 12 (bookworm).
  • GPU: NVIDIA card with 8GB to 16GB VRAM. An RTX 3060 12GB, RTX 4060 Ti 16GB, or used RTX 3090 24GB will comfortably run 8B models (Llama 3.1, Mistral 7B, DeepSeek-R1-Distill-Qwen-8B). Modern DDR5 system RAM works for smaller 1B–3B models, but GPUs provide the best experience.
  • System Memory: 32GB RAM minimum to prevent out-of-memory errors during concurrent document embedding and vector indexing.
  • Storage: Fast NVMe SSD with at least 100GB of free space. Model weights take 4GB to 15GB each, plus storage for Open WebUI’s ChromaDB vector store.
  • Runtime: Docker Engine 24.0+ with NVIDIA Container Toolkit.

Step-by-Step Implementation

We will set up a three-container stack using Docker Compose: Ollama with GPU acceleration, Open WebUI as the frontend, and SearXNG for privacy-first web grounding.

Step 1: Install the NVIDIA Container Toolkit

If you have an NVIDIA GPU, install the container toolkit so Docker can talk directly to CUDA:

# Add the NVIDIA Container Toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install the toolkit and configure Docker
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 2: Write the Docker Compose Configuration

Create a project folder for your stack and create the compose definition:

mkdir -p ~/homelab/local-ai && cd ~/homelab/local-ai
nano docker-compose.yml

Add the following service configuration:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ./ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - SEARXNG_BASE_URL=http://searxng:8080/
    volumes:
      - ./searxng_data:/etc/searxng

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=generate_a_random_32_char_secret_here
      - ENABLE_RAG_WEB_SEARCH=True
      - RAG_WEB_SEARCH_ENGINE=searxng
      - SEARXNG_QUERY_URL=http://searxng:8080/search?q=<query>&format=json
    volumes:
      - ./open_webui_data:/app/backend/data
    depends_on:
      - ollama
      - searxng

Step 3: Launch Containers and Download Models

Bring the stack up in the background:

docker compose up -d

Next, pull an 8B chat model and a dedicated embedding model into Ollama:

# Pull Llama 3.1 8B (~4.7 GB)
docker exec -it ollama ollama pull llama3.1:8b

# Pull an embedding model for document retrieval (~274 MB)
docker exec -it ollama ollama pull nomic-embed-text

Step 4: Configure Open WebUI Settings

  1. Open your browser and go to http://<your-server-ip>:3000.
  2. Register your admin account. The first registered user automatically gains full administrative access.
  3. Head to Admin Panel > Settings > Documents. Set Embedding Model Engine to ollama and select nomic-embed-text from the dropdown. This offloads document chunking to a specialized model rather than wasting main model compute.
  4. Go to Admin Panel > Settings > Web Search. Verify that search is enabled with the engine set to SearXNG and the query URL configured to http://searxng:8080/search?q=<query>&format=json.

Step 5: Test Document RAG and Live Web Grounding

Start a new chat to verify both features:

  • Document Retrieval (RAG): Click the attachment icon (+) and upload a PDF, text file, or code repository export (e.g., a 20-page server manual). Ask specific questions about its contents. Open WebUI embeds the document chunks, queries the local vector database, and injects the top scoring snippets directly into your prompt context.
  • Real-Time Web Search: Toggle the globe icon in the chat bar and ask for current info, such as “What are the latest security advisories for Debian 12?”. Open WebUI queries SearXNG, parses the search results, and cites sources directly in the answer.
Share:
Tags: