Posted inSecurity

Deploy TheHive and Cortex on Linux: Self-Hosted Security Incident Response Platform

Security tutorial - IT technology blog
Security tutorial - IT technology blog

Context & Why: The Operational Chaos of Incident Handling

During a security incident, every wasted second helps the attacker. Most small and midsize security teams drown in alert noise. A single firewall spike or endpoint alert triggers a flood of uncoordinated notifications across Slack, Teams, and shared inboxes. Analysts scramble. They manually copy-paste IP addresses, domains, and SHA-256 hashes into VirusTotal, AbuseIPDB, or ANY.RUN one by one.

Doing this for 40 alerts a shift burns out analysts fast. Tracking active incidents in spreadsheets or Jira creates blind spots, drops forensic context, and destroys the chain of custody. Containment drags from minutes into hours.

A 2:00 AM SSH brute-force attack on my own jump host drove this lesson home. Over 14,000 failed attempts filled the logs before anyone reacted. Manual triage under pressure will fail you. A dedicated Security Incident Response Platform (SIRP) solves this problem. Pair TheHive for case management with Cortex for automated observable analysis. This duo keeps sensitive telemetry on your own hardware while running multi-engine threat checks with a single click.

Installation: Deploying TheHive and Cortex via Docker Compose

Docker Compose provides the cleanest deployment path on Ubuntu 22.04 or Debian 12. It prevents dependency conflicts across Cassandra, OpenSearch, and Python runtimes. Minimum host requirements are 4 vCPUs, 16 GB of RAM, and 50 GB of SSD storage.

Step 1: System Preparation and Prerequisites

Update your base packages and raise kernel virtual memory limits. OpenSearch will fail to start if vm.max_map_count remains at default Linux values:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl git apt-transport-https ca-certificates gnupg lsb-release

# OpenSearch requires at least 262,144 memory map areas
echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Install Docker Engine and the Compose plugin
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
newgrp docker

Step 2: Prepare Project Directory and Compose Specification

Create dedicated directories for persistent volumes and Cortex job execution:

mkdir -p ~/thehive-cortex/{cortex-data,thehive-data,opensearch-data,cortex-jobs}
cd ~/thehive-cortex
# Ensure write access for the search backend container UID (1000)
sudo chown -R 1000:1000 opensearch-data

Create docker-compose.yml:

version: '3.8'

networks:
  soc-net:
    driver: bridge

services:
  opensearch:
    image: opensearchproject/opensearch:2.11.0
    container_name: opensearch
    environment:
      - discovery.type=single-node
      - plugins.security.disabled=true
      - "OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - ./opensearch-data:/usr/share/opensearch/data
    networks:
      - soc-net
    restart: unless-stopped

  cortex:
    image: thehiveproject/cortex:3.1.8
    container_name: cortex
    depends_on:
      - opensearch
    environment:
      - http.port=9001
      - job.directory=/tmp/cortex-jobs
    volumes:
      - ./cortex-jobs:/tmp/cortex-jobs
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - soc-net
    ports:
      - "127.0.0.1:9001:9001"
    restart: unless-stopped

  thehive:
    image: thehiveproject/thehive:5.2.5
    container_name: thehive
    depends_on:
      - opensearch
      - cortex
    environment:
      - http.port=9000
    networks:
      - soc-net
    ports:
      - "127.0.0.1:9000:9000"
    restart: unless-stopped

Launch the stack in detached mode:

docker compose up -d
docker compose ps

Configuration: Connecting TheHive to Cortex and Enabling Analyzers

Both applications bind to local loopback (127.0.0.1) for safety. To reach the web consoles without exposing raw ports to the internet, open an SSH tunnel from your workstation:

ssh -N -L 9000:127.0.0.1:9000 -L 9001:127.0.0.1:9001 user@<server-ip>

Step 1: Initialize Cortex and Enable Analyzers

Open http://localhost:9001 in your local browser.

  1. Click Update Database to initialize ES/OpenSearch indices.
  2. Set up your SuperAdmin account with a 16+ character passphrase.
  3. Navigate to Organizations and create an org named SOC_Core.
  4. Add a user to SOC_Core with the orgadmin role to manage analyzer keys.
  5. Add an automation account named thehive-connector. Assign the read, analyze roles, click Create API Key, and save the token securely.

Sign in as your orgadmin user, navigate to Organization > Analyzers, and search for AbuseIPDB. Click Enable, enter your free or commercial API key, and set check_days to 30.

Step 2: Configure TheHive to Communicate with Cortex

Head to http://localhost:9000. Initialize the database schema when prompted. Log in using default credentials ([email protected] / secret), then change that temporary password right away.

Next, link Cortex through TheHive’s admin panel:

  1. Navigate to Platform Management > Organizations and configure your primary SOC team.
  2. Go to Cortex Connectors and select Add Cortex Server.
  3. Set the endpoint URL to http://cortex:9001. The containers share Docker’s internal bridge network, so name resolution works out of the box.
  4. Paste the API Key generated for thehive-connector.
  5. Click Test & Save. The health check icon will turn green.

Verification & Monitoring: Running an End-to-End Incident Workflow

Test the full pipeline with a live triage scenario.

Step 1: Create a Case and Attach Observables

  1. In TheHive, go to Cases > New Case.
  2. Input case details: Title: Suspicious Inbound SSH Brute-Force, Severity: Medium, TLP: Amber.
  3. Open the case and click the Observables tab.
  4. Click Add Observable, set type to ip, and input an external test address like 198.51.100.24.

Step 2: Trigger Cortex Analyzers

Check the box next to the new IP observable, click Actions > Run Analyzers, and pick AbuseIPDB_1_0.

Cortex launches a short-lived container via the Docker socket, queries AbuseIPDB, parses the JSON payload, and appends the taxonomy report directly to the observable drawer in under 4 seconds:

{
  "status": "Success",
  "observable": "198.51.100.24",
  "taxonomy": [
    {
      "level": "malicious",
      "namespace": "AbuseIPDB",
      "predicate": "ConfidenceScore",
      "value": "100%"
    }
  ]
}

Step 3: Service Health and Monitoring Checks

Keep containers stable by monitoring resource utilization and queue logs regularly:

# Review real-time CPU and memory footprints
docker stats --no-stream

# Stream connector and indexing logs
docker logs -f thehive --tail 50
docker logs -f cortex --tail 50

# Verify OpenSearch cluster health status
curl -s -X GET "http://localhost:9200/_cluster/health?pretty"

You now run an independent, self-contained incident response stack. Ingest alerts from Wazuh, Suricata, or Python webhooks. Let Cortex handle background intelligence gathering, and give your analysts the exact context they need to isolate threats before damage spreads.

Share:
Tags: