RAPTOR: Building a Hierarchical RAG System with Recursive Document Summarization in Python

AI tutorial - IT technology blog
AI tutorial - IT technology blog

Quick Start: RAPTOR RAG in 5 Minutes

Standard RAG breaks down badly on long documents. You chunk a 200-page PDF, stuff it into a vector store, and when a user asks a high-level question, the retriever pulls back three random paragraphs from page 47. No context. No structure. Just noise.

RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) fixes this by building a tree of progressively more abstract summaries. Your retriever can then grab both the fine-grained detail and the big-picture context — depending on what the question actually needs.

Technical documentation, legal contracts, research papers — these are the document types where flat retrieval collapses in production. If you’ve hit that wall with a 200-page spec and wondered why the answers were garbage, this is the technique you’re missing.

Install the dependencies first:

pip install anthropic sentence-transformers scikit-learn numpy tiktoken umap-learn

Then grab a quick sanity check to confirm everything’s working:

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.mixture import GaussianMixture

model = SentenceTransformer('all-MiniLM-L6-v2')
test_emb = model.encode(["RAPTOR test"])
print(f"Embedding shape: {test_emb.shape}")  # Should be (1, 384)

If that runs without errors, you’re ready to build the tree.

Deep Dive: How RAPTOR Actually Works

The Problem with Flat Chunking

When you read a 100-page architecture document, you naturally build a mental model at multiple levels — chapter themes, section arguments, paragraph details. Standard RAG only stores paragraph-level chunks, so it misses the chapter-level understanding entirely. Ask “what’s the overall database strategy?”, and you get back a snippet about connection pooling from page 63.

RAPTOR mirrors the way you actually read. It creates a tree where the leaves are your original chunks, and each parent node is a summary of its children. At query time, you search the entire tree — not just the leaves.

The Three-Step Build Loop

The algorithm runs in a loop until the tree collapses to a single root node:

  1. Embed — encode all chunks at the current level into vectors
  2. Cluster — group similar chunks using Gaussian Mixture Models
  3. Summarize — generate one summary per cluster using an LLM

Those summaries become the new “chunks” for the next iteration. Keep looping until you’re down to one node.

import anthropic
from dataclasses import dataclass, field
from typing import List, Optional

@dataclass
class TreeNode:
    text: str
    level: int
    embedding: Optional[np.ndarray] = None
    children: List['TreeNode'] = field(default_factory=list)
    node_id: str = ""

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env

def summarize_cluster(texts: List[str]) -> str:
    combined = "\n\n---\n\n".join(texts)
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        system="You are a technical summarizer. Produce a concise, information-dense summary that captures the key points from all provided text segments.",
        messages=[{
            "role": "user",
            "content": f"Summarize these related text segments into one coherent paragraph:\n\n{combined}"
        }]
    )
    return response.content[0].text

Clustering with Gaussian Mixture Models

Unlike K-Means, GMM gives soft cluster assignments — a chunk can partially belong to multiple clusters, which matters when a paragraph straddles two topics. K-Means forces a hard assignment and loses that nuance.

def cluster_embeddings(embeddings: np.ndarray, max_clusters: int = 10) -> np.ndarray:
    n_samples = len(embeddings)
    n_clusters = min(max(2, n_samples // 3), max_clusters)
    
    if n_samples <= 2:
        return np.zeros(n_samples, dtype=int)
    
    # Reduce dimensions before clustering — GMM struggles in high-dim space
    from sklearn.decomposition import PCA
    n_components = min(10, n_samples - 1, embeddings.shape[1])
    reduced = PCA(n_components=n_components).fit_transform(embeddings)
    
    gmm = GaussianMixture(n_components=n_clusters, random_state=42)
    gmm.fit(reduced)
    return gmm.predict(reduced)

Advanced Usage: Full RAPTOR Tree Builder

Here’s a complete implementation that takes raw text, builds the tree, and exposes a retrieval function:

import uuid
from collections import defaultdict

class RAPTORIndex:
    def __init__(self, embed_model='all-MiniLM-L6-v2'):
        self.embedder = SentenceTransformer(embed_model)
        self.all_nodes: List[TreeNode] = []
    
    def _chunk_text(self, text: str, chunk_size: int = 400) -> List[str]:
        words = text.split()
        chunks = []
        for i in range(0, len(words), chunk_size):
            chunk = " ".join(words[i:i + chunk_size])
            if chunk.strip():
                chunks.append(chunk)
        return chunks
    
    def build(self, document: str, max_levels: int = 3) -> None:
        # Level 0: leaf chunks from the raw document
        chunks = self._chunk_text(document)
        current_nodes = []
        
        for i, chunk in enumerate(chunks):
            emb = self.embedder.encode([chunk])[0]
            node = TreeNode(
                text=chunk,
                level=0,
                embedding=emb,
                node_id=str(uuid.uuid4())
            )
            current_nodes.append(node)
        
        self.all_nodes.extend(current_nodes)
        
        # Iteratively build higher levels
        for level in range(1, max_levels + 1):
            if len(current_nodes) < 2:
                break
            
            embeddings = np.array([n.embedding for n in current_nodes])
            cluster_labels = cluster_embeddings(embeddings)
            
            # Group nodes by cluster
            clusters = defaultdict(list)
            for node, label in zip(current_nodes, cluster_labels):
                clusters[label].append(node)
            
            next_nodes = []
            for label, cluster_nodes in clusters.items():
                # Summarize this cluster
                cluster_texts = [n.text for n in cluster_nodes]
                summary = summarize_cluster(cluster_texts)
                
                summary_emb = self.embedder.encode([summary])[0]
                parent_node = TreeNode(
                    text=summary,
                    level=level,
                    embedding=summary_emb,
                    children=cluster_nodes,
                    node_id=str(uuid.uuid4())
                )
                next_nodes.append(parent_node)
                self.all_nodes.append(parent_node)
            
            print(f"Level {level}: {len(current_nodes)} chunks → {len(next_nodes)} summaries")
            current_nodes = next_nodes
    
    def retrieve(self, query: str, top_k: int = 5) -> List[TreeNode]:
        query_emb = self.embedder.encode([query])[0]
        
        # Score all nodes across all tree levels
        scores = []
        for node in self.all_nodes:
            if node.embedding is not None:
                similarity = np.dot(query_emb, node.embedding) / (
                    np.linalg.norm(query_emb) * np.linalg.norm(node.embedding) + 1e-9
                )
                scores.append((similarity, node))
        
        scores.sort(key=lambda x: x[0], reverse=True)
        return [node for _, node in scores[:top_k]]
    
    def answer(self, query: str, top_k: int = 5) -> str:
        relevant_nodes = self.retrieve(query, top_k=top_k)
        context = "\n\n".join([
            f"[Level {n.level}] {n.text}" for n in relevant_nodes
        ])
        
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            system="Answer questions based solely on the provided context. Cite which level (leaf detail vs summary) your answer draws from.",
            messages=[{
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}"
            }]
        )
        return response.content[0].text

Test it with a real document:

with open("your_document.txt", "r") as f:
    doc_text = f.read()

index = RAPTORIndex()
index.build(doc_text, max_levels=3)

print(f"Total nodes in tree: {len(index.all_nodes)}")
print(f"  Leaf nodes: {sum(1 for n in index.all_nodes if n.level == 0)}")
print(f"  Summary nodes: {sum(1 for n in index.all_nodes if n.level > 0)}")

answer = index.answer("What is the overall architecture approach?")
print(answer)

Practical Tips for Production

Tune Chunk Size to Your Document Type

400-word chunks work well for technical docs. For legal contracts with dense clause references, drop to 150-200 words so each chunk stays semantically atomic. For narrative text (reports, whitepapers), 600-800 words gives the summarizer more to work with per cluster.

Cache the Tree — Don’t Rebuild Every Request

Building the RAPTOR tree hits the LLM API for every cluster summary at every level. On a 200-page doc, that’s 50-100 API calls. Build once, serialize the tree with pickle or write node text+embeddings to SQLite, and reload on startup.

import pickle

# Save
with open("raptor_index.pkl", "wb") as f:
    pickle.dump(index.all_nodes, f)

# Load
with open("raptor_index.pkl", "rb") as f:
    index.all_nodes = pickle.load(f)

Mix Level Weights During Retrieval

When retrieving, bias slightly toward higher-level summaries for broad questions, and toward leaf nodes for specific fact-finding. A simple heuristic: if the query is under 8 words, boost leaf node scores by 1.1×. If it’s over 15 words, boost summary nodes.

Watch Your Token Budget

The summarization loop compounds fast. Set a hard max_tokens on each summarize call — 512 is usually enough. Use a cheaper model for the intermediate levels. Reserve your best model for the root summary and final QA step. This cuts costs by 60-70% on large documents.

Validate Tree Quality Before Shipping

After building, spot-check a few parent-child pairs. The parent summary should actually capture the key points from all its children — if it reads like a copy of just one child, your clusters are too imbalanced. Bump up max_clusters or add a minimum cluster size check.

Share: