Posted inAI

Optimize RAG Accuracy with Query Expansion and HyDE: A Practical Guide

AI tutorial - IT technology blog
AI tutorial - IT technology blog

The Frustrating Reality of Basic RAG

I remember building my first Retrieval-Augmented Generation (RAG) application using a standard vector database and a simple similarity search. On paper, it looked perfect. I fed it my company’s internal documentation, and for simple questions like “What is the holiday policy?”, it worked flawlessly. However, as soon as my teammates started asking more complex, nuanced questions like “How do I troubleshoot intermittent latency in the authentication service?”, the system fell apart.

The LLM would confidently state that it couldn’t find any relevant information, or worse, it would hallucinate an answer based on unrelated snippets of code. If you have built a RAG pipeline, you have likely hit this wall. You have the data, and you have a powerful LLM, but the bridge between the user’s question and the right document is broken.

The Root Cause: The Semantic Gap

The fundamental problem with basic RAG is the reliance on a single, often poorly phrased user query to find matches in a massive vector space. This is known as the “Semantic Gap.” There are three main reasons why a simple search fails:

  • Vocabulary Mismatch: A user might ask about “slowness,” but your documentation uses the term “latency” or “bottleneck.” Traditional vector embeddings are good, but they aren’t always perfect at mapping these synonyms in a high-dimensional space.
  • Lack of Context: User queries are typically short (3-5 words). Documents are long and detailed. Comparing a short vector to a long vector often leads to low similarity scores.
  • Underspecified Queries: Users often assume the AI knows what they are talking about. A query like “How do I fix the login error?” is useless if your database contains logs for five different login systems.

Comparing Strategies: Query Expansion vs. HyDE

To solve these issues, we need to move beyond simple retrieval. I have experimented with several techniques, but two stand out as the most effective for junior developers to implement: Multi-Query Expansion and Hypothetical Document Embeddings (HyDE).

1. Multi-Query Expansion

The idea here is simple: instead of searching with one query, use an LLM to rewrite the user’s question into 3 or 5 different versions. This covers different perspectives and terminologies.

Pros: Easy to implement, catches synonyms effectively.
Cons: Increases latency (multiple LLM calls) and can pull in noisy, irrelevant data if the expansion is too broad.

2. HyDE (Hypothetical Document Embeddings)

HyDE takes a different approach. Instead of rewriting the query, it asks the LLM to write a “fake” or hypothetical answer to the query first. We then take that hypothetical answer, turn it into an embedding, and use that to search the database.

Pros: Extremely effective for complex technical questions because the hypothetical answer will likely share the same structure and technical vocabulary as the real documents.
Cons: If the LLM’s hypothetical answer is completely wrong (hallucination), it might lead the search to the wrong part of the database.

The Best Approach: Implementing HyDE with LangChain

After testing both, I found that HyDE often provides the most significant boost in accuracy for technical documentation. I have applied this approach in production and the results have been consistently stable, especially when dealing with specialized domain knowledge that doesn’t appear often in general training data.

Step 1: Setting up the Environment

You will need Python and a few libraries. I recommend using LangChain as it simplifies the orchestration of these steps.

pip install langchain openai chromadb tiktoken

Step 2: Implementing the HyDE Logic

Here is how you can set up a HyDE chain. We will use OpenAI’s GPT-3.5 or GPT-4 to generate the hypothetical document and then use it to query a local ChromaDB instance.

import os
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langchain_community.vectorstores import Chroma
from langchain_core.output_parsers import StrOutputParser

# Initialize your LLM
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

# 1. Define the prompt to generate a hypothetical document
hyde_prompt = ChatPromptTemplate.from_template("""
As an expert software engineer, please write a brief technical paragraph 
that would answer the following question. 
Focus on technical terms and potential solutions.

Question: {question}

Answer:""")

hyde_chain = hyde_prompt | llm | StrOutputParser()

# 2. Example usage
user_query = "How do I handle connection pooling in a Python PostgreSQL app?"
hypothetical_doc = hyde_chain.invoke({"question": user_query})

print(f"Hypothetical Document: {hypothetical_doc}")

Step 3: Integrating with Retrieval

Now, instead of passing the user_query to the vector store, we pass the hypothetical_doc. This ensures the vector search is comparing “document-like” vectors with other “document-like” vectors.

# Assuming you already have a vectorstore populated
# vectorstore = Chroma(persist_directory="./db", embedding_function=OpenAIEmbeddings())

# Retrieve documents based on the hypothetical answer
retrieved_docs = vectorstore.similarity_search(hypothetical_doc, k=3)

for doc in retrieved_docs:
    print(f"Source: {doc.metadata['source']}")
    print(f"Content snippet: {doc.page_content[:100]}...")

Refining the Strategy: Multi-Query as a Safety Net

While HyDE is powerful, it can occasionally miss the mark if the LLM doesn’t know the specific jargon of your company. To make the system even more robust, I often combine Multi-Query Expansion with HyDE. I generate three variations of the query, generate one hypothetical document, and search for all four. This “ensemble” approach minimizes the chance of missing the right context.

Practical Implementation Tips

  • Temperature Control: When generating hypothetical documents, keep the LLM temperature at 0. You want consistent, technical output, not creative writing.
  • Prompt Engineering: Tell the LLM specifically what kind of document it is writing (e.g., “Write a README section” or “Write a StackOverflow-style answer”). This helps align the hypothetical text with your actual data format.
  • Cost Management: Remember that HyDE adds an extra LLM call for every search. If you are on a tight budget, use a cheaper model like GPT-3.5-Turbo for the generation and reserve GPT-4 for the final synthesis.

Summary

Standard RAG is often insufficient for real-world enterprise needs. By implementing techniques like Query Expansion and HyDE, you bridge the gap between how humans ask questions and how documentation stores information.

Start with HyDE; it provides the biggest “bang for your buck” in terms of accuracy improvements. Once you have that running, you can experiment with multi-query variations to catch the edge cases. Moving from a basic search to a sophisticated retrieval strategy is what separates a prototype from a production-ready AI assistant.

Share:
Tags: