8 Ways to Reduce Hallucinations in RAG Systems

RAG systems rely on retrieved documents to generate answers. Outdated, inaccurate, or contradictory information can lead to unsupported answers even when retrieval works correctly. Validate source quality and check document freshness before adding content to the knowledge base.

A Python freshness check can flag documents that have not been updated within a defined period.

from datetime import datetime, timedelta, timezone

def is_stale(updated_at, max_age_days=90):
    if not updated_at:
        return True

    updated = datetime.fromisoformat(updated_at)
    if updated.tzinfo is None:
        updated = updated.replace(tzinfo=timezone.utc)

    cutoff = datetime.now(timezone.utc) - timedelta(
        days=max_age_days
    )
    return updated < cutoff

Use this check to flag potentially outdated documents for review before indexing or retrieval. The 90-day limit is an example, not a universal rule. Frequently changing information may need daily checks, while stable reference material can remain useful for years.

Also preserve source URLs, publication dates, and update timestamps as metadata. These details help trace claims back to their origins and assess whether the retrieved evidence remains trustworthy.

RAG systems split documents into smaller chunks before storing them in a vector database. Poor chunk boundaries can separate facts from their explanations, causing the retrieval process to miss important context. Very large chunks can also introduce irrelevant information into the model’s context.

A practical approach is to use a text splitter with configurable chunk size and overlap. For example, LangChain’s RecursiveCharacterTextSplitter attempts to split text along natural boundaries before falling back to smaller units.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=150,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_text(document_text)

Here, each chunk targets 800 characters, with 150 characters of overlap between adjacent chunks. The overlap helps preserve context when a relevant passage crosses a chunk boundary.

These values are starting points, not universal settings. Test different chunk sizes against your documents and queries. Preserve headings and document metadata when possible, since they help provide context for retrieved passages.

Initial retrieval can return relevant documents alongside passages that only loosely match the query. A reranker scores each query and passage together to prioritize the most useful evidence before it reaches the language model.

The reranking process follows this flow:

1. Initial retrieval: Fetch a set of candidate passages from the vector database or search index.
2. Reranking: Score each passage against the original query using a reranking model.
3. Passage selection: Keep the highest-ranked passages and pass them to the LLM as context.
4. Answer generation: Generate a response from the selected evidence.

The sentence-transformers library provides a CrossEncoder class for reranking retrieved passages.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder(
    "cross-encoder/ms-marco-MiniLM-L-6-v2"
)

query = "How do I fix ModuleNotFoundError in Python?"

passages = [
    "Python virtual environments isolate dependencies.",
    "ModuleNotFoundError occurs when Python cannot find an imported module.",
    "Python lists support indexing and slicing."
]

scores = reranker.predict([
    (query, passage) for passage in passages
])

ranked_passages = sorted(
    zip(passages, scores),
    key=lambda item: item[1],
    reverse=True
)

top_passages = [
    passage for passage, score in ranked_passages[:2]
]

Here, the reranker scores each passage against the query and selects the top two results. The actual ranking depends on the model’s scores.

Reranking can improve the evidence supplied to an LLM, but it adds inference cost. It also cannot correct false information in the source documents, so source quality and evidence validation remain important.

Even when a RAG system retrieves relevant passages, an LLM may generate claims that the evidence does not support. A grounded prompt instructs the model to answer from the retrieved context and acknowledge when information is missing.

First, format the retrieved documents with stable IDs and source URLs. This preserves traceability when the model cites evidence.

def format_context(documents):
    return "\n\n".join(
        f"[Source ID: {doc['id']}]\n"
        f"URL: {doc['url']}\n"
        f"Content: {doc['content']}"
        for doc in documents
    )

Next, pass the formatted context and the user’s question to the LLM with clear grounding instructions.

prompt = f"""
Answer using only the evidence in the context.
If evidence is insufficient, say so.
Cite source IDs for factual claims.
Do not invent facts or citations.

Context:
{format_context(documents)}

Question: {question}
"""

For example, if the retrieved passages explain how to install a Python package but do not specify its minimum supported version, the model should acknowledge the missing information rather than guess.

A RAG system can only retrieve information available in its knowledge base. If the source documents lack relevant details or contain outdated information, the generated answers may be incomplete or inaccurate.

Geekflare’s Search and web scraping APIs can help address this limitation by discovering relevant pages and extracting their content.

Geekflare provides a Python SDK through the geekflare-api package. The Search API finds relevant web pages, and the Web Scraping API extracts page content that can be reviewed and prepared for indexing.

Install the required packages:

pip install geekflare-api python-dotenv

Store your API key in a .env file and initialize the client. Never commit API keys to source control.

Use the Search API to discover potential sources:

from geekflare_api.client import GeekflareClient
from geekflare_api.models import SearchDto
import os

with GeekflareClient(api_key=os.environ["GEEKFLARE_API_KEY"]) as client:
    results = client.search(
        SearchDto(
            query="official Python 3.13 release notes",
            limit=10,
            source="web",
            format="json"
        )
    )
    print(results)

Review the results and select trustworthy pages before scraping them. The Web Scraping API can then extract their content for further processing.

The extracted material should be checked for relevance, accuracy, and freshness before indexing. Preserve source URLs and update timestamps so retrieved claims can be traced to their origins.

Citations do not guarantee factual accuracy. An LLM can misinterpret a source or attach a citation to an unsupported claim. Claim verification checks whether each statement matches its cited evidence.

A separate LLM can compare each claim against its cited passages and classify it as supported, unsupported, or contradicted. Unsupported claims can trigger another retrieval attempt or be removed.

Also verify that cited source IDs exist and point to the correct documents. Automated checks can make mistakes, so test them against manually reviewed examples

When retrieved documents lack sufficient evidence or contain contradictory claims, a RAG system should communicate uncertainty rather than generate a confident answer. This requires checking whether the available context supports a reliable response.

A simple decision flow can guide this behavior:

• Sufficient, consistent evidence: Generate an answer with citations.
• Insufficient evidence: Abstain or request clarification.
• Conflicting evidence: Retrieve additional sources or explain the disagreement.

Source metadata can help resolve conflicts. Compare publication dates, prefer authoritative sources for the relevant topic, and check whether newer information supersedes older claims.

Abstention is especially important for questions outside the knowledge base. A clear response that the available evidence is insufficient is safer than a plausible but unsupported answer.

Test these scenarios with questions that have known answers, missing evidence, and deliberately conflicting sources.