Hallucination Mitigation & Context Window Management

Building reliable LLM-powered applications requires addressing two critical issues: factual correctness (mitigating hallucinations) and context optimization (fitting relevant data into the model’s active memory without performance loss).

🛡️ Hallucination Mitigation Strategies

Hallucination occurs when a model generates text that is factually incorrect, self-contradictory, or unsupported by its training data or input context. Engineers mitigate this using programmatic prompt patterns:

1. Chain-of-Verification (CoVe)

The Chain-of-Verification pattern splits generation into separate stages to prevent the model from compounding its own errors:

[Input Query] ➔ [1. Generate Base Response] ➔ [2. Draft Verification Questions] 
                      │
                      ▼
[4. Output Refined Answer] 🖎 [3. Answer Verification Questions Independently]

• Step 1: Generate Base Response: Ask the model to draft a response to the prompt.
• Step 2: Design Verification: Have the model identify key facts in the base response and generate list-based verification questions to cross-check them (e.g., “What is the exact founding date of Company X?”).
• Step 3: Independent Execution: Answer the verification questions. To prevent self-priming bias, execute each question in a fresh context window, or instruct the model to answer them relying strictly on verified databases/APIs.
• Step 4: Refined Generation: Feed the base response and the verified answers into a final prompt, producing a corrected response.

2. Self-Consistency (CoT-Voting)

Self-consistency is a technique that generates multiple reasoning paths for a single query:

• Generate a batch of responses (typically $N = 5$ or $10$) at temperature > 0.5.
• Instruct the model to show its work using Chain-of-Thought (CoT) reasoning.
• Extract the final answer from each path and apply a majority-vote filter. The most common answer is returned.
• Best For: Math word problems, database queries, and logical tasks where the end result is deterministic.

3. Active Guardrails & Grounding Verifiers

Implement input/output verification steps before sending data to models or clients:

• Input Filtering: Use lightweight classifiers (e.g., Llama Guard) to check if the prompt violates security guidelines.
• Entity Grounding: Extract names, dates, and numbers from the generated output. Use database queries to verify if these entities exist in the source document. If a mismatch is found, trigger a retry loop or flag the output.

🗄️ Context Window Management

Modern foundation models boast massive context windows (up to 128k to 2M+ tokens). However, filling these windows with raw text introduces latency and leads to information retrieval degradation.

1. The “Lost in the Middle” Effect

Research shows that LLMs are highly sensitive to the layout of information in their context windows. Models recall facts located at the very beginning and very end of their prompts with high accuracy, but their retrieval rate drops significantly for facts located in the middle:

[System instructions]     [Irrelevant history/sources]     [Active User Query]
   (High Attention) ➔ ➔ ➔      (Low Attention)    ➔ ➔ ➔    (High Attention)
   [Start of Prompt]             [Middle]                  [End of Prompt]

Optimization Guidelines

• Anchor System Prompts: Place core behavioral rules, rules of engagement, and schema requirements at the very beginning.
• Place Context in the Center: Insert retrieved documents, historical logs, or search results in the middle.
• Position the Query at the End: Always place the target question or parsing instruction at the very end of the prompt. This forces the model to focus on the active request.

2. Context Compression (LLMLingua)

Instead of sending raw source documents to the LLM, compress them first. Frameworks like LLMLingua use small language models (like Llama-3-8B) to calculate the perplexity of individual tokens in the prompt.

• Tokens that are highly predictable (low perplexity) are redundant and can be pruned.
• Tokens with high perplexity contain the core information and are kept.
• This approach can reduce prompt sizes by 30% to 50% without degrading performance, leading to lower API costs and faster Time to First Token (TTFT).

3. Summarization Hierarchies for Agent Memory

For long conversations, appending the complete history to every request will quickly exceed token limits. Instead, implement a hierarchical summarization pattern:

# System state tracking diagram
# Raw Chats (Recent) -> Keep unmodified (e.g., last 5 turns)
# Raw Chats (Older)  -> Summarize dynamically (e.g., every 10 turns)
# Summary Block      -> Appended as static system state metadata

• Active Window: Keep the last 5 turns of conversation raw and unmodified.
• Archived History: Summarize the conversation’s first $N$ turns into a structured bullet-point state block (e.g., “User is querying API specs for project X”).
• Injection: Inject the state block at the top of the prompt, discarding the older raw chat logs.