RAG for Prompting

RAG (Retrieval-Augmented Generation) as a prompting technique — supplementing prompts with retrieved knowledge at inference time without fine-tuning the model.

Definition (confidence: high)

RAG combines information retrieval with text generation: the system retrieves relevant document chunks from an external knowledge base and injects them into the prompt context. This allows the LLM to answer with up-to-date or private information without retraining.

The Prompt Engineering Guide describes the Meta AI approach (Lewis et al., 2021): a seq2seq model uses a dense vector index of Wikipedia as a non-parametric memory, while the model’s own weights serve as parametric memory. Retrieved documents are concatenated with the original prompt before generation.

When to Use RAG in Prompting (confidence: high)

• Questions requiring up-to-date knowledge the model wasn’t trained on
• Domain-specific queries where private documents are the source of truth
• Reducing hallucination by grounding answers in retrieved evidence
• Tasks where factuality matters more than stylistic fluency

Related

• prompt-engineering-guide — source guide
• chain-of-thought — complementary reasoning technique often paired with RAG
• RAG — main wiki page on RAG as a retrieval pattern (contrasted with LLM Wiki)

Prompting for RAG (confidence: medium)

The captured survey adds a prompt-side framing for RAG grounded in retrieval context:

Context Information:
[Retrieved Document 1]
[Retrieved Document 2]
[Retrieved Document N]

Task: Answer the following question using ONLY information from the context above.

Question: [User query]

Instructions:
- Quote specific passages when making claims
- If the context doesn't contain the answer, say "Not found in provided context"
- Do not use external knowledge

Challenges the survey flags: context-window limits, relevance ranking, and the lost-in-the-middle effect (Liu et al., 2023) — models attend less to the middle sections of long contexts, so critical information should be placed at the beginning or end of the retrieved block.

Implications

The “cite-only-context / say not-found” constraints are the same class of behavioral constraints covered by constraint-based-prompting — they turn RAG grounding into an explicit instruction rather than a hope. Combined with the placement guidance for long contexts, this is a concrete, reusable RAG prompt shape.

Sources

• Prompt Engineering Guide: RAG technique page
• raw/external/huggingface-co-advanced-prompt-engineering-bc976d6b.md