Memory Retrieval Patterns

Memory Retrieval Patterns is a multi-stage hybrid retrieval pipeline for LLM agent memory that combines semantic search (embeddings), BM25 keyword matching, Reciprocal Rank Fusion (RRF) fusion, cross-encoder re-ranking, and MMR (Maximal Marginal Relevance) diversity filtering. Instead of relying on a single retrieval method, each strategy runs and its results are fused, so different stages catch what the others miss.

At a Glance

• What it is: A multi-stage pipeline that fuses embedding-based semantic search, BM25 keyword matching, RRF rank fusion, cross-encoder re-ranking, and MMR diversity into a single HybridRetriever.
• When you need it: Basic vector search misses keyword-dependent matches or returns too many near-duplicates; you need higher recall across a memory store of moderate-to-large size.
• The trade-off: Each stage adds latency (cross-encoder re-ranking adds 50–200ms; a full pipeline takes 200–500ms vs 20–50ms for cosine-only), and the system has many tunable parameters (RRF k, MMR lambda, candidate counts) that create misconfiguration risk.

Definition

A memory system is only as good as its ability to surface the right information at the right time. Memory Retrieval Patterns brings the same multi-strategy approach used by search engines to agent memory: keyword matching, meaning analysis, re-ranking precision, and diversity filtering each catch things the others miss.

The pipeline operates in stages:

1. Optional HyDE transformation — the query is optionally rewritten into a hypothetical answer document, bridging the vocabulary gap between a short query and longer stored memories.
2. Parallel retrieval — two indices run at once: semantic search (embeddings + cosine similarity) and BM25 (term-frequency keyword matching).
3. Rank fusion — Reciprocal Rank Fusion (RRF) merges the two ranked lists into a single fused ranking.
4. Re-ranking — a cross-encoder model re-scores the fused top candidates with higher precision than embedding similarity alone.
5. Diversity selection — MMR selects the final top-K results while maximizing diversity, reducing repetitive near-duplicates.
6. Injection — the results are formatted and injected into the agent’s context window.

Each component is implemented as a standalone piece before being wired together, so stages can be added incrementally as retrieval quality problems appear.

Key Concepts

• Semantic similarity — Using embedding vectors to find memories whose meaning is close to the query, measured by cosine similarity.
• BM25 lexical search — A term-frequency-based ranking function that excels at exact keyword matching and handles domain-specific terminology well.
• Hybrid retrieval — Combining semantic and lexical search results using RRF. Captures both meaning and keyword matches.
• Maximal Marginal Relevance (MMR) — A re-ranking strategy that balances relevance with diversity, reducing repetitive results.
• Cross-encoder re-ranking — A model that scores (query, document) pairs together for higher precision than embedding similarity alone.
• HyDE (Hypothetical Document Embeddings) — A query transformation that generates a hypothetical answer, then searches for similar documents, bridging the vocabulary gap.

Architecture

The retrieval pipeline works as a multi-stage funnel. Each stage narrows the candidate set while increasing quality.

Data flow: The query enters at the left. An optional query transformer (HyDE or expansion) rewrites it. The transformed query goes to two indices in parallel: a Semantic Index (embedding-based search) and a BM25 Index (keyword search). Score Fusion (Reciprocal Rank Fusion) combines both ranked lists into one. A metadata filter narrows candidates by time range, memory type, or source. The Re-ranker (a cross-encoder) re-scores the remaining candidates with higher precision. Finally, MMR selects the top-K results while maximizing diversity. These results enter the agent’s context window.

Implications

This architecture makes retrieval quality composable: each stage is optional and can be toggled based on latency budget. The funnel design means early stages (embedding + BM25) run cheaply on pre-computed indices, while expensive stages (cross-encoder, HyDE) run only on small candidate sets. The tradeoff is explicit: more stages = better recall but higher latency, with measurable cost per stage.

Implementation

The full pipeline is implemented as 9 standalone components wired into a single HybridRetriever class. Each piece can be tested and deployed independently:

1. Memory Store — A Memory dataclass (text, embedding, metadata, timestamp) and MemoryStore class that holds memories and computes embeddings in batch.
2. Embedding Helper — get_embeddings() wraps the OpenAI embeddings API with batching.
3. Semantic Search — cosine_similarity() and semantic_search() perform brute-force vector similarity against all memories.
4. BM25 Lexical Search — BM25Index class wraps rank_bm25.BM25Okapi for keyword-based retrieval.
5. Reciprocal Rank Fusion — reciprocal_rank_fusion() merges ranked lists using score = sum(1 / (k + rank)) with k=60.
6. Cross-Encoder Re-ranking — Reranker class loads a cross-encoder model (cross-encoder/ms-marco-MiniLM-L-6-v2) and re-scores (query, document) pairs.
7. Maximal Marginal Relevance — _mmr_greedy_select() greedy loop + mmr_selection() wrapper that normalizes scores to [0,1] range. MMR formula: lambda * relevance - (1 - lambda) * max_similarity_to_selected.
8. HyDE Query Transformation — hyde_transform() calls an LLM to generate a hypothetical answer, then searches for similar documents.
9. Full HybridRetriever — Orchestrates all 8 components: optional HyDE → parallel semantic + BM25 → RRF fusion → cross-encoder re-ranking → MMR diversity selection.

Implications

The modular implementation means you can adopt stages incrementally. Start with semantic search (steps 1-3), add BM25 (step 4) when keyword misses appear, add re-ranking (step 6) when precision matters, add MMR (step 7) when results are repetitive, add HyDE (step 8) when queries are ambiguous. The cost of each stage is measurable and additive, so you can tune based on latency budget.

When to Use It

• Any agent system that retrieves from memory — retrieval quality matters everywhere.
• Systems where naive semantic search returns repetitive or off-topic results.
• Domains with specialized terminology where pure embedding search underperforms keyword matching.
• Large memory stores (hundreds to thousands of entries) where a multi-stage pipeline improves precision.

Limitations

• Latency: Each stage adds measurable latency. Embedding lookup: milliseconds. BM25: fast. Cross-encoder re-ranking: 50-200ms depending on candidate count. HyDE: 200-1000ms (full LLM call). A full pipeline can take 200-500ms compared to 20-50ms for vector search alone. Real-time chat may need to skip stages.
• HyDE risk: The hypothetical answer can mislead retrieval when the LLM generates an incorrect answer — the search then finds documents similar to the wrong answer instead of the right one. HyDE works best for open-ended queries, not precise factual lookups.
• Small stores: Memory stores with fewer than 50 entries see little benefit from the full pipeline. Basic cosine similarity works well enough at that scale.
• Parameter tuning: The system has many tunable parameters: RRF’s k constant (typically 60), MMR’s lambda_param (0.0=diversity, 1.0=relevance, 0.5-0.7 works well in practice), initial_k_multiplier (how many candidates to fetch), candidate counts passed between stages. More moving parts means more misconfiguration risk.
• Complexity grows: Each component adds configuration surface. The pipeline has at least 6 knobs (RRF k, MMR lambda, candidate counts at each stage, HyDE on/off, reranker on/off, MMR on/off).

Further Reading

• Robertson, S., & Zaragoza, H. (2009). “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval, 3(4), 333-389. The definitive reference for BM25.
• Carbonell, J., & Goldstein, J. (1998). “The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries.” ACM SIGIR, 335-336. The original MMR paper.
• Gao, L., et al. (2022). “Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE).” arXiv:2212.10496. Shows that generating a hypothetical answer and embedding it improves zero-shot retrieval.
• Ma, X., et al. (2023). “Fine-Tuning LLaMA for Multi-Stage Text Retrieval.” arXiv:2310.08319. Explores LLMs as retrievers and re-rankers in multi-stage pipelines.

Open Questions

• What is the optimal RRF k value for agent memory stores of different sizes?
• Can MMR lambda be auto-tuned based on query type (factual vs open-ended)?
• How does the pipeline perform on non-English languages where embedding models are weaker?
• Is there a principled way to decide which stages to skip based on query characteristics?

Challenges

Three practical exercises to deepen understanding, each 10-30 minutes:

1. Fusion weight tuning: Adjust the relative weight of semantic vs. BM25 results in RRF. Try ratios of 0.3/0.7, 0.5/0.5, and 0.7/0.3. For each setting, run 10 queries and measure MRR (Mean Reciprocal Rank). Identify which ratio works best for your data.
2. Reranker impact: Measure Recall@10 and Precision@10 before and after applying the Reranker. Run 20 queries and record the position change of the top result after reranking. Compute how often reranking promotes a better result to position 1.
3. HyDE for memory retrieval: Apply hyde_transform() to 10 memory queries. Compare the top-5 results from raw queries vs. HyDE-transformed queries using semantic_search(). Count how many additional relevant memories the HyDE approach surfaces.

Relationship to This Vault

The Vault’s own hybrid_query_helper.py already implements a three-stage RRF fusion of BM25 + vector search + personalized PageRank for wiki page retrieval. The Memory Retrieval Patterns pipeline from the agent-memory ecosystem is a runtime retrieval strategy for agent memory stores; this vault uses a compile-time ingestion strategy instead. The shared insight is that fusion across retrieval methods consistently outperforms any single method.

Shared Techniques Across Sources

The three sources converge on a common toolkit for hybrid retrieval, each contributing a layer:

• HippoRAG uses Personalized PageRank seeded by query entities as its graph-traversal signal, performing multi-hop retrieval in a single step by exploring related graph neighborhoods. This is the graph-based retrieval signal that complements lexical and semantic matching.
• LightRAG introduces a dual-level structure: low-level retrieval over specific entities and their relations, and high-level retrieval over broader topics and abstract themes, combining both via hybrid retrieval. This mirrors the Memory Retrieval Patterns idea of running multiple retrievers in parallel and fusing results.
• Memory Retrieval Patterns formalizes the fusion: BM25 keyword matching catches token-exact hits, semantic search catches meaning matches, RRF merges the rankings, cross-encoder re-ranking filters false positives, and MMR selects diverse final results.

Together these show that hybrid retrieval is not a single algorithm but a composition: graph traversal (HippoRAG), dual-level indexing (LightRAG), and multi-stage fusion with re-ranking + diversity (Memory Retrieval Patterns) are all stages that can be mixed and matched depending on the latency budget.

The architectural contrast is direct:

Aspect	This Vault (LLM Wiki)	Memory Retrieval Patterns
When is synthesis performed?	Ingest time — wiki pages are compiled ahead of queries	Query time — memories are fused fresh per request
Persistent unit	Linked Markdown wiki pages	Memory entries (vectors + keywords)
What compounds?	The synthesized wiki itself	The memory store; no accumulated human-readable layer
Can a human inspect it?	Yes, directly as files	Partially, via retrieval traces and indices

Implications

The vault’s artifact-first approach and the source’s retrieval-first approach optimize the same goal — better retrieval — on different axes. The vault pays the cost at ingest (writing human-readable pages) to make queries cheap and auditable. The source pays the cost at query time (running multiple retrievers per question) to keep the memory store flexible. For a personal second brain, the artifact-first trade-off favors long-term durability and human inspection; for a multi-session agent with opaque memory, the runtime pipeline favors recall and flexibility.

Sources

• ^[raw/external/github-com-20-memory-retrieval-patterns-4f3658d2.md] — Memory Retrieval Patterns README (NirDiamant/Agent_Memory_Techniques, 847 stars)
• ^[raw/external/github-com-memory-retrieval-patterns-notebook-98ad0849.md] — Full Jupyter notebook with implementation code, tradeoffs, and challenges
• ^[raw/papers/hipporag-neurobiologically-inspired-long-term-memory.md] — HippoRAG paper: Personalized PageRank for multi-hop retrieval
• ^[raw/papers/lightrag-simple-and-fast-rag.md] — LightRAG paper: dual-level graph + vector hybrid retrieval

Related

• agent-memory-systems — the graph/retrieval-first family this pattern belongs to
• rag — classic retrieval-augmented generation, the simpler predecessor
• query — this vault’s own BM25+vector+PageRank RRF fusion implementation at query time
• llm-wiki-vs-memory-and-graph-rag — comparison of artifact-first vs retrieval-first vs memory-first families