Ingest

Definition

Ingest is the core operation of adding a new source to the LLM Wiki. The LLM reads the source, analyzes the existing wiki, and creates or updates wiki pages accordingly. It is the mechanism by which knowledge accumulates and compounds in the system.

Ingest is the write operation, not a synonym for index.md. The index is the catalog consulted to locate existing pages and updated after the work; it does not replace the source reading, routing, and page updates that make up ingest. ^[raw/articles/karpathy-llm-wiki-gist.md]

Key Points

• The 5-step ingest pipeline: Resolve (detect source type) → Route (find relevant existing pages) → Synthesize (update or create pages) → Embed (re-index) → Update (index and log). [confidence: medium — single implementation report]
• One source typically generates 5-15 wiki pages depending on content richness. [confidence: low — single source (VentureBeat)]
• Wyndo’s production experience reports 5-50 wiki pages per source, a wider range than the 5-15 reported by VentureBeat. [confidence: medium — single source (GenAI Unplugged)]
• The LLM identifies concepts, entities, relationships, and comparisons from each source
• Existing pages are extended, never overwritten (the compounding invariant)
• Ingest respects guardrails: max 5 pages updated, forward-only linking, entity threshold (2+ mentions)
• Provenance is recorded from source to page: raw sources remain immutable and the mutable wiki page records which raw files contributed to its synthesis.
• The L0-L3 lifecycle profiles described alongside the OKF draft are an atomicstrata extension; they should not be treated as a mandatory step for every ingest.

Wyndo’s Ingest Process

Wyndo (of genai-unplugged) describes a simpler 5-step ingest process that differs from the formal pipeline above:

1. Drop a source (PDF, URL, text) into the raw/ folder
2. Tell the LLM to ingest it
3. The LLM reads the source and the existing wiki
4. The LLM decides what pages to create/update
5. The LLM writes the pages and updates the index/log

This process is intentionally simple from the human’s perspective — the complexity is entirely on the LLM side.

Implications

Wyndo’s process is less formalized than the 5-step pipeline (Resolve → Route → Synthesize → Embed → Update) but achieves the same outcome. The difference is in how much of the decision-making is explicit vs implicit. Wyndo’s approach trusts the LLM to handle routing and synthesis without explicit sub-steps.

Event-Driven Ingest

LLM Wiki v2 extends ingest with event-driven hooks: auto-ingest on new source, auto-consolidate on session end, and scheduled lint runs. ^[raw/articles/llm-wiki-v2-rohitg00-2026.md] This transforms ingest from a manual operation into an automated background process.

Implications

Event-driven ingest reduces the maintenance burden to near zero — the human only intervenes for curation and override. However, critics argue that auto-ingest assumes reliable LLMs, which introduces silent corruption risk. The debate is between manual (verified but slow) and automated (fast but potentially noisy).

Crystallization

LLM Wiki v2 introduces crystallization: treating completed chains of work (research threads, debugging sessions, analyses) as sources themselves. ^[raw/articles/llm-wiki-v2-rohitg00-2026.md] The system auto-distills what was learned into structured digests that become first-class wiki pages.

Implications

Crystallization makes every interaction a potential source. This vault currently treats only external material as sources — adopting crystallization would mean treating session outputs as warrants for new pages or claim updates.

Page Volume Contradiction

Wyndo reports that one source might generate 5-50 wiki pages depending on content richness. ^[raw/articles/genai-unplugged-llm-wiki.md] This is a wider range than the 5-15 pages reported by VentureBeat. ^[raw/articles/venturebeat-llm-wiki.md] The difference likely reflects the diversity of source types — a dense research paper generates more pages than a short blog post.

Implications

The 5-50 range suggests that page volume is highly dependent on source density and complexity. The existing 5-15 estimate may be conservative or based on shorter sources. This matters for capacity planning: a single dense source could require updating the maximum allowed pages per ingest.

Related Concepts

• llm-wiki — Ingest is one of the three core LLM Wiki operations
• query — Complementary operation to ingest (read the compiled wiki)
• lint — Health check operation that follows ingest
• open-knowledge-format — format baseline and consumer-tolerance rules distinct from local governance
• toolboxmd-karpathy-wiki — implementation with background auto-capture + detached ingest
• ai-memex-cli — dual-vault CLI that distills sessions into the wiki
• mycelium-hq-ai-brain-starter — verification harness for Claude Code memory

Open Questions

• What is the maximum practical source size for a single ingest? Sources describe individual articles but not book-length inputs or large GitHub repositories.
• How are ingest conflicts resolved when two sources provide contradictory information in the same ingest batch? The contradiction step handles cross-batch issues but within-batch conflicts are unaddressed.
• What metrics determine when an ingest is “complete”? Sources describe the pipeline steps but not quality gates or stopping criteria.

^[raw/articles/google-okf-spec.md] (L3 Compiled requires gaps surfaced)

Sources

^[raw/articles/karpathy-llm-wiki-gist.md] ^[raw/articles/levelup-llm-wiki-deep-dive.md]