Ingest

July 1, 2026 · View on GitHub

Ingest is the primary LLM Wiki operation: process a new raw source into the bundle so knowledge compounds rather than being re-derived per query. The defining principle: the wiki is the compiled artifact, not a cleaned-up copy of the source. You extract entities, themes, and signals — you do not restate the note.

Flow

A typical ingest, distilled from the the personal work wiki:

  1. Read & classify the source (transcript, email, note, document, image). Sources may be any format the agent can read.
  2. Extract entities and signals. For each meaningful entity, create or update the appropriate concept document in the right section of the bundle.
  3. Write valid frontmatter on every new or touched concept — at minimum a non-empty type; set timestamp on meaningful change.
  4. Add cross-links in both directions between related concepts — a person named in a deal links to their page and back.
  5. Re-synthesize overviews. Any section that changed gets its roll-up/overview rewritten to reflect the new state — an overview is a synthesis, not a file listing.
  6. Update index files so progressive disclosure stays reliable.
  7. Append to the log — one dated entry recording source, concepts created/updated, and themes found. Append-only; never edit past entries.
  8. Retire the source. Move the raw source to a processed location. Raw sources are immutable — move, never modify.
  9. Commit (when the bundle is a git repo) with a message summarizing what was ingested.

Supervision

Ingest one source at a time with a human in the loop (read the summaries, guide emphasis) or batch-ingest many with less supervision. A single rich source can touch 10–15 concepts. The right cadence is a per-project choice and belongs in the schema layer.

Citations

  1. Karpathy — LLM Wiki gist
  2. personal work wiki