Cloud-Native KB Ingestion

August 31, 2026 · View on GitHub

Status — operational model. This guide documents how a cloud tenant turns its own file formats into Knowledge Base chunks through push parsing or a tenant-local pull profile. A worked parser ships alongside this guide under ai/examples/cloud-deployment/.

Source vs Parser

The KB ingestion substrate splits content acquisition into two roles (see Overview):

  • A Source locates and reads content from a territory — used by the full-corpus build (ai:sync-kb). See Custom Sources.
  • A Parser transforms one file format into chunk content. A parser is what the push path (ingest_source_files, ai:ingest-tenant — see Hook Wiring) invokes to turn a tenant's raw file into parsed-chunk-v1 records.

The legacy full-corpus path registers parsers in SourceRegistry. Multi-tenant data-tier declarations do not: they resolve per tenant from a deployment-pinned root, preventing one tenant's parser id from overwriting another's process-global entry. This guide covers the format layer a cloud tenant most often needs to extend.

The tenant-level choice of which source families use server parsers, client-side parsed-chunk-v1, or an explicit unsupported status is defined in Tenant Ingestion Model.

flowchart TD
    PushFile[Push raw file] --> TenantResolver[Tenant-local parser resolver]
    PullRepo[Pulled repository revision] --> ProfileRoute[ParserSource profile route]
    ProfileRoute --> TenantResolver
    TenantResolver --> Parser[Operator-authorized parser]
    Parser --> ParsedChunks[Validated parsed chunks]
    ClientParser[Client-side parser] --> ParsedChunks
    ParsedChunks --> ServerEmbed[Server-owned embedding]

The parsed-chunk-v1 contract

Every chunk entering the KB through the push path conforms to parsed-chunk-v1 — the ingest contract. A parser's job is to emit records of this shape. The required fields:

FieldMeaning
schemaVersionThe constant "1.0.0". The ingestion service validates an exact match.
tenantIdThe tenant's claim — server-overwritten with the authoritative value. Lowercase kebab.
repoSlugThe tenant repo this chunk belongs to.
rootKindneo-workspace | bare-repo | external-source.
sourcePathPath relative to the repoSlug root, forward-slash normalized, no leading slash.
contentThe chunk text — what gets embedded.
hashInputsThe field names that compose the chunk's content hash. The server prepends tenantId + repoSlug implicitly.
parserIdThe id of the parser that produced the chunk (provenance + protocol versioning).
parserVersionA semver string for the parser.
kindOpen-enum semantic category (doc-section, method, class-config, …).
nameHuman-readable chunk name.

Optional: line_start, line_end, className, extends, extractorId, extractorVersion, extractionIdentity, and customMeta (an open extension slot). The schema is additionalProperties: false — unknown top-level keys are rejected; put parser-specific extras in customMeta. Pull-mode extractor and extraction fields are server-derived. Extraction identity participates in the chunk hash so a profile change creates a replacement generation instead of becoming metadata on an id the vector store would skip.

One field is forbidden: embedding. A record carrying an embedding is a restore record (backup-record-v1), not an ingest record. The ingestion service rejects any parsed-chunk-v1 record with an embedding field (KB_PARSED_CHUNK_EMBEDDING_REJECTED) — embeddings are always generated server-side. Every record is Ajv-validated against the schema at ingest; a non-conforming record is rejected (KB_PARSED_CHUNK_INVALID) without aborting its sibling records in the same push.

Two places a parser can run

A tenant's file format can be parsed in either of two places — the choice is a trust decision (see Security):

The tenant runs the parser in its own environment and pushes already-formed parsed-chunk-v1 records. No parser code runs on the deployment. The push envelope carries the records directly:

  • ingest_source_files — a files entry of the shape {sourcePath, parsedChunks: [ /* parsed-chunk-v1 */ ]}, or a files entry that is itself a parsed-chunk-v1 record (schemaVersion: "1.0.0").
  • ai:ingest-tenant — one parsed-chunk-v1 record per JSONL line.

This is the only path for a non-JS source format: a tenant with C++, Python, or .proto files writes a parser in whatever language and tooling it likes (tree-sitter, a language-native AST library, a regex pass), emits parsed-chunk-v1 JSON, and pushes it. The deployment never executes the tenant's parser — it only validates and embeds the records. Non-JS parser distribution is therefore a tenant-side concern; Neo's substrate sees only the JSON output.

Server-side (operator-gated)

A parser class runs in the deployment's process when a tenant pushes a raw file with parserId, or when a pull profile routes repository files through ParserSource. Because this executes parser code in the shared server process, it is operator-gated. JS config may carry a live ParserClass; graph/YAML tiers carry {parserId, parserModule, exportName?}, resolved below the deployment's tenantParserRoot. Empty root disables module loading, and failures never fall back to raw text.

Authoring a server-side parser

A parser has a stable parserId. Two declaration modes exist:

  • Legacy/global: list a live ParserClass in JS aiConfig.customParsers, or call SourceRegistry.registerParser for the full-corpus/push compatibility path.
  • Tenant-local: store {parserId, parserModule, exportName?} in the tenant's graph/YAML config and pin tenantParserRoot at deployment level. The loaded value is cached by tenant plus full declaration and never registered globally.

aiConfig.useDefaultParsers (default true) controls whether Neo's built-in parsers are present; a deployment serving only non-Neo content can set it false.

A registered parser implements one of two methods — the ingestion service (resolveFileChunks) dispatches on whichever is present:

  1. parseIngestionFile(file, {tenantContext}) → parsed-chunk-v1[] (recommended for new parsers). Receives the push envelope's files entry plus the resolved tenant context (tenantId, repoSlug, visibility, …), and returns parsed-chunk-v1 records directly — no adapter, no signature ambiguity.
  2. parse(content, sourcePath, type, hierarchy) → legacy chunks (the contract Neo's built-in SourceParser uses). Returns chunks of the legacy {type, kind, name, content, source, …} shape; the ingestion service adapts each into parsed-chunk-v1 via legacyChunkToParsedRecord (it defaults rootKind: 'external-source' and hashInputs: ['kind','name','content','sourcePath','parserId','parserVersion']).

Whatever you register is what gets dispatched. resolveFileChunks calls the parse method on the registered value and never instantiates it, so the method has to be callable on that value. Three shapes work:

  • a singleton — export default Neo.setupClass(MyParser) with singleton: true exports an instance, so ordinary instance methods are reachable. This is the idiom every built-in Source in ai/services/knowledge-base/source/ uses;
  • a constructor carrying static methods;
  • a plain object literal with the methods on it.

The shape that fails is a plain, non-singleton class whose parse method sits on the prototype: the constructor is what gets registered, nothing instantiates it, and the method is unreachable. That parser used to fall through to raw-text — which ingests successfully, leaving whole-file chunks, no error, and a plausible chunk count with nothing to notice. A tenant-declared parser in that shape is now refused with KB_TENANT_PARSER_NOT_DISPATCHABLE instead.

If a pushed file names a parserId that is not registered, the ingestion service returns KB_PARSER_NOT_REGISTERED for that file. A raw file with no parserId falls through to the built-in raw-text handling — the whole file becomes a single chunk.

Pull-mode compatibility uses the same failure semantics. An absent repository profile with a declared parserId synthesizes a ParserSource route whose canonical options contain exactly parserId and parserVersion. An unresolvable parser throws KB_PARSER_NOT_REGISTERED; it never silently relabels a RawRepoSource whole-file chunk as parser output.

The parser-execution boundary

The trust rule is invariant (see Security for the full model): untrusted parsing happens tenant-side; server-side parser execution is operator-gated. A cloud tenant extending the KB with a new format defaults to the client-side path — it needs no operator coordination, runs no code in the shared process, and supports any source language. The server-side path is reserved for parsers the operator has explicitly vetted.

A runtime sandbox for in-process execution of tenant-supplied parser code (WASM / tree-sitter isolation) is out of scope for V1 — it graduates via a separate Discussion if the need materializes.