FlexDoc and DocGraph: Design Specification

July 20, 2026 · View on GitHub

Status: Specifies the document model in the flexdoc package: the FlexDoc Python core and the DocGraph serialized projection. Design decisions are recorded in this document; §14 states what is implemented versus specified but not yet built. Dated planning documents under docs/project/specs/ track work toward this spec and reference it; this spec does not depend on them.

1. Purpose

The document model consolidates, in one source-anchored structure, what a document analysis or editing task normally needs from separate tools, and anchors every piece back to the source by exact character offset:

  • Markdown block structure: headings, paragraphs, lists/list items, tables, code, blockquotes, HTML, footnotes, thematic breaks.
  • Markdown inline structure: links, code spans, inline HTML.
  • Language structure: paragraphs, sentences, and words, with the exact spacing between them.
  • Document structure: heading hierarchy and table of contents.
  • Document sizes at every grain: bytes, characters, lines, raw and normalized words, sentences, paragraphs—and approximate LLM token estimates—all derived on demand, never stored. Sizing matters because the model’s main consumers window, chunk, and budget documents for LLM processing.

A Markdown parser gives you a block/inline AST but no sentences, sizes, or section rollups. An NLP toolkit gives sentences but no Markdown structure and no exact source mapping. This model is both, anchored to one retained source string.

Two surfaces, one design:

  • FlexDoc is the Python core: the in-process object for parsing, analysis, rollups, transforms, and editable reassembly.
  • DocGraph is the serialized, language-neutral projection of the same content: a JSON contract for frontends, cross-language clients, and annotations. It is derived from FlexDoc, not a competing model.

Terminology

Terms used throughout; each is detailed in the section noted.

  • Source text—the single immutable input string a FlexDoc retains. Everything else is derived from it (§4.1).
  • Offset / span—a position, or [start, end) half-open range, in the source text, measured in Unicode code points (§4.1).
  • Layer—one independent parse dimension over the same source: textual, markdown, document, synthetic (§3).
  • Node / node table / node kind—the id-addressed record of one parsed element; the flat table of all of them across layers; and the element’s type tag (heading, paragraph, link, section, ...) (§4.3).
  • Block—a Markdown block-level element. §6 disambiguates the three precise senses: block element (the CommonMark class), block node (a node in the recursive structural tree), and base block (a unit of the flat sequential partition).
  • Paragraph / sentence (editing view)—the blank-line-delimited units and their sentences; the mutable view edits and reassemble() operate on (§4.4, §12).
  • Section—a heading plus the content it owns, nested by heading level (§7).
  • Frontmatter—a leading ----delimited YAML block, treated as a non-content region (§3).
  • Sizes and TextUnit—the units a document or any unit can be measured in: bytes, chars, lines, raw_words, words, wordtoks, sentences, paragraphs, tokens. raw_words is a literal whitespace-delimited count. words always means logical words: the default normalized word-equivalent count across natural languages and code. It equals raw_words for non-wide text averaging 3–6 non-whitespace characters per whitespace-delimited word. Longer averages increase it, shorter averages decrease it, and wide/fullwidth characters contribute 0.5 each. The tokens unit is an approximate LLM token estimate derived from this measure, useful for windowing and budgeting; it is never a provider-exact count.
  • Wordtok—the model’s lowest-level lexical unit, used by the editing view and by downstream word-oriented diff/window machinery: a word, a punctuation character, a whitespace break, or an embedded HTML tag kept whole, with sentence/paragraph breaks represented as sentinel tokens. Wordtoks are an implementation-level unit for exact word-level alignment; they are not LLM tokens (see tokens above for that).
  • DocRef—a validated locator for the source document (§11).
  • SpanRef—quote, context, and optional position evidence for one non-empty source range (§11).
  • TextRef—a portable document reference with an optional source hash and optional span, point, or section selector (§11).
  • Annotation—a consumer-owned body and motivation targeting a TextRef; annotation identity and state are separate from reference identity (§11).
  • DocGraph—the serialized JSON projection of a parse (§10).

2. Principles and Goals

The foundational principles are the spine; the representation and pragmatic principles follow from them. Decisions throughout this spec cite the principle they serve.

Principles

Tier 1—Foundational (the spine):

  • P1. One immutable source, one shared offset space, is the canonical substrate. Every structure aligns to source_text by exact [start, end) in Unicode code points. This, not any one parsed structure, is what unifies the model.
  • P2. Source is canonical; every structural model is a derived, re-derivable projection. Edits go through the source/editing view and re-derive; the parsed model is read-mostly and never drifts from the text.
  • P3. Layered parsing; cross-layer relationships are offset-containment queries, not stored edges. The same source is parsed along independent dimensions (textual, markdown, document, synthetic), each tagged by layer; relations between layers are interval containment, never persisted edges.
  • P4. Mechanism over menu. One general query primitive (collect()) and two composable serialization axes (include layers / detail payloads); no blessed per-kind rollups or fixed detail ladders. New capability is one additive layer or detail, not an API refactor.
  • P5. Model ≠ format ≠ implementation. The contract is a language-neutral JSON schema (Pydantic-authored); Python today, Rust/TypeScript later, implement one frozen contract.

Tier 2—Representation (what the model must faithfully carry):

  • P6. Exact ground-truth references (source_text[s:e] == original_text).
  • P7. Faithful, complete Markdown structure—all modern block and inline kinds (tables, footnotes, fenced code, blockquotes, inline/block HTML, images); one top-level type per block; recursive, fully-populated nesting; bullet vs. ordered distinct.
  • P8. Textual structure with clean round-trip—paragraphs, sentences, and wordtoks (see Terminology) as the editing view; edit units and reassemble().
  • P9. Document structure—heading hierarchy, sections, and TOC, with section containment ("which paragraphs are in which section").
  • P10. Synthetic tag structure—marker-tag regions as a first-class layer (§3).
  • P11. Clean whole-tree round-trip editing—after arbitrary edits, reproduce a document with Markdown-object-level equivalence, and byte-exact equivalence when Flowmark-normalized or reconstructed from retained offsets (two distinct equivalence levels).

Tier 3—Pragmatics (cost, ergonomics, robustness):

  • P12. Single canonical form, derived views, no stored counts—sections, slices, and tallies are calculated fields; if a view is hard to derive, refine the form.
  • P13. Complete base-block partition—a flat, ordered, non-overlapping, depth-annotated cover of the whole document that reassembles to the source.
  • P14. Serializable projections—full parse or any slice to language-neutral JSON for frontends.
  • P15. Parse cost ≈ one Markdown parse; expensive views are lazy—pay for costly rollups on demand, cached on the immutable source.
  • P16. Approximation where cheap and sufficient—fast regex sentence segmentation and heuristic token sizing are accepted; exactness is reserved for offsets/spans.
  • P17. Lenient input, strict contracts, visible degradation—parsing never throws on malformed input; it degrades to deterministic, documented best-effort structure (see Error posture below).
  • P18. Explicit evolution. Serialized contracts carry format versions; runtime APIs model the current contract directly instead of accumulating version-suffixed types. New layers and details remain composable additions when their semantics are additive.

Error posture

P17, expanded into the three rules that govern errors throughout the model. Each layer’s section ends with its specific cases.

  1. Input is handled leniently and deterministically. Parsing any string yields a model; no input—malformed Markdown, broken tables, unclosed fences or tags, headingless or structureless documents—raises an exception. Every degradation is deterministic and documented in the owning layer’s error-handling subsection, and the golden test corpus pins the behavior (including a dedicated malformed-input document).
  2. Degradation is visible, never silent. When the model falls back, the fallback is observable in the structure itself: a block that fails to parse as a table is a paragraph node; an unlocatable reference link carries span=None; a headingless document has sections() == []. Consumers can always inspect what was actually recognized; nothing is patched over or guessed invisibly.
  3. Internal contracts are validated strictly. Builder invariants (layer nesting guarantees, deterministic node-id assignment) and serialization contracts (JSON-safe attrs) are checked and raise on violation—these indicate bugs in the model, never bad input, so failing loudly is correct. Where opt-in strictness on input exists it is explicit (e.g. the HTML tag finder’s strict=True raises on unparsable candidates instead of skipping them). A uniform opt-in strict-validation / diagnostics pass over a whole parse is specified direction, not yet built (§14).

Goals

Each goal realizes the principle(s) noted.

  • One consolidating structure (P7, P8, P9). Markdown block, inline, language, and document structure in one object, not four tools stitched together.
  • Exact source anchoring (P1, P6). Every unit references the source by [start, end); spans round-trip verbatim; no copies, no drift. Offsets are Unicode code points.
  • Normalized form, derived views (P12). One canonical form anchored to the offset space; section hierarchy, block-type slices, element rollups, and tallies are calculated fields, never stored. If a view is hard to derive, refine the form.
  • Simplicity with flexibility (mechanism over menu) (P4). One general query primitive (collect()), composable serialization layers (include), not a fixed menu of blessed rollups or detail levels, so the model serves many downstream uses without per-need API changes.
  • Markdown-correspondent block types (P7). One-to-one with Markdown kinds (bullet vs. ordered lists distinct); each block has one top-level type; nesting is recursive and fully populated.
  • Two block views (P13). A recursive structural block tree (for nesting/slicing/queries) and a flat sequential base-block list (a complete, ordered, non-overlapping partition whose reassembly reproduces the document), so a pipeline can process or resequence a document block by block.
  • Density-invariant lists (P7, P12). Tight and loose lists produce identical tallies.
  • Source-canonical portable references (P1, P2). Any locatable document object can produce a TextRef. Quotes and boundary context recover targets across edits; positions are hints unless a matching source hash binds them to the current source.
  • Cross-language contract (P5, P14). DocGraph is a plain, parser-agnostic JSON schema (Pydantic-authored); Python and any future TypeScript/Rust client are implementations of one contract.
  • Dual use (P8, P11). Analysis of a fixed document and an editable model: modify units, reassemble, serialize a clean normalized document.
  • LLM/agent-friendly, Python-first, Rust-portable (P5). Ergonomic from Python and from LLM/agent code; a tight spec and thorough tests make a Rust port feasible.
  • Minimal dependencies; explicit evolution (P18). Keep the model small, version wire contracts visibly, and keep compatibility history out of the current runtime vocabulary.

3. The Normalized Form

Everything is aligned by span into a single retained source_text. The canonical substrate is the source text plus its offset space (P1): every structure references the source by exact [start, end) (Unicode code points) and is a derived projection. The node table is the primary such projection—a stable set of nodes addressable by id and span—and is what the serialized contract and cross-layer queries are built on:

  1. Source: source_text plus exact [start, end) spans (Unicode code points); each unit’s original_text is a computed slice, exact by construction.
  2. Node table: one node per block, inline element, and heading: Node{id, kind, parent, children, source_span, attrs}. Block containment is parent/children; this is taken from marko’s parse and referenced, not duplicated.
  3. Language structure: paragraphs, sentences, and the wordtok view, with spans and spacing tokens (the editing view).

A leading YAML frontmatter block (----delimited) is a non-content region: it is excluded from the node table, the block/section views, and the editing view (and so from every size/prose count), and exposed verbatim via FlexDoc.frontmatter. source_text retains it, so spans stay absolute and the document still round-trips. Opening and closing delimiters begin at column 0 and may have trailing spaces or tabs; leading whitespace disqualifies a delimiter. The returned frontmatter preserves those delimiter lines verbatim. A leading --- line with no closing --- line is not frontmatter; it parses as an ordinary thematic break (a deterministic, lenient reading of the ambiguity).

Why a node table and not a single tree: a document has several hierarchies that overlap and do not nest: a section spans sibling blocks and is not a subtree of the block tree; links are inline ranges; annotations target arbitrary spans. A single canonical tree privileges one hierarchy and forces the rest into bespoke overlays. A node table gives one id space for blocks and inline items, so the containment tree, section tree, block list, link index, and token stream all share one id space, and the serialized JSON still presents a document root with children as its top-level shape.

The ergonomic Python views (blocks(), sections(), links(), base_blocks()) and the node table both derive from the same memoized structural parse over the shared offset space; they are sibling projections, not a store and lookups into it. This keeps parse cost at roughly one Markdown parse (P15) without coupling every view to the node-table schema.

Views over the form. Several ways to walk the same source_text, none of them the canonical store:

  • the blank-line Paragraph/Sentence editing view (the unit set used by word-level diffing and windowing);
  • the structural block tree (the recursive Markdown backbone: slicing, nesting, per-item access);
  • the sequential block list (the flat, non-recursive base-block partition, §6);
  • the section tree (heading hierarchy);
  • the inline/link index.

The editing view’s block boundaries are unchanged by the structural tree, so there is no forced migration of the editing unit.

Parse layers. Those views are not ad-hoc; each is one parse layer over the shared offset space. flexdoc parses the same source_text along several independent dimensions, each contributing nodes tagged with their layer:

LayerProducesDepends onNesting guarantee
textualparagraphs, sentences (wordtoks are a stream view, not nodes)—ordered list
markdownblock elements (recursive) and inline (links, code spans, images, inline HTML, footnote refs; emphasis is not modeled as nodes)—tree
documentsection / heading hierarchy and TOCmarkdown (headings)tree
syntheticmarker-tag regions (see below)—tree

Two consequences define how layers interact:

  • Cross-layer relationships are offset-containment queries, not stored edges. Within a layer, navigate parent/children; across layers, use interval containment/overlap ("which markdown blocks are inside this region", “which section contains this link”). This is what lets layers overlap and cross-cut without contradiction (a section is not a subtree of the block tree; a marked region may open mid-block).
  • Each layer declares a nesting guarantee: well-nested layers project to a tree view, ordered-only layers to a sequential list view (the §6 tree-vs-partition distinction, generalized). The SpanRef-targeted annotation layer (§11) is the out-of-band layer, anchored to the same offset space.

The synthetic layer

The synthetic layer carries structure that authors or tools introduce into the text with marker tags: a configured subset of XML-style elements whose open/close pairs delimit regions for chunking, grouping, and in-band metadata. Each recognized region becomes a node in the synthetic layer with the tag’s name and attributes in attrs, so “which markdown blocks fall inside this region” is the same offset-containment query as any other cross-layer relationship.

The tag vocabulary is configuration, not code: it is a defined whitelist of tag names, and adding a tag is adding an entry. In practice the vocabulary takes a few common shapes:

  • custom extension-style tags, typically lowercase and hyphenated in the HTML custom-element convention (<my-chunk id="a">...</my-chunk>), or simple semantic XML names such as <chunk>;
  • generic HTML containers—<div> and <span> are the most common in existing documents;
  • comment-delimited directives such as <!-- chunk id="foo" -->, which carry structure in Markdown without rendering.

Tags outside the configured vocabulary are inert: they remain ordinary text/HTML in the other layers and produce no synthetic node. Because layers are compositional (P3, P18), future extensions can introduce further parsing-based layers of the same shape—any mechanism that yields spans over the shared offset space can contribute a layer—without changes to the node table, collect(), or the serialization contract.

Layers are enabled à la carte: a configuration, not a fork of the model.

Status: the synthetic layer is specified here but not yet implemented in this package; §14 states what exists today and where the migration is tracked.

Error handling—synthetic layer (specified behavior). Unknown tags: inert, by definition of the whitelist. An unclosed marker tag forms no region; the tag itself remains visible to the markdown layer as inline/block HTML (rule 2: the degradation is observable). Regions that fail to nest (overlapping open/close pairs) violate the layer’s declared tree guarantee; the implementation must either reject the offending region (lenient: drop it, keep the text) or relax the layer’s guarantee to ordered-list—this is an open implementation decision recorded with the migration plan. Builder-side, whichever policy is chosen is then enforced strictly at node-table build like every other layer invariant.

4. Core Types, Nodes, and Offsets

4.1 The source and its offset space

A parse begins from one immutable string, the source text (source_text). All positions in the model are offsets into this string, counted in Unicode code points (Python’s native string indexing—not bytes, and not UTF-16 code units). A span is a half-open offset pair [start, end), so source_text[start:end] is exactly the spanned text, empty spans are representable, and adjacent spans share a boundary without overlapping.

The defining invariant of the whole model (P1, P6) is that every located, source-backed span selects the unit’s authoritative text:

unit_text = source_text[unit.span[0] : unit.span[1]]

Paragraphs and sentences also expose this slice as original_text. Blocks, sections, links, and nodes expose their spans; identities that cannot be located use span=None and do not claim an exact source slice.

Code points are the canonical unit because they are the one offset basis every language runtime can reproduce exactly; byte offsets (UTF-8) and UTF-16 units (browsers) are derivable on demand and may be exposed by DocGraph as secondary coordinates, but the canonical source_span is always code points. This matches the W3C Text Quote Selector requirement that text selection use Unicode code points rather than code units.

Documents built with from_wordtoks use their initial reassembled text as source_text. Later editing operations, including append_sent, change the editing view but do not rewrite source-backed spans or source_text; callers reparse doc.reassemble() to establish a new coordinate space (§12). Similarly, from_text normalizes line endings (\r\n and lone \r become \n) before retaining source_text, so the invariant holds against the normalized string (§4.5).

4.2 The document object: FlexDoc

FlexDoc is the package’s entry point and the owner of one parse:

  • Construction: FlexDoc.from_text(text) normalizes line endings (§4.1, §4.5) and retains the result as source_text, isolates any leading frontmatter, and builds the editing view (paragraphs and sentences) eagerly. All other projections are built lazily on first use and cached against the immutable source (P15).
  • Owned views: the paragraphs list (the editing view, §4.4); and the derived, lazily-cached projections—blocks() (§6), base_blocks() (§6), sections() (§7), links() (§8), node_table() (§4.3), collect() (§9), and graph() (§10).
  • Offset inversion: paragraph_at_offset(o) and sentence_at_offset(o) map an absolute offset back to the editing-view unit containing it, and block_at_offset(o) to the innermost structural Block (each None for offsets in inter-unit whitespace or outside the document). Structural blocks are otherwise addressed by their own spans or via collect(overlaps=...).
  • Frontmatter: FlexDoc.frontmatter is the leading YAML block from normalized source_text, or None; delimiter whitespace rules are defined in §3.
  • Sizing: size(unit) and size_summary() measure the document in any TextUnit (see Terminology), including the approximate LLM tokens estimate. A summary's words field uses logical-word semantics. Document and section word totals measure the full paragraph sequence before rounding, so they can differ from a sum of independently rounded sentence counts.
  • Prose projection: prose_text(include_tables=False) returns prose-only text for editorial linting and metrics—prose-bearing blocks (paragraphs/headings, and table cells when include_tables=True) with inline code/footnote refs dropped, links/images reduced to their text/alt, inline-HTML tags removed (wrapped text kept), and heading/blockquote/list markers and reference-definition lines stripped. Slices are verbatim (line wrapping preserved, never reflowed), so editorial spacing (e.g. a spaced em-dash) survives.

4.3 Nodes, kinds, layers, and the node table

A node is the uniform record of one parsed element, from any layer:

  • id—a string id, unique within the parse. Ids are assigned by a single contiguous preorder counter (n0001, n0002, ...) over a fixed build order (markdown block tree, then document sections, then textual paragraphs/sentences, then inline elements), so two parses of the same source produce identical ids. This determinism is part of the cross-language contract and is pinned by test.
  • kind—the element’s type: the Markdown block kinds of §5, the inline kinds of §8 (link, code_span, image, inline_html, footnote_ref, link_ref_def), the document-layer section, and the textual-layer sentence.
  • layer—which parse dimension produced it (§3).
  • parent / children—within-layer containment edges (node ids). Cross-layer relationships are never stored; they are offset queries (P3).
  • source_span—the node’s exact span (§4.1), or None for the few elements that have identity but no locatable position (e.g. an unresolvable reference link).
  • attrs—typed metadata as a JSON-safe mapping (AttrValue: strings, numbers, booleans, None, and lists/maps of the same). Examples: a heading’s level, a list’s tight/ordered, a link’s url/title, a code block’s language. JSON-safety is validated at serialization (§10); parser-internal objects never appear in attrs.

The node table (NodeTable) is the flat, id-addressed collection of all nodes in a parse, plus the list of root ids per layer, over the shared source_text. It is a projection like the others—built from the parses, not the store they read from—but it is the projection queries (§9) and serialization (§10) operate on.

Each layer declares a nesting guarantee (§3’s table): tree layers promise that a child’s span lies within its parent’s; ordered-list layers promise siblings are ordered and non-overlapping. These guarantees are validated strictly when the table is built—a violation raises, because it can only mean a bug in a layer builder, never malformed input (Error posture, rule 3).

4.4 The editing view: paragraphs and sentences

The editing view is the mutable face of the model (§12): the units whose text can be edited and reassembled.

  • Paragraph—one blank-line-delimited unit of the source. Carries original_text (the verbatim slice), its sentences, offsets, a span, a cached Markdown classification block_type (§5) with heading helpers (heading_level, heading_title), typed code_info/table_info/list_info conveniences, and links().
  • Sentence—one sentence within a paragraph. text is the normalized, editable content (what reassembly uses); original_text is the verbatim source slice; span is exact when original_text is present. Sentence boundaries come from flowmark’s span-aware splitter, which never bisects a link, code span, autolink, or URL, so sentence spans are exact for all content.
  • Offsets(doc_offset, block_offset)—every paragraph and sentence carries both its absolute offset in the document and its offset relative to its enclosing unit (the document for a paragraph, the paragraph for a sentence).
  • SentIndex(para_index, sent_index)—the stable address of a sentence within a FlexDoc, used by editing and diff/window machinery.

A custom sentence splitter may be supplied; offsets for its sentences are recovered by search and are best-effort where the splitter normalized whitespace (a documented, visible degradation—original_text is absent for such sentences, and span falls back to the normalized length).

4.5 Error Handling: Textual Layer

The textual layer accepts any string; there is no invalid input.

  • Empty or whitespace-only input parses to a document with zero paragraphs; sizes are zero; iteration yields nothing (boundary sentinels are still emitted for the wordtok stream so downstream alignment has stable endpoints).
  • Line endings: \r\n and lone \r are normalized to \n by from_text, and source_text retains the normalized string, so every layer shares one offset space. (The underlying Markdown parser computes positions against LF-only text; retaining \r would desynchronize structural spans from the source.) Callers anchoring offsets to an external CRLF original must normalize it the same way first.
  • Sentence segmentation is heuristic (P16): abbreviations or unusual punctuation can mis-split. The degradation is visible, not corrupting—every sentence still carries an exact verbatim span, so a “wrong” boundary is a presentation choice, never a wrong offset.
  • Custom splitters degrade as described in §4.4: best-effort offsets, observable via the missing original_text.
  • No strict mode is needed at this layer: there is nothing to reject—the layer’s output is a total function of the input string.

5. Block-Type Model

BlockType corresponds one-to-one to Markdown block kinds: heading, paragraph, list (bullet/unordered), ordered_list, list_item, table, code, blockquote, html, footnote, thematic_break.

  • Bullet vs. ordered lists are distinct types. marko’s List carries ordered; list is the bullet list, ordered_list is enumerated, list_item is shared.
  • One top-level type per block, from its outer element: a blockquote wrapping a table classifies as blockquote at the top level. This is what the top-level views and default rollups key on.
  • Nesting is recursive and fully populated. Every container (blockquote, list item) has its block children in the node table, so a table nested inside a blockquote is a node and is found by a recursive collect(). The inner table keeps its own table kind; default (shallow) rollups attribute it to its enclosing top-level block, recursive rollups count it directly.

List density must not change tallies. A dense list and the same list written sparsely are the same list. In the structural tree a list always decomposes into list_item children regardless of density: a loose list is one list block with blank-line-separated items, not N lists, so len(list.children) and any tally are density-invariant. Density is metadata, not structure: a tight: bool on the list (CommonMark semantics); the flag never enters a tally.

Typed per-block metadata. Code, table, list, and heading blocks carry parser-authoritative typed metadata (flexdoc.docs.block_info): CodeInfo (language, line_count), TableInfo (rows, cols, cells, alignments), ListInfo (ordered, start, max_depth, item_count), and HeadingInfo (level, title). It is computed once where the marko element is in hand and exposed on the structural Block (Block.code_info/.table_info/.list_info/.heading_info—the density-invariant source of truth) and, as a convenience carrying the editing-view density caveat, on Paragraph. The same facts are flattened into the markdown node’s attrs, so they flow into collect()/DocGraph. Extraction is parser-authoritative (marko element attributes, never a regex over source); a table column with no alignment marker is "default", not None, so alignments is always explicit strings of length cols. TableInfo.alignments is a tuple so this metadata remains immutable when its owning block is cached.

6. Block Views: Structural Tree and Sequential Base-Block List

Terminology. To avoid overloading “block”:

  • block element / inline element: the Markdown element class (CommonMark/mdast sense): block-level (heading, paragraph, blockquote, list, list item, table, code, …) vs inline (link, code span, emphasis, …).
  • block node: a node with a block kind in the recursive structural block tree (blocks()); containers (blockquote, list, list item) contain child block nodes.
  • base block: a unit of the flat sequential block list (base_blocks()): a complete, ordered, non-overlapping partition of the document into the units a pipeline processes or a UI resequences.

These are two views over the same shared parse, for two different jobs (and they must not be conflated; see §9: the tree supports queries that may overlap; the base-block list is a partition with a cover invariant).

Structural block tree: FlexDoc.blocks() -> list[Block]

The recursive view (lazy, cached on the immutable source_text):

  • Block(type, span, children, tight): a frozen record whose span is trimmed so source[start:end] is the exact text; children is an immutable tuple of nested blocks. A list/ordered_list block’s children are its list_items; containers fully populate their block children (a blockquote’s or list item’s nested blocks are present). tight carries CommonMark list density on list blocks (None elsewhere).
  • Resolves what blank-line splitting cannot: a fenced code block stays whole even with internal blank lines; a list decomposes into items with nested sublists; a table inside a blockquote is reachable.

Block boundaries and spans come straight from flowmark’s parser: every block element carries an authoritative element.span = (start, end) read from marko’s own source positions (flowmark.markdown_ast.block_span), so flexdoc runs no block-detection regex of its own and makes no block-boundary decisions. The structure is cross-checked against marko in tests. FlexDoc.blocks() returns a fresh root list over the shared, recursively immutable graph, so callers can reorder the result but cannot mutate cached block state.

Sequential block list: FlexDoc.base_blocks() -> list[BaseBlock]

A base block is a BaseBlock wrapping a block node (Block) with a depth; the base-block list is a partition of the document: the ordered sequence of base blocks, each carrying its depth. FlexDoc.base_blocks() is a thin method over the flexdoc.docs.base_blocks.base_blocks(text, *, item_partition_depth=6) free function (the partition lives in its own module, distinct from the recursive tree in block_tree.py). It is the view for block-by-block pipelines and outline UIs that move/resequence blocks (e.g. Notion-style drag-and-drop, where every item is a draggable unit). flexdoc does not implement such UIs, but the model supports addressing and reordering at this granularity. Document manipulation and processing happen base block by base block.

Frontier. Leaf and atomic blocks (heading, paragraph, table, code, thematic break, HTML, and a whole blockquote) are each one base block. Lists decompose: each list item, at every nesting level, is its own base block with increasing depth (flat-with-depth). An item holding a nested list contributes a list_item head block (the marker and lead content) at depth d, then its nested items at d+1; any continuation content (paragraphs after or between sublists) follows as base blocks carrying their own real block type (e.g. paragraph) at depth d—never relabeled list_item, so a consumer can tell a continuation paragraph apart from an independent list item.

How deep lists decompose is one numeric parameter, base_blocks(item_partition_depth=6):

  • item_partition_depth = N (default 6, deep enough for normal nested lists): split list items down to N nesting levels; list content nested deeper than N stays whole inside its depth-N base block (avoids pathological fan-out on very deep lists).
  • item_partition_depth = -1: unlimited; split at every nesting level.
  • item_partition_depth = 0: lists are not split; each list is a single base block (coarse mode, i.e. the top-level blocks).

Blockquotes are always one base block regardless of item_partition_depth.

Invariants (validated and documented): the base-block list is ordered by position, non-overlapping by span, and together the spans cover every non-whitespace character exactly once (the gaps are inter-block and structural whitespace). Every base block retains its exact source_span, so exact source reconstruction is by slicing the source at those spans (or via the structural blocks() tree). Reassembling the rendered base-block text is lossy for list-item continuation content—list markers and continuation indentation are whitespace outside the trimmed spans, so naive text concatenation normalizes them; reconstruct from offsets when exactness matters. A pipeline may process, edit, or resequence base blocks; depth is mutable metadata, so promoting a depth-2 item to depth-1 on a move just changes its rendered nesting, not a violation.

Error Handling: Markdown Layer

The Markdown parse is total: every input yields a block tree, with CommonMark’s own recovery semantics (via marko) deciding how malformed constructs degrade. The common cases, all deterministic and pinned by the golden corpus (which includes a dedicated malformed-input document):

  • Unclosed fenced code block: the fence runs to end of document; everything after the opening fence is one code block (CommonMark semantics). Visible: the block’s span shows exactly what was swallowed.
  • Malformed table: rows that do not parse as a table degrade to paragraph blocks; a valid table region keeps its table kind and explicit alignments.
  • Broken or unclosed HTML: block-level HTML that marko cannot classify remains an html block; a single-line tag that marko reads as an inline-HTML paragraph is classified html by an explicit markup fallback. Tags are never “repaired.”
  • Reference links without definitions / unlocatable constructs: identity is kept, span=None marks the unlocatable position, and offset-scoped views exclude them (rule 2: visible, not guessed).
  • Inconsistent list markers / indentation: CommonMark’s list-interruption and lazy-continuation rules apply; the result may split or merge lists, but spans and types always describe what the parser actually decided.
  • Strictness: there is no strict mode at this layer today—CommonMark itself is defined to be total—but classification is fully observable, so a caller can layer its own validation (e.g. “this document must contain no html blocks”) over collect(). A uniform diagnostics pass is future work (§14).

7. Sections and TOC

The document layer derives a heading hierarchy from the markdown layer’s headings—no re-parse, no stored state.

Construction rules

  • What starts a section: exactly the top-level structural heading blocks of blocks(). Gating on structural blocks is what makes the layer robust: a #-prefixed line inside a fenced code block is not a heading (the structural tree keeps the fence whole), and headings nested inside blockquotes or list items are not top-level blocks, so neither starts a document section.
  • Ownership: a section owns the content from its heading up to (not including) the next heading of equal or higher level. Content between a heading and a deeper heading belongs to the shallower section directly (content); the deeper heading starts a nested child section.
  • Nesting: sections nest strictly by level using stack semantics—an incoming heading of level n closes every open section of level ≥ n and attaches to the nearest open section of level < n, or becomes a root if none is open. Multiple top-level headings yield multiple roots.
  • Preamble: content before the first heading belongs to no section. It remains fully present in every other view (paragraphs, blocks, sizes); it is simply not section-owned.

The Section type

  • heading—the heading’s editing-view Paragraph; title is its text without markers (an empty string for a bare #).
  • level—the heading level, 1–6, exactly as authored.
  • content—the section’s own paragraphs (excluding the heading line and excluding everything owned by child sections).
  • children—nested Sections, in document order.
  • own_paragraphs() / subtree_paragraphs()—the heading plus content; the same plus all descendants’ paragraphs, in document order.
  • blocks()—the structural block tree (§6) restricted to the section’s own content span; density-invariant like the whole-document tree, so per-section block-type tallies are spacing-independent.
  • span—[heading start, end of last subtree paragraph): the full extent of the section including its subsections.
  • size(unit, subtree=True|False) / size_summary(...)—sizes in any TextUnit, rolled up over the subtree by default or restricted to own content; computed by the same machinery as FlexDoc.size, so every unit (including the approximate LLM tokens estimate) aggregates uniformly.
  • links()—links in the section’s subtree, attributed by span containment; links with span=None are excluded (they cannot be placed by offset).

Document-level accessors

  • FlexDoc.sections()—the list of root sections (computed once and cached internally; each call recursively copies the section tree and its editable paragraphs, so caller mutation cannot affect the cache or a later result).
  • FlexDoc.toc()—the flat table of contents: (level, title, span) per heading, in document order, by walking the section tree.
  • FlexDoc.section_size_tree(units=...)—a rendered, indented size rollup per section, for quick structural inspection; its default unit is words, using logical-word semantics.

Error Handling: Document Layer

Documents are under no obligation to be well-structured; the section layer is total and its degradations are visible:

  • No headings at all: sections() == [] and toc() == []. The document is still fully usable through every other view; “no sections” is a true statement about the document, not a failure.
  • Preamble-only or mostly-unstructured documents: the preamble rule covers them — content simply belongs to no section, and per-section rollups cover whatever sections do exist.
  • Skipped levels (e.g. an ### directly under a #): no intermediate sections are synthesized; the ### nests directly under the #, and its level remains 3 as authored. Authors’ level choices are preserved, never “corrected.”
  • Out-of-order levels (a document starting at ##, or an # appearing after ###): handled by the same stack rule—a shallower heading closes deeper open sections and becomes a root or sibling as the rule dictates. Nothing raises.
  • Malformed near-headings: #Title without a space, or seven-plus # characters, are not CommonMark headings—they parse as paragraphs and therefore start no section (consistent with the markdown layer’s classification, which is the single source of truth). A heading with no text (# alone) is a real heading with an empty title.
  • Setext ambiguity: a text line underlined with ===/--- is a setext heading (level 1/2); a bare --- with no text above is a thematic break; an opening --- at offset 0 with a closing --- line is frontmatter (§3). All three readings are deterministic and mutually exclusive.
  • Duplicate titles are legal; sections are identified by position and span, never by title.
  • Strictness: none is imposed—but the layer’s output makes validation trivial to express externally (e.g. assert toc() levels start at 1 and never skip), and a built-in opt-in diagnostics pass is specified future work (§14).

Inline elements (links, code spans, images, inline HTML, footnote references, …) are first-class nodes whose parent is their containing block, with computed section/sentence associations, so block↔inline relationships are node edges, and “links in section 3” is a scoped collect(kinds={link}).

  • Link(text, url, title, span, link_form): identity from flowmark.markdown_ast.extract_links (reference links resolved, escapes honored), an AST walk for images, and marko’s LinkRefDef elements for definitions. Each carries a LinkForm discriminator—inline, reference, autolink, bare_url, image, or reference_definition—so consumers count by form without heuristics. flexdoc recovers each exact [start, end) by reconciling the ordered identities with the name-tagged atomic spans from flowmark.atomic_spans.iter_atomic_spans (and a trimmed block_span for definitions); an identity that cannot be located keeps its identity with span=None.
  • links() returns navigable links only by default (NAVIGABLE_LINK_FORMS: inline, reference, autolink, bare_url); links(link_forms=…) selects any forms and images() is the convenience for LinkForm.image. Reference definitions ([id]: url) are surfaced as NodeKind.link_ref_def nodes—parented to their containing block, so a block-scoped collect() finds them—and via links(link_forms={LinkForm.reference_definition}), never the default links() (a definition is not a link occurrence).
  • link → sentence via sentence_at_offset(link.span[0]).
  • footnote_ref: a footnote reference [^label] is a first-class inline node (NodeKind.footnote_ref) carrying its label in attrs and an exact span, collected like any inline kind (collect(kinds={NodeKind.footnote_ref}, recursive=True)). A footnote definition ([^label]:) is a footnote block, not a reference.

Error handling—inline elements. Inline parsing inherits the markdown layer’s total, lenient posture. The cases specific to this sublayer: an identity that cannot be located in the source keeps its identity with span=None and is excluded from offset-scoped rollups; a URL or link text that appears multiple times resolves in document order (a forward cursor prevents one unlocatable identity from desyncing the rest); escaped constructs are honored as escapes, not links. Nothing raises.

9. Derived Views and Rollups

All derived from the canonical source/offset substrate (the node table is the id-addressed projection used for queries); nothing stores counts. These structural/query views describe the parsed source_text; after editing, re-parse with from_text(doc.reassemble()) before structural analysis. The surface is one general query primitive, no blessed per-kind rollups:

collect(*, subtree_of=None, within=None, overlaps=None,
        kinds=None, where=None, recursive=False, inline=None, layer=None) -> list[Node]

Available as doc.collect(...) (and as the free collect(table, ...) over a node table). Two distinct relations select candidates. The tree relation subtree_of= takes a node id and restricts to that node’s within-layer parent/child subtree (recursive descends it). The interval relations are cross-layer and offset-based, each accepting a node id or (start, end) span: within= keeps nodes whose span is contained in the region (e.g. within=section_id for everything inside a section); overlaps= keeps nodes whose span merely intersects the region. Supplying an interval relation scans the whole document, so within=section_id needs no recursive=True. kinds= selects by node kind (the typed common case); where= is a Node -> bool predicate escape hatch. When inline is omitted, recursive traversal and an explicit inline kinds selection include inline nodes; inline=False explicitly excludes them, and inline=True includes them for any query. layer= restricts parse layers. It returns nodes (each with span, attrs, edges). Counts, values, and groupings are standard Python over the result, documented with worked examples, not separate methods:

doc.collect(kinds={NodeKind.table}, recursive=True)        # table values and spans
len(doc.collect(kinds={NodeKind.table}, recursive=True))   # how many
Counter(n.kind for n in doc.collect(recursive=True))       # tally by kind
doc.collect(within=section_id, kinds={NodeKind.link})      # links in a section

Slice-by-block-type, per-section rollups, and element rollups are all expressions of this one primitive; relationships are node edges. There are no tables()/code_blocks() shortcuts to maintain.

Query vs. partition; do not conflate. collect() is a query: it gathers matching nodes and the results may overlap their containers (a table nested in a blockquote is returned by collect(kinds={table}, recursive=True) alongside the blockquote). That is correct for counting/gathering. The base-block list (§6) is a partition: a complete, ordered, non-overlapping cover for linear processing. Use collect() to ask “how many / which”; use base_blocks() to iterate the document’s content units.

10. DocGraph: The Serialized Projection

DocGraph is the strict, language-neutral JSON projection of one FlexDoc source snapshot. It contains one node table, derived node-id views, and enough shared source identity to turn every locatable node back into a complete TextRef (§11). Python class names and parser implementation details are not part of the wire contract.

The current shape is:

DocGraph = {
  schema: "DocGraph/v0.2",
  source: {
    format,
    offset_unit: "unicode_code_points",
    document,
    source_hash,
    text?
  },
  nodes: [Node, ...],
  views: {toc, blocks, links, paragraphs, sentences},
  annotations: [AnnotationSetEntry, ...],
  layout: [],
  provenance: []
}

There is one runtime DocGraph model and one builder. The version belongs in the serialized schema field, not in Python class names. document and source_hash are required even when annotations is empty because they define what every source_span means outside the creating process. source_hash is the algorithm-qualified hash defined in §11.1; there is no second, redundant digest field.

FlexDoc.graph(document=..., include=..., detail=..., annotations=...) builds the projection. The caller supplies the document locator because source text alone cannot identify where another process should retrieve it. If annotations are embedded, their sidecar document and non-null source hash must match the graph source exactly.

Source Spans as a Reference Basis

Every locatable node carries source_span: {start, end} in the graph's shared Unicode code-point offset space. Together with source.document and source.source_hash, this is a compact reference basis, not a second reference format. To materialize a TextRef for a node, a client:

  1. Uses source.text when present, or retrieves canonical source through source.document.
  2. Verifies that source against source.source_hash.
  3. Slices [start, end) to obtain the exact quote and immediate context.
  4. Constructs the span or semantic section selector described in §11.

The same derivation is exposed in Python by TextRefContext.for_target(). A client must not persist only a node id or raw offsets: node ids are graph-local and positions alone lose meaning when the source changes. An unlocatable node has source_span: null and cannot be converted to a source TextRef without another explicit adapter.

Projection Controls

Two composable axes control what is built, without changing the graph type:

doc.graph(document="docs/design.md")
doc.graph(document="docs/design.md", include={Layer.markdown})
doc.graph(
    document="docs/design.md",
    include={Layer.markdown, Layer.document, Layer.textual},
    detail={Detail.text, Detail.inline},
    annotations=annotation_set,
)
  • include selects parse layers. Enabling document also enables its markdown dependency.
  • detail selects payloads within enabled layers. text includes the canonical source and per-node slices; inline includes inline nodes. tokens and coords are reserved payload categories.
  • annotations optionally embeds consumer-owned annotation entries. It is data in the same graph contract, not a different graph model.

The structural core—node ids, kinds, layers, tree edges, and source spans—is always present for included nodes. layout and provenance remain reserved until their schemas are specified.

Error handling—serialization. A DocGraph is produced only when it satisfies the schema. Invalid JSON values in attrs, dangling view or tree ids, malformed source identity, a source-text/hash mismatch, or mismatched annotation context raises. These are contract violations, not malformed-Markdown degradation. Node ids and emission order are deterministic and contract-tested.

11. TextRef, SpanRef, and Annotations

A source offset is useful inside one parse but is not a portable reference: it omits the document, becomes stale after edits, and provides no evidence for recovery. TextRef solves that problem with one small composition:

TextRef = DocRef + optional source_hash + optional selector
selector = span | point | section

The four target kinds are exhaustive: whole_document, span, point, and section. Omitting selector means the whole document. TextRef identifies where; annotation bodies, workflow state, edits, and presentation remain separate values.

11.1 Canonical Source and Document Identity

TextRefs address canonical source text, not rendered HTML or visual layout. FlexDoc's canonical source profile is:

  • Unicode scalar values only
  • line endings normalized to \n
  • offsets measured as zero-based Unicode code points
  • spans represented as half-open [start, end) ranges
  • hashing over the UTF-8 encoding of the complete normalized source

No Unicode normalization is applied. Except for line-ending conversion, the original Unicode code-point sequence is preserved; canonically equivalent NFC and NFD text are different source snapshots and quote evidence must come from the canonical source.

DocRef is a non-empty, opaque locator supplied by the consumer. It may be a repository locator, URL, database key, path, or application identifier. FlexDoc validates the string but does not perform I/O or decide that ./design.md and design.md are equal. A document store owns retrieval and any locator canonicalization.

source_hash is optional in a detached TextRef and has the form sha256:<64 lowercase hexadecimal digits>. It proves that position evidence was made against one exact canonical source snapshot. It does not retrieve that snapshot and does not make a mutable DocRef immutable; reproducible citations need an immutable DocRef or consumer-owned archival provenance.

11.2 Selector Model

Whole Document

Selector absence refers to the complete canonical source:

{
  "format": "textref/0.1",
  "document": "docs/design.md",
  "source_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
}

Span

A span selects any non-empty source range. The quote-anchored form carries exact selected text, optional immediate prefix and suffix context, and an optional starting position:

{
  "format": "textref/0.1",
  "document": "docs/design.md",
  "source_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "selector": {
    "type": "span",
    "exact": "Canonical source is authoritative.",
    "prefix": "The parser guarantees that ",
    "suffix": " Derived views are secondary.",
    "start": 1842
  }
}

The resolved range is [start, start + len(exact)) when that position is trusted. exact and context are durable recovery evidence; start is a hint unless the source hash matches.

The compact snapshot-bound form omits quote evidence and carries both positions:

{
  "format": "textref/0.1",
  "document": "docs/design.md",
  "source_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "selector": {
    "type": "span",
    "start": 1842,
    "end": 1876
  }
}

This form requires a source hash and resolves only when that hash matches; against any other snapshot it reports missing rather than trusting positions. It is appropriate when compact snapshot identity matters more than recovery after edits. Boundaries in either form may fall inside Markdown syntax or cross nodes, blocks, and lines. SpanRef is the lower-level quote/context/position value used to construct and resolve quote-anchored selectors over caller-supplied text.

Point

A point selects one boundary between code points for insertions, bookmarks, and zero-width comments. It is not an empty span:

{
  "format": "textref/0.1",
  "document": "docs/design.md",
  "source_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "selector": {
    "type": "point",
    "position": 1881,
    "affinity": "before",
    "prefix": "authoritative.",
    "suffix": "\nDerived views"
  }
}

prefix ends immediately before the boundary and suffix begins immediately after it. Affinity says which side owns the point across an insertion: before owns the prefix side and after owns the suffix side. A point normally requires context. The only context-free point is hash-bound position=0, the document-start sentinel and the sole boundary of empty source.

Section

A section selects a complete CommonMark heading-owned section, including nested subsections. Its start anchor identifies the complete heading. An optional end anchor identifies the next equal-or-higher-level heading, which is excluded:

{
  "format": "textref/0.1",
  "document": "docs/design.md",
  "selector": {
    "type": "section",
    "syntax": "commonmark",
    "start_anchor": {
      "exact": "## TextRef Design",
      "start": 7421
    },
    "end_anchor": {
      "exact": "## Open Decisions",
      "start": 12684
    }
  }
}

The resolver locates the start heading, derives the current range from the parsed heading hierarchy, and uses the end anchor as corroborating boundary evidence. It does not freeze all section text into the selector, so interior edits preserve section identity. A text-only resolver reports unsupported rather than guessing the range.

Headings without owned content, tables, rows, cells, list items, code blocks, links, and ordinary nodes use span selectors over their complete source ranges. New structural selector kinds are justified only when a real identity requirement cannot survive as a span.

11.3 Constructing TextRefs From FlexDoc

FlexDoc.references(document=...) binds one DocRef to one immutable FlexDoc snapshot, computes its source hash once, and returns a TextRefContext:

refs = doc.references(document="docs/design.md")

paragraph_ref = refs.for_target(doc.paragraphs[0])
section_ref = refs.for_target(doc.sections()[0])
link_ref = refs.for_target(doc.collect(kinds={NodeKind.link})[0])
arbitrary_ref = refs.for_span(start, end)
insertion_ref = refs.for_point(position, affinity="after")
document_ref = refs.whole_document()

By default, generated spans include complete exact quote evidence. Applications that need bounded reference size can choose their own policy once for the context, for example doc.references(document=..., max_exact_chars=1024). Spans above that limit use the compact hash-bound form. for_span(..., include_exact=...) and span-producing for_target(..., include_exact=...) override the context policy for exceptional cases. There is no format-level or library-default span-length cutoff.

The central mapping is:

FlexDoc valueTextRef target
Paragraph, sentence, block, base block, located link, ordinary nodeSpan over its complete source range
Section or document-layer section nodeSemantic section selector
Explicit source rangeSpan
Explicit source boundaryPoint
Complete documentWhole document

Every public value with a valid source span must map through this table. TextRefs are derived on demand rather than stored on parsed objects: this avoids repeating document and hash data and prevents stale reference objects after a reparse. Unlocatable values such as Node.source_span=None, objects from another document, reversed spans, and empty spans fail construction visibly.

Extraction and processing results remain consumer-owned. A result may retain one TextRef or tuple[TextRef, ...] in a source_refs field, allowing a scalar, summary, classification, chunk, search hit, diagnostic, or citation to resolve back to all of its evidence without FlexDoc prescribing the result schema.

11.4 Persisted and Transport Forms

The strict JSON object is normative. format and document are required; selector objects are a discriminated union; unknown fields are rejected; extension keys must be namespaced under extensions. JSON integers are limited to the interoperable safe range.

TextRef.to_uri() produces a canonical, reversible URI projection:

textref:0.1?doc=docs%2Fdesign.md&hash=sha256%3A...&type=span&exact=Canonical%20source&start=1842

The codec uses UTF-8 percent encoding, emits fields in canonical order, encodes spaces as %20, rejects duplicates and incompatible fields, performs no I/O, and refuses URIs over its defined size limit. The URI contains one TextRef only. It never contains an annotation body or workflow state.

Quote-anchored reference size is O(span length). Its URI form is therefore intended for modest spans and may be unavailable for large paragraphs, code blocks, tables, or chunks; use structured JSON when the full quote is required. Applications needing compact URIs can configure exact-quote capture or omit it per span, producing the hash-bound start/end form. Section selectors remain preferable for complete heading-owned regions because they preserve identity across interior edits without copying the section body.

Restricted YAML is a convenience projection for annotation sidecars. It must parse to the same JSON-compatible value tree before strict model validation; YAML tags, duplicate keys, and YAML-specific value semantics are not part of the format.

11.5 Resolution

Resolution separates document retrieval, snapshot validation, and selector matching so callers can distinguish stale content from missing or ambiguous targets:

AxisValues
Documentresolved, unavailable, invalid
Source validationabsent, matched, mismatched
Selectorwhole_document, resolved, missing, ambiguous, boundary_mismatched, unsupported
Methodsource_position, context_position, exact_quote, context_quote, point_context, point_affinity, section_structure, section_anchors, none

Consumers must inspect the document axis first. The selector axis is meaningful only when the document is resolved: a missing or different document prevents selector evaluation, and renderers present that document status instead of the unevaluated selector status.

A consumer first retrieves the requested DocRef. Failure to retrieve it is unavailable; supplying a different bound document is invalid. The resolver then normalizes and hashes the source, records whether the optional hash is absent, matched, or mismatched, and applies the selector's evidence ladder:

  • Span (exact-less): hash-bound start/end; a mismatched hash reports missing.
  • Span (quote-anchored): hash-bound start, position corroborated by quote/context, unique exact quote, then exact quote disambiguated by immediate context.
  • Point: hash-bound position, corroborated position, two-sided boundary context, then sufficiently strong affinity-owned context.
  • Section: resolve the heading anchors, derive the range from CommonMark structure, and verify the optional exclusive-end anchor.
  • Whole document: return [0, len(source)) after document and source validation.

Resolution never chooses an arbitrary duplicate. Missing, ambiguous, unsupported, and boundary-mismatched results are ordinary typed outcomes because a caller may display, retry, or reconcile them. Exact resolution is deterministic. Future normalized or fuzzy policies may consume the same persisted evidence but must report their method and must not silently rewrite the stored target.

11.6 Source Context and LLM-Friendly Rendering

TextRefContext.context() resolves a reference and returns structured selected source, the resolved range, surrounding source lines, one-based line and code-point-column labels, and resolution evidence. Lines and columns are display values, not persisted selectors.

render_context() produces a bounded Markdown-compatible view containing the document, target kind, TextRef URI, resolution status, selected source, and line-numbered source window. render_annotations() merges overlapping windows, places stable annotation IDs and bodies beside their targets, uses explicit elision, and groups unresolved targets separately. These views are deterministic and readable by humans and LLMs, but they are derived presentation and are never parsed back as authoritative records.

This supports two distinct operations with the same reference:

  • retrieve context around a cited or extracted source span without an annotation
  • render an annotation together with its target and surrounding document context

11.7 Annotations

TextRef answers where. A consumer-owned annotation answers what is said or proposed there. TextAnnotation contains an independent ID, one complete TextRef target, one or more motivations, an optional discriminated plain-text body, and optional style, tags, captured_text, and import provenance. Multiple annotations may share one target; multiple targets remain multiple TextRefs.

AnnotationSet is a strict one-document JSON/YAML sidecar. It hoists document and optional source_hash, stores bare selectors under each annotation, and expands them mechanically to complete TextRefs before resolution. A null target represents the whole document. Annotation IDs are unique within a set. Every entry must satisfy the complete TextRef evidence contract under the hoisted identity, and the set enforces this at construction: an exact-less span requires the shared source hash, and a context-free point is valid only as the hash-bound position=0 sentinel.

Annotations remain outside mutable FlexDoc state. Callers pass them to context rendering or FlexDoc.graph(). A detached, hash-less sidecar may resolve conservatively through quote and context. Embedding a sidecar in DocGraph requires its non-null hash and document to match the graph snapshot because the graph's shared source identity then binds every bare selector.

Annotation workflow status, review history, synchronization state, edit operations, and publication metadata are not part of TextRef. Suggested edits use TextRef as their target but define replacement, validation, ordering, and application semantics in a separate typed envelope. Automatic edit application requires a matching base source hash.

11.8 Representation Boundaries and Errors

Source TextRefs never silently become rendered-page selectors. Browser text fragments, DOM paths, PDF coordinates, and Git line links address different representations and require explicit directional adapters with visible refusal when a faithful mapping is not possible.

Construction, schema parsing, and URI parsing raise on invalid contracts: malformed DocRefs or hashes, empty required quotes, invalid selector combinations, unknown fields, unlocatable source objects, and out-of-range spans. Resolution failures are typed values. Document retrieval remains consumer-owned. TextRef does not perform I/O, store annotation state, define an edit language, or choose a fuzzy match.

12. Editing and Serialization

Sentence.text is the editable content: edits change what reassemble() produces while the fixed source references (original_text, offsets, cached block_type) keep describing the original. So FlexDoc doubles as an editable model: modify units, then reassemble() to serialize a new document (optionally normalized by flowmark). The diff/sliding-window/wordtok machinery operates on this editing view unchanged.

The structural node table is a pure function of the immutable source_text (sentence edits touch the editing view, not source_text), so it and its derived views are lazily cached; the operative contract is “do not reassign source_text after parse.” Edit by editing the FlexDoc/source and re-deriving DocGraph; an editor bridge resolves annotations through TextRef. Render helpers emit data-node-id / data-source-span so a rendered selection resolves to a node and thence to source. Selections of rendered substrings require an explicit rendered-to-source adapter.

FlexDoc.graph(document=...) always returns the current DocGraph contract. Supplying consumer-owned annotations populates its annotation list after verifying that the sidecar document and source hash match. Detached annotations carry complete TextRefs; embedded entries use the graph's shared source identity.

13. Invariants and Non-Goals

Invariants: offset-anchored (code points), the source and offset space being the canonical substrate (P1); node ids stable within a parse; derived views over one shared parse (no duplicated content, no stored counts), the node table among them; every locatable value has a deterministic TextRef mapping; references are source-grounded and resolve conservatively; annotations do not change target identity; wire evolution is explicit.

The promoted flexdoc.docs namespace is the document-model surface. Lower-level word-token/search and diff/mapping machinery remains available only through its owning modules (wordtoks, search_tokens, token_diffs, and token_mapping), keeping those pipeline internals out of the model’s advertised namespace.

Non-goals: a parallel runtime BlockDoc/SectionDoc Python model (DocGraph is a projection, not a competing editable model). Naming note: an abandoned design branch used “FlexDoc” for that competing runtime model; the name now refers only to the package’s entry-point class. Other non-goals: blessed per-kind rollups or fixed detail levels; DOM/XPath/CSS/PDF selectors in TextRef's source-text profile; CommonMark/GFM rendering (flowmark covers normalization); stored cross-layer edges (cross-layer relationships are offset-containment queries, §3); exact provider-keyed token counts (estimate_tokens is a heuristic); a thread-safety layer.

Later phases, not non-goals. The synthetic layer (§3) and cross-layer structural edits (move/wrap/splice anchored on TextRef, generalizing today’s tag-region edit helpers) are deferred phases, not excluded. The operation, provenance, and layout layers are likewise schema-reserved and built later. The hooks already in place (the layer field, offset-containment collect(), and shared source coordinates) let those layers compose with the current design. §14 states each phase’s current status and where it is tracked.

Pitfalls and key decisions

Non-obvious choices, each grounded in a principle:

  • The shared offset space—not the node table—is the canonical substrate (P1). The node table is one projection (the id-addressed, layer-tagged, serialization-friendly one); it is built from the parses, and blocks()/sections()/links() derive from the same memoized parse rather than from the table’s id space. “Single canonical form” holds at the parse and offset space.
  • Cross-layer overlap is expected (P3). The same logical paragraph appears as distinct nodes in distinct layers (a markdown block node and a textual paragraph node over the same span), so a query that does not restrict layer returns both. This is honest, not a bug; scope with collect(layer=…).
  • Tight vs. loose lists are structurally identical (P7, P12). Density is Block.tight metadata only and never enters a tally.
  • Base blocks decompose lists recursively to item_partition_depth (default 6); blockquotes are always atomic (P13).
  • Fast/approximate sentence segmentation is accepted (P16): the regex splitter avoids a heavy NLP dependency; offsets stay exact via the span-aware splitter.
  • words normalizes cross-language volume (P16): every core words field means the logical measure. Non-whitespace Unicode wide and fullwidth characters contribute 0.5 word each. For the remaining text, the raw whitespace-delimited count is clamped to an average of 3–6 non-whitespace characters per word: it equals the raw count inside that range, increases above six, and decreases below three. Long identifiers and URLs, short-token sequences, Markdown, code, and symbolic runs can therefore differ from an ordinary expected word count. Non-visible HTML markup is removed before both measures. The combined non-negative result is rounded to the nearest integer with ties upward, and aggregates are measured before rounding. raw_words remains available when a caller specifically needs whitespace-split behavior.
  • Fast/approximate token sizing is accepted (P16): estimate_tokens is a heuristic, not provider-keyed. It rounds up logical_word_count(text) * 1.6 by default; callers can calibrate a different positive finite multiplier for another model family.
  • Reference links and other unlocatable identities carry span=None and are excluded from offset-scoped rollups (e.g. Section.links()), since they cannot be attributed by offset.
  • Offsets are Unicode code points (P1); byte/UTF-16 are derived on demand, never canonical.
  • Round-trip is Markdown-object-exact, not byte-exact (P11), except byte-exact via retained offsets or after Flowmark normalization—two distinct equivalence levels.

14. Implementation Status

Implemented (in this package, verified by the unit and golden suites):

  • Exact spans over the shared offset space; the editing view (paragraphs/sentences with exact verbatim spans); frontmatter isolation as a non-content region.
  • The structural block tree blocks() (boundaries and spans from flowmark, no regex scanner); the base_blocks() sequential partition with its non-overlapping cover invariant; ordered_list and density-invariant lists; typed per-block metadata (CodeInfo/TableInfo/ListInfo, §5).
  • Sections/TOC/size rollups built from structural headings (§7), cached like the other derived views; inline-link rollups and link-aware sentences; inline kinds including footnote_ref (§8). Raw word sizes and normalized words fields, summary defaults, and logical-word token estimates are implemented across sentences, paragraphs, sections, and documents.
  • The recursive node table with deterministic contiguous-preorder ids (contract-tested) and strict layer-nesting validation at build (§4.3); JSON-safe attrs validated at serialization; the single collect() query primitive (§9); composable include/detail serialization; and the unified DocGraph/v0.2 Pydantic schema with required document identity, source hash, and optional typed annotations (§10).
  • The SpanRef, DocRef, and TextRef contracts; span, point, section, and whole-document construction; canonical JSON and URI forms; document-bound mapping and typed exact resolution; contextual rendering; the annotation envelope and sidecar; and graph embedding (§11).

Specified here, not yet implemented (tracked by active plans and repo beads and summarized in TODO.md):

  • The synthetic layer (§3). Today’s implementation of marker-tag regions (TextNode/parse_divs, currently <div>/<span>-focused) lives in the chopdiff package as a standalone subsystem, not keyed into the node table. Migrating it here and re-expressing regions as synthetic-layer nodes is mapped concretely in the extraction plan (Stage 4): a builder pass over a configurable tag whitelist, an overlap/nesting policy decision, fixtures for regions that cross block boundaries, and the moved test suite. Moderate difficulty; no changes to the node table, collect(), or the schema are expected (the synthetic layer value is already reserved).
  • Cross-layer structural edits (§13): operations anchored on TextRef.
  • Normalized and fuzzy TextRef re-anchoring (§11), after exact resolution has an edited-document corpus.
  • A uniform opt-in strict-validation / diagnostics pass over a parse (Error posture): today strictness exists piecemeal (builder invariants, serialization validation, per-API strict= flags); a whole-document diagnostics surface is direction, not yet designed.
  • Operation, provenance, and layout layers: schema-reserved only.

15. Background and Further Reading

This spec stands alone; the following are background, not dependencies.

  • Research surveys (authored during the model’s design, in chopdiff; copied here as history): the cross-language document-model survey research-2026-05-29-document-model.md, the span-references survey research-2026-05-30-span-references.md (background for §11), and the layered-parsing brief research-2026-05-30-multilayer-parsing.md (background and prior art for §3).
  • The portable reference protocol brief research-2026-07-10-text-reference-microformat.md supplies the evidence and protocol rationale for §11. The focused implementation plan is plan-2026-07-14-native-textref-integration.md.
  • Dated planning documents under docs/project/specs/ (active and archived) track the incremental work toward this design and reference this spec.
  • The external Logical Word Count definition and validation gives the reference algorithm, rationale, and multilingual examples for words.
  • flowmark v0.7.1 API relied on for spans and splitting: flowmark.atomic_spans (iter_atomic_spans, split_sentences_with_spans, named AtomicSpans) and flowmark.markdown_ast (block_span, walk_elements, extract_links, Link).
  • Source: src/flexdoc/docs/flex_doc.py (the FlexDoc core), with the editing units in paragraphs.py, link extraction in links.py, sections in sections.py, and the structural layer in block_tree.py, block_types.py, block_info.py. The node/query surface is node.py (Node, NodeKind, Layer, NodeTable), node_table.py (build_node_table), collect.py (the collect() primitive), and interval_index.py; serialization is doc_graph.py (the DocGraph schema and builder) with the render helpers in render.py and reports in debug.py; references are span_ref.py (SpanRef), text_ref.py (the wire values and exact resolver), text_ref_context.py (FlexDoc mapping, context, and rendering), and text_annotations.py (annotation envelopes and sidecars). Supporting modules: base_blocks.py (the sequential partition), sizes.py (TextUnit), frontmatter.py (frontmatter isolation), and the wordtok/diff machinery in wordtoks.py, token_diffs.py, token_mapping.py, search_tokens.py.