Overture Schema Codegen

August 26, 2026 · View on GitHub

Generates documentation from Overture Maps Pydantic schema definitions.

Pydantic's model_json_schema() flattens the schema's domain vocabulary into JSON Schema primitives. NewType names disappear, constraint provenance is lost (which NewType contributed which bound), custom constraint classes lose their identity (a GeometryTypeConstraint becomes an anonymous enum array), and discriminated union structure collapses into anyOf arrays with duplicated fields.

Navigating Python's type annotation machinery -- NewType chains, nested Annotated wrappers, union filtering, generic resolution -- is complex. The codegen does it once. analyze_type() unwraps an annotation into a FieldShape, a tree-shaped target-independent representation. Extractors build specs from FieldShape. Renderers consume specs without touching the type system. New output targets (Arrow schemas, PySpark expressions) add renderers, not extraction logic.

Usage

# Generate markdown documentation for all themes
overture-codegen generate --format markdown --output-dir docs/schema/reference

# Generate for a single theme
overture-codegen generate --format markdown --tag overture:theme=buildings --output-dir out/

# List discovered models
overture-codegen list

The generator discovers models via overture.models entry points (provided by theme packages like overture-schema-theme-buildings), extracts type information, and renders output pages with cross-page links, constraint descriptions, and validated examples.

Architecture

Four layers with strict downward imports -- no layer references the one above it:

Rendering            Output formatting, all presentation decisions
    ^
Output Layout        What to generate, where it goes, how outputs link
    ^
Extraction           FieldShape, FieldSpec, RecordSpec, UnionSpec
    ^
Discovery            discover_models() from overture-schema-system

Discovery loads registered Pydantic models via entry points. The return dict includes both concrete BaseModel subclasses (like Building) and discriminated union type aliases (like Segment). Both satisfy the ModelSpec protocol and flow through the same pipeline.

Extraction unwraps type annotations into specs. analyze_type() is the central function -- a single iterative loop that peels NewType, Annotated, Union, and container wrappers, accumulating constraints tagged with the NewType that contributed them. Domain-specific extractors (model_extraction, union_extraction, enum_extraction, newtype_extraction, numeric_extraction) call analyze_type() for field types and produce spec dataclasses.

Output Layout determines what artifacts to generate and where they go. Supplementary type collection walks expanded feature trees to find referenced enums, NewTypes, and sub-models. Path assignment maps every type to an output file path mirroring the Python module structure. Link computation and reverse references enable cross-page navigation.

Rendering consumes specs and owns all presentation decisions. Markdown output uses Jinja2 templates for feature pages (with field tables, constraint sections, and examples), enum pages, NewType pages, and aggregate numeric/geometry reference pages.

markdown/pipeline.py orchestrates the full pipeline without I/O, returning list[RenderedPage]. The CLI writes files to disk with Docusaurus frontmatter.

Programmatic use

analyze_type() returns a 3-tuple: the annotation's FieldShape, whether the field accepts None, and the first Field(description=...) encountered while unwrapping.

from overture.schema.buildings import Building
from overture.schema.codegen.extraction.type_analyzer import analyze_type

annotation = Building.model_fields["version"].rebuild_annotation()
shape, nullable, description = analyze_type(annotation, owner=Building)

# NewTypeShape(name='FeatureVersion', ref=..., inner=Primitive(base_type='int32', ...))
assert shape.name == "FeatureVersion"
assert shape.inner.base_type == "int32"
assert nullable is False

FieldShape is a tree, not a flat record: NewTypeShape, ArrayOf, and MapOf wrap an inner shape, and the three terminals (Primitive, LiteralScalar, AnyScalar) sit at the leaves. Nesting order is meaningful -- NewTypeShape(inner=ArrayOf(...)) is a NewType over list[X], while ArrayOf(element=NewTypeShape(...)) is a list of NewType-wrapped values.

Constraints attach to the layer they target and carry the NewType that contributed them:

for source in shape.inner.constraints:
    print(f"{source.constraint} from {source.source_name}")
# Ge(ge=0) from FeatureVersion
# Ge(ge=-2147483648) from int32
# Le(le=2147483647) from int32

Fetching sample data

Theme packages include example records in their pyproject.toml files under [[examples.<Type>]] sections. The codegen validates these against Pydantic models and renders them in feature pages.

To fetch a fresh sample from the latest Overture release using DuckDB:

duckdb -json \
  -c "load spatial" \
  -c "attach 'http://labs.overturemaps.org/data/latest.ddb' as overture" \
  -c "select to_json(columns(*))
      from (
        select * REPLACE ST_AsText(geometry) as geometry
        from overture.place
        USING SAMPLE 1
      )" \
  | jq .

The latest.ddb database always points to the current release. Tables use the type name directly (overture.place, overture.segment, overture.building, etc.). Convert the JSON output to TOML for inclusion in the theme's pyproject.toml.

Usage

Architecture

Programmatic use

Fetching sample data

Further reading