notcrawl Spec

September 22, 2026 ยท View on GitHub

Goals

  • build a local-first Notion crawler
  • mirror Notion pages, blocks, databases, comments, and workspace metadata
  • store normalized records in SQLite
  • preserve raw source records for future re-rendering
  • render normalized Markdown blobs into an organized file tree
  • support fast text search and raw SQL
  • support one-shot backfill and incremental repair
  • publish and subscribe private git-backed snapshots

Product Summary

notcrawl is a Go CLI that turns Notion workspace memory into a local SQLite archive plus normalized Markdown files.

V1 scope:

  • macOS Notion Desktop cache discovery
  • read-only desktop snapshot ingestion
  • official Notion API sync
  • targeted repair through a preconfigured Codex Notion MCP app
  • pages and blocks
  • databases/data sources as collections, including current data-source API endpoints
  • database rows as pages linked to their collection
  • comments where available
  • users and spaces/workspaces
  • FTS5 search over rendered page/comment text
  • raw SQL access
  • archive status, activity reporting, and SQLite maintenance commands
  • Markdown export
  • CSV/TSV export for database rows
  • git-backed archive publishing and subscription

Out of scope for V1:

  • write-back actions
  • modifying Notion local storage
  • bypassing workspace permissions
  • full attachment blob mirroring by default
  • public integration Marketplace hardening

Data Sources

Desktop Source

Default macOS path:

~/Library/Application Support/Notion/notion.db

Desktop sync must:

  1. locate Notion Desktop storage
  2. snapshot notion.db into the cache dir
  3. open the snapshot read-only
  4. ingest supported tables into the local archive
  5. record unsupported source records in raw_records

Desktop cache coverage is opportunistic. It only includes what Notion has cached, downloaded, or recently touched locally.

notion.desktop.space_ids optionally limits future Desktop ingestion to listed workspace IDs (case-insensitive, with or without UUID hyphens). An absent or empty list preserves all-workspace ingestion. The filter covers spaces, teams, collections, pages, blocks, raw block records, and comments, including explicit tombstones. With a nonempty list, records without a workspace ID are skipped. Excluded existing archive rows are retained unchanged, remain searchable, and remain eligible for Markdown export. Shared users and complete source snapshots are not filtered. API and MCP coverage is unchanged; this is not an archive cleanup or access-control feature.

API Source

API sync uses NOTION_TOKEN by default. It must:

  1. search/list pages and data sources visible to the integration
  2. recursively fetch block children
  3. fetch users
  4. fetch comments where the integration has access
  5. obey Retry-After on rate limits
  6. store raw JSON plus normalized rows
  7. after a successful, nonempty full discovery, retire API page sources absent from both page search and all collection query results

Omission reconciliation records complete-authoritative-enumeration on the API page source, retires its API blocks and comments with parent-delete-event, and clears API page-block completion. Source retirement, fallback promotion, and FTS updates commit together. Other live sources survive; rediscovered API records clear their tombstones. Canonical and raw recovery rows are retained. Discovery with no pages or rows (including database-only results), malformed discovery responses, errors, cancellation, incomplete block walks, and targeted ingestion never trigger omission reconciliation. Restricted user listing does not prevent complete page discovery. Collection rows participate, but database containers do not have omission tombstones. This mirror policy treats omission as unavailable API coverage, not proof of deletion; Notion search does not guarantee an exhaustive inventory.

Official API success responses are limited to 8 MiB per HTTP response; oversized bodies fail without retrying and incomplete pages remain eligible for repair. This limit does not apply to whole-page Notion MCP responses.

Official API pagination treats cursors as opaque values and rejects repeated cursors within each listing, without imposing a fixed page-count limit. Pagination failures must not mark an incomplete page sync complete or retire cached blocks that were not reached. Error messages identify the operation without including cursor values.

Block listings at every depth and pagination step must contain a boolean has_more, a results array, and objects with nonempty string IDs. Invalid listings fail sync without retiring unseen blocks or pages or marking the page complete; previously committed batches remain searchable and exportable. A valid empty final listing still permits retirement after the whole walk succeeds.

sync --verbose enables per-invocation stderr diagnostics for source phases and counts, with official API request attempts, endpoint classes, elapsed time, numeric HTTP statuses, and retry delays. Diagnostics use fixed labels and numeric fields, never credentials, headers, payloads, raw URLs, cursors, page identifiers, or upstream error text. Verbose mode replaces warning text with counts and sanitizes sync failures; stdout and non-verbose behavior remain unchanged.

New configs should use the current Notion API version. Existing configs pinned to legacy 2022-06-28 must continue using deprecated database query endpoints.

Notion MCP Source

Notion MCP sync uses the Notion app already connected in Codex through the ChatGPT apps gateway. It must:

  1. read Codex auth at request time without persisting or logging the bearer token
  2. dynamically resolve read-only Notion search and fetch tools
  3. fetch explicit page IDs/URLs or bounded targeted search results
  4. automatically repair incomplete known Desktop pages and API pages whose block sync did not complete
  5. preserve connector enhanced Markdown, including properties, without lossy block reconstruction
  6. strip signed URL credentials before storing or exporting content
  7. avoid claiming complete workspace enumeration or incremental coverage

The Codex gateway transport is experimental and OpenAI-specific. Official API sync remains the stable integration path.

SQLite Archive

SQLite is canonical. Markdown is generated output.

Store startup must enable WAL, foreign keys, a busy timeout, normal synchronous writes, in-memory temp storage, and the crawler query indexes needed for common page, collection, comment, raw-record, and sync-state lookups.

report must provide a SQL-free archive summary: total records, recent edited page/comment windows, top databases, top spaces, and recently edited pages.

sql opens an existing archive read-only, without creating tables or migrating its schema. SQLite enforces write rejection; the CLI accepts only one SELECT, WITH, or PRAGMA statement so input cannot disable connection safeguards and execute further statements. Quoted strings, identifiers, and comments retain SQLite syntax. Run sync to initialize or upgrade an archive before querying newer fields.

Core tables:

  • spaces
  • users
  • teams
  • pages
  • blocks
  • collections
  • comments
  • raw_records
  • record_sources
  • record_revisions
  • sync_state
  • page_fts
  • comment_fts

Markdown Archive

Markdown export writes deterministic Unicode-safe paths. Path components keep readable letters, numbers, and CJK text while replacing emoji, filesystem path separators, and unsafe punctuation with dashes:

pages/<space-slug>/<team-slug>/<page-title>-<short-id>.md

The team slug is omitted when no teamspace can be resolved.

Each export removes stale generated .md files under the Markdown root while leaving unrelated Markdown and non-Markdown sidecar files alone. Cleanup recognizes only files beginning with notcrawl's generated front matter and a closing delimiter; quoting the ownership marker in a document body does not make that file generated.

Each file starts with YAML front matter. String values are double-quoted with backslashes, quotes, control characters, and Unicode line separators escaped. Ordinary newline characters in metadata are folded to spaces.

Example fields:

---
id: ...
space_id: ...
title: ...
source: desktop
notion_url: ...
created_time: ...
last_edited_time: ...
---

The body renders blocks into normalized Markdown. Unsupported blocks should be represented with concise placeholders, not silently dropped.

Git Share

Git share mode exports:

manifest.json
data/*.jsonl.gz
pages/**/*.md

publish writes a snapshot and optionally commits/pushes it. --tag attaches an immutable Git checkpoint to the resulting commit.

subscribe clones a snapshot repo, writes reader config, and imports data into SQLite without requiring Notion credentials.

update pulls the latest snapshot and imports it. update --ref REF reads the manifest and table objects at a tag, commit, or branch without changing the share checkout.

Database Export

API sync discovers databases/data sources visible to the integration, stores metadata in collections, queries each collection for row pages, and links those pages through pages.collection_id.

export-db renders row properties into delimited text:

notcrawl export-db --database <database-id> --format csv --output rows.csv
notcrawl export-db --database <database-id> --format tsv --output rows.tsv
notcrawl export-db --all --dir exports/csv

The first columns are stable metadata:

  • page_id
  • page_title
  • url

Remaining columns come from the database schema, with any extra row properties appended alphabetically. Properties whose keys match page_id, page_title, or url remain separate from the fixed metadata columns. Duplicate column names are ordered by property key and disambiguated with that key. Desktop title inference prefers title, Name, then name; other properties are considered in key order so repeated exports choose the same fallback title and path.