Warp-Ingest Parser (REST API)

Intro

The Warp-Ingest Parser is an alternative PDF parser for OpenContracts based on Warp-Ingest — a deterministic, rule-based PDF parser. Instead of a machine-learning layout model, it derives structure from text coordinates, graphics, and font metadata (via pdfplumber), with optional RapidOCR for scanned pages. There is no GPU requirement and no per-page image rasterization, so parses are fast and reproducible.

Like the Docling Parser, it runs as a microservice and is accessed over REST, keeping the heavy parsing dependencies out of the Django image. Its defining feature for OpenContracts is that it renders directly to the OpenContracts structural export format — PAWLS word tokens, per-block structural annotations, and a heading hierarchy expressed as parent_id links plus explicit relationships — so no lossy format translation is required.

Docling remains the default PDF parser; Warp-Ingest is opt-in and selected per-corpus / file-type in the admin System Settings UI (or by seeding PipelineSettings).

Architecture

sequenceDiagram
    participant U as User
    participant WP as WarpIngestParser (REST Client)
    participant WS as Warp-Ingest Service (Microservice)
    participant PL as pdfplumber / RapidOCR
    participant DB as Database

    U->>WP: parse_document(user_id, doc_id)
    WP->>DB: Load Document + read PDF from storage
    WP->>WS: POST /api/parse?render_format=opencontracts (multipart file)
    WS->>PL: Extract word boxes + fonts (OCR scanned pages)
    WS->>WS: Rule-based layout → blocks → OpenContracts export
    WS-->>WP: {"page_dim", "num_pages", "result": <OpenContractDocExport>}
    WP->>WP: Unwrap "result"
    WP->>DB: save_parsed_data (annotations, relationships, PAWLS)
    WP-->>U: OpenContractDocExport

Implementation

• Parser: opencontractserver/pipeline/parsers/warp_ingest_parser.py::WarpIngestParser
• Base class: opencontractserver/pipeline/base/parser.py::BaseParser (non-chunked — see Chunking below)
• Auto-discovered by opencontractserver/pipeline/registry.py; no manual registration needed.

Configuration

Settings live in the PipelineSettings DB singleton — the runtime source of truth — and are edited via the admin System Settings UI. There are two ways a value gets there:

1. Seeded from config/settings/base.py by migrate_pipeline_settings. The command resolves each setting's env_var as a Django settings attribute (getattr(settings, env_var)), so only these three — which have matching attributes in base.py — are populated from the environment at migrate time:

Env var (in base.py)	Setting field	Default	Purpose
WARP_INGEST_PARSER_SERVICE_URL	service_url	http://warp-ingest:5001/api/parse	Warp-Ingest /api/parse endpoint
WARP_INGEST_API_KEY	api_key	""	Sent as X-API-Key; must match the service's WARP_API_KEY
WARP_INGEST_PARSER_TIMEOUT	request_timeout	600	HTTP request timeout (seconds)

2. Component-setting defaults, changed via the admin System Settings UI. The remaining fields exist only as PipelineSetting metadata on the parser's Settings dataclass. Their env_var names are metadata; because they have no base.py attribute, setting them in .env does not change them via migrate_pipeline_settings — edit them in the admin UI (this matches the other parsers, e.g. Docling's image/OCR fields):

Setting field	env_var metadata	Default	Purpose
use_cloud_run_iam_auth	WARP_INGEST_USE_CLOUD_RUN_IAM_AUTH	False	Force Google Cloud Run IAM Authorization bearer
apply_ocr	WARP_INGEST_APPLY_OCR	False	Force OCR on every page
disable_ocr	WARP_INGEST_DISABLE_OCR	False	Disable OCR (mutually exclusive with apply_ocr)
semantic_units	WARP_INGEST_SEMANTIC_UNITS	False	Append the Semantic-Unit clause annotation layer
include_images	WARP_INGEST_INCLUDE_IMAGES	False	Embed extracted images in the export
max_file_size_mb	WARP_INGEST_MAX_FILE_SIZE_MB	200	Reject PDFs larger than this (checked against storage size before reading)

The single request-timeout constant lives at opencontractserver/constants/document_processing.py::WARP_INGEST_PARSER_REQUEST_TIMEOUT_SECONDS and backs both the Django setting default and the dataclass field default.

Authentication

Warp-Ingest always requires an API key (its own default is abc123; the server logs a loud warning when serving on that fallback). OpenContracts sends the key in the X-API-Key header — deliberately not Authorization: Bearer, which Warp-Ingest also accepts — so the Authorization header stays free for a Google Cloud Run IAM id_token when the service runs behind Cloud Run IAM (use_cloud_run_iam_auth). Set WARP_INGEST_API_KEY on the OpenContracts side to the same value as the service's WARP_API_KEY.

Microservice Setup

Run the official image, ghcr.io/open-source-legal/warp-ingest. Both local.yml and production.yml define a warp-ingest service behind the opt-in warp-ingest compose profile (it is not started by default because the image is ~2.5 GB):

# Start the stack with Warp-Ingest available
docker compose -f local.yml --profile warp-ingest up

# local.yml / production.yml (abridged)
services:
  warp-ingest:
    image: ghcr.io/open-source-legal/warp-ingest:1.0.0
    container_name: warp-ingest
    profiles:
      - warp-ingest
    environment:
      - WARP_API_KEY=${WARP_INGEST_API_KEY:-abc123}

Health endpoints: GET / and GET /healthz (the latter reports version and ocr_available). The container listens on port 5001 (override with WARP_PORT).

Input / Output

Input: a PDF stored in Django's storage, plus a user ID and document ID.

Request: POST /api/parse?render_format=opencontracts with the PDF as a multipart file field and the OCR / semantic-unit / image flags as query params.

Response: {"page_dim": ..., "num_pages": ..., "result": <export>}. The parser unwraps result (falling back to the top-level body if a future API revision returns the export unwrapped) and returns an OpenContractDocExport:

{
    "title": str,
    "content": str,                 # full text
    "description": str | None,
    "pawls_file_content": list,     # PAWLS token pages
    "page_count": int,
    "doc_labels": list,
    "labelled_text": list,          # structural annotations (annotationLabel, rawText, parent_id, ...)
    "relationships": list,          # heading hierarchy + group relationships
    "file_type": str,
}

The export already uses OpenContracts field names (snake_case top-level keys, camelCase annotationLabel / rawText within annotations), so no key normalization is performed — it flows straight into save_parsed_data.

Why not chunked?

Unlike the Docling parser (which splits large PDFs into overlapping page-range chunks via BaseChunkedParser), Warp-Ingest sends the whole PDF in one request:

• It is CPU-only (no per-page GPU layout model), so bounded per-request cost — the main motivation for Docling chunking — does not apply.
• Its native /api/parse API accepts a whole file.
• It performs cross-page structure joining (heading hierarchies, tables and lists spanning page boundaries) that page-range chunking would fragment.

For very large scanned documents, raise WARP_INGEST_PARSER_TIMEOUT rather than introducing chunking.

Memory footprint: because the whole PDF is buffered in the worker and POSTed in one request, peak memory per concurrent parse scales with the file size. WARP_INGEST_MAX_FILE_SIZE_MB (default 200) caps this — the parser checks the file's storage size before reading it, so a PDF above the limit is rejected with a permanent DocumentParsingError and is never buffered at all. Raise it if you routinely ingest larger scans, sizing it against worker memory / expected concurrent parses.

Error Handling

The REST client raises DocumentParsingError with an is_transient flag the ingestion pipeline uses to decide whether to retry:

• Timeout / connection errors → transient (retryable).
• 4xx (e.g. 415 unsupported media type, 401 bad key, 422 both apply_ocr and disable_ocr set) → permanent (no retry).
• 5xx → transient.
• A apply_ocr + disable_ocr conflict is caught client-side and fails fast (permanent) before any HTTP call.

Testing

• Unit tests (mocked HTTP): opencontractserver/tests/test_doc_parser_warp_ingest.py
• End-to-end smoke test against the real container: docs/test_scripts/warp_ingest_parser_smoke_test.md

See Also

• Pipeline Overview
• Docling Parser — the default PDF parser
• LlamaParse Parser — cloud-based alternative
• PDF Data Layer Architecture
• Warp-Ingest