Bulk ZIP Import

July 18, 2026 · View on GitHub

The bulk ZIP import allows uploading many documents at once, preserving folder structure from the ZIP archive. It also supports optional metadata and relationship CSV files to set document titles/descriptions and create inter-document relationships automatically.

Overview

Upload a ZIP file containing documents organized in folders. The folder structure is preserved in the target corpus. Two optional CSV files at the ZIP root provide metadata overrides and document-to-document relationships.

GraphQL Mutation: ImportZipToCorpus

Parameter	Type	Required	Description
`base64FileString`	String	Yes	Base64-encoded ZIP file
`corpusId`	ID	Yes	Target corpus
`targetFolderId`	ID	No	Subfolder to import into
`titlePrefix`	String	No	Prefix prepended to all document titles
`description`	String	No	Description applied to all documents
`customMeta`	GenericScalar	No	Custom metadata applied to all documents
`makePublic`	Boolean	Yes	Public visibility for imported documents

The import runs asynchronously via Celery and returns a jobId for progress tracking.

Bulk-loading a very large local collection (100k+ files)? Use the resumable CLI driver at scripts/bulk_import/oc_bulk_import.py. It walks a local directory tree, packs the files into right-sized ZIPs that feed this same zip-to-corpus endpoint (mirroring the folder structure), paces itself against the corpus parse backlog, and records progress in a SQLite ledger so the job is crash-resumable. See scripts/bulk_import/README.md for the full operational runbook.

ZIP File Structure

A typical ZIP for import:

my-import.zip
+-- contracts/
|   +-- legal/
|   |   +-- agreement.pdf
|   +-- financial/
|       +-- report.pdf
+-- docs/
|   +-- amendment.pdf
+-- readme.txt
+-- meta.csv             (optional)
+-- relationships.csv    (optional)

Supported Document Types in ZIP

All file types registered in the pipeline are accepted. Currently:

PDF (.pdf)
Word (.docx)
Plain Text (.txt)

Skipped Files

The following are automatically skipped:

Hidden files (names starting with .)
macOS metadata (__MACOSX/, .DS_Store)
Files exceeding size limits
Unsupported file types
The meta.csv and relationships.csv control files themselves

Metadata File (`meta.csv`)

An optional CSV file at the ZIP root that overrides document titles and descriptions. Accepted filenames (priority order): meta.csv, META.csv, metadata.csv, METADATA.csv.

Schema

Column	Required	Description
`source_path`	Yes	Relative path within the ZIP
`title`	No	Custom document title
`description`	No	Custom document description
`external_id`	No	Durable identifier in the producing system, stored on the document's `DocumentPath.external_id` (max 512 chars)

Example

source_path,title,description,external_id
contracts/legal/agreement.pdf,Master Services Agreement,The main services contract,
contracts/financial/report.pdf,Q4 Financial Report,Quarterly financial summary,
rulings/H022844.txt,Plastic serving trays; classification,CROSS HQ ruling,cross:H022844

Behavior

Empty cells are ignored (defaults are used instead)
Not all documents need metadata entries -- partial coverage is fine
If a titlePrefix is provided in the mutation, it is prepended to the metadata title (e.g., "2024 - Master Services Agreement")
external_id values should be namespaced by the producer (e.g. cross:H022844); unlike the display title, they survive later renames. Consumers resolve through the namespace — the enrichment engine treats a cross:-namespaced id as a document's canonical ruling number, ahead of its path or title (opencontractserver/enrichment/resolver.py::document_identity_candidates). Overlong values are rejected per-row with an import warning, never truncated.

Relationships File (`relationships.csv`)

An optional CSV file at the ZIP root that creates document-to-document relationships. Must be named relationships.csv or RELATIONSHIPS.csv.

Schema

Column	Required	Description
`source_path`	Yes	Path to source document (relative to ZIP root)
`relationship_label`	Yes	Relationship label text (e.g., "AMENDS")
`target_path`	Yes	Path to target document (relative to ZIP root)
`notes`	No	If present, creates a NOTES relationship instead of RELATIONSHIP

Example

source_path,relationship_label,target_path,notes
contracts/legal/agreement.pdf,AMENDS,docs/amendment.pdf,Amendment to main contract
docs/amendment.pdf,AMENDED_BY,contracts/legal/agreement.pdf,
contracts/legal/agreement.pdf,REFERENCES,contracts/financial/report.pdf,

Path Normalization

Paths in CSV files are normalized to a canonical form:

Backslashes converted to forward slashes
Leading ./ removed
Leading slashes normalized
Duplicate slashes collapsed

All of these reference the same document:

contracts/agreement.pdf
/contracts/agreement.pdf
./contracts/agreement.pdf
contracts\agreement.pdf

Path traversal (..) is rejected for security.

Cross-batch Relationships

An endpoint that does not match a file inside the current ZIP is resolved against documents already in the corpus (matched by their path within the corpus, which equals the ZIP-relative path when no target folder is used). This lets a relationships.csv connect documents that were imported in separate batches — you can upload a ZIP containing only a relationships.csv to wire together documents already present in the corpus.

Resolution order for each endpoint:

A document imported in the current ZIP (same archive), then
A current, non-deleted document already in the corpus that you can access.

Only documents you can already see in the corpus are eligible, so cross-batch resolution never exposes a document you lacked access to. An endpoint that matches neither is skipped (and reported), exactly as before.

Annotation Sidecars

A document may ship a co-located JSON sidecar (same name, .json extension — e.g. agreement.pdf + agreement.json) describing producer annotations to attach. Each annotation is a dumb anchor: a label + the rawText to find, plus a location hint (PDF page + bbox, or text start/end). The document is parsed normally and the annotations are re-anchored onto the freshly-produced text/PAWLs layer after ingest, so the anchors survive re-parsing.

Sidecar Schema

{
  "annotations": [
    {
      "id": 1,
      "label": "OC_SECTION",
      "rawText": "Article 1 — Definitions",
      "page": 0,
      "bbox": {"left": 50, "top": 50, "right": 300, "bottom": 70},
      "link_url": "https://example.com/ref",
      "data": {"canonical_name": "France", "lat": 46.0, "lng": 2.0}
    },
    {
      "id": 2,
      "label": "OC_CLAUSE",
      "rawText": "indemnification obligations",
      "start": 1234,
      "end": 1261
    }
  ],
  "doc_labels": ["Contract"],
  "relationships": [
    {
      "id": "r1",
      "relationshipLabel": "OC_PARENT_CHILD",
      "source_annotation_ids": [1],
      "target_annotation_ids": [2]
    }
  ]
}

Field	Scope	Description
`annotations[].label`	required	Resolves against `labels.json`/the corpus label set
`annotations[].rawText`	required	The text to (re-)anchor onto — the source of truth
`annotations[].page` + `bbox`	PDF	Location hint for token-based anchoring
`annotations[].start` + `end`	text	Character-offset hint for span anchoring
`annotations[].link_url`	optional	Click-through target for `OC_URL` hyperlink annotations
`annotations[].data`	optional	Structured sidecar persisted to `Annotation.data` (e.g. geocoded `OC_COUNTRY`/`OC_STATE`/`OC_CITY` payloads)
`relationships[]`	optional	Annotation-to-annotation edges (see below)

Annotation-to-annotation Relationships

The optional relationships list declares edges between annotations in the same sidecar. Each entry carries a relationshipLabel (auto-created as a RELATIONSHIP_LABEL if it doesn't exist — e.g. OC_PARENT_CHILD, OC_SUBTREE_GROUP) and source_annotation_ids / target_annotation_ids that reference the sidecar annotations' own ids. Relationships are wired after the annotations are re-anchored, using the producer-id → new-annotation map. An edge whose endpoints did not survive anchoring is dropped and recorded on the import report, never silently lost.

Security Constraints

Constraint	Default	Description
`ZIP_MAX_FILE_COUNT`	1,000	Max files per ZIP
`ZIP_MAX_TOTAL_SIZE_BYTES`	500 MB	Max uncompressed total size
`ZIP_MAX_SINGLE_FILE_SIZE_BYTES`	100 MB	Max single file size
`ZIP_MAX_COMPRESSION_RATIO`	100:1	Zip bomb detection threshold
`ZIP_MAX_FOLDER_DEPTH`	20	Max nesting depth
`ZIP_MAX_FOLDER_COUNT`	500	Max folders created
`ZIP_MAX_PATH_COMPONENT_LENGTH`	255	Max filename length
`ZIP_MAX_PATH_LENGTH`	1,024	Max total path length

All limits are configurable via Django settings.

Import Phases

The import proceeds through four phases:

Phase 1: Validation

The ZIP is scanned for security violations (size limits, zip bombs, path traversal) before any files are extracted. A ZipManifest is produced listing valid files, skipped files, and any errors.

Phase 2: Folder Creation

Folder structure from the ZIP is created in the corpus in a single transaction. Existing folders with matching names are reused, not duplicated.

Phase 3: Document Import

Each valid file is extracted and created as a document. If a document path already exists in the corpus, a new version is created (upversioning) rather than failing.

Phase 4: Relationship Creation

If a relationships.csv is present, document relationships are created. Annotation labels are created or reused via the corpus's label set. Missing source/target documents are logged and skipped -- a malformed relationships file does not fail the overall import.

Flat ZIP Upload (Without Corpus)

There is also a simpler UploadDocumentsZip mutation that uploads a ZIP of documents without targeting a specific corpus and without preserving folder structure. Documents are created as standalone items. The same security constraints apply.

Error Handling

The import uses graceful degradation:

Validation failures (zip bomb, too many files) block the entire import
Individual file errors (unsupported type, too large) skip that file and continue
Relationship errors (missing source/target document) skip that relationship and continue
The import result includes detailed counts of processed, skipped, errored, and upversioned files

For implementation details (code snippets, task result schema, file locations, and testing commands), see the Bulk Import Architecture.