Worker Uploads (REST API)

The worker upload system provides a REST API for external pipelines and scripts to push pre-processed documents -- with annotations, labels, and embeddings already attached -- directly into a corpus. Unlike the other upload methods, worker uploads bypass OpenContracts' built-in parsing pipeline entirely because the documents have already been processed externally.

When to Use Worker Uploads

• External processing pipelines -- your organization has a custom NLP or document processing pipeline that produces structured output
• Bulk ingestion -- migrating thousands of pre-annotated documents from another system
• Pre-computed embeddings -- documents where text extraction, layout analysis, and embedding generation have already been performed

How It Works

Worker uploads use a token-scoped REST API with a database-backed processing queue:

1. An admin creates a WorkerAccount (a service account with no web login)
2. A corpus owner creates a CorpusAccessToken scoped to a specific corpus
3. The external pipeline sends documents via POST /api/worker-uploads/documents/ with the token in the Authorization: WorkerKey <token> header
4. Uploads are staged in a database queue and processed asynchronously by a Celery batch processor
5. Each upload becomes a Document with annotations, labels, and embeddings in the target corpus

Key Differences from Other Upload Methods

Feature	Worker Uploads	Other Methods
Authentication	Scoped API token (WorkerKey)	Session/Auth0
Processing pipeline	Skipped (data pre-processed)	Runs parser, thumbnailer, embedder
Annotations	Included in upload metadata	Created during parsing or added manually
Embeddings	Can be pre-computed and included	Generated by the embedder pipeline stage
Folder assignment	Via target_folder_path in metadata	Via UI or mutation parameter
Document ownership	Corpus creator (not the worker)	Uploading user

REST API Endpoints

Method	Endpoint	Description
POST	/api/worker-uploads/documents/	Upload a document with metadata
GET	/api/worker-uploads/documents/<upload-id>/	Check status of a single upload
GET	/api/worker-uploads/documents/list/	List uploads for this token (paginated)

All endpoints require the Authorization: WorkerKey <token> header.

Upload Request

The upload is a multipart/form-data POST with two fields:

Field	Type	Description
file	File	The document file (PDF, DOCX, TXT, etc.)
metadata	String	JSON string with document text, annotations, labels, and embeddings

Required Metadata Fields

Field	Type	Description
title	string	Document title
content	string	Full extracted text
page_count	integer	Number of pages
pawls_file_content	array	PAWLs token data (one entry per page with page dimensions and tokens)

Optional Metadata Fields

Field	Type	Description
description	string	Document description
file_type	string	MIME type (defaults to application/pdf)
target_folder_path	string	Folder path to place the document in (auto-created if missing)
doc_labels	array of strings	Document-level label names to apply
doc_labels_definitions	object	Label definitions for document-level labels
text_labels	object	Label definitions for text annotations
labelled_text	array	Text annotations (same format as corpus export annotations)
relationships	array	Annotation-to-annotation relationships
embeddings	object	Pre-computed vectors with embedder_path, optional document_embedding, and annotation_embeddings map

Upload Lifecycle

PENDING  -->  PROCESSING  -->  COMPLETED
                   |
                   +-------->  FAILED

• PENDING -- staged, waiting for the batch processor
• PROCESSING -- batch processor has claimed this upload
• COMPLETED -- document created and added to the corpus
• FAILED -- check the error_message field for details

!!! warning "A Celery worker must drain the queue" The batch processor (process_pending_uploads) runs on the worker_uploads Celery queue, and the post-upload ingest stages (thumbnail, parse, unlock) run on the doc_parse queue. The target deployment must run a Celery worker consuming all three queues (the stock images start it with -Q celery,worker_uploads,doc_parse) plus Celery Beat. With no worker draining worker_uploads, uploads are accepted (HTTP 202) but stay PENDING indefinitely and no documents are created -- a silent failure for self-hosted/custom deployments. See Celery Configuration for Worker Uploads for the full operations reference (queues, Beat, scaling, verification).

Uploads stuck in PROCESSING for longer than WORKER_UPLOAD_STALE_MINUTES (default: 15 minutes) are automatically reset to PENDING by a recovery task (recover_stalled_uploads, scheduled by Celery Beat).

Security Model

• Token storage: Only a SHA-256 hash is stored; the plaintext is shown exactly once at creation
• Scope: Each token is scoped to exactly one corpus
• Worker accounts: Django users with unusable passwords -- cannot log in via web UI
• Ownership: Documents are owned by the corpus creator, not the worker account
• Revocation: Revoking a token immediately prevents authentication; existing queued uploads continue processing

Rate Limiting

Rate limits are set per token via rate_limit_per_minute (0 = unlimited). When exceeded, the API returns HTTP 429 with a Retry-After: 60 header. The check is best-effort; for strict enforcement, use a reverse proxy.

Configuration

Setting	Default	Description
MAX_WORKER_UPLOAD_SIZE_BYTES	256 MiB	Max file size per upload
MAX_WORKER_METADATA_SIZE_BYTES	500 MiB	Max metadata JSON size
WORKER_UPLOAD_BATCH_SIZE	50	Uploads claimed per batch processor run
WORKER_UPLOAD_STALE_MINUTES	15	Minutes before stalled uploads reset to PENDING

Managing Worker Accounts and Tokens

Worker accounts are created by superusers. Corpus access tokens can be created by superusers or corpus owners. Both are manageable through the UI (Admin Settings for accounts, Corpus Settings for tokens) and via GraphQL mutations.

For the complete setup walkthrough with examples, UI screenshots, and the full metadata schema reference, see the Worker Upload System Walkthrough.

Higher-Level Driver: The Remote Ingest Worker

You usually do not call this REST API by hand. The Remote Ingest Worker (scripts/remote_ingest) is a turnkey, resumable CLI + docker-compose bundle that runs the real Docling parser and embedder on your own hardware, optionally calculates extra metadata and annotations, and streams the finished documents to this endpoint -- so the upload metadata above is produced faithfully for you rather than assembled by hand. Also mints tokens with one command: python manage.py mint_worker_token.