Context CI

Context CI is the local/GitHub-native proof loop for DocPull context packs. It answers one question: can an agent safely depend on this context right now?

The context can come from documentation sites, API references, OpenAPI specs, changelogs, product pages, pricing pages, filings, PDFs, typed extraction packs, partner-owned corpora, or any other source that DocPull can normalize into a pack. Docs are a high-value lane, not the product boundary.

docpull ci --prepare
docpull ci packs/vendor-context --predictions agent-output.jsonl

The command is network-free by default. Project mode checks the latest project run under .docpull/runs/<run_id>/; pack mode checks the given directory. Add --sync only when CI should refresh a project before checking.

Gates

docpull ci writes context-ci.report.json and CONTEXT_CI.md next to the checked pack. The v1 gates are:

• project lockfile validity, in project mode;
• pack.score.json and pack.audit.json are current for documents.ndjson;
• pack score is at least 80;
• audit score is at least 80;
• coverage confidence is not low;
• citation coverage is at least 0.90;
• eval-grade artifacts exist, or are generated by --prepare;
• rights metadata warns by default, or fails when --require-rights is used;
• evidence basis quality warns by default and fails under --strict or with prediction grading when claims are unsupported, low-confidence, or missing;
• context prediction pass rate is checked when --predictions is provided.

Optional project config:

ci:
  min_pack_score: 80
  min_audit_score: 80
  min_citation_coverage: 0.9
  min_context_pass_rate: 0.8
  fail_on_medium_coverage: false
  max_age_days: 14
  require_rights:
    - eval_generation

GitHub Actions

name: Context CI
on:
  pull_request:
  workflow_dispatch:

jobs:
  context:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install docpull
      - run: docpull ci --prepare

The same workflow is checked in at docs/examples/context-ci-github-action.yml.

To grade agent output, write JSONL predictions with id or task_id plus an answer, output, text, or response field:

docpull pack prepare packs/vendor-context --eval-grade
docpull pack validate packs/vendor-context --level eval
docpull ci packs/vendor-context --predictions agent-output.jsonl

Prediction grading is part of docpull ci when predictions are provided. The pack must already satisfy the eval-grade contract, or CI must be run with --prepare.

Prediction examples are available under docs/examples/:

context-predictions-pass.jsonl
context-predictions-missing-citation.jsonl
context-predictions-stale-answer.jsonl
context-predictions-insufficient-evidence.jsonl

Context Pack Contract

A CI-ready context pack should include:

documents.ndjson
corpus.manifest.json
sources.md
context.lock.json
coverage.report.json
acquisition.routes.json
pack.score.json
pack.audit.json
rights.manifest.json
provenance.graph.json
citation.index.json
basis.ndjson
basis.report.json
BASIS.md
PACK_CARD.md
evals/tasks.public.jsonl
evals/answers.hidden.jsonl
evals/grader.py
context-ci.report.json
CONTEXT_CI.md

Use this refresh loop for public demo packs:

docpull https://example.com --max-pages 50 -o packs/example
docpull pack prepare packs/example --eval-grade
docpull pack validate packs/example --level eval
docpull ci packs/example --prepare

For context-infrastructure demos, prioritize packs that show different source classes: documentation sites, OpenAPI specs, product/pricing pages, changelogs, PDFs, filings, and pages that require explicit agent-browser rendering.

The durable artifact shape is documented in Context Pack Contract v3.

Human Review Protocol

Human review is a protocol, not a marketplace in v1. Reviewers inspect hidden answers and traps before a pack is treated as commercially eval-ready.

Queue item shape:

{
  "task_id": "eval_...",
  "pack": "stripe-docs",
  "task_type": "version-drift",
  "source_url": "https://docs.example.com/api",
  "expected_claims": [],
  "review_state": "needs_review"
}

Allowed review states:

needs_review
approved
needs_source_fix
needs_question_fix
rejected

Reviewer rubric:

• expected claims are supported by the cited source;
• the task cannot be answered correctly from stale memory alone;
• hidden answers do not include unsupported claims;
• traps identify concrete stale, deprecated, or uncited failure behavior;
• coverage gaps are represented as coverage-aware tasks instead of invented answers.

Reviewed output should preserve the generated task ID and add reviewer, reviewed_at, review_state, and notes fields. Do not redistribute raw source content or sell training/eval use unless rights.manifest.json explicitly allows it.

Future hosted scheduling, hidden eval storage, context PRs, signed releases, and webhooks should stay outside the OSS evidence contract and use the same artifacts generated by the local runner.