Jev Review

September 16, 2026 ยท View on GitHub

A small autonomous PR review bot built around TypeSafe's Jev. It consumes structured PR metadata and patches, asks finite typed questions, then applies deterministic approval gates. Uncertain or risky reviews route to trusted owners with a structured explanation. Suggestions are advisory; the bot never edits or merges code.

Status: GitHub App installed on thiago-ss/jev-review; scheduled evidence comments enabled; approval writes disabled. Production approval is gated on a named repository, trusted deployment configuration and representative held-out calibration evidence. The included synthetic examples do not satisfy that gate.

Review X-ray and experiment atlas

The pilot adds three typed perspectives to comment-only reviews: correctness, security and verification. Reports preserve disagreements, per-file hypotheses, missing evidence and exact CI links. These are correlated prompts, not independent reviewers.

Explore the self-contained interactive atlas, visual experiment plate, live observations, and validation evidence. The ten synthetic bug/repair and metadata-injection probes reveal false alarms as well as successful holds; they do not establish production calibration.

Run locally

Python 3.9+; runtime uses the standard library.

python3 -m venv .venv
.venv/bin/python -m unittest discover -s tests -v
.venv/bin/python -m jev_review --help

review --input FILE accepts a JSON packet containing pr, optional review, config and calibration. Supplying a review exercises the pipeline offline; omitting it calls Jev using TYPESAFE_API_KEY. The CLI reads environment variables and does not automatically load .env.

.venv/bin/python -m jev_review review --input examples/review.json
.venv/bin/python -m jev_review calibrate --input examples/calibration.json --config examples/calibration-config.json

The calibration example is intentionally synthetic and reports ready: false; production readiness requires the held-out evidence described below.

Dry-run is the default. review --execute only simulates active policy evaluation locally; it never calls GitHub. Only github and poll can write externally: --comment-only posts evidence comments; --execute additionally permits gated approvals and reviewer requests. CLI defaults remain dry-run.

Installable GitHub App

Use the GitHub App setup guide to register predefined permissions, install on selected repositories and run with short-lived scoped tokens. The scheduler remains disabled until configured. Once enabled, it posts evidence comments by default (JEV_COMMENTS=false opts out); JEV_EXECUTE=true separately enables gated approvals and reviewer requests.

GitHub integration

export TYPESAFE_API_KEY='...'
export GITHUB_TOKEN='...'
.venv/bin/python -m jev_review github --repo OWNER/REPO --pr 123 --config examples/github-config.json --dry-run
.venv/bin/python -m jev_review poll --repo OWNER/REPO --config examples/github-config.json --dry-run

Use trusted configuration from the bot's default branch. Do not load credentials, policy or calibration from a PR branch. Review the deployment runbook before enabling writes.

Confidence and calibration

Jev returns probability distributions and a vendor confidence statistic. These are preserved separately. Model certainty alone never authorizes approval. Production evidence must match the concrete response model, prompt/schema and frozen policy; synthetic, stale, duplicated or mismatched evidence fails closed.

The evaluation protocol defines safe autonomous coverage and the false-approval bound. The final ten-case real API run matched all synthetic approval labels; it does not establish production accuracy or calibration.

Project map

Matt Pocock's selected engineering skills are installed under .agents/skills/, with upstream provenance and license. Implementers use Luna high; recorded evidence distinguishes local tests, real API observations and missing production evidence.