Jev Review Action

September 19, 2026 · View on GitHub

Review submissions and classify pull requests with TypeSafe Jev. You define the criteria and categories; Jev returns typed judgments; code applies the policy and updates one PR comment. Jev can be reached through TypeSafe, Vercel AI Gateway or Cloudflare Workers AI, with an ordered fallback chain (see Jev providers).

Only Jev is used. Comments come from a fixed template. There is no text-generation model, autonomous agent, or hosted bot server.

An independent personal open-source project by fatwang2, not an official TypeSafe product. The first consumer is Awesome Jev.

Two modes, one engine

Mode	Evidence	Example
`catalog`	1–10 submitted JSON entries, each with its public GitHub repository's README and source files at an immutable commit	Awesome lists, integration directories, project showcases
`pull-request`	PR title, body and changed-file patches	Scope checks and change classification for an ordinary repository

Policies are JSON. The action does not hardcode Jev ecosystem criteria: replace the categories and questions to review another topic. Examples: catalog policy, PR policy.

Quick start

Copy an example policy to .github/jev-review.json on your default branch.
Set the repository secret TYPESAFE_API_KEY to your own TypeSafe key, or configure another Jev provider.
Add the workflow below to the default branch. It only runs trusted base-branch code. For production, replace the action version with the full commit SHA from the release.

name: Jev review
on:
  pull_request_target:
    types: [opened, synchronize, reopened]
permissions:
  contents: read
  pull-requests: write
concurrency:
  group: jev-review-${{ github.event.pull_request.number }}
  cancel-in-progress: false
jobs:
  review:
    if: github.event.pull_request.base.ref == github.event.repository.default_branch
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
        with:
          ref: ${{ github.event.pull_request.base.sha }}
          persist-credentials: false
      - uses: fatwang2/jev-review-action@v0.2.0
        id: review
        with:
          typesafe-api-key: ${{ secrets.TYPESAFE_API_KEY }}
      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        if: always()
        with:
          name: jev-review-${{ github.event.pull_request.number }}
          path: jev-report.json
          if-no-files-found: ignore
          retention-days: 14

External fork submissions work through pull_request_target. Never check out, install dependencies from, or execute the PR head in this privileged workflow. The action verifies that the local checkout matches the event's base SHA. It reads submitted data through the GitHub API, rejects symlinks and submodules, and does not execute it.

Policy

Each check is a positive yes/no criterion with id, title, question, yes, no, accept, and reject. All checks are required. One Choice question selects among categories, which must include other. Set categoryConfidence to your review boundary.

recommended: all checks pass, a category is supported, and evidence is complete within the configured collector.
needs-review: ambiguous answers, category disagreement, missing/truncated integration evidence, archive status, an unrecognized license, or a pipeline limit such as source-selection overflow (add up to six evidence paths).
not-recommended: at least one check fails, with no evidence collection warning.
error: invalid submission or operator/infrastructure failure; the action fails and does not approve anything.
skipped: no catalog entry changed, or the PR changed/closed before publication.

recommended is advisory, not a GitHub review approval. The action never merges, closes, edits submissions, or applies labels. Outputs let a caller implement additional behavior explicitly. Example thresholds are provisional: use human-labeled examples to measure false acceptance, false rejection, and category agreement before relying on them.

Catalog comments also include GitHub star and owner-follower counts as maintainer context. These are not review criteria and are not sent to Jev.

Changing the policy requires a separate maintainer PR. A catalog submission changing non-entry files, removing/renaming entries, or exceeding ten entries is rejected before inference.

Batch catalog reviews (unreleased)

Batch support requires the commit containing this change; the v0.2.0 quick-start pin above still accepts only one entry. Publish the updated Action and pin its immutable commit before enabling batch submissions in a consumer.

Each project gets independent evidence collection and a separate Jev call, with at most two projects processed concurrently. One invalid entry or provider failure does not discard sibling results. One comment contains a summary table and expandable per-project details; oversized details remain available in the JSON report.

The batch decision has precedence error → not-recommended → needs-review → recommended. Only an all-recommended batch is recommended; any error fails the Action. This is not majority voting or automatic merge enforcement. GitHub merges the entire PR, so resolve, remove, or split unresolved entries before merging.

Single-entry reports retain schemaVersion: 1 and their existing top-level fields. Multi-entry reports use schemaVersion: 2, with ordered reports containing each entry's full result, entryPath, and (when valid) projectRepository. The top-level repository remains the PR host repository. Batch category output is empty; categories belong to individual results. Every rerun reviews all entries again; results are not cached. Reports remain Actions artifacts with the caller's retention period, not a permanent archive.

Catalog entry

Use entries/owner--repository.json (lowercase). Submit one file per project, up to ten projects per PR with batch support:

{
  "name": "Example project",
  "repository": "owner/repository",
  "description": "A concrete description whose claims can be checked against source.",
  "category": "search"
}

evidence is optional. Omit it (or use []) to let the Action find integration source files. If you know the relevant files, add up to six relative paths, for example "evidence": ["src/client.ts"]. Supplied paths still receive strict validation and priority.

The file name is the canonical repository identity and prevents duplicate filenames. The directory's own validation should also check its full catalog for duplicates. Only public GitHub source repositories are supported in catalog mode. Non-GitHub projects and arbitrary webpages need a future evidence adapter, not unrestricted URL fetching.

The collector resolves the submitted repository's default branch to an immutable commit. Jev selects up to six source files using the complete root README, up to two root manifests, and candidate paths/sizes, without seeing source contents or submission evidence hints. Each candidate receives a Noul judgment; candidates scoring at least 0.5 are ranked by probability, then path. Dependencies, generated output, tests, fixtures, symlinks, and submodules are excluded from automatic candidates; source files are limited to 100 KB. Up to six optional evidence paths are read in addition to the selected files. Explicit evidence can still point to a regular supported test or example file.

Selection is a retrieval hint, not proof of integration. A separate Jev call judges the actual source. The selector targets TypeSafe/Jev; it is not a general-purpose ecosystem selector. No selected or supplied source means maintainer review. Provider errors fail visibly rather than falling back to favorable rule-based results. The old rule selector remains available internally for offline comparisons only.

Catalog files are never truncated: at most ten complete files and 48,000 file-content characters reach the review. A file that cannot fit is omitted with a visible warning requiring maintainer review; smaller subsequent files can still fit. File selection sends all eligible paths in one shared-state request with one short Noul question per path, not one API call per file. There is no candidate-count or serialized-byte cutoff for selection; the provider enforces model context limits. A context-limit rejection is a pipeline limit, not a PR defect: the review asks for evidence paths instead of failing the Action. Other provider errors still fail visibly without truncation or automatic splitting. Ordinary pull-request diff mode retains its separate excerpt behavior.

The collector also sends a bounded facts block (GitHub Action file existence, license, top-level paths, and package.json dependency names) so description claims can be checked against filesystem facts even when those files were not selected as source evidence.

The JSON report's discovery field records selection candidates, probabilities, selected/included paths, model, token usage, latency, and state/question hashes. Top-level usage describes the final review call only; add discovery.usage for total model usage. Missing evidence still requires maintainer review; no second retrieval round is implemented. Discovery does not inspect an entire repository or prove that a project works.

Inputs and outputs

Input	Default	Purpose
`jev-providers`	`typesafe`	Enabled Jev providers in order of preference: `typesafe`, `vercel`, `cloudflare`
`typesafe-api-key`		TypeSafe key, sent only to `api.typesafe.ai`; enables `typesafe`
`ai-gateway-api-key`		Vercel AI Gateway key, sent only to `ai-gateway.vercel.sh`; enables `vercel`
`cloudflare-account-id`, `cloudflare-api-token`		Cloudflare account and Workers AI token, sent only to `api.cloudflare.com`; enable `cloudflare`
`github-token`	`github.token`	Read evidence and write PR comments
`policy`	`.github/jev-review.json`	Trusted policy file
`model`	`jev-latest`	Jev model ID sent to TypeSafe; pin a version for comparisons
`ai-gateway-model`	`typesafe-ai/jev`	Model ID sent to Vercel AI Gateway
`cloudflare-ai-model`	`typesafe/jev`	Model ID run through Cloudflare Workers AI
`comment`	`true`	Set `false` for report-only use
`report-path`	`jev-report.json`	JSON report in the workspace

Outputs: decision, category, and report-path. The JSON report includes raw typed answers, token usage, resolved model, the provider that answered (judge), source commit, PR head, policy/state hashes, thresholds, evidence URLs and follow-up reasons. It does not contain the API key or complete source files. Changing a threshold can be evaluated against saved answers without another provider call.

Every run costs provider tokens; each catalog project normally uses one Jev file-selection call and one review call containing its checks and category question. Only transient HTTP failures retry, at most twice per provider. Evidence reads, HTTP response sizes, and request timeouts remain bounded. No API keys are needed for CI tests.

Jev providers

Jev is served by three services that answer the same questions. Any one is enough; the others are optional fallbacks.

Provider	How it is called	Inputs
`typesafe`	TypeSafe's own API, `api.typesafe.ai/v1/systemone`	`typesafe-api-key`
`vercel`	Vercel AI Gateway evaluation-model endpoint, model `typesafe-ai/jev`	`ai-gateway-api-key`
`cloudflare`	Cloudflare Workers AI REST API, model `typesafe/jev`	`cloudflare-account-id`, `cloudflare-api-token`

jev-providers is the switch. Only listed providers are used, in the order given; the default is typesafe alone. Providers not listed stay off even when their credentials are set, so adding a secret never changes behavior by itself. The first listed provider with credentials is primary and the rest are fallbacks; a listed provider without credentials is skipped, and a run with no configured provider fails before any GitHub or model call.

      - uses: fatwang2/jev-review-action@v0.2.0
        with:
          jev-providers: typesafe,vercel,cloudflare
          typesafe-api-key: ${{ secrets.TYPESAFE_API_KEY }}
          ai-gateway-api-key: ${{ secrets.AI_GATEWAY_API_KEY }}
          cloudflare-account-id: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          cloudflare-api-token: ${{ secrets.CLOUDFLARE_API_TOKEN }}

A request moves to the next provider only when the current one fails with HTTP 402 (no credit), 429 (throttled) or 5xx after its own transient retries. Client errors such as 400 or 401 are not retried elsewhere, and answers from every provider pass the same validation, so a fallback can never produce a more favorable result than a malformed answer would. Each hop is logged as Jev provider <name> returned HTTP <status>; retrying with <next>, and the report's judge field and the comment's Model: line name the provider that answered.

Provider notes:

TypeSafe bills per token to your TypeSafe organization; without credit it returns 402.
Vercel free-tier teams are rate-limited per model and return 429 after a few requests; any purchased AI Gateway credit moves the team to the paid tier. The gateway's boolean answers map to Jev's noul probabilities and Jev's confidence is read from the gateway's provider metadata; a choice answer without a probability distribution is rejected like any other malformed answer.
Cloudflare needs an API token with Workers AI permission and the account ID; there is no Workers binding in GitHub Actions. Jev is billed to Cloudflare AI Gateway credits; an exhausted balance is reported inside the REST envelope and treated as 402.
Model inputs accept only Jev model IDs in each provider's naming (jev-*, typesafe-ai/jev*, typesafe/jev*).

The local CLI reads the same switches from the environment: JEV_PROVIDERS, TYPESAFE_API_KEY, AI_GATEWAY_API_KEY, CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN and the optional AI_GATEWAY_MODEL/CLOUDFLARE_AI_MODEL.

Local review

Node.js 22+; the GitHub Action uses Node.js 24. There are no npm dependencies and no generated bundle to audit.

npm test
export TYPESAFE_API_KEY=your_key
export GITHUB_TOKEN=your_read_only_token # optional, raises GitHub rate limits
npm run review -- --policy examples/catalog.json --entry /path/to/entry.json

The CLI only writes a local report. It does not post to GitHub. --repository owner/repo can be used instead of --entry for exploratory checks without a proposed description/category.

Data and permissions

PR title/body/diff in PR mode, or submitted descriptions and public repository excerpts in catalog mode, are sent to the Jev provider that answers: TypeSafe directly, or Vercel AI Gateway or Cloudflare Workers AI, which forward to TypeSafe. Do not enable it on private PR data unless this transfer is intended. GitHub requests go only to api.github.com, model calls only to api.typesafe.ai, ai-gateway.vercel.sh and api.cloudflare.com for the providers you enable, and redirects are rejected. Retrieved text is untrusted evidence; model judgments are advisory and cannot grant permissions.

Use the standard GITHUB_TOKEN for one-comment updates: the action only edits comments authored by github-actions[bot] with its marker. A fresh PR-head check suppresses stale results; keep the workflow concurrency group to serialize reviews of one PR. Reports identify the reviewed head so changes remain visible.

Development and validation

npm test covers policy decisions, malformed model answers, bounded evidence, symlinks, unsafe paths, private-source refusal, comment ownership/pagination, stale PR suppression, trusted checkout enforcement, and a mocked end-to-end action run. These tests validate software behavior, not Jev's classification accuracy.

Live calibration is not yet published. Configure a dedicated Jev provider key and compare saved reports against human labels before claiming accuracy or enabling downstream automation.

See CONTRIBUTING.md and SECURITY.md. MIT licensed; TypeSafe and Jev names belong to their respective owners.