Try Jev and inspect the results

Run the actual examples first, then use the offline checks to verify the SDK mechanics.

Six labeled support cases

From a source checkout, run the evaluation with the offline baseline:

python examples/evaluate_support.py

To evaluate Jev, with your key configured:

python examples/evaluate_support.py --provider typesafe

This makes at most six requests without retries. The JSON report in reports/support-evaluation.json retains every attempted case, full distributions, expected labels, model identity, request times, and provider-reported usage. Expected labels never enter model inputs. A failed run retains its error and traces but produces no aggregate quality metrics.

The cases cover billing, outages, degraded service, sparse evidence, missing impact, and conflicting monitors. Metrics include accuracy, abstention rate, recall on missing-evidence labels, and multiclass Brier score (sum across classes, then mean across questions). The lexical provider's Brier value measures heuristic score error. These six author-labeled cases are a recipe, not a held-out benchmark or calibration study. Edit the labels/rubrics for your domain before using a larger independent set.

Check Jev on the actual examples

Complete the Jev setup. With TYPESAFE_API_KEY set:

RUN_TYPESAFE=1 python scripts/probe_examples.py

At most four requests: the support, ranking, and evidence examples, plus a deliberately sparse support message. A missing-information example matters as much as a clear one. The script records all attempted cases in reports/jev-examples.json, including disagreements, full distributions, model IDs, request time, and provider-reported usage. It stops on an error without retrying. It also runs the same ranking input through the offline lexical provider. Expected labels are separate from model inputs.

These are handwritten examples with author-assigned expectations, not a held-out benchmark. A successful result establishes behavior on those inputs only. In particular, the token-overlap baseline is not an embedding model, reranker, or LLM. No LLM accuracy, latency, cost, or calibration advantage has been measured here. No authenticated results are included unless explicitly recorded with their scope.

After the development setup, you can also run a one-request adapter smoke test using all three primitives:

RUN_TYPESAFE=1 python -m pytest -q -m live

That smoke test checks response shape/provenance, not correctness. Both live commands are opt-in and separate from CI's offline gate.

Offline verification

The offline suite checks typed values, complete distributions, provider batching, budgets, timeouts, cancellation, ranking, crawler scope, snapshot replay, synchronous session cleanup, and record export. TypeSafe contract tests use the installed SDK with an in-memory HTTP transport. They check serialization and validation, not model quality. The gate also builds and inspects both wheel and source distributions.

After the development setup, use the active environment:

python scripts/check.py

Crawler regression fixture

decido eval --output reports/local-eval.json

This compares BFS, keyword ranking, and the decision runtime on a bundled synthetic graph. With the default lexical provider, keyword and runtime ranking use the same signal. Their agreement checks the integration; it does not demonstrate semantic intelligence. The report records fetch order, relevance metrics, failures, budgets, fixture identity, provenance, and usage. Labels never enter the model's evidence.

Measure your actual workflow

Use an independently labeled, representative set, including ambiguous cases and missing evidence. Keep inputs and allowed actions the same across systems. Evaluate the application's mistakes and abstentions as well as ranking/classification quality; record latency and total cost including retries. Choose policy thresholds separately from the model's output. Preserve full distributions so Brier scores and calibration can be evaluated against labels for the exact questions asked.

For historical research, only use information available at the decision time. A newly crawled old article is not automatically a historical snapshot. No live market feeds, trained trading model, or financial-performance study is part of this package.