TypeSafe/Jev evaluation contract

This is a documented interface, not a live client. The supplied request file is synthetic and is never sent by the offline tests.

Request boundary

The public API reference defines POST https://api.typesafe.ai/v1/systemone with a Bearer token, a model, a state, and a map of typed questions. A Choice question uses type, instructions, and a map of option descriptions. See the API reference.

Use request.example.json to inspect the shape. Option descriptions are strings containing what, not_for, and examples, which keeps them compatible with the API reference. Every instruction includes the literal claim rather than relying on a question ID to convey meaning.

The state contains permitted source material and observations. It excludes benchmark answers, the host's desired verdict, and the routing threshold. A hypothesis may be included as a labeled claim. Actual test results remain observations, not opinions to strip from the payload.

Provider identity

The example uses the documented alias jev-latest. Record both the requested identifier and the model returned by the service. The historical benchmark reported jev-1.13.0; its past use does not prove that the same version remains available to every account.

A frozen-version experiment must request its declared identifier and stop or mark a protocol deviation when it is unavailable. Never silently replace a pin and present the run as identical. A version pin also does not promise identical responses on repeated requests.

Response boundary

Preserve the raw choice, complete probabilities, and provider confidence. Validate the expected question IDs, option keys, finite probability range, normalization tolerance, and consistency between the selected option and the distribution. Keep model and usage as returned. Missing telemetry is null with a reason, not a fabricated zero.

The current example routing uses max(probabilities), called top_p, rather than the API's distribution-derived confidence. A low top probability and an explicit insufficient-evidence answer must have distinct routing reasons. See TypeSafe confidence.

Batching and limits

Questions about the same frozen material may share one request. They should each be interpretable on their own; do not assume one answer becomes evidence for another. Add only the questions the assessment needs. Extra questions still consume tokens, as described in the Choice guide.

The reference profile uses one request, no automatic retries, and a 10-second end-to-end request deadline including body delivery. This is our conservative experiment policy, not the provider's universal recommendation. A timeout returns to normal lead review and grants no extra work.

Credentials and egress

A real client must obtain TYPESAFE_API_KEY from its authorized execution environment. It must not write the value, authorization header, environment dump, or unrelated secrets into evidence or logs. This repository does not ask for a key and does not search for one.

A project must separately authorize both external calls and the data they send. Synthetic examples are suitable for a transport smoke test, but such a test does not establish the privacy or correctness of a production collector.

Instructional skill

The official TypeSafe skill is maintained separately. It is not included, altered, or installed here. Projects decide whether and where to load it. Loading that skill is not evidence that this handoff protocol is enforced.