JudgeCalibrationKit Design Decisions

This document records decisions that implementation work may rely on. It is a living record, but changes to a frozen decision must include the evidence that caused the change.

Canonical xceval format dependency

Starting with JudgeCalibrationKit 0.2.0, JudgeCalibrationXCEval depends on the public XCEvalFormat product. Version 0.3.0 moves the reviewed dependency range to xceval 0.4.x. XCEvalFormat owns the normalized envelopes, document models, schema and command checks, sample-count validation, and JSON Lines framing. JudgeCalibrationKit owns only the downstream calibration interpretation: stable sample identity, explicit metric bindings, scale validation, abstentions, agreement analysis, and gate policy.

This is intentionally a graph-wide package dependency even though only the xceval bridge target links the product. The cost is that all package consumers resolve Evaluations-Framework-CLI and its swift-argument-parser dependency, Swift 6.0 support ends, and the macOS floor rises from 13 to 14. The benefit is one canonical format implementation in one repository, without a compatibility decoder or a separately versioned bridge package. JudgeCalibration itself continues to link no third-party module.

The rejected alternative was a second JudgeCalibrationXCEval repository, which would preserve the old core package graph but create another release, Swift Package Index entry, and cross-repository compatibility surface. That is the documented escape hatch if the core later gains a meaningful non-Apple consumer base for which graph-wide resolution becomes a demonstrated cost. Package traits, vendored upstream models, and conditional duplicate decoders are not part of this design.

The dependency uses an up-to-next-minor range beginning at 0.4.0. xceval is pre-1.0, so every new minor version is a potentially breaking boundary that requires an explicit compatibility review.

xceval 0.4 adds many typed agent-workflow documents. JCK imports its complete report rows and retains inspect/report artifactID and byteDigest provenance. It deliberately rejects evidence, selection, comparison, gate, and dataset-lifecycle documents: those forms are summary-only or may contain filtered samples, so treating them as a complete calibration population would bias agreement and uncertainty estimates.

Slice-analysis semantics

Slice analysis is opt-in and operates only on explicit single metadata keys. It never searches arbitrary metadata or cross-products, and its output is not a fairness or bias certification. Missing metadata is a distinct slice by default because missing labels often identify new or difficult samples; it can never collide with a real string value. Boolean, finite-number, and string values are supported, with a default maximum of 32 distinct values per key. Exceeding the cap is an error rather than a reason to truncate silently.

The report distinguishes slice analysis not requested from requested analysis with no eligible slices. Worst-slice gates support judge-human comparisons and evaluate coverage and overlap inside each slice. Human-panel slice gates are deferred because within-slice panel eligibility and alpha uncertainty require separate treatment.

The 0.2.0 worst-slice minimum is defined only for higher-is-better metrics. Mean absolute error is lower-is-better and mean signed error is directional; both produce unsupportedWorstSliceDirection instead of a misleading minimum. Supporting them requires a future maximum-value or magnitude-aware gate API.

A worst-slice confidence interval bootstraps the minimum across the originally eligible slice family. It is not the ordinary pointwise interval for the slice that happened to be worst in observed data. Every result reports the number of eligible slices, and unavailable bootstrap replicates record which slice or slices caused the failure. This preserves the existing rule that insufficient successful replicates yields an explicit unavailable interval.

Evaluations-capable Xcode discovery

The package must not encode an Xcode prerelease application name. Scripts resolve an installation dynamically and print its canonical Contents/Developer path; they never call xcode-select --switch.

Resolution precedence is:

1. JCK_XCODE_PATH, when set. It may name an .app or its Contents/Developer directory and is authoritative.
2. Apple's standard DEVELOPER_DIR, when set. It is also authoritative.
3. The active xcode-select -p installation, if it contains Evaluations.
4. Other Xcode bundles found through Spotlight, /Applications, $HOME/Applications, and $HOME/Downloads. The highest semantic Xcode version is selected; equal-version ambiguity requires JCK_XCODE_PATH.

An authoritative override that lacks a macOS Evaluations.swiftinterface is an error; the resolver does not silently substitute another compiler. Without an override, an older active Xcode is skipped so a separately installed capable Xcode can be found. Callers select a particular installation by setting the package-specific override.

Scripts/resolve-evaluations-xcode.sh writes only the resolved developer directory to standard output. Scripts/check-evaluations-interface.sh is a read-only seed check. A nonzero result means that no capable installation was found, an adapter-critical declaration disappeared, or Apple added a possible agreement/calibration API that requires human review.

The relative framework location is stable input to discovery:

Platforms/MacOSX.platform/Developer/Library/Frameworks/
  Evaluations.framework/Versions/A/Modules/Evaluations.swiftmodule/

An absolute beta path may appear below as dated evidence. It is not build configuration and must never be copied into a script, package manifest, or CI workflow. Build scripts should resolve the path, then pass it as DEVELOPER_DIR for that process only.

Current toolchain and interface evidence

Read-only discovery on July 25, 2026 produced the following result:

• The active /Applications/Xcode.app/Contents/Developer is Xcode 26.0 (17A324) with Apple Swift 6.2 (swiftlang-6.2.0.19.9) and does not contain Evaluations.
• Dynamic discovery selected the separately installed Xcode 27.0 beta 4 at /Applications/Xcode-27.0.0-Beta.4.app/Contents/Developer. This is observed evidence, not a pinned path.
• The selected Xcode build is 27A5228h; its toolchain reports Apple Swift 6.4 (swiftlang-6.4.0.27.1).
• The arm64 macOS textual interface has 885 lines, module version 25061.1, compiler version Apple Swift version 6.4 effective-5.10 (swiftlang-6.4.0.27.1 clang-2100.3.27.1), a macOS 14 target, and Swift 5 language mode with library evolution enabled.
• Evaluations imports Foundation, FoundationModels, OSLog, TabularData, Testing, and Swift concurrency/testing support modules. Its declarations are guarded by @available(anyAppleOS 27.0, *) and marked unavailable on tvOS.
• No case-insensitive match for kappa, agreement, calibrat, reliability, annotator, or human appears in this interface. Apple has not added an obvious reusable agreement API in this seed.

This interface check establishes compile-time API shape only. Runtime behaviour is covered separately by Scripts/check-evaluations-simulator.sh, which runs a deterministic, model-free Evaluation on an iOS 27 simulator and imports the result through the typed adapter. It tests the package's own JudgeCalibrationKit-Package scheme, so the test bundle runs without an application host — relevant on iOS 27, where an app built against the SDK must adopt the UIScene life cycle or refuse to launch.

That check needs the preview xcode-27 runner image, so it runs weekly and on demand rather than on every pull request, and it skips cleanly when no iOS 27 runtime is installed. No live adapter execution has been recorded yet, so the adapter ships as beta and compile-time verification is not presented as runtime verification.

Adapter-critical symbol checklist

Every Xcode seed must retain all of the following capabilities before adapter work proceeds:

Capability	Public interface relied upon
Execute and identify an evaluation	Evaluation, run(info:), and the originating evaluation value
Read result rows	EvaluationResult.detailed
Recover typed samples	Evaluation.inputColumn and the typed DataFrame column subscript
Recover dimension scores	ScoreDimension.metric and the metric DataFrame subscript
Preserve scale metadata	ScoreDimension.scale, ScoringScale.options, and ScaleOption.value
Interpret a score	Metric.name, Metric.value, Metric.doubleValue, and Metric.rationale
Preserve abstention	Metric.Value.ignore
Use persisted fixtures without parsing Apple's schema	EvaluationResult.loadJSON(from:) and saveJSON(to:)
Create a deterministic model-free fixture	closure-based Evaluator and ModelSubject.init(value:transcript:)

The importer therefore takes both an EvaluationResult and the originating generic Evaluation; a result alone does not provide its typed input column. Dimension bindings remain explicit, and measurement level is never inferred from numeric-looking scale values. .ignore becomes an explicit abstention and the metric rationale is preserved.

Run the checklist after installing each Xcode seed:

bash Scripts/check-evaluations-interface.sh

If overlap terms appear, inspect the new declarations before continuing. A complete Apple agreement API changes the package's positioning; a partial API should be delegated to where doing so preserves JudgeCalibrationKit's stronger uncertainty, multi-rater, abstention, and reporting behavior.

Unavailable-result semantics

Invalid data and statistically undefined results are different states:

• Structural violations such as duplicate observations, a non-finite score, an unknown scale value, or an unsupported interchange version reject input with a validation error.
• A structurally valid analysis whose statistic is undefined returns .unavailable(reason). It never substitutes zero or NaN, and it does not destroy other metrics or dimensions in the report.
• Cohen's kappa is unavailable when its actual denominator 1 - p_e is zero, not merely because one rater is constant. Spearman rho is unavailable when a rank vector is constant. Each later metric must freeze its own exact rule in its documentation and oracle tests.
• A missing human-panel baseline with one human is reported as unavailable with a singleHuman reason while applicable judge-to-human comparisons continue.
• Missing observations and explicit abstentions reduce overlap and coverage. They make a result unavailable only when that metric lacks enough usable observations; abstentions are never silently dropped from coverage reporting.
• Bootstrap resamples whole sample clusters. Degenerate replicates are excluded and counted. A confidence interval is unavailable when fewer than 90 percent of requested replicates succeed, without erasing an otherwise valid point estimate.
• A gate whose requested metric or requested confidence-bound basis is unavailable fails with that reason and the relevant overlap/coverage counts. No unavailable result can accidentally pass through floating-point comparison behavior.

This distinction keeps reports complete and diagnostic while ensuring malformed input is rejected early.

Frozen JSON interchange schema

The judge-calibration/v1 dataset and judge-calibration-report/v1 report are language-neutral contracts. They never expose Swift's synthesized associated value keys such as _0, and identifier wrappers encode as JSON strings rather than objects containing rawValue.

Rating values are ordinary JSON Booleans, finite numbers, or strings. A scored rating is explicit:

{
  "sample": "book-01",
  "rater": "judge-v1",
  "dimension": "relevance",
  "outcome": { "status": "scored", "value": 4 },
  "metadata": {}
}

Outcomes use one of these shapes:

• { "status": "scored", "value": ... }
• { "status": "abstained", "reason": "..." }, where reason may be omitted
• { "status": "invalid", "reason": "..." }

Measurement scales use an explicit type discriminator:

• binary: { "type": "binary", "negative": false, "positive": true }
• nominal: { "type": "nominal", "values": ["a", "b"] }
• ordinal: { "type": "ordinal", "values": [1, 2, 3, 4, 5] }
• interval: { "type": "interval" }

Metric and confidence-interval results use status: "estimate" or status: "unavailable" with named payload fields. Warnings, gate comparison targets, and gate failures similarly use code or type discriminators.

Tests/JudgeCalibrationTests/Fixtures/dataset-v1.json is deliberately hand-authored and must decode, validate, re-encode, and round-trip without a key beginning with _ or named rawValue. Any incompatible change requires a new schema version rather than silently changing v1.