Implementation parity
September 2, 2026 · View on GitHub
This checks whether TT-Bio reproduces each model's original reference implementation on the same input. The device fold is compared to the reference fold across seeds; it is device-vs-reference parity within the model's own seed-to-seed noise, not a benchmark against experiment. Model accuracy (does the fold match the native structure) is out of scope.
Verdict
| model | target | verdict | reason |
|---|---|---|---|
| ESMC-300m | 4 proteins, L20–129 | PASS | deterministic encoder; emb PCC 0.9987–0.9996, residual is bf16 rounding |
| ESMC-600m | 4 proteins, L20–129 | PASS | same path; emb PCC 0.9994–0.9996 |
| ESMC-6b | 4 proteins, L20–129 | PASS | same path at 6b; emb PCC 0.9990–0.9997 (opt-in for load time, not accuracy) |
| ESMFold2 | trp-cage, L20 | PASS | CA-RMSD 0.61 Å inside the 0.51 Å floor |
| ESMFold2 | GB1, L56 | PASS | CA-RMSD 0.33 Å inside the 0.29 Å floor |
| ESMFold2 | ubiquitin, L76 | PASS | CA-RMSD 0.75 Å inside the 0.92 Å floor; device closer to ref than ref to itself |
| ESMFold2 | lysozyme, L129 | PASS | CA-RMSD 0.136 Å inside the 0.139 Å floor (X/floor 0.98); seed-wiring fix applied (per-leg evidence) |
| ESMFold2-Fast | trp-cage, L20 | PASS | 24-block checkpoint through the same harness; CA-RMSD 0.66 Å inside the 0.76 Å floor |
| ESMFold2-Fast | GB1, L56 | PASS | CA-RMSD 0.44 Å inside the 0.46 Å floor |
| ESMFold2-Fast | ubiquitin, L76 | PASS | CA-RMSD 1.46 Å against a 1.25 Å reference floor, inside that floor's 1.76 Å +1sd band; plddt_pcc 0.9976, distogram_pcc 0.9995 |
| ESMFold2-Fast | lysozyme, L129 | PASS | CA-RMSD 0.27 Å inside the 0.31 Å floor (X/floor 0.86); plddt_pcc 0.9997 |
| ESMFold2-Fast | cdk2x2, L256 | PASS | same CDK2 construct as the L512 row, so the two differ only in length; plddt_pcc 0.9939, distogram_rel_l2 0.045, ptm 0.9673 device against 0.9683 reference, same-seed plDDT gap 0.0033 against a 0.0024 reference seed spread |
| ESMFold2-Fast | cdk2x2, L512 | GAP-evidenced: structure PASS, confidence FAIL | the size the perf page publishes. Coordinates inside the floor (Kabsch 0.92 Å against 1.19 Å, distance-matrix 1-pcc 0.00080 against 0.00108, distogram_rel_l2 0.125 against a 0.25 bound), but mean plDDT 0.8987 device against 0.9197 reference on the same seed, a 0.0210 gap at 4.9x the reference's own 0.0043 seed spread, and ptm 0.7679 against 0.7980. Device-side and specific to the 24-block trunk: the 48-block row on the same fixture and stack agrees with its reference to 0.0005. Onset is between L256 and L512. site/data/perf-512aa.json keeps parity_pending on this row (detail) |
| Protenix-v1 | 7ROA, L117, MSA | PASS | external CPU reference (upstream Protenix v0.5.0, fp32, the checkpoint's own 4 recycles); all-atom Kabsch RMSD X 1.714 Å inside the 1.840 Å reference noise floor (X/floor 0.93). Same target and the same MSA bytes as the Protenix-v2 row below, so the two differ only in checkpoint |
| Protenix-v2 | 7ROA, L117, MSA | PASS (legacy R/D/X); GAP-evidenced under the envelope gate | CA-RMSD 2.63 Å inside the 2.94 Å floor; confidence-head under-ranking shared with reference (model property). The envelope test GAPs this leg since v0.6.2 (numerator 1.168 Å vs a collapsed 0.042 Å envelope): the in-range AttentionPairBias unfusing shifts device trajectories at bf16 scale; root-caused below |
| Protenix-v2 | ubiquitin, L76, MSA | PASS (legacy R/D/X); GAP-evidenced under the envelope gate | CA-RMSD 1.73 Å inside the 1.92 Å floor; passes on TM-score and CA-lDDT too. The envelope test GAPs this leg since v0.6.1 (numerator 2.015 Å vs a collapsed zero envelope): the in-range MSA row-chunking puts this target's 20826-sequence MSA on the chunked trunk path, measured non-bit-exact by design; root-caused as the path change, not a regression — see below |
| Protenix-v2 | HSA, L585, MSA | PASS | on-device fp32 diffusion matches the reference's own fp32 boundary; CA-RMSD 0.685 Å inside the 0.695 Å floor (was GAP-evidenced in bf16, X 1.03 Å) |
| OpenFold3 | ubiquitin, L76, MSA | PASS | external CPU reference (official openfold3 0.4.4, fp32); all-atom RMSD X 1.46 Å inside the 1.64 Å reference noise floor (X/floor 0.89) |
| OpenFold3 | 7ROA, L117, MSA | PASS | external CPU reference; all-atom RMSD X 2.02 Å inside the 1.97 Å floor |
| OpenFold3 | 7XI5, L133, MSA + templates ON | PASS | external CPU reference; all-atom RMSD X 4.38 Å inside the 4.16 Å floor (X/floor 1.05); templates verified active (on/off structures differ up to 6.1 Å); 1-lDDT above its tighter floor (per-leg evidence) |
| OpenFold3 | 7XI5, L133, MSA, templates OFF | PASS at commit; GAP-evidenced since the fp32 diffusion boundary | external CPU reference; all-atom RMSD X 5.40 Å vs the 2.87 Å reference floor (was X 4.64 within a device-noise-inflated floor, D 3.76 → 0.70 under the intended fp32 boundary). All five device seeds land 0.59–0.61 Å vs the experimental structure, better than the CPU reference's own 0.42–0.90 Å spread, root-caused below |
| OpenFold3 | 8HEL construct, L77, MSA, no templates | PASS | external CPU reference; all-atom RMSD X 5.52 Å inside the 7.59 Å floor (X/floor 0.73); 1-lDDT above its tighter floor (per-leg evidence) |
| OpenFold3 | 8HEL construct, L77, single-sequence | PASS | external CPU reference; all-atom RMSD X 11.31 Å within the floor (R 11.57, D 8.14; single-sequence de-novo helix is intrinsically ill-determined) |
| OpenFold3 | 9BK6 heterodimer (2xL~104/60), per-chain MSA | PASS | external CPU reference; all-atom RMSD X 1.663 Å within the 5-seed noise floor under the on-device fp32 diffusion boundary (OF3_DIFFUSION_FP32_DEVICE, default on — the Protenix HSA lever); bf16 diffusion missed this floor (X 1.889 Å) — root cause and A/B in openfold3-port.md; 1-TM and 1-lDDT above their tighter floors (per-leg evidence) |
| OpenBind-0 | ubiquitin, L76, MSA | PASS | external CPU reference (upstream aqlaboratory/openfold-3 v0.5.0 clone, fp32, OpenBind checkpoint); all-atom RMSD X 0.969 Å inside the 1.033 Å reference noise floor (X/floor 0.94), and 1-PCC, 1-TM and 1-lDDT inside theirs. Same sequence, same MSA bytes and same 200-step / 5-sample / 4-recycle settings as the OpenFold3 ubiquitin row above, so the two checkpoints are comparable leg to leg |
| OpenBind-0 | FKBP12 + SB3 (1FKG), L107 + 33 ligand atoms, MSA | PASS-caveated | external CPU reference; all-atom RMSD X 0.602 Å inside the 0.551 Å floor (X/floor 1.09), so full_parity_gate.py records it PASS on the gate metric. The residual is the ligand pose, not the fold: split under the same global superposition the protein half is X 0.582 Å against a 0.551 Å floor and the ligand half X 0.969 Å against the reference's own 0.524 Å ligand spread. 1-TM (1.21) and 1-lDDT (1.29) sit above their tighter floors. The device is 2.9× TIGHTER than the reference on the ligand (D 0.183 Å vs R 0.524 Å) since the CCD stereo fix; before it, our own reference conformers drew a random handedness per centre and the device's ligand spread was 0.630 Å, which inflated the floor and flattered the ratio. All 864 atoms pair between device and reference, the 33 SB3 atoms included (per-leg evidence) |
| Boltz-2 | trp-cage, L20, no MSA | PASS | wide no-MSA floor; absolute X 0.60 Å |
| Boltz-2 | 7ROA, L117, no MSA | PASS (legacy R/D/X); GAP-evidenced under the envelope gate | wide no-MSA floor (R 4.98 Å); absolute X 4.21 Å. The tighter envelope test GAPs this leg at the pinned seed (ratio 2.04); root-caused as seed-0 chaotic-trajectory amplification, not a precision bug — see below |
| Boltz-2 | 7ROA, L117, MSA | PASS | CA-RMSD 0.94 Å inside the 0.81 Å floor |
| Boltz-2 | ubiquitin, L76, MSA (production default) | PASS | all 4 metrics within the tight MSA-backed GPU-reference floor (CA-RMSD X/floor 1.03, 1-lDDT X/floor 0.97); residual systematic bf16 (same-seed diagonal) |
| Boltz-2 | HSA, L585, no MSA | PASS | CA-RMSD 1.47 Å inside the 1.50 Å floor; first L585 target |
| Boltz-2 | 9NCY AbAg complex (3 chains, 505 tokens), no MSA | GAP-evidenced (envelope gate) | shared-draws envelope: CA-RMSD numerator 6.71 Å vs bound 2.539 Å on qb1 (ttnn 0.67.4); the 0.760 Å PASS numerator was recorded on pc (ttnn 0.68.0) at fixture birth and the whole commit range since is byte-inert on qb1. Root-caused 2026-08-16 as stack-sensitive basin selection on a chaotic no-MSA target, not a code regression (device, bf16 ref, fp32 ref land in equally accurate 6.4-6.6 Å basins vs ground truth, TM 0.92-0.94) — see below. First leg inside the former [385,506]aa L1 crash band (fixed by c06bd76cf); campaign-relevant size (AbAg-XM median 509) |
| Boltz-2 (affinity) | FKBP12 + SB3, L107, no MSA (non-default) | PASS (legacy R/D/X); PASS under the envelope gate | device-fp32 hybrid diffusion vs the GPU bf16 reference: pocket-lDDT X 0.014 within the GPU noise floor (X/floor 1.25); affinity scalar, affinity probability, and ligand-pose RMSD also pass (X/floor 0.79 / 1.38 / 0.92). Under the envelope gate (gate of record) the affinity scalar passes at ratio 1.35 (numerator 0.0505 vs bound 0.0660) and affinity_probability at 3.65 (bf16-floor evidence) |
| Boltz-2 (affinity) | FKBP12 + SB3, L107, MSA (production default) | PASS-caveated | Under the envelope gate (gate of record) the affinity scalar passes at ratio 0.32 (numerator 0.0456 vs bound 0.226) and affinity_probability at 0.18; pocket-lDDT GAPs under legacy R/D/X (X/floor 4.48), the same narrower-basin systematic-bf16 property as the other affinity legs (seed-independent same-seed diagonal). The legacy R/D/X scalar "GAP" (X/floor 2.27) was a stale-fixture artifact, measured against pre-shared-draws refs before the fixture regen, and is superseded by the envelope pass |
| Boltz-2 (affinity) | DHFR + MTX, L187, no MSA (non-default) | PASS-caveated | affinity scalar and ligand-pose RMSD pass (X/floor 0.68 / 1.36); pocket-lDDT GAPs (4.72), proven a genuine bf16-BACKEND floor by three-backend triangulation (GPU-bf16 and CPU-bf16 references disagree on the pocket by the same ~0.13 lDDT margin the device does), not a port defect |
| Boltz-2 (affinity) | DHFR + MTX, L187, MSA (production default) | PASS-caveated | affinity scalar (1.32), affinity_probability (0.95), and ligand-RMSD (1.61) PASS; pocket-lDDT GAPs (13.35), systematic bf16 by the same-seed diagonal |
| Boltz-2 (affinity) | trypsin + BAM, L223, no MSA (non-default) | PASS-caveated (legacy R/D/X); GAP under the envelope gate | affinity scalar and ligand-pose RMSD pass under legacy R/D/X (X/floor 0.94 / 0.95); pocket-lDDT GAPs (10.13), proven a genuine bf16-BACKEND floor by three-backend triangulation (GPU-bf16 vs CPU-bf16 pocket-lDDT X/floor 7.51, both NO), not a port defect. Under the envelope gate (gate of record) the affinity scalar GAPs at ratio 2.81 (numerator 0.0537 vs bound 0.0387): the device predicts 2.552 where the CPU reference gives 2.606, about 2%. Pose is unaffected (ligand-RMSD ratio 0.11, pocket-lDDT bit-identical). Pre-existing, un-masked in v0.6.5 when the reference was regenerated on the conformer-seeding fix — see the bar move |
| Boltz-2 (affinity) | trypsin + BAM, L223, MSA (production default) | PASS-caveated | affinity scalar (0.79), affinity_probability (0.92), and ligand-RMSD (0.78) PASS; pocket-lDDT GAPs (2.75), systematic bf16 by the same-seed diagonal |
| OpenDDE | trp-cage, L20, no MSA | PASS (legacy R/D/X); GAP-evidenced under the envelope gate | CA-RMSD 0.51 Å inside the 0.52 Å floor. The envelope test GAPs this leg since v0.6.2 (numerator 0.198 Å vs a collapsed zero envelope), same AttentionPairBias-unfusing root cause as the other v0.6.2 GAPs, root-caused below |
| OpenDDE | 7ROA, production | PASS (legacy R/D/X); GAP-evidenced under the envelope gate | wide device-dominated floor (D 6.04 Å); absolute X 4.67 Å. The envelope test GAPs this leg since v0.6.2 (numerator 2.149 Å vs a collapsed zero envelope; still inside the 6.04 Å floor and below the committed cross term), same root cause, root-caused below |
| OpenDDE-abag | 1AHW Ab–Ag | PASS | global DockQ 0.864; per-interface iRMSD 0.65/0.70/1.20 Å, all sub-Å-to-low-Å |
| BoltzGen | binder vs 7ROA chain A | PASS | designability 93.8% (≤2 Å scRMSD) vs reference 68.75%; device meets-or-exceeds |
| SaProt-35m | ubiquitin, L76 | PASS | deterministic encoder; emb PCC 0.99914, in the ESMC band |
| SaProt-650m | ubiquitin, L76 | PASS | deterministic encoder; emb PCC 0.99964, in the ESMC band |
| RFdiffusion3 | IAI protein motif-scaffold, I40/L419 | PASS | host featurizer 43/43 f keys bit-exact vs the committed upstream foundry reference capture; card-free, in-process (scripts/rfd3_port/parity_gate.py) |
| RoseTTAFold3 | ubiquitin, L76, MSA | PASS | vendored torch CPU reference, bf16 both sides, shared draws; CA-RMSD X 0.221 Å inside the 0.418 Å reference noise floor (X/floor 0.53) and all-atom X 0.219 Å inside a 0.467 Å floor (X/floor 0.47). Four of the five seeds reproduce the reference trajectory to 0.087-0.097 Å; seed 4 alone reads 0.736 Å and it is the reference that moved, its own sample sitting 0.86-0.91 Å from its siblings while the device’s stays inside the normal device spread — a bifurcation, like boltz2-prot-nomsa at seed 0. scripts/rf3_port/accuracy_cell.py, five seeds, shipped arm (10 recycles, 50 sampling steps, one diffusion sample). The shipped arm is now the fused SDPA with the ragged key tail masked (fp32_softmax=False), which reads 0.221 Å against the materialised route’s 0.229 Å, and 0.0920 Å against 0.0955 Å on the four non-bifurcating seeds |
| RoseTTAFold3 | 7ROA, L117, MSA | PASS | same harness and convention as the row above; CA-RMSD X 0.178 Å inside the 0.382 Å noise floor (X/floor 0.47) and all-atom X 0.229 Å inside a 0.632 Å floor (X/floor 0.36). No outlier seed here: all five X land 0.128-0.254 Å, so the ubiquitin leg’s seed-4 bifurcation is a per-seed trajectory property and not a length- or target-independent one. This is the rung the shipped-arm flip moves most: the materialised route read 0.203 Å against a 0.361 Å floor, so masking the ragged tail cut the error 1.14x while the floor widened to the device’s own spread. The unmasked fused route reads 1.64 Å here, outside the floor, which is why the pad and the arm ship together and never separately |
| RoseTTAFold3 | 7EIP, L997, MSA | PASS | the 1024 aa rung's absolute cell, on a real target instead of the tiled-CDK2 chimera every other ~1000-residue artifact in this repo is built from: chondroitin sulfate ABC endolyase I, 1.88 Å X-ray, R/R-free 0.1711/0.2128, one chain, monomeric assembly, ligand-free, folded as the mature enzyme (SEQRES 25-1021). Same harness and convention as the two rows above; the reference ran on a rented H200 (--ref-device cuda — the sampler and the RNG stay on the host, so the recorded draws are the same stream the device half re-harvests). CA-RMSD X 0.641 Å inside a 2.357 Å floor (X/floor 0.27) and all-atom X 0.638 Å inside a 2.260 Å floor (0.28). Read the ratio carefully: X grew 3.6x from the 7ROA row while the floor grew 6.2x, because the floor is the model's own five-seed spread and that widens with length — the cell is not tighter here, the model is looser. The absolute reading is the one that settles it: against the deposited coordinates over the 966 modelled residues the device sits 1.890 ± 0.153 Å and the H200 reference 1.951 ± 0.174 Å, so the port is no further from the experiment than the reference it replaces. 997 mod 32 = 5, so this is also the first RF3 anchor where the shipped arm's ragged key-tail masking does any work: 968 padded calls, all at tri_att, counted in the run and not argued from the flag, on a rollout an independent re-run reproduced bit-for-bit. The rf3-1024aa leg of scripts/full_parity_gate.py re-folds seed 0 of this row every release and fails the gate if the device drifts past 4.0 Å from the crystal. |
| RoseTTAFold3 | CDK2 ladder 128/256/512/768/1024 aa, plus 5vht (real homodimer, paired MSA, atomized NCAA) | PASS | supporting evidence for the row above, not a second absolute number: scored against the vendored torch reference, ceiling-relative with the bf16-vs-fp32 ceiling measured in the same run; no trend with size. Confidence reductions (pTM / ipTM / ranking) 12/12 against upstream's own code. Bit-exact run-to-run AND cross-process at 128 / 256 / 512. The pairformer s-track defect is fixed: ttnn.softmax returns rows summing to 0.9769 rather than 1, and an RF3-scoped accurate softmax takes the s-track from 0.021263 to 0.003290 against a 0.001869 reference (11.4x -> 1.76x); opt-in, not flipped for shared sites. The 1024 aa accuracy cell is measured, on 7EIP and not on this construction — see the row above. The shipped arm is the fused SDPA with the ragged key tail masked; at 128 aa the five-seed result is bit-identical across a main merge that touched the fused-SDPA route and across two different cards, and at 512 aa the CIF digest reproduces across two processes. TT_BIO_TRIATT_FUSED_HIFI is not additive with this arm and is not used: it lives on the materialised-softmax route, which is the other side of the same switch. |
| Nesso-1 | tyr48 + tyrosine, 61 tokens | PASS | host featurizer bit-exact vs the committed upstream capture; torch model 22/22 activations bit-identical and 11/11 output scalars exact. On device the worst of the 11 scalars sits at 3.43x upstream's own 0.058 featurization-draw spread, inside the 5.0x floor, and the device is deterministic (spread 0.0 across 3 repeats). On DAVIS the within-target Pearson is 0.662 against the upstream/H200 arm's 0.636 on the same 30 compounds per target |
| PXDesign | PD-L1 quick-start target, 196 tokens, design featurizer | PASS | 25/25 arms bit-exact vs a committed capture of the upstream featurizer, over the conditioning template, the xpb binder-placeholder exclusion, the 36-way restype and the whole 18-key model input built from the target file. One arm scores the capture rather than the port, so a fixture whose conditioning coordinates are wrong fails the leg instead of agreeing with it; card-free, in-process (scripts/pxdesign_port/parity_gate.py) |
| PXDesign | LacZ-C 768aa target + 80-mer binder, 848 tokens, generated coordinates | PASS | design coordinates bit-exact across all six warm repetitions of three independent device legs, structure_sha16 cd80f8e274306706, including across the fp32-softmax L1-padded plan arm — the lever changes the shard's extent and not the arithmetic |
| AF2-IG (PXDesign filter) | LacZ-C 128aa + 80-mer binder, host featurizer | PASS | 60/60 featurizer keys bit-exact vs a captured ColabDesign forward pass, over both the complex and monomer stages; card-free, in-process (scripts/af2_port/parity_gate.py) |
| AF2-IG (PXDesign filter) | same fixture, torch trunk, complex stage (templates on) | PASS | 94/94 activation taps and all 6 filter scalars inside their bars vs the captured JAX taps; card-free, needs params_model_1_ptm.npz |
| AF2-IG (PXDesign filter) | same fixture, torch trunk, monomer stage (model_3_ptm) | PASS-caveated | 90 taps, 0 failed, 32 adjudicated inside the reference own float32-vs-bfloat16 envelope: on those taps the reference sits closer to our bfloat16 arm than our float32 arm does |
| AF2-IG (PXDesign filter) | same fixture, ttnn trunk on card, complex stage | GAP-evidenced | 9 of 94 taps and 3 of 6 scalars miss (worst envelope ratio 12.3; pLDDT 0.0028, pTM 0.0020, pAE 0.0021). A bfloat16 realisation floor in the trunk residual chain, amplified about 3x by the structure module; three trunk-precision levers were measured dead against it. The leg gates that floor instead of a PASS bar (scripts/af2_port/device_floor.py): GAP only while the failing taps, their envelope ratios and the scalar deltas all reproduce the committed record within 1.10x, and both tap_gate.py --mutate controls break every one of those conditions. Bit-identical on qb1 cards 3 and 0 |
| Net: 37 PASS, 7 PASS-caveated, 2 GAP-evidenced (boltz2-9ncy-nomsa, root-caused below; af2ig-trunk-device, a measured bfloat16 floor the leg gates as such). The three Boltz-2 affinity | |||
| legs were re-run with MSA (Boltz-2's production default — a pharma user folds a | |||
| target whose homologs are known, so the MSA is fed); the earlier single-sequence | |||
rows are retained and relabeled non-default. The MSA legs score 9 PASS / 3 GAP | |||
| across their 12 metric-cells (bf16-floor evidence): the consistent GAP is 1-pocket-lDDT on all | |||
| three targets, the same narrower-basin systematic-bf16 property the no-MSA legs | |||
| show; all three MSA affinity scalars PASS the envelope gate (FKBP12 0.32, DHFR | |||
| 1.32, trypsin 0.79). The FKBP12 MSA affinity scalar — previously recorded | |||
| GAP-evidenced at legacy X/floor 2.27 and asserted a bf16 floor by GPU-vs-CPU | |||
| triangulation on the scalar — PASSES the envelope gate (gate of record) cleanly | |||
| on the current committed fixtures; the legacy 2.27 was a stale-fixture artifact | |||
| (measured against pre-shared-draws refs before the fixture regen, never | |||
| re-measured after), and the scalar triangulation that was built on it is moot as | |||
| a verdict justification (the cross-backend bf16 divergence it measured is real | |||
| but is absorbed by the envelope gate's measured bf16 denominator; see the | |||
| affinity-scalar triangulation). | |||
| The no-MSA affinity rows use the device-fp32 hybrid diffusion path: FKBP12 PASSes | |||
| cleanly there (pocket-lDDT X 0.011 within the 0.011 GPU floor, X/floor 0.94), and | |||
| the two PASS-caveated no-MSA entries (DHFR, trypsin) are proven a genuine | |||
| bf16-BACKEND precision floor, not a port defect, by three-backend triangulation | |||
| (the pinned GPU-bf16 and CPU-bf16 references disagree on the pocket by the same | |||
| ~0.09-0.13 lDDT margin the device does, so no single-backend lever or reference | |||
| switch can manufacture a PASS). All Boltz-2 legs were re-measured with the seed-wiring fix live (2026-07-21); every verdict held (see the seed-fix remeasure note in the details doc). Protenix-v2 HSA was GAP-evidenced under bf16 | |||
| diffusion; running that model's diffusion sampler in fp32 on device — matching | |||
| the reference's own fp32 boundary rather than a blanket precision bump — closed | |||
| it to a clean PASS. The full measured R/D/X | |||
| table and per-leg evidence are in | |||
| Implementation parity — details. |
Method in one line. R = reference-vs-reference across seeds, D =
device-vs-device across seeds, X = device-vs-reference; the floor is
max(R, D); a leg passes when X is no larger than the floor within sampling
uncertainty. Deterministic legs (ESMC, SaProt) are bit-exact by construction
(R = D = 1.0). Diffusion legs (Boltz-2, Protenix-v2, OpenDDE, Boltz-2 affinity)
share one CPU torch.randn stream between device and reference at a matched
seed, so the comparison is RNG-fair; both sides run bf16 where the reference
does. BoltzGen is scored by designability (fraction of designs re-folding
within 2 Å scRMSD), not by a distance. OpenDDE-abag by global DockQ and
per-interface iRMSD.
Correctness method — integration-parity envelope (supersedes the R/D/X floor)
Nesso-1 is the one model here scored on neither a distance nor an envelope. It predicts no coordinates, so there is nothing to align and no diffusion draw to share; its leg compares the eleven output scalars against the torch reference and normalises by upstream's own featurization-draw spread, because upstream roto-translates every conformer off the global RNG and so differs from itself run to run by up to 0.058.
The R/D/X floor above answers a distribution question ("is X within the run-to-run spread?") with
a point comparison against a GUESSED floor max(R, D). Because R, D and X each compare INDEPENDENT
stochastic samples (device and reference drew different diffusion noise), X conflates real backend
arithmetic divergence with ordinary sample-to-sample chaos: a correct port can fail by construction
(different noise basin) and a subtle bug can hide under a loose floor. The "GAP-evidenced" and
"PASS-caveated" verdicts above are the floor telling us it cannot separate a bug from noise — and
each was ultimately cleared only by an ad-hoc cross-backend triangulation (GPU-bf16 vs CPU-bf16
references disagreeing by the same margin the device does). The integration-parity envelope test
turns that triangulation into the systematic pass criterion.
A diffusion model is a deterministic function of its input noise. Feed byte-identical noise to
three CLOSED-LOOP runs — device_bf16 (TT), reference_fp32 and reference_bf16 (both tt-bio's
own CPU torch path, --no_kernels, the second under TT_BIO_REF_BF16=1). The single --seed is
NOT enough to share draws: the device (ttnn) trunk and the CPU (torch) trunk consume the global RNG
differently before the sampler, so the gate sets TT_BIO_SHARED_DRAW_SEED on all three runs, which
re-seeds in AtomDiffusion.sample right before the first torch.randn so all three draw
byte-identical noise (verified bit-exact). Then, per leg per metric d:
d(device_bf16, reference_fp32) <= d(reference_bf16, reference_fp32) * (1 + margin) + abs_floor
The floor is the intrinsic bf16 cost of the full trajectory (chaotic amplification included),
MEASURED from a bf16 recomputation of the reference itself, not guessed. Scorer:
scripts/integration_envelope.py; see RELEASING.md for the full rationale and the pass criterion.
Shared draws require a sampler-entry re-seed (TT_BIO_SHARED_DRAW_SEED). The single up-front
seed is not enough: the device (ttnn) and CPU (torch) trunks consume the global RNG differently
before the sampler, so without the re-seed the device and reference draw DIFFERENT diffusion noise
(measured bit-for-bit). With it they draw byte-identical noise, so the numerator is arithmetic-only.
The numbers below are with the fix (the first valid head-to-head).
Three-leg head-to-head (no-MSA affinity, seed 0, shared draws).
| leg | affinity_pred_value ratio | ligand-RMSD ratio | 1-pocket-lDDT | verdict |
|---|---|---|---|---|
| trypsin | 2.81 | 0.11 | 0.00 (bit-identical) | GAP |
| DHFR | 1.44 | 0.19 | 0.00 (bit-identical) | PASS |
| FKBP12 | 1.35 | 0.32 | 0.00 (bit-identical) | PASS |
Structure/pose parity is excellent on all three legs: the device structure is bit-identical to fp32
in pocket-lDDT and ligand pose is well inside the bf16 envelope. FKBP12 and DHFR pass on the
affinity scalar too. Trypsin does not, at ratio 2.81, since its CPU references were regenerated in
v0.6.5 on the conformer-seeding fix (the row above, and the bar move below). The earlier table
recorded FKBP12 at ratio 1.90 / GAP, DHFR at 1.22 and trypsin at 0.96; those numbers were measured
against pre-shared-draws reference fixtures (commit c0529ca79) and never re-measured after the
fixtures were regenerated with TT_BIO_SHARED_DRAW_SEED=0 ~37 min later (commit fb3bd0075). The
trypsin 0.74 recorded here was itself stale by one fixture generation: against the reference this
doc shipped with (b48876d9d) the leg read 1.74, marginally over its bound, and the row was never
refreshed. FKBP12's and DHFR's stale-doc GAPs were real stale-doc artifacts and stay
closed. The standing "on-device fp32, not host fallback" directive
for the affinity pairformer+heads remains a separate portability refactor (the head already runs
fp32 on host, BOLTZ2_AFFINITY_FP32_HOST=1 default ON); it is not an accuracy fix and is not needed
for the scalar to pass.
The trypsin affinity bar moved in v0.6.5
The unseeded RDKit ETKDG conformer draw fixed in v0.6.5 (1190a1daa) also fed the CPU reference
path, so this leg's committed ref_fp32/ref_bf16 pair carried conformer noise on top of the bf16
difference the pair exists to measure. Regenerating both references on the fixed code (1e1dc7c4,
--accelerator cpu --no_kernels) shrank the envelope by more than half and left the
device-vs-reference residual with nothing to hide behind.
old (b48876d9d) | new (1e1dc7c4) | |
|---|---|---|
reference fp32 affinity_pred_value | 2.628256 | 2.605899 |
| envelope (bf16 vs fp32 reference) | 0.043619 | 0.019101 |
| bound (envelope x 1.5 + 0.01 floor) | 0.075429 | 0.038652 |
| numerator (device vs fp32 reference) | 0.076088 | 0.053731 |
| ratio | 1.744 | 2.813 |
| verdict | GAP, 0.9% over bound | GAP |
The device value did not move: 2.552168 on three independent folds, bit-identical. So the leg was
already GAP before the regen, marginally; the regen made it unambiguous and worth chasing. The bound
was deliberately not widened to keep the leg green. The tight envelope is the correct bar, the leg is
recorded GAP in docs/implementation-parity-data/boltz2-affinity-tryp-nomsa-envelope.json, and
full_parity_gate.py fails on it until the ~2% is explained. Pose is unaffected: ligand-RMSD 0.0174
against a 0.2901 bound, pocket-lDDT bit-identical. The other five affinity references still carry
pre-fix conformer noise and are regenerated separately.
Wired into the gate of record. The envelope test is the default correctness criterion for
every diffusion (structure/affinity) leg in scripts/full_parity_gate.py: the gate folds the
device once at the reference seed, reads the leg's cached ref_fp32 + ref_bf16 CPU references,
and scores with integration_envelope.py through the one finalize_leg verdict path. The two CPU
references are the cached fixture (--regen-refs generates them, fingerprinted like the old ones,
so only the device fold + scoring re-run per release); a leg without them reports
BLOCKED-REF-REGEN-NEEDED rather than a false pass (and a run where EVERY leg is blocked
this way prints GATE INCONCLUSIVE and exits nonzero — nothing scored means nothing verified).
The retired R/D/X floor is still available as
an opt-in device self-consistency (D) diagnostic via --legacy-rdx.
Full matrix complete as of 2026-07-24. Every envelope leg's ref_fp32/ref_bf16 CPU
reference is now regenerated (the last 9: boltz2-hsa-nomsa, all 3 protenix-v2-*-msa
structure legs, all 3 boltz2-affinity-*-msa legs, and both opendde-*-nomsa legs).
Correction (issue #10): the 5 Protenix-v2/OpenDDE legs in that regen list never got CPU
references. --accelerator cpu was silently ignored for ttnn-only models, so their
ref_fp32/ref_bf16 pairs are device folds (autocast off/on) and every envelope verdict
measured against them is an instrument artifact, not a floor measurement. Those legs are
legacy_rdx now (the 2026-08-11 paragraph below) and their fixture meta.json no longer
claims a tt-bio-cpu-torch reference. The Boltz-2 legs in the same list are unaffected:
Boltz-2 has a real torch CPU path, so its regenerated references are genuine.
A full non-dry full_parity_gate.py run against every one of the 21 wired
legs gives:
Tally: 20 PASS, 1 GAP (esmc/saprot/esmfold2/boltzgen/opendde-abag all reproduce committed)
The lone GAP is boltz2-prot-nomsa (7ROA, no-MSA structure leg): envelope worst kabsch_rmsd
ratio 1.83 (exceeds the 1.5× bound), reproduced bit-for-bit on a second --fresh re-fold (not
noise or a flaky measurement). This DRIFTS from the leg's legacy R/D/X-methodology verdict
(PASS) — plausibly the same seed-0-chaos amplification root-caused below for this leg, not a
precision bug. Root-caused 2026-07-27 — see below; gate metric intentionally not loosened to hide
it. The boltz2-affinity-fkbp12-msa leg, separately, was recorded GAP-evidenced in the legacy
R/D/X verdict table (scalar X/floor 2.27); the envelope test (gate of record) PASSES its scalar
cleanly (ratio 0.32) against the current committed fixtures. That is not a drift or a
contradiction: the legacy 2.27 was a stale-fixture artifact, measured against pre-shared-draws
refs before the fixture regen and never re-measured after, so the envelope pass is the live verdict
and the legacy GAP-evidenced row is corrected above and in the
affinity-scalar triangulation.
Prior state: a 2026-07-23/24 pass first wired the automated gate end-to-end but had only 9 of the
21 legs' CPU references generated (9 PASS, 12 BLOCKED-REF-REGEN-NEEDED). That pass closed every
remaining reference gap, but this claim went stale two days later without being corrected here:
regen_envelope_refs wholesale-overwrote each fixture's root meta.json, which for 5 legs
(opendde-{trpcage,prot}, protenix-v2-{prot,ubq,hsa}-msa) destroyed the harvested "official
Aureka-OpenDDE/ByteDance-Protenix" provenance + settings_tag the separate legacy R/D/X scorer
needs — so a 2026-07-26 fix restored that provenance, which silently put those 5 legs back to
BLOCKED-REF-REGEN-NEEDED under this (envelope) gate. Root-caused and fixed 2026-07-27:
regen_envelope_refs now nests its own cache-key bookkeeping under meta["envelope"] instead of
replacing the file, so a fixture can carry both an envelope reference and legacy R/D/X provenance
at once — regenerating one can no longer break the other. All 8 previously-BLOCKED legs
(boltz2-{trpcage,prot,hsa}-nomsa, protenix-v2-{prot,ubq,hsa}-msa, opendde-{trpcage,prot})
were regenerated and re-verified against this fix:
Tally (8 re-verified legs): 7 PASS, 1 GAP — boltz2-prot-nomsa, envelope worst kabsch_rmsd
ratio 2.04 (same pre-existing open item above, not a new regression)
The legacy R/D/X path (--legacy-rdx) was independently re-verified for the 5 "ttnn-only" legs
whose provenance the fix preserves: opendde-prot-prod PASS (X=4.824 R=1.499 D=6.319, matching
the 2026-07-26 verified number X=4.678 R=1.499 D=6.103 within noise), protenix-v2-{prot,ubq,hsa}
all PASS (within_noise_floor=true on every metric); opendde-trpcage-nomsa reports
BLOCKED-REF-REGEN-NEEDED under --legacy-rdx for an unrelated, pre-existing reason (its committed
seed dirs lack structures/*.cif — not touched by this pass). Gate of record —
pending Moritz's sign-off before merge.
opendde-trpcage-nomsa closed 2026-08-24: PASS, X=0.361 R=0.374 D=0.342, within_noise_floor
true. The reference was never missing. All five seeds of the official Aureka OpenDDE CPU run
(commit a0d5134) were committed with full provenance; only the CIFs were absent, because
pharma_harvest_ref_fixtures.py wrote them under the reference's own structure id (trpcage)
while the gate leg asks for the tt-bio target id (trpcage_no_msa). SeedSpec now carries a
dst_id that renames the CIF and the results.json id and nothing else. The recovered reference
reproduces the committed reference-vs-reference floor exactly (R 0.374 against the committed
0.3736), which is what confirms these are the runs the provenance describes.
Three envelope legs are collapsed, found during the v0.5.0 release run (2026-07-27). For
opendde-trpcage-nomsa, opendde-prot-prod and protenix-v2-ubq-msa the published fixture's
ref_fp32 and ref_bf16 structures are BYTE-IDENTICAL, so the envelope denominator
d(ref_bf16, ref_fp32) is exactly 0 and the leg can only pass on the abs_floor. For the two
OpenDDE legs the device fold is byte-identical to both references as well (numerator = envelope =
~4e-15, and the freshly-folded device CIF has the same md5 as the cached reference), so those two
legs currently compare a fold against itself and measure nothing. protenix-v2-ubq-msa still
discriminates (numerator 0.0374 Å, nonzero) but only against the 0.05 Å floor, not a measured
envelope. Cause: the envelope test needs a torch CPU path that can be recomputed in fp32 and in
bf16, and these are ttnn-only ports — the same "envelope methodology is torch-capable-only"
constraint recorded above, which --regen-refs does not detect, so it emits a device fold labelled
as a CPU reference. This is NOT a coverage hole: OpenDDE and Protenix-v2 are scored by the legacy
R/D/X device-vs-reference diagnostic against their official upstream implementations, which is the
correct scorer for a ttnn-only port and which passes (see the paragraph above).
Fixed 2026-08-11, and the cause was one line. tt_bio/main.py built the worker slots for every
non-Boltz-2 model with a hardcoded "tenstorrent", so --accelerator cpu was accepted and then
folded on the card: a reference fold was caught holding /dev/tenstorrent/0 open. Four changes.
predict now refuses --accelerator cpu/gpu for a model with no torch path instead of ignoring it.
The three Protenix-v2 and two OpenDDE structure legs carry legacy_rdx, like the OpenFold3 legs and
for the same reason, so --regen-refs skips them and they score R/D/X against their upstream
reference. --regen-refs refuses to write a reference pair whose two arms are byte-identical. And a
report whose every metric has a zero envelope reads NO-DATA, never PASS. Boltz-2 is unaffected:
it has a real torch path, its references differ as they should (fp32 0.911573 against bf16 0.871069
on hsa-nomsa), and its legs are still envelope-scored.
boltz2-prot-nomsa GAP root-caused 2026-07-27 (GAP-evidenced, not fixed): seed-0 chaos, not a
precision bug. Two on-device fp32 levers were tried and neither closes it: the reference's own
selective-fp32-softmax boundary (BOLTZ2_FP32_SOFTMAX=1, ratio 2.04→2.00) and a new fp32-storage/
native-bf16-SDPA hybrid diffusion lever ported from the affinity path to plain structure prediction
(BOLTZ2_STRUCTURE_DIFFUSION_FP32_DEVICE=1, ratio 2.04→2.07), stacked or alone. The real cause:
re-running the identical closed-loop triple (device_bf16, ref_fp32, ref_bf16) at two more seeds
shows the envelope itself — d(ref_bf16, ref_fp32), the reference's own bf16-recompute drift from
its fp32 self — swings from 3.45 Å at seed 0 to 2.16 Å at seed 1 to 0.81 Å at seed 2, and the leg
PASSes cleanly at both (ratio 0.98, 0.48). ENVELOPE_SEED=0 (fixed for every leg, by gate design)
happens to land this one no-MSA target in an unusually chaotic reverse-diffusion trajectory where a
bf16 perturbation early in sampling amplifies into a large final-structure divergence — the same
"wide no-MSA floor" property the legacy R/D/X row above already named (R 4.98 Å), now visible even
in the tighter single-seed envelope test because that pinned seed happens to be this target's worst
draw. Not a port defect and not a precision-boundary mismatch to match — closing it would require
either accepting per-leg seed selection (rejected: gaming the pinned seed per-leg is indistinguishable
from cherry-picking and undermines the gate's uniformity) or a chaos-damping change to the model
itself (out of scope for a parity gate). Verdict stays GAP-evidenced, margin not loosened. Both
new gates are retained as documented negative infrastructure, default OFF (same policy as
BOLTZ2_FP32_SOFTMAX) — harmless, reversible, and available for the next leg that might need them.
boltz2-9ncy-nomsa GAP root-caused 2026-08-16 (GAP-evidenced): the numerator is stack-sensitive on this target, no tt-bio code regressed. The committed 0.760 Å PASS was scored on pc (p150a, ttnn 0.68.0) at fixture birth (2026-08-11, main 7b9d0b427). Every scoring on qb1 (p150a, ttnn 0.67.4) gives 6.45-6.71 Å instead, and an endpoint A/B over the full 676-commit window 4a89d38d..8c878dd0 — fixture commit itself, midpoint probe, and tip, on both qb1 cards — is byte-inert: no commit in the window moves the qb1 number at all, so git bisect over tt-bio code is moot. The drift entered the gate record when scoring moved hosts, not when any lever merged. Adjudication against experimental ground truth (examples/ground_truth_structures/9ncy.cif): device fold 6.49 Å (TM 0.935, lDDT 0.791), CPU bf16 reference 6.43 Å (TM 0.924), CPU fp32 reference 6.55 Å (TM 0.931) — three different, equally accurate basins of a 505-token no-MSA antibody-antigen complex with a chaotic diffusion landscape, and the device fold has the best lDDT of the three. On this target the envelope numerator measures basin identity, which flips with the software stack, not correctness: same accepted class as the boltz2-prot-nomsa seed-0 chaos above. The qb1 numerator is not even stable on one host (6.4474 vs 6.71027 Å measured 3.5 h apart at one fixed commit, no package or cache change), so the exact value carries no gate signal here. Verdict stays GAP-evidenced, margin not loosened, reference side untouched.
protenix-v2-prot-msa on Wormhole root-caused 2026-07-31: the structure metric was scoring a
confidence-rank coin flip, and the metric is now rank-invariant. The leg read 2.403 Å against a
0.042 Å envelope (ratio 56.7) on a Wormhole Galaxy while passing on Blackhole. Cause: tt-bio names
structure output by confidence rank, so <tid>.cif is whichever of the five diffusion samples has
the highest pTM, and on 7ROA the top two pTM values are tied to within 3.1e-4 (1.3e-4 on Blackhole,
3.0e-6 on Wormhole) — far below the resolution of the arithmetic being tested. The scorer compared
rank 0 to rank 0, so a reordering read as a 2.4 Å structural error. Measured with the full 5×5
CA-Kabsch matrix between every pair of samples: the device reproduces the reference's five samples
at 0.076–0.63 Å, and its counterpart of the reference's top structure is 0.139 Å away, not
2.403 Å. The reordering is not a device property at all — two tt-bio CPU fp32 folds of the same
code, seed and MSA on different hosts also come out reordered, with the swapped pair bitwise
identical (prot_model_2 of one equals prot_model_1 of the other, sha256).
integration_envelope.py now anchors on the reference's top structure and compares it to its
counterpart in the run under test, requiring the match to be at least 3× closer than the next
candidate (a diffusion ensemble's samples sit 1–3 Å apart, a reproduced trajectory lands ~0.1 Å
away, so a true counterpart wins by 10×+; below the threshold it falls back to the strict rank-0
compare). Single-sample legs are unaffected bit-for-bit. Across the protenix-v2 legs it changes
exactly one number — prot 2.4033 → 0.1399 — leaving hsa at 0.0506 (PASS) and ubq at 0.1008
(GAP, the collapsed-envelope leg above). Every report now carries a sample_match block naming
the matched rank and flagging a flip.
Rank-corrected, the leg reads numerator 0.139 Å against a 0.042 Å envelope (ratio 3.28, bound 0.114) — still GAP, by 22%, and the Blackhole control on the identical fixture and seed reads 0.082 Å (PASS). Both sit inside the leg's honest noise: refolding the fp32 reference itself with the same code and seed on a different host moves the top structure by 0.044 Å, i.e. a same-dtype cross-host recompute already spends the whole measured bf16 envelope, and the device side is compared against a fixture generated on a different machine. So the residual is a bf16 + cross-host floor at ~0.1 Å on a 117-residue target, not a Wormhole port defect. It has not been driven under the bound; no fp32-boundary lever has been tried for it yet.
protenix-v2-ubq-msa GAP root-caused 2026-08-06 (GAP-evidenced, not fixed): the chunked MSA path, not a numerics regression. The v0.6.1 gate run read numerator 2.015 Å against this leg's collapsed zero envelope (abs_floor 0.05), a 20x jump from its v0.6.0 reading of 0.011 Å. The cause is the abag-xm MSA row-chunking that landed in this range (tt_bio/protenix.py, MSA_ROW_CHUNK_BUDGET_BYTES = 0.25 GiB): ubiquitin's MSA is 20826 sequences deep, so its m_feat (0.377 GiB) is the only protenix gate leg that crosses the budget and takes the chunked trunk path, which is measured non-bit-exact by design (whole-vs-chunked mean 0.738 Å / max 3.98 Å on the same chip and seed, commit c49dd6a4; the changed depth extent re-plans each matmul's K-blocking, changing the bf16 summation order over tokens). prot (166 sequences) and hsa (0.139 GiB) stay on the whole path and reproduce at 0.045 / 0.025 Å. The device ensemble is the reference ensemble: 3 of 5 device samples match reference samples at 0.093 / 0.430 / 0.453 Å, all five sit inside the reference's own inter-sample spread (0.856–2.851 Å), and the rank-invariant matcher finds the reference top structure's counterpart at 1.189 Å (unsafe at 1.69x, so the report falls back to the strict rank-0 compare). 2.015 Å is inside this target's committed noise floor (ref floor max 2.993 Å). Verdict stays GAP-evidenced, margin not loosened; the accuracy content of the port is unchanged.
Measurement bounds and non-gated variants
Two facts a skeptical reader should have that do not appear as a verdict row above. Both are recorded here because this doc, not the public JapanFold accuracy page, is where the full detail belongs.
BoltzGen designability carries a sampling bound. The leg is n=16 per side
(two batches of 8, docs/implementation-parity-data/boltzgen.json), and the
reference's own two batches are 12.5 points apart on the ≤2 Å bar (batch_a 75%,
batch_b 62.5%). So part of the 93.75%-vs-68.75% margin is sampling noise, not
port quality. What the leg establishes is the direction, and the direction holds
across both batch pairings: device median scRMSD 0.78 Å vs reference 1.05 Å, and
device pass-rate on the favorable side of the reference's spread in every
pairing. BoltzGen designs new sequences, so there is no 1:1 correspondence to
score — parity is in the designability distribution, not a per-design RMSD.
SaProt-1.3B is served but is not a clean PASS. On ubiquitin (L76) it measures
per-residue embedding PCC 0.995076 and MLM-logits PCC 0.998952 (R = D = 1.00000,
deterministic), which lands just below the 0.9987–0.9996 band the 35M and 650M
variants hit, so docs/saprot-parity.md records it as a near-pass and claims no
PASS row. The residual tracks depth rather than a port defect: 1.3B is the 650M
width at twice the layers (66 vs 33), so it accumulates about twice the bf16
rounding. It has no leg in full_parity_gate.py and is therefore absent from
the tally above; the 650M leg is the gated SaProt path.
Reproduce
Each leg's reproduce command is in Implementation parity — details.
The one-command runner for the full story is scripts/full_parity_gate.py (fans
the device side across cards, reuses the committed reference fixtures, and
emits the verdict table + tally); the per-leg scorers it dispatches to are
scripts/pharma_parity.py (structures / embeddings / saprot) and
scripts/boltz2_affinity_parity.py (affinity). Reference fixtures live under
docs/implementation-parity-data/ref-fixtures/.
protenix-prot-msa, opendde-prot-prod, opendde-trpcage-nomsa GAPs root-caused 2026-08-07 (GAP-evidenced, not fixed): the intended AttentionPairBias unfusing, not a numerics regression. The v0.6.2 gate run read numerators 1.168 / 2.149 / 0.198 Å against collapsed envelopes (0.0424 / ~0 / ~0; these legs discriminate only against abs_floor 0.05). The cause is ba6ede96, which replaced the fused ttnn SDPA with an unfused matmul + softmax + matmul in the shared token-level AttentionPairBias because the fused kernel measurably flattens near-degenerate attention distributions (per-call PCC 0.98128 fused vs 0.99993 unfused on identical tensors; the same fix moved OF3 7XI5 from 8.61 Å to 0.54 Å). Every model on the shared path shifted by a small bf16-scale perturbation that the diffusion rollout amplifies, and only collapsed-envelope legs trip: every real-envelope leg reproduces committed on the same tree, and the already GAP-evidenced protenix-ubq-msa improved 2.015 → 1.111 Å. OpenDDE's shapes cross none of the in-range chunk gates (pair channel 384 is outside the transition big-chunk envelope, gate sequence lengths are below the tri-attention chunk band), so its 0.0 → 0.198 / 2.149 Å movement isolates the unfusing as the cause. Same ensemble, not wrong structures: the protenix-prot 5×5 CA-Kabsch matrix puts every device sample's nearest reference at 0.960–2.029 Å against the reference's own inter-seed spread of 1.388–2.922 Å, and the opendde-prot device sample sits 0.857–2.443 Å from the five committed references (spread 0.389–2.131 Å). All three numerators sit inside their committed noise floors (protenix-prot R 2.94 Å; opendde-prot floor 6.04 Å, whose committed cross term was itself 4.67 Å; opendde-trpcage ref floor max 0.666 Å). Verdicts stay GAP-evidenced, margins not loosened; the accuracy content of the ports is unchanged.
openfold3-7xi5-notmpl GAP root-caused 2026-08-07 (GAP-evidenced, not fixed): the fp32 diffusion boundary collapsed the floor prop, not the fold. This leg's committed record (added at f8af5943, bf16 diffusion module) passed with X 4.64 Å against floor max(R, D) = 3.76 Å, where D was the device's own pre-P15 seed noise. The tree now runs the on-device fp32 diffusion boundary (c1a55cd9, default-on, the reference's own precision recipe), and the device converges tightly on this target (D 3.76 → 0.70 Å), so the floor is now the reference spread alone (R 2.87 Å) and the same-band X (5.40 Å) reads over it. All-atom X on 7XI5 is dominated by flexible-tail/side-chain placement, not the fold: measured directly against the experimental structure (RCSB 7XI5, sequence-aligned CA Kabsch over 57 resolved positions), all five v0.6.2 device seeds land at 0.589–0.609 Å while the five committed CPU reference seeds themselves sprawl 0.422–0.895 Å. The templates-ON sibling leg on the same target passes on the same tree (X 4.42 Å inside R 4.16 Å). Not a regression; verdict recorded GAP-evidenced with the evidence in openfold3-7xi5-notmpl.json.
Where the reference fixtures come from
The verdict numbers a reader checks are the small committed JSONs at the top of
docs/implementation-parity-data/ (the score/verdict files, ~160 KB total) plus
the per-fixture meta.json/results.json provenance under ref-fixtures/. The
large binary fixtures — the reference CIF structures and A3M MSAs that back each
diffusion leg — are externalized to GitHub Release assets to keep the repo
small, and are no longer committed going forward (see .gitignore).
A fresh checkout reproduces a leg end-to-end by restoring the binaries:
scripts/fetch_parity_fixtures.sh # default tag = parity-fixtures-latest
# or a pinned pass: scripts/fetch_parity_fixtures.sh --tag parity-fixtures-2026-07
The fixtures are harvested from real reference runs (multi-hour GPU/CPU legs via
scripts/pharma_harvest_ref_fixtures.py); they are not regenerable on demand,
which is why they are versioned as release assets rather than rebuilt. The
fixtures already committed in the repo today (the 33 MB present at the time of
this change) stay tracked — no history rewrite was performed; only future binary
additions are externalized.