MinT-2M Limitations
July 24, 2026 ยท View on GitHub
The public release state is review_only_not_runnable. CPU contracts, the 2M
payload constructor, direct-Ray planning, ownership-scoped cleanup, and
fail-closed release tooling are reviewable. They do not prove that this tree can
perform a 32-GPU training update from a fresh clone. External artifacts now
prove one exact-2M 32-GPU online update, but those artifacts are not source-bound
to this public tree. This distribution limitation is separate from the
method-level ready_for_2m_training decision: the validated private environment
can begin 2M training.
Direct backend and licensing
The candidate repository-local Megatron rank port, resident runner, backend,
and direct-Ray actor launcher are checked in, and candidate execution is enabled
with IMPLEMENTATION_COMPLETE=True. The direct CLI still exits before
ray.init until run-scope doctor accepts the immutable runtime,
execution/redistribution grant, model, fixture, and deployment. Separately, no
source-bound current-tree 32-rank GPU promotion chain exists in the public
checkout, so the public validation gate remains blocked. The sanitized
exact-2M online and 32K gradient receipts are external summaries. Private HTTP,
scheduler, and session code is not a supported runtime dependency.
The repository-local CPU fake backend validates lifecycle ordering,
transactional page append, bounded GradientProbe slices, token autograd
events, and finalize/optimizer receipt consistency. Its forward_nonce is a
backend-attested uniqueness value within one session, not independent proof
that an arbitrary port executed a fresh GLM forward. The port contract forbids
raw .data/storage/pointer writes to captured prefix pages and model parameters;
the bounded value/version witness cannot detect a transient raw write that is
restored before a boundary. Real forward provenance and rank-local probe
availability remain part of the source-bound public acceptance chain. External
numerical comparison is now available at 32K, while a 64K repeat was not run.
The optional optimizer probe manifest is also backend-attested: the CPU fake omits it and
reports optimizer_ownership_attested=false. The live path derives it from the
optimizer parameter groups; even then it is a bounded wiring witness, not
full trainable-shard or cross-rank parity evidence. The live port must also
recompute audit_policy_logprobs in FP32 from the same returned logits using
the adapter's canonical log_softmax(...).gather(...) formula. Quantized or
fused-kernel logprobs are parity evidence, not this exact consistency field.
The direct Ray plan is also fail-closed on runtime identity. A live plan must
bind the deployment's immutable runtime_image and its derived
mint2m_image_<64hex> image_resource; the top-level node_ips and
gpus_per_node values are checked against target Ray records before the
placement group is created. Coordinator and rank bundles bind both the target
node resource and digest resource. A review plan with runtime_image=None is
never executed, and the driver rejects it before the ownership journal or Ray
connection. Ray workers must already run the pinned image; no per-actor image
override is supplied.
If the driver process exits after placement-group creation starts but before
Ray returns an ID that can be journaled, a new process cannot safely distinguish
that group from an unknown same-name group. Recovery proves absence when the
name is gone, but it reports cleanup_failed and requires operator inspection
while the group still exists; it never deletes an ID-less group by name alone.
CUDA 13 runtime
The runtime lock is structurally validated but incomplete. A publishable CUDA 13 base digest, external immutable artifact origins and hashes, a reproducible Apex wheel/build proof, OCI digest, SBOM/provenance, and consensus that all four target Ray workers run that image and advertise its digest resource are still required. The current build command fails before any download while those fields are unresolved.
Inputs and distributed validation
Approved, hash-bound 32K and 64K parity fixtures are not checked in. No current
source tree has passed the required doctor, single-GPU kernel, 32-GPU NCCL,
direct actor materialization, 32K parity, 64K parity, and fresh 2M G2 sequence
on 32 H20 GPUs. Process exit zero or a self-declared passed field is not
acceptance; rank identities, numerical metrics, artifacts, optimizer state, and
cleanup must all be independently bound.
Outside that public promotion chain, one exact-2M external run did complete a
real policy rollout and finalized optimizer update, and the associated 32K
gradient comparison passed. Their content hashes and
limitations are recorded in evidence/online_grpo_2m.json and
evidence/gradient_parity_32k.json. Both measured parity cases passed; 64K
gradient parity was not run. The online receipt also records the real
multi-node Tinker sampling backend and a bounded 16-sample decode audit. That audit rules
out deterministic encoding corruption at exact 2M, but occasional degeneration
in an untrained extension remains a quality target rather than a runtime
failure.
Validation and training intentionally use different fixture contracts. The
validation deployment accepts mint2m.fixture-set/v1; the training deployment
requires mint2m.training-fixture/v1, its adjacent content-addressed payload,
and a real model_manifest_sha256 for the mounted HF snapshot. A successful
candidate run additionally produces rank-sessions.json and
checkpoint-manifest.json; the receipt validator independently checks all 32
phase transitions, planned/observed Ray placement, cleanup, and the live
checkpoint tree. None of these CPU contracts substitutes for 32x H20 evidence.
Historical and external evidence
The retained deterministic 2M execution receipt and 32K forward canary came
from predecessor source/runtime states. Newer external artifacts contain an
actual exact-2M online update and matched 32K gradients with 0.9999336 cosine
and 0.0117/0.0116 relative L2. Both cases pass the project gradient-parity
acceptance contract. None of these source-unbound summaries can promote the
current release.
Repository boundary
This repository contains no model weights, datasets, approved rollout payloads, raw traces, credentials, external registry image, or GPU allocation. Sampling and reward production remain outside the training actor; the checked-in online coordinator only validates and transfers their bounded outputs. The deployment owner must provide those inputs and a 32x H20 validation window.