MinT-2M Limitations

July 24, 2026 ยท View on GitHub

The public release state is review_only_not_runnable. CPU contracts, the 2M payload constructor, direct-Ray planning, ownership-scoped cleanup, and fail-closed release tooling are reviewable. They do not prove that this tree can perform a 32-GPU training update from a fresh clone. External artifacts now prove one exact-2M 32-GPU online update, but those artifacts are not source-bound to this public tree. This distribution limitation is separate from the method-level ready_for_2m_training decision: the validated private environment can begin 2M training.

Direct backend and licensing

The candidate repository-local Megatron rank port, resident runner, backend, and direct-Ray actor launcher are checked in, and candidate execution is enabled with IMPLEMENTATION_COMPLETE=True. The direct CLI still exits before ray.init until run-scope doctor accepts the immutable runtime, execution/redistribution grant, model, fixture, and deployment. Separately, no source-bound current-tree 32-rank GPU promotion chain exists in the public checkout, so the public validation gate remains blocked. The sanitized exact-2M online and 32K gradient receipts are external summaries. Private HTTP, scheduler, and session code is not a supported runtime dependency.

The repository-local CPU fake backend validates lifecycle ordering, transactional page append, bounded GradientProbe slices, token autograd events, and finalize/optimizer receipt consistency. Its forward_nonce is a backend-attested uniqueness value within one session, not independent proof that an arbitrary port executed a fresh GLM forward. The port contract forbids raw .data/storage/pointer writes to captured prefix pages and model parameters; the bounded value/version witness cannot detect a transient raw write that is restored before a boundary. Real forward provenance and rank-local probe availability remain part of the source-bound public acceptance chain. External numerical comparison is now available at 32K, while a 64K repeat was not run. The optional optimizer probe manifest is also backend-attested: the CPU fake omits it and reports optimizer_ownership_attested=false. The live path derives it from the optimizer parameter groups; even then it is a bounded wiring witness, not full trainable-shard or cross-rank parity evidence. The live port must also recompute audit_policy_logprobs in FP32 from the same returned logits using the adapter's canonical log_softmax(...).gather(...) formula. Quantized or fused-kernel logprobs are parity evidence, not this exact consistency field.

The direct Ray plan is also fail-closed on runtime identity. A live plan must bind the deployment's immutable runtime_image and its derived mint2m_image_<64hex> image_resource; the top-level node_ips and gpus_per_node values are checked against target Ray records before the placement group is created. Coordinator and rank bundles bind both the target node resource and digest resource. A review plan with runtime_image=None is never executed, and the driver rejects it before the ownership journal or Ray connection. Ray workers must already run the pinned image; no per-actor image override is supplied.

If the driver process exits after placement-group creation starts but before Ray returns an ID that can be journaled, a new process cannot safely distinguish that group from an unknown same-name group. Recovery proves absence when the name is gone, but it reports cleanup_failed and requires operator inspection while the group still exists; it never deletes an ID-less group by name alone.

CUDA 13 runtime

The runtime lock is structurally validated but incomplete. A publishable CUDA 13 base digest, external immutable artifact origins and hashes, a reproducible Apex wheel/build proof, OCI digest, SBOM/provenance, and consensus that all four target Ray workers run that image and advertise its digest resource are still required. The current build command fails before any download while those fields are unresolved.

Inputs and distributed validation

Approved, hash-bound 32K and 64K parity fixtures are not checked in. No current source tree has passed the required doctor, single-GPU kernel, 32-GPU NCCL, direct actor materialization, 32K parity, 64K parity, and fresh 2M G2 sequence on 32 H20 GPUs. Process exit zero or a self-declared passed field is not acceptance; rank identities, numerical metrics, artifacts, optimizer state, and cleanup must all be independently bound.

Outside that public promotion chain, one exact-2M external run did complete a real policy rollout and finalized optimizer update, and the associated 32K gradient comparison passed. Their content hashes and limitations are recorded in evidence/online_grpo_2m.json and evidence/gradient_parity_32k.json. Both measured parity cases passed; 64K gradient parity was not run. The online receipt also records the real multi-node Tinker sampling backend and a bounded 16-sample decode audit. That audit rules out deterministic encoding corruption at exact 2M, but occasional degeneration in an untrained extension remains a quality target rather than a runtime failure.

Validation and training intentionally use different fixture contracts. The validation deployment accepts mint2m.fixture-set/v1; the training deployment requires mint2m.training-fixture/v1, its adjacent content-addressed payload, and a real model_manifest_sha256 for the mounted HF snapshot. A successful candidate run additionally produces rank-sessions.json and checkpoint-manifest.json; the receipt validator independently checks all 32 phase transitions, planned/observed Ray placement, cleanup, and the live checkpoint tree. None of these CPU contracts substitutes for 32x H20 evidence.

Historical and external evidence

The retained deterministic 2M execution receipt and 32K forward canary came from predecessor source/runtime states. Newer external artifacts contain an actual exact-2M online update and matched 32K gradients with 0.9999336 cosine and 0.0117/0.0116 relative L2. Both cases pass the project gradient-parity acceptance contract. None of these source-unbound summaries can promote the current release.

Repository boundary

This repository contains no model weights, datasets, approved rollout payloads, raw traces, credentials, external registry image, or GPU allocation. Sampling and reward production remain outside the training actor; the checked-in online coordinator only validates and transfers their bounded outputs. The deployment owner must provide those inputs and a 32x H20 validation window.