MinT-2M: 2M-Context Response-Only GRPO

July 17, 2026 ยท View on GitHub

MinT-2M is MindLab's long-context training system for capturing a 2M-token prompt once, keeping compact MLA/DSA state resident, and replaying only the trainable response for each GRPO group member.

The intended release path attaches directly to an operator-managed Ray cluster and does not start an HTTP control plane. The checked-in release state is review_only_not_runnable: the repository contains the candidate megatron_backend.py/megatron_rank_port.py path and direct-Ray lifecycle code, and candidate execution is enabled with IMPLEMENTATION_COMPLETE=True. Separately, the immutable CUDA 13 image, execution/redistribution grant, approved fixtures, and current-tree 32x H20 evidence are still missing, so the public release cannot be promoted to runnable.

Historical scale context

TrackConfigurationObserved result
Standard full sequence32K, 32x H20, TP4/CP8/EP32passed; model create 115s, forward/backward 35s, optimizer 5s
Standard full sequence2,097,152 prompt tokens, same hardware/topologyOOM at 139.74GiB process memory with 61.94MiB free before the next 2.54GiB allocation
Resident-prefix scale run2,097,152 prompt tokens, 32x H20, TP1/CP32/EP32, 78 layers, LoRA rank 8, G2prefix capture, two policy backwards, one terminal optimizer invocation on every rank
Global-CP forward gate32K, CP32selected-logprob max abs 9.84669e-5; rank consensus 0

These retained results describe earlier private-runtime runs; they are not current-tree runnable evidence. The resident path reached 64x the verified full-sequence baseline length. The recorded run used three response tokens per group member, scored two tokens, kept prefix pages on CPU, returned 32 rank reports, and emitted 128 layer/rank traces. Its peak recorded device allocation was 145.15GB.

The fresh 2M actor used the checked-in optimizer contract: Adam, LR 1e-4, minimum LR 0, betas 0.9/0.999, epsilon 1e-8, weight decay 0.01, decoupled weight decay, clip grad 1, distributed optimizer, BF16, and a constant schedule. The live runner applies one optimizer step after the two G2 backwards; it does not inherit an ambient learning rate.

Core contributions

  1. Resident MLA/DSA prefix state. Prompt state is captured once in paged rank-local storage with explicit ownership, transactional append/pop, and a metadata/storage/version boundary witness. Byte-content correctness is established by parity gates, not claimed by the hot-path witness.
  2. Global context-parallel sparse composition. Rank-local DSA candidates are gathered, merged deterministically across CP32, and materialized back into the replay layout.
  3. Serial response-only GRPO. G responses reuse one prefix, freeze old/reference scores before policy autograd, accumulate LoRA gradients, finalize distributed gradients, and apply exactly one optimizer step.
  4. 2M memory profile. Chunked LM head and cross entropy, activation input offload, DSA holder release, MoE replay checkpointing, and bounded expert kernels make the working set explicit.
  5. Distributed parity instrumentation. Conventional and resident paths can export every trainable shard, routing metadata, layer traces, global cosine, relative L2, zero-mask mismatches, and optimizer surfaces.

Training data flow

prompt tokens
     |
     v
CP32 prefix forward -----------------> CPU-resident MLA/DSA pages
                                               |
                         +---------------------+---------------------+
                         |                     |                     |
                    response 0            response 1            response G
                    old/policy/ref         old/policy/ref         replay
                         |                     |                     |
                         +---------- serial LoRA backward ---------+
                                               |
                                  finalize_model_grads
                                               |
                                      one optimizer step

Rollout fixtures are engine-neutral JSON. Sampling, reward computation, model weights, datasets, and credentials remain outside this training repository.

NVIDIA review and run flow

Do not treat a clone by tag name or a mutable OCI tag as an authenticated release. v0.1.0 is unsigned and review-only. For a later signed release, obtain the release bundle plus policy and Sigstore trusted root through separate channels, then verify them before installing or running anything:

python3 scripts/verify_release_bundle.py \
  --archive /nvidia/input/mint2m-v0.2.0-release-bundle.tar.gz \
  --checkout "$PWD" \
  --policy /nvidia/trust/mint2m-policy.json \
  --trusted-root /nvidia/trust/sigstore-trusted-root.json \
  --cosign /nvidia/tools/cosign-v3.1.1 \
  --verified-output /nvidia/verified/mint2m-v0.2.0

The exact bundle layout, signing policy, and offline trust model are in docs/release-signing.md. Signature verification is an additional prerequisite; it does not clear the actor, image, licensing, fixture, or 32x H20 gates below.

From a fresh clone, install and inspect the CPU/release surface:

python -m pip install -r requirements/test-cpu.txt
make check
make payload-2m
python -m glm52 doctor --scope checkout --root .

The checkout doctor currently exits 2 and emits one JSON object with "status":"blocked". The implementation check passes, but the independent runtime, licensing, input, and validation gates remain unresolved. Doctor reports all blockers and a deterministic primary_failure; it does not import Ray, Torch, Megatron, or the model.

The external run contract is four nodes with eight H20 GPUs each (32x H20), an operator-managed Ray 2.51.1 cluster, one externally pullable image reference with an immutable digest, the pinned GLM-5.2 snapshot, an approved hash-bound fixture set, runtime identity plus SPDX SBOM, image-provenance sidecar, and a pre-created writable output directory outside the source tree. Use two deployment JSON files: a validation deployment for the legacy mint2m.fixture-set/v1 doctor/kernel/NCCL chain, and a training deployment for the strict mint2m.training-fixture/v1 manifest plus adjacent payload. They must share image, Ray, node, model, runtime, and output identities; the training file additionally carries the observed model_manifest_sha256. Put both files outside the source checkout. Start from the checked schema-compatible template:

sudo mkdir -p /etc/mint2m
sudo cp environment/deployment.example.json /etc/mint2m/nvidia-validation.json
sudo cp environment/deployment.training.example.json /etc/mint2m/nvidia-training.json
sudo "${EDITOR:-vi}" /etc/mint2m/nvidia-validation.json /etc/mint2m/nvidia-training.json
export VALIDATION_DEPLOYMENT_FILE=/etc/mint2m/nvidia-validation.json
export TRAINING_DEPLOYMENT_FILE=/etc/mint2m/nvidia-training.json

The example's all-zero digest and fixture hash, reserved IPs, and example paths are placeholders. Replace every one; doctor rejects the all-zero digest and does not treat this example as runnable. Plan mode has no Ray side effects:

export ATTEMPT=nvidia-review-0001

python -m glm52 plan \
  --config configs/run/glm52_2m_grpo_cp32.json \
  --deployment "$TRAINING_DEPLOYMENT_FILE" \
  --run-id nvidia-2m \
  --instance-nonce "$ATTEMPT" \
  --output artifacts/direct-plan.json

Successful plan output has "mode":"plan", "world_size":32, 32 unique rank actor names, 32 node-affined bundles, and a plan_sha256. The instance nonce is hashed and is not written to the plan.

The plan's top-level cluster identity is explicit: node_ips, gpus_per_node, and image_resource are carried alongside runtime_image. A live plan derives image_resource as mint2m_image_<64hex> from the immutable image digest; its placement-group bundles, coordinator, and every rank actor bind the target node:<ip> resource plus that digest resource. A review plan with runtime_image=None and image_resource=None is inspection-only: the DirectRayDriver rejects it before creating the journal or connecting to Ray. The Ray worker cluster must already be running the deployment image at that digest; actors do not receive a per-actor image override.

Before a live run, the run-scope doctor validates the digest-pinned external registry reference, four IPs, exact model hashes and 282-shard manifest, fixture manifest SHA, runtime identity/SBOM/image provenance, writable output, and Ray package without connecting to the cluster:

python -m glm52 doctor --scope run --root . \
  --deployment "$TRAINING_DEPLOYMENT_FILE" \
  --output /operator/output/doctor.json

After the release owner supplies a validation-plan.json bound to this source tree, runtime lock, published image digest, backend, model, optimizer, fixtures, and topology, NVIDIA can produce the first three receipts in the fixed chain doctor -> kernel -> NCCL -> actor -> 32K -> 64K -> 2M:

export VALIDATION_PLAN=/operator/input/validation-plan.json
export VALIDATION_RUN_ID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
export EVIDENCE_DIR="/operator/evidence/$VALIDATION_RUN_ID"
export TILELANG_CACHE_DIR=/operator/cache/tilelang
export NCCL_MASTER_PORT=29500
export NCCL_SOCKET_IFNAME=eth0
# Optional example; set only when the fabric requires an explicit HCA selector.
# export NCCL_IB_HCA=mlx5_0

make doctor-validation \
  VALIDATION_DEPLOYMENT_FILE="$VALIDATION_DEPLOYMENT_FILE" \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR"

make kernel-smoke \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR" \
  CUDA_DEVICE=0 \
  TILELANG_CACHE_DIR="$TILELANG_CACHE_DIR"

make nccl-smoke \
  VALIDATION_DEPLOYMENT_FILE="$VALIDATION_DEPLOYMENT_FILE" \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR" \
  NCCL_MASTER_PORT="$NCCL_MASTER_PORT" \
  NCCL_SOCKET_IFNAME="$NCCL_SOCKET_IFNAME" \
  NCCL_IB_HCA="${NCCL_IB_HCA:-}"

EVIDENCE_DIR must be outside the source checkout. The TileLang cache must be precompiled, nonempty, and unchanged during the bounded one-GPU run. Doctor writes doctor.json, doctor-ranks.json, doctor-artifacts.json, and doctor-validation.json; the kernel step then writes kernel-smoke.json, kernel-ranks.json, kernel-artifacts.json, and kernel-validation.json. NCCL then consumes that exact parent and writes nccl-smoke.json, nccl-ranks.json, nccl-artifacts.json, and nccl-validation.json. Each command returns exit 0 only for a passed, identity-bound bundle and exit 2 for blocked or invalid input. Do not run the next stage after exit 2.

Before NCCL, the four target Ray worker cluster processes themselves must run inside deployment.image at the same immutable image digest; merely pulling the image onto each host is insufficient. Every target advertises mint2m_image_<64hex>=1, where <64hex> is the digest without sha256:, plus Ray's node:<ip> resource. NCCL actors do not use runtime_env.image_uri, so Ray 2.51.1 does not attempt a nested Podman container without host GPU flags. Before creating its placement group, the launcher requires exactly one live Ray record for each deployment target IP, exactly 8 GPUs, at least 8 CPUs, a positive node:<ip> resource, and the matching digest resource. Unrelated Ray nodes may remain in the cluster; every NCCL bundle binds both target-node and digest resources so the 32 ranks can use only the four approved workers.

Reserve NCCL_MASTER_PORT (default 29500) and make it reachable between the selected NCCL_SOCKET_IFNAME interfaces. NCCL_IB_HCA is optional and is omitted when empty. The deployment runtime_identity must be an absolute runtime_identity path available to the CLI driver and at that same path inside each target worker cluster image, with the same bytes everywhere. The CLI hashes it before and after doctor, and all 32 actors hash it again; any drift or failed consensus blocks NCCL.

The bounded collective is expected to take 5-10 minutes on a healthy cluster after image pull and Ray placement (a blocked attempt can consume the fixed launcher timeout), and it does not load the model. It accepts only 32 unique H20 ranks mapped eight per node, complete all-gather/all-reduce/barrier results, runtime/fabric consensus, destroyed process groups, and zero owned Ray resources. Failure triggers automatic cleanup of the run-scoped actors and placement group. If the driver disappears before the JSON result, explicitly remove only resources in namespace mint2m-nccl-$VALIDATION_RUN_ID, verify they are absent, and use a new UUID; never continue from an incomplete bundle.

This checkout currently has no live 32-GPU NCCL receipt. The commands and receipt validators are reviewable, but 5.4 implementation does not make the release runnable: the locked image, approved validation plan and TileLang cache, execution/redistribution authorization, and target-H20 evidence are still external gates. The candidate rank port and actor launcher are checked in and candidate execution is enabled; public promotion remains blocked by those independent gates.

The train command remains fail-closed on every independent prerequisite:

python -m glm52 train \
  --plan artifacts/direct-plan.json \
  --ownership artifacts/ownership.json \
  --deployment "$TRAINING_DEPLOYMENT_FILE" \
  --root .

At the current candidate commit the backend gate no longer stops execution. Before ray.init, train validates the content-addressed plan and its embedded canonical configuration, reruns run-scope doctor against the training deployment, and requires the deployment Ray address and ordered node IPs to match the plan. After ray.init, the direct driver validates each target Ray record and its matching mint2m_image_<64hex> resource before placement-group creation. Until the runtime, licensing, and input prerequisites are accepted, run-scope doctor exits 2 without constructing Ray. A missing, incomplete, or explicitly disabled backend still returns direct_megatron_backend_unavailable. Success requires a versioned receipt with 32 unique ranks, exact phase order, placement attestation, two G2 backwards, one distributed gradient finalization, one applied/zeroed optimizer step, complete artifacts, unchanged source identity, and verified cleanup. Process exit zero alone is never acceptance. The receipt and CLI result carry claim_scope=2m_execution_only_not_numerical_correctness; 32K/64K parity and release validation remain separate gates.

The built worker image also installs a checkout-independent entry point:

mint2m-nvidia-run \
  --deployment "$TRAINING_DEPLOYMENT_FILE" \
  --instance-nonce "nvidia-$(date -u +%Y%m%dT%H%M%SZ)"

It executes strict training doctor, plan, and train in order and writes its control files under the deployment output root. It does not weaken any release gate or turn the current review tree into a runnable release.

Run-scope doctor intentionally does not require the public validation gate: the live chain produces that evidence. Checkout doctor and the release checker continue to report it blocked until the accepted 32K, 64K, and 2M receipts are published.

The training receipt is independently checkable. It references artifacts/rank-sessions.json (all 32 rank transitions, rank-local and driver cleanup, and planned/observed Ray placement) and artifacts/checkpoint-manifest.json (a regular-file tree hash). The validator re-hashes the live checkpoint tree and rejects symlinks, additions, deletions, or tampering. The strict training fixture binds model revision, model manifest, policy/profile hashes, and an adjacent content-addressed payload; a self- declared approved field is not an authorization grant.

If a live process is interrupted after ownership is written, recover only that run's deterministic namespace, actor names, IDs, and placement group:

python -m glm52 cleanup \
  --plan artifacts/direct-plan.json \
  --ownership artifacts/ownership.json

Cleanup succeeds only after the coordinator, placement group, and every owned rank actor are positively absent. Historical timing was about 115 seconds for model creation, 35 seconds for a 32K forward/backward, and 5 seconds for an optimizer step on 32 H20 GPUs. No current-tree 2M duration is claimed; reserve a dedicated multi-hour window for the ordered kernel, NCCL, actor, 32K, 64K, and 2M gates.

The exact fresh-clone runtime build, operator-managed Ray contract, direct commands, artifacts, and interruption cleanup are in docs/runbook.md.

Repository map

glm52/                 resident-prefix and response-replay implementation
runtime_patches/       opt-in backend/Bridge/verl worker hooks
integrations/          model preparation and legacy private-path compatibility
configs/run/           profiles consumed by the launch tools
scripts/               release checks, runtime build planning, and legacy tools
fixtures/              non-claiming parser/harness examples
tests/                 CPU contracts and distributed-interface tests
evidence/              compact historical receipts and acceptance status
environment/           dependency and runtime provenance
docs/                  runbook, limitations, and upstream integration

Validation status

The current tree contains global-CP composition, distributed gradient finalization, transport-neutral session contracts, a candidate repository-local Megatron rank/backend path, direct-Ray ownership and cleanup, a fail-closed runtime lock, and validation-chain contracts. The candidate is intentionally disabled and has no current-tree 32-rank GPU evidence or redistribution grant. The historical 2M receipt predates current global-CP and finalization fixes; current 32K gradient/optimizer parity, 64K parity, and fresh 2M execution all remain required. Exact blockers are tracked in STATUS.md, not hidden behind the scale context.

Compact evidence is indexed in evidence/README.md, and source files are bound by SHA-256 in evidence/manifest.json.