MinT-2M Reproduction Levels

The complete deployment and execution procedure is runbook.md. This file defines what each command proves.

Level 1: local contracts

python3 -m pip install -r requirements/test-cpu.txt
make check

This validates page ownership, replay bookkeeping, GRPO math, global-CP merge contracts, worker hooks, run tooling, and release metadata without loading the model.

The sanitized external validation receipts are locally inspectable without GPU dependencies:

python3 -m json.tool evidence/online_grpo_2m.json >/dev/null
python3 -m json.tool evidence/gradient_parity_32k.json >/dev/null

They establish the recorded exact-2M online update and 32K engineering parity, not reproducibility from this checkout. Their raw artifacts, runtime, model, dataset, and source binding remain external; see glm52_2m_validation.md.

Level 2: 2M payload construction

make payload-2m

This constructs deterministic Python request data for 2,097,152 tokens. It does not contact a service or serialize a complete HTTP request, and it does not run prefix capture, forward, backward, or an optimizer.

Level 3: direct-Ray plan and prerequisite proof

Generate a plan without connecting to Ray, then run the training deployment doctor. Keep a separate validation deployment for the legacy fixture-set used by the kernel/NCCL promotion stages:

export TRAINING_DEPLOYMENT_FILE=/etc/mint2m/nvidia-training.json
export VALIDATION_DEPLOYMENT_FILE=/etc/mint2m/nvidia-validation.json
make doctor-run DEPLOYMENT_FILE="$TRAINING_DEPLOYMENT_FILE"
make direct-plan TRAINING_DEPLOYMENT_FILE="$TRAINING_DEPLOYMENT_FILE" \
  INSTANCE_NONCE=nvidia-review-0001

Plan success proves only explicit 4x8 placement, unique names, configuration identity, timeout, and ownership scope. It does not create actors. Candidate execution is enabled, but make direct-train still requires a passed run-scope doctor before Ray initialization.

The profile also binds the historical fresh-actor optimizer values. Each worker must observe Adam/LR 1e-4/constant schedule and the remaining reviewed fields from the actual backend optimizer before the step; unavailable or conflicting values fail the run before weights are updated.

The checked-in candidate runner refuses altered plan/parent identities and marks a receipt passed only after validating all 32 rank reports, placement, artifacts, optimizer invariants, unchanged source, and cleanup. Missing runtime, licensing, or input prerequisites stop the current checkout before Ray without disabling the reviewed backend implementation.

Once the external image, authorization, fixture, and 32x H20 gates are accepted, the same training flow can be run as one command:

mint2m-nvidia-run \
  --deployment "$TRAINING_DEPLOYMENT_FILE" \
  --instance-nonce "nvidia-$(date -u +%Y%m%dT%H%M%SZ)"

# Source-checkout equivalent:
make nvidia-run \
  TRAINING_DEPLOYMENT_FILE="$TRAINING_DEPLOYMENT_FILE" \
  INSTANCE_NONCE="nvidia-$(date -u +%Y%m%dT%H%M%SZ)" \
  PLAN_FILE=/outputs/mint2m/direct-plan.json \
  OWNERSHIP_FILE=/outputs/mint2m/ownership.json

The 2M profile is machine-readable at configs/run/glm52_2m_grpo_cp32.json.

Level 3a: doctor, single-GPU kernel, and 32-GPU NCCL receipts

For the first three promotion stages, use a release-owner supplied validation plan and keep all generated evidence outside the checkout:

export VALIDATION_PLAN=/operator/input/validation-plan.json
export VALIDATION_RUN_ID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
export EVIDENCE_DIR="/operator/evidence/$VALIDATION_RUN_ID"
export TILELANG_CACHE_DIR=/operator/cache/tilelang
export NCCL_MASTER_PORT=29500
export NCCL_SOCKET_IFNAME=eth0
# Optional example; omit when no explicit HCA selector is required.
# export NCCL_IB_HCA=mlx5_0

make doctor-validation \
  VALIDATION_DEPLOYMENT_FILE="$VALIDATION_DEPLOYMENT_FILE" \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR"
make kernel-smoke \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR" \
  CUDA_DEVICE=0 TILELANG_CACHE_DIR="$TILELANG_CACHE_DIR"
make nccl-smoke \
  VALIDATION_DEPLOYMENT_FILE="$VALIDATION_DEPLOYMENT_FILE" \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR" \
  NCCL_MASTER_PORT="$NCCL_MASTER_PORT" \
  NCCL_SOCKET_IFNAME="$NCCL_SOCKET_IFNAME" \
  NCCL_IB_HCA="${NCCL_IB_HCA:-}"

The plan binds source/runtime-lock/image identities, and the TileLang cache is an explicit precompiled input. Doctor writes the four doctor-* files; kernel writes kernel-smoke.json, kernel-ranks.json, kernel-artifacts.json, and kernel-validation.json. NCCL requires that exact parent and writes nccl-smoke.json, nccl-ranks.json, nccl-artifacts.json, and nccl-validation.json. Exit 0 means the complete identity-bound bundle was accepted. Exit 2 means blocked or invalid prerequisites and must stop the chain. Promotion remains ordered doctor -> kernel -> NCCL -> actor -> 32K -> 64K -> 2M.

For NCCL, start the four target Ray worker cluster processes themselves inside the same immutable image digest with GPU devices attached. Each target advertises mint2m_image_<64hex>=1 (digest hex without sha256:); Ray supplies its node:<ip> resource. The launcher permits unrelated nodes, but each deployment target IP must have exactly one live Ray record, exactly 8 GPUs, at least 8 CPUs, and both node and digest resources before placement. Every bundle binds those two resources. NCCL actors do not use runtime_env.image_uri, avoiding a nested Ray 2.51.1 Podman launch without GPU flags.

Make NCCL_MASTER_PORT (default 29500) reachable over the required NCCL_SOCKET_IFNAME, and set optional NCCL_IB_HCA only when the fabric needs an explicit selector. The deployment's absolute runtime_identity path must resolve in the driver and at the same path inside every target worker cluster image with the same bytes. CLI pre/post hashes and all 32 rank hashes must agree. The bounded run is expected to take 5-10 minutes on a healthy cluster after image pull; a blocked attempt can consume the fixed launcher timeout. Rank, collective, runtime, fabric, and cleanup acceptance are independent; a process exit alone is insufficient. The launcher performs automatic cleanup, while a hard driver loss requires explicit cleanup of only the mint2m-nccl-$VALIDATION_RUN_ID namespace before retrying.

There is currently no live 32-GPU NCCL receipt in this checkout. 5.4 implementation does not make the release runnable; this level is a reproducible operator contract until current-tree H20 evidence and the other release gates pass. The candidate actor path is checked in and enabled for those validation runs.

Level 4: ordered numerical correctness

The required promotion order is doctor, single-GPU kernels, 32-GPU NCCL, direct actor materialization, 32K parity, 64K parity, then 2M. The commands are not advertised as runnable until authorization, the immutable image, approved fixtures, and current-tree evidence are accepted. The predecessor private commands remain available only as legacy-parity-* targets for internal comparison:

make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_32k_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-parity-forward-32k \
  RUN_DIR=artifacts/parity-forward ROLLOUT_JSON=/shared/real-rollout.json
make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_32k_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-parity-gradient-32k \
  RUN_DIR=artifacts/parity-gradient ROLLOUT_JSON=/shared/real-rollout.json

The full gate compares every trainable LoRA shard and exits nonzero when any logprob, loss, gradient cosine, relative-L2, zero-mask, or nonfinite threshold fails. After 32K passes, run the corresponding parity-forward-64k and parity-gradient-64k targets with the 64K profile and rollout. A 2M execution run is not a substitute for these bounded correctness gates.

Rollout fixture contract

{
  "prompt_tokens": [1, 2, 3],
  "responses": [[10, 11, 12], [20, 21, 22]],
  "rewards": [-1.0, 1.0],
  "old_logprobs": [[-2.0, -3.0], [-2.5, -3.5]],
  "ref_logprobs": [[-2.1, -3.1], [-2.6, -3.6]]
}

Every response must contain at least three tokens for the active-clip parity case, and token-level old/reference arrays must contain response_tokens - 1 values. The fixture must match the exact policy and base-model revision used to produce it.