MinT-2M Runbook

This runbook starts from a fresh checkout and defines the direct, operator-owned Ray path toward source-bound 32K/64K correctness gates and a 2M resident-prefix GRPO run. It does not require or start the private HTTP control plane. The current checkout is review_only_not_runnable, but candidate execution is enabled with IMPLEMENTATION_COMPLETE=True. mint2m train reaches Ray only after run-scope doctor accepts the runtime, licensing, model, fixture, and deployment prerequisites. This execution state is separate from the missing 32-rank validation result.

1. Required infrastructure

The checked-in 2M profile uses the same resource shape as the recorded run:

• four dedicated nodes;
• eight H20-class GPUs and 160 CPUs per node;
• 200,000,000,000 bytes of Ray object-store memory per node;
• 32 total GPUs with TP1/PP1/CP32/EP32/ETP1;
• a worker-visible shared filesystem for source, runtime, model, and artifacts;
• Python 3.13, CUDA 13, a compatible PyTorch build, Ray 2.51, and the training dependencies recorded in environment/versions.template.json;
• GLM-5.2 revision b4734de4facf877f85769a911abafc5283eab3d9.

Deployment inputs are supplied by the NVIDIA/MindLab release process rather than guessed by this repository: the immutable Python 3.13/CUDA 13 image, the approved actor-code grant, the exact model revision, parity fixture set, and a 32x H20 window. The commands below bind and validate those inputs; the repo does not invent a digest, grant, fixture, or source revision.

The recorded GLM-5.2 artifacts are a fresh base model plus a new LoRA adapter. No optimizer-bearing 2M resume checkpoint is required or implied by this flow.

All live profiles pin the optimizer configuration observed in the fresh 2M training actor launched by MinT Runtime: Adam, LR 1e-4, minimum LR 0, betas 0.9/0.999, epsilon 1e-8, weight decay 0.01, decoupled weight decay, clip grad 1, distributed optimizer, BF16, constant decay over 100,000 configured decay steps, and zero warmup steps. GRPO performs exactly one optimizer step after two accumulated G2 backwards. These values are machine-readable under optimizer in each run profile.

2. Verify the checkout

For a signed release after v0.1.0, follow docs/release-signing.md first. NVIDIA must verify the bundle with scripts/verify_release_bundle.py using a policy and trusted root supplied out-of-band and outside this checkout. The current v0.1.0 tag is unsigned and review-only, so it cannot satisfy that step. A tag name, GitHub tag archive, or mutable image tag is insufficient.

git clone https://github.com/MindLab-Research/longstraw.git
cd longstraw
python3 scripts/check_release_tree.py
python3 -m pip install -r requirements/test-cpu.txt
make check
make payload-2m
make doctor-checkout

payload-2m builds the deterministic 2M request data structures only. It does not contact Ray, load a model, capture a prefix, run backward, or update weights. The existing v0.1.0 review tag predates the direct CLI. The release owner must publish a new immutable tag only after source binding and the required validation receipts pass; check_release_tree.py rejects an unbound checkout.

3. Plan the locked runtime and direct run

The checked runtime lock is intentionally incomplete. The first command exits 2 without downloading or building until the CUDA 13 base digest and every artifact hash are externally resolved. The second command prints the exact OCI build without executing it; only --execute starts the builder:

export IMAGE_TAG=registry.example/mint-2m-runtime:candidate
python3 scripts/check_runtime_lock.py --require-complete
python3 scripts/build_cuda13_image.py --tag "$IMAGE_TAG"
python3 scripts/build_cuda13_image.py --tag "$IMAGE_TAG" --execute
docker push "$IMAGE_TAG"

# Set these from the registry response and metadata copied from the built image.
export IMAGE_DIGEST=registry.example/mint-2m-runtime@sha256:<manifest-sha256>
export RUNTIME_IDENTITY=/operator/path/runtime-identity.json
export RUNTIME_SBOM=/operator/path/sbom.spdx.json
python3 scripts/generate_image_provenance.py \
  --lock environment/runtime.lock.json \
  --identity "$RUNTIME_IDENTITY" \
  --sbom "$RUNTIME_SBOM" \
  --image "$IMAGE_DIGEST" \
  --output /etc/mint2m/image-provenance.json

Record the immutable registry manifest reference returned after publication, not the mutable tag. The release process must bind that digest to the runtime lock, generated identity, and SPDX SBOM, and all four nodes must be able to pull it before doctor can pass. Do not infer a digest from a local image ID.

Create two operator-owned deployment files from the versioned template. The validation file is used by doctor/kernel/NCCL and must reference the legacy mint2m.fixture-set/v1; the training file is used by plan/train and must reference strict mint2m.training-fixture/v1 plus its adjacent payload.

sudo mkdir -p /etc/mint2m
sudo cp environment/deployment.example.json /etc/mint2m/nvidia-validation.json
sudo cp environment/deployment.training.example.json /etc/mint2m/nvidia-training.json
sudo "${EDITOR:-vi}" /etc/mint2m/nvidia-validation.json /etc/mint2m/nvidia-training.json
export VALIDATION_DEPLOYMENT_FILE=/etc/mint2m/nvidia-validation.json
export TRAINING_DEPLOYMENT_FILE=/etc/mint2m/nvidia-training.json

The all-zero digest and fixture hash, reserved IPs, and example paths are placeholders. Replace all of them, including the published image provenance, before invoking the direct flow. The two files must share the Ray address, ordered node list, image, model snapshot/manifest, runtime identities, and output root; only the fixture schema/path is stage-specific. The training file must include the real model_manifest_sha256 observed from the HF snapshot.

The direct plan records node_ips, gpus_per_node, runtime_image, and the derived image_resource (mint2m_image_<64hex>) at top level. Its placement bundles, coordinator, and rank actors bind both node:<ip> and the digest resource. A review plan with runtime_image=None is inspection-only; the DirectRayDriver rejects it before creating the journal or connecting to Ray. The target Ray workers must already run the deployment image at the immutable digest, so direct actors inherit the cluster image and receive no per-actor image override.

3.1 Produce the bounded kernel and NCCL evidence

The release owner must provide a validation-plan.json whose common identity matches this checkout, environment/runtime.lock.json, the immutable OCI digest, model/backend/optimizer inputs, and the ordered stage topology. Keep the evidence directory outside the checkout and provide a precompiled, non-empty TileLang cache visible to the selected runtime.

Before the distributed stage, the container/scheduler layer must start the four target Ray worker cluster processes themselves inside the exact deployment.image digest. The container runtime must have already attached all 8 GPUs, the host network, and required mounts before ray start; the commands below execute inside that pinned image, never directly on the host. A host-side pull alone is insufficient. Each target must advertise a custom mint2m_image_<64hex> resource with value 1, where <64hex> is the image digest without sha256:. The equivalent dedicated-host Ray bootstrap, run inside that pinned image, is:

export DEPLOYMENT_IMAGE='registry.example/mint-2m-runtime@sha256:<64hex>'
export IMAGE_DIGEST="${DEPLOYMENT_IMAGE##*@}"
export IMAGE_RESOURCE="mint2m_image_${IMAGE_DIGEST#sha256:}"
export HEAD_IP=192.0.2.1

# Run once in the pinned image on the head/first GPU worker.
ray start --head --node-ip-address="$HEAD_IP" --port=6379 \
  --ray-client-server-port=10001 --num-gpus=8 \
  --resources="{\"$IMAGE_RESOURCE\":1}"

# Run once in that same pinned image on each of the other three workers.
export NODE_IP=192.0.2.2
ray start --address="$HEAD_IP:6379" --node-ip-address="$NODE_IP" \
  --num-gpus=8 --resources="{\"$IMAGE_RESOURCE\":1}"

Repeat the worker command for the remaining ordered deployment IPs. Ray creates the node:<ip> resource and auto-detects CPU capacity; do not fake either through --resources. The launcher allows unrelated cluster nodes, but each of the four deployment target IPs must have exactly one live Ray record, exactly 8 GPUs, at least 8 CPUs, a positive node:<ip> resource, and the matching digest resource. Its placement bundles bind both node:<ip> and mint2m_image_<64hex>, so the 32 ranks cannot spill to an unrelated node. NCCL actors do not use runtime_env.image_uri; this avoids Ray 2.51.1 nested Podman launches without the host GPU device flags.

Verify the selected TCP master port is reachable between the selected socket interfaces. Then use one UUID for the doctor, kernel, and NCCL receipts:

export VALIDATION_PLAN=/operator/input/validation-plan.json
export VALIDATION_RUN_ID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
export EVIDENCE_DIR="/operator/evidence/$VALIDATION_RUN_ID"
export TILELANG_CACHE_DIR=/operator/cache/tilelang
export NCCL_MASTER_PORT=29500
export NCCL_SOCKET_IFNAME=eth0
# Optional example; leave unset when the fabric should select its HCA.
# export NCCL_IB_HCA=mlx5_0

make doctor-validation \
  DEPLOYMENT_FILE="$DEPLOYMENT_FILE" \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR"

make kernel-smoke \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR" \
  CUDA_DEVICE=0 \
  TILELANG_CACHE_DIR="$TILELANG_CACHE_DIR"

make nccl-smoke \
  DEPLOYMENT_FILE="$DEPLOYMENT_FILE" \
  VALIDATION_PLAN="$VALIDATION_PLAN" \
  VALIDATION_RUN_ID="$VALIDATION_RUN_ID" \
  EVIDENCE_DIR="$EVIDENCE_DIR" \
  NCCL_MASTER_PORT="$NCCL_MASTER_PORT" \
  NCCL_SOCKET_IFNAME="$NCCL_SOCKET_IFNAME" \
  NCCL_IB_HCA="${NCCL_IB_HCA:-}"

doctor-validation runs the run-scope prerequisite checks and emits doctor.json, doctor-ranks.json, doctor-artifacts.json, and the generic doctor-validation.json. kernel-smoke verifies that parent bundle, then executes the bounded CUDA/Apex/TE/TileLang/flash_mla path on exactly the requested CUDA device and emits the corresponding four kernel-* files. The raw receipt is independently checked for source/runtime/image identity and numerical acceptance; a generic status line is not enough.

nccl-smoke requires the existing external evidence directory and its exact kernel-validation.json parent. NCCL_MASTER_PORT defaults to 29500 and must be free and reachable on the ordered first node; NCCL_SOCKET_IFNAME is required and must name the routable interface on every node. NCCL_IB_HCA is optional; the Make target appends --ib-hca only when it is nonempty. The stage writes nccl-smoke.json, nccl-ranks.json, nccl-artifacts.json, and nccl-validation.json.

The deployment.runtime_identity value must be an absolute runtime_identity path which resolves in the CLI driver and at the identical path inside every target Ray worker cluster image (normally /opt/mint2m/runtime-metadata/runtime-identity.json). Every location must expose the same bytes. The CLI hashes those bytes before and after current doctor; the launcher and all 32 actors hash them again and require consensus. A missing path, changed file, or rank hash mismatch fails NCCL before acceptance.

Reserve 5-10 minutes for this bounded 4x8 H20 stage on a healthy cluster after all nodes have already pulled the digest; a blocked attempt can consume the fixed launcher timeout. Success requires ranks 0 through 31 exactly once, eight H20s per ordered node, matching runtime and fabric identities, accepted all-gather, all-reduce, and barrier results, destroyed process groups, and zero owned Ray resources. It does not load GLM-5.2. The launcher performs automatic cleanup of its 32 named actors and placement group on success or failure. After a hard driver interruption with no final JSON line, explicitly remove only namespace mint2m-nccl-$VALIDATION_RUN_ID, its -rank-00 through -rank-31 actors and -pg placement group through the operator Ray control surface, verify absence, and restart with a new UUID.

Exit 0 means a complete, identity-bound bundle was written. Exit 2 means blocked prerequisites, invalid input, stale identity, or a failed smoke; stop the chain and inspect the single JSON status line. The required promotion order is doctor -> kernel -> NCCL -> actor -> 32K -> 64K -> 2M. This checkout has no live 32-GPU NCCL receipt yet. 5.4 implementation does not make the release runnable; these commands are documented interfaces and validators until target H20 evidence and all other release gates pass.

glm52/actor_smoke.py is packaged as a contract-only validator for the next stage. It requires an injected launcher backed by the checked-in candidate Megatron actor; its schema and CPU tests do not provide live GPU evidence and therefore cannot be used as runnable actor evidence.

With the operator-managed 4x8 H20 Ray cluster available and the external gates accepted, the installed image runs the training attempt without a source checkout or Makefile:

mint2m-nvidia-run \
  --deployment "$TRAINING_DEPLOYMENT_FILE" \
  --instance-nonce "nvidia-$(date -u +%Y%m%dT%H%M%SZ)"

The source-checkout equivalent is:

export INSTANCE_NONCE=nvidia-$(date -u +%Y%m%dT%H%M%SZ)
export PLAN_FILE=/outputs/mint2m/direct-plan.json
export OWNERSHIP_FILE=/outputs/mint2m/ownership.json

make nvidia-run \
  TRAINING_DEPLOYMENT_FILE="$TRAINING_DEPLOYMENT_FILE" \
  INSTANCE_NONCE="$INSTANCE_NONCE" \
  PLAN_FILE="$PLAN_FILE" \
  OWNERSHIP_FILE="$OWNERSHIP_FILE"

make direct-plan has no Ray side effects. make direct-train consumes the content-addressed plan, reruns run-scope doctor against the training deployment, validates each target node's live Ray record and digest resource before placement-group creation, then connects to Ray. Every coordinator and rank actor binds both its target node and mint2m_image_<64hex> resource; the worker cluster itself supplies the image. The candidate execution is enabled, while run-scope doctor stops before ray.init until authorization and the image/fixture execution prerequisites are accepted. Run-scope doctor deliberately excludes the public validation gate, because this run is what produces that evidence. The release checker still blocks runnable promotion until validation passes.

The expanded equivalent is make doctor-run, then make direct-plan, then make direct-train; make nvidia-run only packages those same three steps.

A passed run must contain artifacts/rank-sessions.json with all 32 exact phase transitions, rank-local and driver cleanup, and planned/observed Ray placement. It also contains artifacts/checkpoint-manifest.json; the receipt validator re-hashes the live regular-file tree and rejects symlinks or any add/delete/ modify drift.

If a started attempt is interrupted and the ownership file exists, clean only the identities recorded for that attempt:

make direct-cleanup

Cleanup passes only when the coordinator, placement group, and all 32 owned rank actors are positively absent. A training exit code is accepted only after the CLI independently validates the hash-bound receipt, exact ranks, two G2 backwards, one gradient finalization, one optimizer step, artifacts, unchanged source identity, and zero-resource cleanup.

4. Legacy private reference path

The remainder of this document records the predecessor private-service flow for internal parity comparison only. NVIDIA should not use it as onboarding. Every corresponding Make target is prefixed legacy-.

Create the local configuration and replace every /shared placeholder with a path visible at the same logical location on all workers:

cp scripts/training.env.example scripts/training.env
set -a
source scripts/training.env
set +a

Point GLM52_RUNTIME_PYTHON at the Python 3.13 interpreter inside the immutable CUDA 13 image. The Ray CLI must come from the same bin directory. Set the four source-root variables to paths that either do not exist yet or already contain the exact declared Git worktree; an arbitrary empty directory is not a valid source target. Then run the checked source bootstrap:

make legacy-sources-prepare
make legacy-runtime-build

The source contract is machine-readable in environment/versions.template.json:

Source	Base commit	Declared state
MinT	12c83d904df5faf3e2cd60633b448a8317d84ee0	clean MindLab checkout
Megatron-LM	03db8324007ed7b33edffc147160bebf9552846c	clean checkout
Megatron-Bridge	22edeb2a487d6a9cc0dcea567827826cc76427c2	bundled GLM-5.2 TileLang/loss patch
verl	d2916f5a0ed346464d8999e040e0ebb05bb8fadf	bundled MCore/THD compatibility patch

The bootstrap fetches each full SHA directly, checks origin and HEAD, applies only the declared patch, and verifies the repository-independent canonical git diff --binary --full-index hash. It is idempotent: a correct existing tree is accepted, while any other tracked or untracked state is rejected.

The runtime builder revalidates that source state, binds the selected Python 3.13 site-packages, creates relative source links, and records every base, patch, and diff identity. For a compatible prebuilt MinT Runtime worker manifest, make legacy-runtime-overlay adds only this repository's opt-in overlay and an idempotent manifest entry.

MINT_SERVER_ROOT is the target for the exact MinT checkout. make legacy-sources-prepare creates it when absent, or validates its origin, full commit, and clean diff when it already exists. make legacy-preflight-local then validates the native mint_server interfaces used by this release and binds their fingerprint into the runtime identity.

Prepare and verify the exact model snapshot before cluster preflight. The download command is online only for this step; normal execution stays offline:

mkdir -p "$GLM52_HF_MODULES_PATH"
HF_HUB_OFFLINE=0 TRANSFORMERS_OFFLINE=0 make model-prepare
make model-verify
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1

model-prepare calls snapshot_download with the full revision SHA and checks the returned models--zai-org--GLM-5.2/snapshots/<revision> directory. It does not depend on a mutable refs/main file.

The retained legacy private path uses MinT's local control-plane mode and binds it to 127.0.0.1. There is no API credential and no paid external service. The launcher rejects a non-loopback bind; use an SSH tunnel from the control node if an operator needs remote access instead of exposing the port directly.

4. Provide Ray

If a scheduler already provides Ray, set MINT_RAY_GCS_ADDRESS and skip to the next section. For four dedicated SSH hosts, fill these values in scripts/training.env:

export GLM52_RAY_HOSTS_CSV="gpu-host-a,gpu-host-b,gpu-host-c,gpu-host-d"
export GLM52_RAY_NODE_IPS_CSV="ray-head,ray-worker-1,ray-worker-2,ray-worker-3"
export GLM52_TRAIN_NODE_IPS_CSV="$GLM52_RAY_NODE_IPS_CSV"
export GLM52_RAY_BIN="${GLM52_RUNTIME_PYTHON%/*}/ray"
export GLM52_RAY_OWNER_TOKEN="$(openssl rand -hex 32)"

Keep the same owner token in the untracked scripts/training.env until this cluster is stopped. The launcher combines it with a per-launch ID and writes only the derived SHA-256 marker into each new Ray session. The raw token never enters remote commands or the journal. Ownership progress is checkpointed with mode 0600 under $GLM52_ARTIFACT_ROOT/control/ray-cluster.json.

Inspect the remote command shape before changing cluster state. The plan uses an all-zero ownership-marker placeholder; legacy-ray-start derives the real marker from the secret owner token and a fresh launch ID without printing either:

make legacy-ray-plan

On dedicated hosts only:

GLM52_CONFIRM_CLUSTER=1 make legacy-ray-start

The generated head command includes the dashboard on port 8265; every node advertises 8 GPUs, 160 CPUs, and a 200GB object store. Hosts and node addresses are deployment inputs and are never hardcoded in the repository. legacy-ray-start also rejects a Ray CLI outside the selected Python 3.13 runtime. If a host fails during startup, the launcher rolls back only nodes whose marker was confirmed for this launch and preserves any incomplete ownership journal for a fail-closed retry.

5. Run preflight

make legacy-preflight-local
make legacy-preflight-cluster

The local check validates:

• required paths and the native MinT Ray address;
• the exact base commit, patch, and canonical diff for every training source;
• installation of this repository's runtime overlay;
• Python 3.13 and the recorded Torch/CUDA/NCCL/Ray/training package versions;
• imports of flash_mla, Megatron Core, Megatron Bridge, and verl;
• the exact GLM-5.2 snapshot, four small-file hashes, all 282 nonempty shards, and the Hugging Face blob manifest;
• a content fingerprint for the pinned MindLab MinT source;
• a writable artifact root.

The cluster check requires exactly four alive Ray nodes, 32 advertised GPUs, and all 32 GPUs available. It schedules a zero-CPU/zero-GPU probe on each node and requires the same Python, packages, source/runtime/model identities, eight H20 GPUs, and at least the configured free memory on every device. It does not reserve a GPU or launch the model.

6. Start MinT Runtime

Run this on the control node that can reach the Ray head:

GLM52_CONFIRM_LIVE=1 make legacy-service-start
make legacy-service-status
make legacy-preflight-service PREFLIGHT_PROFILE=configs/run/glm52_2m_grpo_cp32.json

The MinT Runtime launcher owns one PID file and one log under $GLM52_ARTIFACT_ROOT/control/. It never kills a process merely because it is using the configured port; stop only targets the PID whose command line matches this repository's server entry point. Readiness uses a repository-specific identity route and matches the process PID, random instance ID, profile hash, and derived service contract, so an unrelated healthy process on the port is not accepted.

Before importing the API application, legacy-service-start runs the pinned MinT bootstrap_control_plane.py entry point with the exact runtime environment. That step creates or verifies the detached config, task-state, scheduler, maintenance, and model-supervisor actors and fails closed if the bootstrap script is missing, times out, exits nonzero, or returns an invalid summary.

The launcher forces training-only service mode: MinT's shared multi-adapter serving engine is disabled, the model-work scheduler is enabled, usage storage is disabled, and the backend is fixed to Megatron. The runtime overlay registers GLM-5.2 and installs the exact-snapshot and profile-bound optimizer contracts in both the API process and detached execution actors. An omitted session learning rate is filled with 1e-4; a conflicting explicit value is rejected.

legacy-preflight-service proves the live identity without creating an attestation. Each legacy-preflight-all below repeats local, cluster, and service checks immediately before one live command and writes $GLM52_PREFLIGHT_ATTESTATION. The attestation is self-hashed, profile-bound, and valid for 15 minutes by default. Every live runner requires it and rechecks the repository source, runtime manifest, runtime package/backend fingerprints, model snapshot, and service instance before creating a model.

At this point no model actor exists. The next training command calls /api/v1/create_model to register a fresh rank-8 LoRA session, then submits a CP32-aligned 64-token /api/v1/forward canary and waits for it to finish. That request materializes the 32-rank actor and runs forward only; it does not call backward or the optimizer. The driver attaches the resident-prefix path only after the canary succeeds and Ray confirms the actor exists.

7. Pass the 32K execution gate

First inspect the generated command:

make legacy-grpo-32k-plan RUN_DIR="$GLM52_ARTIFACT_ROOT/grpo-32k-plan"

Run a fresh deterministic G2 update:

make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_32k_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-grpo-32k \
  RUN_DIR="$GLM52_ARTIFACT_ROOT/grpo-32k-$(date -u +%Y%m%dT%H%M%SZ)"

The driver captures all 78 layers, performs two response backwards, applies one optimizer step, saves all rank reports, then gracefully removes the actor and deletes the API model record. The wrapper rejects missing or duplicate rank/CP-rank identities, any incomplete per-rank update, missing traces, or an optimizer count other than one even when the child process exits zero. Before the step, every worker resolves the actual backend optimizer parameter groups, sets and reads back LR 1e-4, and attests the full optimizer configuration; missing or mismatched fields fail closed without calling the step.

8. Run the 2M profile

This section is the training-side scale gate. For final acceptance, repeat the fresh 2M command after the numerical parity gates in section 9.

Inspect the exact 2M/G2/rank-8 command:

make legacy-grpo-2m-plan RUN_DIR="$GLM52_ARTIFACT_ROOT/grpo-2m-plan"

Execute the deterministic training-side scale run:

make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_2m_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-grpo-2m-smoke \
  RUN_DIR="$GLM52_ARTIFACT_ROOT/grpo-2m-$(date -u +%Y%m%dT%H%M%SZ)"

The profile uses 2,097,152 prompt tokens, three response tokens per group member, two scored response tokens, G2, beta 0.01, page size 64, all 78 layers, and CP32. To consume an external rollout fixture with the same lengths:

make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_2m_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-grpo-2m-fixture \
  RUN_DIR="$GLM52_ARTIFACT_ROOT/grpo-2m-fixture-$(date -u +%Y%m%dT%H%M%SZ)" \
  ROLLOUT_JSON=/shared/fixtures/glm52-rollout.json

Every run directory contains:

• preflight-attestation.json: the exact short-lived preflight consumed;
• command.json: command, profile/input hash, every execution-source hash, Git identity, and attestation binding;
• events.log: complete driver output;
• ranks.json: all per-rank results;
• traces/: per-group, per-phase, per-rank replay traces;
• ownership.json: model/session ownership for safe interrupted cleanup, when model creation completed;
• run_receipt.json: child exit, independent acceptance summary, pre/post source identity, and hashes/sizes for every artifact.

Only run_receipt.json.status == "passed" is success. The status requires 32 unique ranks and CP ranks, two backwards and two accumulation groups on every rank, the reviewed optimizer configuration on every rank, one applied optimizer step followed by zero-grad, no execution errors, complete rank traces, and an unchanged execution-source tree.

9. Run correctness gates

Parity requires both conventional and resident paths to derive routes with router replay disabled. Stop the API and use a fresh namespace before creating the parity actor because MinT's detached control actors are configuration-bound:

GLM52_CONFIRM_LIVE=1 make legacy-service-stop
export MINT_RAY_NAMESPACE="mint-2m-parity-$(date -u +%Y%m%dT%H%M%SZ)"
export GLM52_ROUTER_REPLAY_MODE=disabled
export MINT_ROUTER_REPLAY_MODE=disabled
GLM52_CONFIRM_LIVE=1 make legacy-service-start

legacy-service-stop stops this repository's API process only. It intentionally does not kill detached actors owned by a shared Ray cluster. Keep one immutable service configuration per namespace; a dedicated cluster is retired in section 10, while a scheduler-owned cluster must apply its normal namespace retention policy.

fixtures/parity.synthetic.json validates harness wiring only. A correctness claim must use a persisted real rollout with distinct responses and reward variation.

Forward gate:

make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_32k_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-parity-forward-32k \
  RUN_DIR="$GLM52_ARTIFACT_ROOT/parity-forward-$(date -u +%Y%m%dT%H%M%SZ)" \
  ROLLOUT_JSON=/shared/fixtures/real-rollout.json

Full gradient gate:

make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_32k_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-parity-gradient-32k \
  RUN_DIR="$GLM52_ARTIFACT_ROOT/parity-gradient-$(date -u +%Y%m%dT%H%M%SZ)" \
  ROLLOUT_JSON=/shared/fixtures/real-rollout.json

After both 32K gates pass, repeat them at 64K with a separately persisted 64K rollout from the same policy/base-model revisions:

make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_64k_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-parity-forward-64k \
  RUN_DIR="$GLM52_ARTIFACT_ROOT/parity-forward-64k-$(date -u +%Y%m%dT%H%M%SZ)" \
  ROLLOUT_JSON=/shared/fixtures/real-rollout-64k.json
make legacy-preflight-all PREFLIGHT_PROFILE=configs/run/glm52_64k_grpo_cp32.json
GLM52_CONFIRM_LIVE=1 make legacy-parity-gradient-64k \
  RUN_DIR="$GLM52_ARTIFACT_ROOT/parity-gradient-64k-$(date -u +%Y%m%dT%H%M%SZ)" \
  ROLLOUT_JSON=/shared/fixtures/real-rollout-64k.json

The gradient command runs ratio1,active_clip, enables the response-kernel shape alignment used by the latest implementation, writes bounded raw shards, produces comparison.json, comparison.csv, manifest.json, and SUMMARY.md, and exits 2 if any acceptance threshold fails.

Issue a fresh selected-profile preflight immediately before each parity command. The service identity includes the router-replay and long-context memory behavior hash, so a service restarted with different switches cannot reuse an older attestation.

10. Cleanup

Normal GRPO and parity drivers clean their model actor and API record. After an interrupted driver, use the ownership file written immediately after model creation:

Cleanup is accepted only after actor absence is positively confirmed and the model-record DELETE succeeds (or returns not found). A lookup, kill, wait, or DELETE failure keeps the model/session ownership record, makes the run receipt fail, and leaves legacy-actor-cleanup as the recovery path.

GLM52_CONFIRM_LIVE=1 make legacy-actor-cleanup \
  OWNERSHIP_FILE="$GLM52_ARTIFACT_ROOT/<run>/ownership.json"

Cleanup first reads the API model record and requires its model ID, session ID, base model, derived actor name, and active namespace to match. A fixed actor name alone is never treated as ownership proof.

Then stop the service:

GLM52_CONFIRM_LIVE=1 make legacy-service-stop

If this run created a dedicated Ray cluster, keep the same GLM52_RAY_OWNER_TOKEN, then stop workers in reverse order and the head last:

GLM52_CONFIRM_CLUSTER=1 make legacy-ray-stop

legacy-ray-stop loads the ownership journal, verifies the token hash, Ray binary, and every remaining launch marker before changing a node, then checkpoints each reverse-order stop. A missing, mismatched, or unreachable marker aborts without stopping an unverified cluster. Do not use this path for a scheduler-owned cluster.

11. Acceptance order

1. Local contracts pass.
2. Runtime, model revision, service fingerprint, and 32-GPU topology pass preflight.
3. Fresh 32K resident GRPO returns 32 complete rank reports.
4. 32K selected-logprob forward parity passes.
5. 32K gradient and optimizer-delta parity passes all recorded thresholds.
6. The same forward, gradient, and optimizer-delta gates pass at 64K.
7. A fresh 2M execution is bound to the reviewed source hashes.

The commands and artifact layout stay the same when a failed correctness gate is fixed; only the evidence status changes.