Modded-NanoGPT

This repository hosts the NanoGPT speedrun, in which we (collaboratively|competitively) search for the fastest algorithm to use 8 NVIDIA H100 GPUs to train a language model that attains 3.28 cross-entropy loss on the FineWeb validation set.

(Note: Besides the main track, there is also an optimization track where we try to minimize steps subject to fixed arch/data/bsz and with unlimited wallclock budget.)

The target (3.28 validation loss on FineWeb) follows Andrej Karpathy's GPT-2 replication in llm.c, which attains that loss after running for 45 minutes. The speedrun code also descends from llm.c's PyTorch trainer, which itself descends from NanoGPT, hence the name of the repo. Thanks to the efforts of many contributors, this repo now contains a training algorithm which attains the target performance in:

• Under 90 seconds on 8xH100 (the llm.c GPT-2 replication needed 45 minutes)
• under 400M tokens (the llm.c GPT-2 replication needed 10B)

This improvement in training speed has been brought about by the following techniques:

• Modernized architecture: Rotary embeddings, QK-Norm, and ReLU²
• The Muon optimizer [writeup] [repo]
• Use FP8 for head, and asymmetric rescale and softcap logits
• Use FP8 on MLP up projection forward pass
• Initialization of projections to zero (muP-like)
• Skip connections from embedding to every block as well as from block 3 to 6
• Extra embeddings which are mixed into the values in attention layers (inspired by Zhou et al. 2024)
• Flash Attention 3 with long-short sliding window attention pattern (inspired by Gemma 2) and window size warmup with YaRN
• Align training batch starts with EoS and set a max document length
• Accumulate gradients for 2 steps for embedding and lm_head before updating parameters
• Single activation input for last 3 attention layers
• Polar Express implementation in Muon
• Smear module to enable 1 token look back
• Sparse attention gate
• NorMuon
• Cautious Weight Decay w/ schedule tied to LR
• Exponential decay of residual stream
• Batch size schedule
• Max seq length schedule
• Partial Key Offset
• Multi token prediction
• Untie embed and lm_head at 2/3 of training
• Additional gating on value embeddings and skip connection
• Paired head attention
• Bigram hash embedding on 1/4 of model_dim w/ sign trick
• MUDD skip connections to residual stream and attention values
• Learnable XSA

As well as many systems optimizations.

Contributors list (growing with each new record): @bozavlado; @brendanh0gan; @fernbear.bsky.social; @Grad62304977; @jxbz; @kellerjordan0; @KoszarskyB; @leloykun; @YouJiacheng; @jadenj3o; @KonstantinWilleke, @alexrgilbert, @adricarda, @tuttyfrutyee, @vdlad; @ryanyang0, @vagrawal, @classiclarryd, @byronxu99, @varunneal, @EmelyanenkoK, @bernard24/https://www.hiverge.ai/, @Gusarich, @li_zichong, @akash5474, @snimu, @roeeshenberg, @ChrisJMcCormick, @dominikkallusky, @acutkosky, @manikbhandari, @andrewbriand, @jrauvola, @soren_dunn_, @photon_mz, @srashedll, @dhrvji, @EmmettBicker, @dualverse-ai, @sisovicm, @moof2x, @samacqua, @Lisennlp, @_djdumpling, @TrianX

---

Running the current record

To run the current record, run the following commands.

git clone https://github.com/KellerJordan/modded-nanogpt.git && cd modded-nanogpt
pip install -r requirements.txt
# downloads only the first 900M training tokens to save time
python data/cached_fineweb10B.py 9
./run.sh

Add torchrun to path if ./run.sh gives error torchrun: command not found.

Note: torch.compile will add around 7 minutes of latency the first time you run the code.

Official records are timed on 8 NVIDIA H100 GPUs from https://app.primeintellect.ai/. PrimeIntellect has generously sponsored recent validation runs.

Alternative: Running with Docker (recommended for precise timing)

For cases where CUDA or NCCL versions aren't compatible with your current system setup, Docker can be a helpful alternative. This approach standardizes versions for CUDA, NCCL, CUDNN, and Python, reducing dependency issues and simplifying setup. Note: an NVIDIA driver must already be installed on the system (useful if only the NVIDIA driver and Docker are available).

git clone https://github.com/KellerJordan/modded-nanogpt.git && cd modded-nanogpt
sudo docker build -t modded-nanogpt .
sudo docker run -it --rm --gpus all -v $(pwd):/modded-nanogpt modded-nanogpt python data/cached_fineweb10B.py 8
sudo docker run -it --rm --gpus all -v $(pwd):/modded-nanogpt modded-nanogpt sh run.sh

To get an interactive docker, you can use

sudo docker run -it --rm --gpus all -v $(pwd):/modded-nanogpt modded-nanogpt bash

---

World record history

The following is the historical progression of world speed records for the following competitive task:

Train a neural network to ≤3.28 validation loss on FineWeb using 8x NVIDIA H100s.

Note: The 3.28 target was selected to match Andrej Karpathy's GPT-2 (small) reproduction.

#	Record time	Description	Date	Log	Contributors
1	45 minutes	llm.c baseline	05/28/24	log	@karpathy, llm.c contributors
2	31.4 minutes	Tuned learning rate & rotary embeddings	06/06/24	log	@kellerjordan0
3	24.9 minutes	Introduced the Muon optimizer	10/04/24	none	@kellerjordan0, @jxbz
4	22.3 minutes	Muon improvements	10/11/24	log	@kellerjordan0, @bozavlado
5	15.2 minutes	Pad embeddings, ReLU², zero-init projections, QK-norm	10/14/24	log	@Grad62304977, @kellerjordan0
6	13.1 minutes	Distributed the overhead of Muon	10/18/24	log	@kellerjordan0
7	12.0 minutes	Upgraded PyTorch 2.5.0	10/18/24	log	@kellerjordan0
8	10.8 minutes	Untied embedding and head	11/03/24	log	@Grad62304977, @kellerjordan0
9	8.2 minutes	Value and embedding skip connections, momentum warmup, logit softcap	11/06/24	log	@Grad62304977, @kellerjordan0
10	7.8 minutes	Bfloat16 activations	11/08/24	log	@kellerjordan0
11	7.2 minutes	U-net pattern skip connections & double lr	11/10/24	log	@brendanh0gan
12	5.03 minutes	1024-ctx dense causal attention → 64K-ctx FlexAttention	11/19/24	log	@KoszarskyB
13	4.66 minutes	Attention window warmup	11/24/24	log	@fernbear.bsky.social
14	4.41 minutes	Value Embeddings	12/04/24	log	@KoszarskyB
15	3.95 minutes	U-net pattern value embeddings, assorted code optimizations	12/08/24	log	@leloykun, @YouJiacheng
16	3.80 minutes	Split value embeddings, block sliding window, separate block mask	12/10/24	log	@YouJiacheng
17	3.57 minutes	Sparsify value embeddings, improve rotary embeddings, drop an attn layer	12/17/24	log	@YouJiacheng
18	3.4 minutes	Lower logit softcap from 30 to 15	01/04/25	log	@KoszarskyB
19	3.142 minutes	FP8 head, offset logits, lr decay to 0.1 instead of 0.0	01/13/25	log	@YouJiacheng
20	2.992 minutes	Merged QKV weights, long-short attention, attention scale, lower Adam epsilon, batched Muon	01/16/25	log	@leloykun, @fernbear.bsky.social, @YouJiacheng, @brendanh0gan, @scottjmaddox, @Grad62304977
21	2.933 minutes	Reduced batch size	01/26/25	log	@leloykun
21	2.997 minutes	21st record with new timing	02/01/25	log	not a new record, just re-timing #21 with the updated rules
21	3.014 minutes	21st record with latest torch	05/24/25	log	not a new record, just re-timing #21 with latest torch
22	2.990 minutes	Faster gradient all-reduce	05/24/25	log	@KonstantinWilleke, @alexrgilbert, @adricarda, @tuttyfrutyee, @vdlad; The Enigma project
23	2.979 minutes	Overlap computation and gradient communication	05/25/25	log	@ryanyang0
24	2.966 minutes	Replace gradient all_reduce with reduce_scatter	05/30/25	log	@vagrawal
25	2.896 minutes	Upgrade PyTorch to 2.9.0.dev20250713+cu126	07/13/25	log	@kellerjordan0
26	2.863 minutes	Align training batch starts with EoS, increase cooldown frac to .45	07/13/25	log	@classiclarryd
27	2.817 minutes	Transpose one of the MLP matrices + add Triton kernel for symmetric matmul	07/18/25	log,PR	@byronxu99
28	2.812 minutes	Sparse attention gate	08/23/25	log,PR	@classiclarryd
29	2.731 minutes	Flash Attention 3, 2048 max_doc_len, update ws schedule	09/03/25	log,PR	@varunneal
30	2.717 minutes	Drop first MLP layer	09/05/25	log,PR	@EmelyanenkoK
31	2.656 minutes	Dynamically incorporate YaRN during training and validation	09/10/25	log,PR	@classiclarryd
32	2.625 minutes	Optimize distributed training, improve skip connection gating, and enhance bfloat16 usage	09/11/25	log,PR	@bernard24 & AI system hiverge.ai
33	2.565 minutes	Asynchronously fetch and index data batches, extend final layer attention window for validation	09/15/25	log,PR	@classiclarryd
34	2.547 minutes	Smear token embeddings 1 position forward	09/18/25	log,PR	@classiclarryd
35	2.527 minutes	Drop first attn layer, extend all long windows for validation, update schedule	09/21/25	log,PR	@classiclarryd
36	2.495 minutes	MuonCustomSizing, perform mlp and attn reduce scatter in shared call	09/23/25	log,PR	@classiclarryd
37	2.483 minutes	Compute cross entropy in BF16 during training	09/27/25	log,PR	@Gusarich
38	2.476 minutes	Polar Express, replacement for Newton-Schulz	09/29/25	log,PR	@varunneal
39	2.447 minutes	Only update Adam params every other step, reduce batch size	09/30/25	log,PR	@classiclarryd
40	2.358 minutes	Backout, misc hyperparameter tuning, optimize lambda padding	10/04/25	log,PR	@classiclarryd
41	2.345 minutes	NorMuon	10/24/25	log,PR	@li_zichong
42	2.313 minutes	Update NorMuon LR, Step Logic	10/27/25	log,PR	@varunneal
43	2.284 minutes	Cautious Weight Decay w/ schedule	11/10/25	log,PR	@varunneal
44	2.269 minutes	Backward hooks on Adam, Profiling 101	11/16/25	log,PR	@akash5474
45	2.248 minutes	Refine skip arch, update exponential decay init	11/18/25	log,PR	@classiclarryd
46	2.203 minutes	Batch size schedule	11/29/25	log,PR	@varunneal
47	2.193 minutes	Multiply attn lambda with weight instead of data, fix warmup	12/10/25	log,PR	@roeeshenberg
48	2.170 minutes	Speed up Muon, additional pre-multiply lambda, reshape matrices, update lr, update NorMuon axis	12/11/25	log,PR	@ChrisJMcCormick
49	2.146 minutes	Partial Key Offset	12/14/25	log,PR	@classiclarryd
50	2.128 minutes	Extend Cautious Weight Decay to Adam parameters	12/18/25	log,PR	@roeeshenberg
51	2.075 minutes	Retie Embed to lm_head, retune fp8 scales	12/19/25	log,PR	@varunneal
52	2.037 minutes	Smooth scalars via beta increase, decrease smear gate lr, freeze scalars during transitions, adam all reduce	12/21/25	log,PR	@ChrisJMcCormick
53	1.988 minutes	Multi-token prediction, untie embed/lm_head at 2/3 training, lr update, tweak CWD	12/22/25	log,PR	@varunneal, feat. @classiclarryd
54	1.940 minutes	Asymmetric Logit Rescale	12/26/25	log,PR	@classiclarryd
55	1.918 minutes	Gates on value embeds and skip connection	12/29/25	log,PR	@classiclarryd
56	1.894 minutes	Optimize and compile Adam, increase Adam buffer precision, move gates from Muon to Adam parameter banks	12/31/25	log,PR	@ChrisJMcCormick
57	1.878 minutes	Bfloat16 attn/mlp weights, mixed precision Muon, interweave Adam/Muon, finer-grain Adam beta	01/04/26	log,PR	@classiclarryd, feat. @YouJiacheng, @ChrisJMcCormick
58	1.820 minutes	Paired Head Attention	01/07/26	log,PR	@classiclarryd
59	1.781 minutes	Fused triton kernel for linear relu square MLP step	01/10/26	log,PR	@andrewbriand8, @Joshrav21
60	1.765 minutes	Fused triton kernel for softcapped multi-token prediction cross entropy step	01/16/26	log,PR	@soren_dunn_ & AI System Locus
61	1.748 minutes	Unified Optimizers and Transposed LM Head	01/18/26	log,PR	@ChrisJMcCormick
62	1.655 minutes	Bigram Hash Embedding	01/19/26	log,PR	@classiclarryd
63	1.650 minutes	Untie Value Embeds	01/26/26	log,PR	@photon_mz
64	1.630 minutes	Tuned nonzero Attn V and O init	01/30/26	log,PR	@srashedll
65	1.613 minutes	Group Value Embeds into single parameter	01/30/26	log,PR	@varunneal
66	1.595 minutes	Torch 2.10	01/31/26	-	-
67	1.540 minutes	Tune fused softcap kernels and fuse fp8 quantization in LM head	01/31/26	log,PR	@andrewbriand8
68	1.535 minutes	Move bigram hash to GPU	01/31/26	log,PR	@dhrvji
69	1.528 minutes	Kernel Optimizations	02/02/26	log,PR	@EmmettBicker & AI System Aster
70	1.521 minutes	Tune value embed layout and ve_gates	02/03/26	log,PR	@photon_mz
71	1.516 minutes	Sparse bigram gradient comms and optimized loading on CPU	02/06/26	log,PR	@roeeshenberg
72	1.496 minutes	Increase minimum lr and add max_seq_len schedule	02/10/26	log,PR	@dualverse-ai & AI System Station
73	1.485 minutes	Partitioned Hyperconnections	02/12/26	log,PR	@sisovicm
74	1.468 minutes	Flattened GPT forward, removed post attention lambdas, added transpose kernels	02/16/26	log,PR	@ChrisJMcCormick
75	1.453 minutes	Cross Entropy Kernel Optimizations	02/23/26	log,PR	@moof2x
76	1.446 minutes	Reuse and tune backward transpose kernel	02/28/26	log,PR	@samacqua
77	1.435 minutes	Replace partitioned hyperconnections with single saved activation	03/06/26	log,PR	@classiclarryd
78	1.426 minutes	Tighten bounds on fa3 max_num_docs to match fineweb distribution	03/22/26	log,PR	@ChrisJMcCormick
79	1.411 minutes	Fuse Cross Entropy Fwd/Bwk Kernel, to avoid recalc on softcap sigmoid	04/04/26	log,PR	@andrewbriand8
80	1.406 minutes	In Muon orthogonize Q and K matrices in pairs of heads, instead of across the full 6 head matrix	04/08/26	log,PR	@samacqua
81	1.363 minutes	MUDD Skip Connections	04/22/26	log,PR	@Lisennlp
82	1.353 minutes	Learnable XSA	04/29/26	log,PR	@_djdumpling
83	1.328 minutes	Sign Trick on Bigram Embed	05/20/26	log,PR	@TrianX
84	1.320 minutes	FP8 on MLP up-projection forward pass	05/21/26	log,PR	@sisovicm

Rules

New records must:

1. Not modify the train or validation data pipelines. (You can change the batch size, sequence length, attention structure etc.; just don't change the underlying streams of tokens.)
2. Attain ≤3.28 mean val loss. (Due to inter-run variance, submissions must provide enough run logs to attain a statistical significance level of p<0.01 that their mean val loss is ≤3.28. Example code to compute p-value can be found here. For submissions which improve speed by optimizing the systems performance, without touching the ML, this requirement is waived.)
3. Not use any extra torch._inductor.config or torch.compile flags. (These can save a few seconds, but they can also make compilation take >30min. This rule was introduced after the 21st record.)
4. Run faster than the prior record when baselined on the same hardware.

Discretionary reasons why a PR may not be accepted:

1. Disproportionately degrades the readability of the codebase. A 200 line kernel to drop 300ms is considered worthwhile. 500 lines that convolute the optimizer layout for a 50ms gain will likely be rejected.
2. The current record is intentionally kept roughly 0.001-0.002 loss below 3.28 to make validation simpler. If a PR substantially consumes this buffer, it should do so in a way that outperforms a simple step count decrease, when measured at equivalent loss.

Note: torch._inductor.config.coordinate_descent_tuning is allowed for GPT-2 Medium track (a.k.a. 2.92 track).

Other than that, anything and everything is fair game!

further clarifications

---

Comment on the target metric

The target metric is cross-entropy loss on the FineWeb val set. To speak mathematically, the goal of the speedrun is *to obtain a probability model of language which assigns a probability of at least math.exp(-3.28 * 10485760) to the first 10,485,760 tokens of the FineWeb valset. Hence, e.g., we allow evaluation at any sequence length, so long as we still have a valid probability model of language.

---

Timing change after record 21

After the 21st record, we made two changes to the timing. First, there used to be an initial "grace period" of 10 untimed steps to allow kernel warmup. We replaced this with an explicit kernel-warmup section which is untimed and uses dummy data. This results in an extra runtime of 850ms from the 10 extra timed steps. Second, we banned the use of torch._inductor.config.coordinate_descent_tuning. This saves ~25min of untimed pre-run compilation, but results in an extra runtime of ~3s.

---

Notable attempts & forks

Notable runs:

• @alexjc's 01/20/2025 2.77-minute TokenMonster-based record. This record is technically outside the rules of the speedrun, since we specified that the train/val tokens must be kept fixed. However, it's very interesting, and worth including. The run is not more data-efficient; rather, the speedup comes from the improved tokenizer allowing the vocabulary size to be reduced (nearly halved!) while preserving the same bytes-per-token, which saves lots of parameters and FLOPs in the head and embeddings.
• @samacqua's 1/23/2026 test time training run. Sam found that prediction accuracy on the later portions of a given document could be improved by performing a training update on Adam parameters based on the early portion of the document. This 'parameter nudging' is repeated independently for each document. Interestingly, these gradient updates prove effective while only using ~500 tokens, substantially less than the over 200k tokens typically used on a normal training step. While technically a valid probability model, we are not allowing untimed backward passes.

Notable forks:

• https://github.com/BlinkDL/modded-nanogpt-rwkv
• https://github.com/nikhilvyas/modded-nanogpt-SOAP

---

Speedrun track 2: GPT-2 Medium

The target loss for this track is lowered from 3.28 to 2.92, as per Andrej Karpathy's 350M-parameter llm.c baseline. This baseline generates a model with performance similar to the original GPT-2 Medium, whereas the first track's baseline generates a model on par with GPT-2 Small. All other rules remain the same.

Note: torch._inductor.config.coordinate_descent_tuning is turned on after the record 6 (*).

#	Record time	Description	Date	Log	Contributors
1	5.8 hours	llm.c baseline (350M parameters)	05/28/24	log	@karpathy, llm.c contributors
2	29.3 minutes	Initial record based on scaling up the GPT-2 small track speedrun	01/18/25	log	@kellerjordan0
3	28.1 minutes	Added standard weight decay	02/08/25	log	@kellerjordan0
4	27.7 minutes	Tuned Muon Newton-Schulz coefficients	02/14/25	log	@leloykun
5	27.2 minutes	Increased learning rate cooldown phase duration	03/06/25	log	@YouJiacheng
6	25.95 minutes*	2x MLP wd, qkv norm, all_reduce/opt.step() overlap, optimized skip pattern	03/25/25	log	@YouJiacheng
7	25.29 minutes	Remove FP8 head; ISRU logits softcap; New sharded mixed precision Muon; merge weights	04/16/25	log	@YouJiacheng
8	24.50 minutes	Cubic sliding window size schedule, 2× max window size (24.84 minutes) 24.5min repro	04/22/25	log	@jadenj3o
9	24.12 minutes	Add two value embeddings	08/28/25	log, PR	@snimu
10	24.07 minutes	Second input embedding	09/11/25	log, PR	@snimu
11	23.45 minutes	Upgrade from torch 2.7 to torch==2.10.0.dev20251210+cu126	-	-	-
12	23.28 minutes	Snoo Optimizer (Outer optimizer around Adam and Muon)	09/16/25	log,PR	@dominikkallusky
13	23.14 minutes	EMA Wrapper on Muon	09/17/25	log,PR	@acutkosky
14	23.08 minutes	Combine both records 12 & 13	09/30/25	log,PR	@acutkosky
15	23.03 minutes	Backout (Skip from 2/3 point to pre-lm_head)	10/04/25	log,PR	@snimu
16	22.99 minutes	Smear-MTP	11/02/25	log,PR	@snimu
17	22.98 minutes	Remove Redundant Mask Op	11/12/25	log,PR	@manikbhandari
18	17.35 minutes	Bulk transfer short track features	12/31/25	log,PR	-

---

Q: What is the point of NanoGPT speedrunning?

A: The officially stated goal of NanoGPT speedrunning is as follows: gotta go fast. But for something a little more verbose involving an argument for good benchmarking, here's some kind of manifesto, adorned with a blessing from the master. https://x.com/karpathy/status/1846790537262571739

Q: What makes "NanoGPT speedrunning" not just another idiosyncratic benchmark?

A: Because it is a competitive benchmark. In particular, if you attain a new speed record (using whatever method you want), there is an open invitation for you to post that record (on arXiv or X) and thereby vacuum up all the clout for yourself. I will even help you do it by reposting you as much as I can.

"Artificial intelligence advances by inventing games and gloating to goad others to play" - Professor Ben Recht

Q: NanoGPT speedrunning is cool and all, but meh it probably won't scale and is just overfitting to val loss

A: This is hard to refute, since "at scale" is an infinite category (what if the methods stop working only for >100T models?), making it impossible to fully prove. Also, I would agree that some of the methods used in the speedrun are unlikely to scale, particularly those which impose additional structure on the network, such as logit softcapping. But if the reader cares about 1.5B models, they might be convinced by this result:

Straightforwardly scaling up the speedrun (10/18/24 version) to 1.5B parameters yields a model with GPT-2 (1.5B)-level HellaSwag performance 2.5x more cheaply than @karpathy's baseline ($233 instead of $576):

[reproducible log]

---

Muon optimizer

Muon is defined as follows:

Where NewtonSchulz5 is the following Newton-Schulz iteration [2, 3], which approximately replaces G with U @ V.T where U, S, V = G.svd().

@torch.compile
def zeroth_power_via_newtonschulz5(G, steps=5, eps=1e-7):
    assert len(G.shape) == 2
    a, b, c = (3.4445, -4.7750,  2.0315)
    X = G.bfloat16() / (G.norm() + eps)
    if G.size(0) > G.size(1):
        X = X.T 
    for _ in range(steps):
        A = X @ X.T
        B = b * A + c * A @ A
        X = a * X + B @ X
    if G.size(0) > G.size(1):
        X = X.T 
    return X.to(G.dtype)

For this training scenario, Muon has the following favorable properties:

• Lower memory usage than Adam
• ~1.5x better sample-efficiency
• <2% wallclock overhead

Provenance

Many of the choices made to generate this optimizer were obtained experimentally by our pursuit of CIFAR-10 speedrunning. In particular, we experimentally obtained the following practices:

• Using Nesterov momentum inside the update, with orthogonalization applied after momentum.
• Using a specifically quintic Newton-Schulz iteration as the method of orthogonalization.
• Using non-convergent coefficients for the quintic polynomial in order to maximize slope at zero, and thereby minimize the number of necessary Newton-Schulz iterations. It turns out that the variance doesn't actually matter that much, so we end up with a quintic that rapidly converges to the range 0.68, 1.13 upon repeated application, rather than converging more slowly to 1.
• Running the Newton-Schulz iteration in bfloat16 (whereas Shampoo implementations often depend on inverse-pth-roots run in fp32 or fp64).

Our use of a Newton-Schulz iteration for orthogonalization traces to Bernstein & Newhouse (2024), who suggested it as a way to compute Shampoo [5, 6] preconditioners, and theoretically explored Shampoo without preconditioner accumulation. In particular, Jeremy Bernstein @jxbz sent us the draft, which caused us to experiment with various Newton-Schulz iterations as the orthogonalization method for this optimizer. If we had used SVD instead of a Newton-Schulz iteration, this optimizer would have been too slow to be useful. Bernstein & Newhouse also pointed out that Shampoo without preconditioner accumulation is equivalent to steepest descent in the spectral norm, and therefore Shampoo can be thought of as a way to smooth out spectral steepest descent. The proposed optimizer can be thought of as a second way of smoothing spectral steepest descent, with a different set of memory and runtime tradeoffs compared to Shampoo.

---

Running on fewer GPUs

• To run experiments on fewer GPUs, simply modify run.sh to have a different --nproc_per_node. This should not change the behavior of the training.
• If you're running out of memory, you may need to reduce the sequence length for FlexAttention (which does change the training. see here for a guide)

---

References

1. Guilherme Penedo et al. "The fineweb datasets: Decanting the web for the finest text data at scale." arXiv preprint arXiv:2406.17557 (2024).
2. Nicholas J. Higham. Functions of Matrices. Society for Industrial and Applied Mathematics (2008). Equation 5.22.
3. Günther Schulz. Iterative Berechnung der reziproken Matrix. Z. Angew. Math. Mech., 13:57–59 (1933).
4. Jeremy Bernstein and Laker Newhouse. "Old Optimizer, New Norm: An Anthology." arxiv preprint arXiv:2409.20325 (2024).
5. Vineet Gupta, Tomer Koren, and Yoram Singer. "Shampoo: Preconditioned stochastic tensor optimization." International Conference on Machine Learning. PMLR, 2018.
6. Rohan Anil et al. "Scalable second order optimization for deep learning." arXiv preprint arXiv:2002.09018 (2020).
7. Alexander Hägele et al. "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations." arXiv preprint arXiv:2405.18392 (2024).
8. Zhanchao Zhou et al. "Value Residual Learning For Alleviating Attention Concentration In Transformers." arXiv preprint arXiv:2410.17897 (2024).
9. Team, Gemma, et al. "Gemma 2: Improving open language models at a practical size." arXiv preprint arXiv:2408.00118 (2024).
10. Alec Radford et al. "Language models are unsupervised multitask learners." OpenAI blog 1.8 (2019).

Citation

@misc{modded_nanogpt_2024,
  author       = {Keller Jordan and Jeremy Bernstein and Brendan Rappazzo and
                  @fernbear.bsky.social and Boza Vlado and You Jiacheng and
                  Franz Cesista and Braden Koszarsky and @Grad62304977},
  title        = {modded-nanogpt: Speedrunning the NanoGPT baseline},
  year         = {2024},
  url          = {https://github.com/KellerJordan/modded-nanogpt}
}