224.md

June 9, 2026 · View on GitHub

Car Gonzalez: All right. And we're back. Episode five, five, five.
Christopher: How's it going? We're just... the graphics are only going to get better and better. Okay?
Car Gonzalez: Yeah.
Christopher: So "Distributed Training 101"... in this episode we're doing a little like mini-seminar on what the heck distributed training is, because we're starting our training runs today, and we're going to be talking about it, we're going to be getting into ML and details and let's all kind of follow along. Now, let's start with just a recap here. Hey, remember four days ago when I was like, "Hey, maybe we'll hit a million this week"? Buh... Bingo. What's that right there? Yes.
Car Gonzalez: Oops. One oh seven. Oh no, that's a million. That's a million sats.
Christopher: Yeah, so we hit the mil-sat, million sats. I like seeing the numbers go up. I like seeing the numbers go up.
Car Gonzalez: Over, over thirteen hundred pylons.
Christopher: Yeah. Some percentage of these are probably people running multiple, which is fine. Like, we want people to game the system. You know, we want you to maximize your earnings. And we'll just be tweaking them. So as of today, we're no longer paying you for being online. We're going to be paying you for the real work that we're sending to your pylon. So in this video, we're going to go into, like, what we're actually sending, why we're doing this, some basics of distributed training. So one like little piece of analysis before we get going... our friend Daniel, a great Bitcoiner, miner, environmental guy... you know, he quote-tweets and says, "Yet another way to acquire more Bitcoin. You can buy it, you can mine it, now you can sell your compute power for it and be paid in Bitcoin." Hey, that's us.
Car Gonzalez: I mean, let's just talk about Daniel for a second. This guy knows about min-... like he literally knows... he literally knows everything about decentralized mining as far as I know, as far as the people I've talked to before. I mean, could you do a little talk about, like, his background and stuff, Christopher?
Christopher: Uh, Daniel's just the main dude countering, like, Bitcoin's environmental FUD and how, like, mining can be this huge force for cleaning up the environment. Okay, we'll do a separate video on that. In fact, we'll probably get Daniel on the show at some point.
Car Gonzalez: Yeah.
Christopher: Um, so one more piece of comment here... so, you know, this AI skeptic guy Mo, developer, has some good posts and he was saying how you've got these big AI models that are trained on, like, the content of the internet, and as a result you get them giving you, like, average "trendslop." A tendency to recommend whatever sounds good on LinkedIn instead of what actually works for your situation. Because the models are trained on the entire internet: every Reddit post, every TED Talk, every Medium article from a guy with eleven subscribers. They don't reason, they regurgitate the most popular opinions in the most convincing voice possible. We thought AI was going to make everyone a genius, instead it's pulling smart people toward the middle. Now, is there a solution for this? Do we have to accept that the way that the big labs have trained their models is the only way that you can do it? No. So imagine new models that are not trained on Reddit and LinkedIn midwittery. Why are you paying more for the big labs' firehose of slop when you can pay a few Bitcoin sats for a fine-tuned scalpel? Imagine lightweight models that keep learning over time because contributors of better data earn Bitcoin. Models should continually learn and improve, and they will, using our approximately limitless pool of "fracked compute" - that twenty gigawatts we showed you a couple episodes ago. We think this idea of continually improving small laser-focused models on top of this retail compute is the basis for a new kind of frontier lab that's going to knock the big guys on their ass. Phase one starts today. Stay tuned for... hey, this exact episode. You're in the right place.
Car Gonzalez: I think the biggest thing that most people aren't thinking about, there's really only like just us and Tau, right? Is that the only other player that's doing this?
Christopher: Prime Intellect has done some training runs. They did one pre-train, one kind of fine-tuned stuff. They've... they've got some decent stuff. Again, there's plenty of these projects that are doing some decent work, all attached to their shitcoin, and we're just going to be harvesting a lot of these ideas en masse and bringing them to Bitcoin. Um, so for the introductory to, like, the base... we're going to do, like, the basics of training and then basics of distributed training. And um, for some of the training stuff, I'm going to show you some of the slides from this gentleman, Nathan Lambert, researcher at Allen AI. He's been talking for a while about "we need an American DeepSeek," and um, I love Nathan and have appreciated what they've been trying to do at Allen, but Allen's one of these companies that just got their core team raided by Microsoft and they no longer have the funding to train models like they were trying to do. Um, but hopefully I-... we-... I hope to offer to the open-source community, like, a model that actually works economically and a pipeline that's ours and, you know, can kind of grow with the network. But we're going to be taking good ideas from brilliant people like this and putting them in. There's our beautiful graph, my favorite graph. So we got over here... the first few slides I'm going to walk through are from this slide presentation from Nathan. He's got a lot of great content on his website and stuff. "Introduction to reinforcement learning from human feedback and post-training." So you've got pre-training and post-training. Um, but let's just step through this real quick because this kind of goes into the basics of... very basic stuff. Okay. So what is a language model? A language model assigns probabilities to text. Chunks of words are broken down as "tokens," which are the internal representation of the model. Given previous tokens, it predicts the next token. Repeating this produces a completion one step at a time. This is called "auto-regressive." That is the original model architecture diagram for the transformer. So pre-training: next-token prediction. You train on trillions of tokens of text from the web, books, code, documents. That's going to be our opportunity later on as we're collecting better data sources to, like, put together our own collections that are opinionated in the way we want them to be. Models are often trained on between five to fifty trillion tokens. One trillion of text tokens is about three to five terabytes of data. Labs gather and filter ten to twenty times more data than is used for the model. Total data funnel targeted for model is on the order of petabytes. Objective is to predict the next token in each sequence. This resulted in incredible, flexible, useful models. Uh, the base model... so usually there's a base model and then there's like a chat-tuned model. The base model is just a completion, auto-complete model, completes text.
Car Gonzalez: Oh cool.
Christopher: "The president of the United States was"... this isn't even fully... oh yeah, oh yeah, I see. "...was George W. Bush, comma, the governor of Florida." So it just kind of completes what you're saying. Then you do post-training to make it answer like a chatbot. The earliest forms of modern post-trained (or reinforcement learning from human feedback tuned) models shifted the continuation format to always conforming to the "answering a question" style. An example of what early conversational models looked like was below: "User: The president was..." and then it kind of puts it into actual sentences. One like fun thing to think about is what are other ways? Everyone's sort of like agreed on this sort of like user assistant thing, but there's potentially all sorts of other ways that text could be generated. Um, building up our own pipeline where we have full control over this entire process is going to let us experiment with a whole bunch of stuff. Stop taking all of the, you know, accumulated best practices as givens and try different stuff. Okay. Can you tell we're into our own slides here? Okay. What is training? So training a model means repeatedly improving its parameters by learning from examples. The core loop is: start with a model with parameters, feed it training examples, it makes predictions, compare the prediction to the target, you measure the error as "loss" (you'll see loss curves that kind of like go down, you want the loss curves to go down), update parameters to reduce future loss, save checkpoints so progress can resume. Those checkpoints can be distributed up and kind of like saved to central storage buckets, but multiple different devices can contribute to checkpoints. So result: training is repeated error measurement plus parameter updates. The training loop: one step of training looks like this. A single step is: the dataset provides a batch of examples; the forward pass is the model predicts outputs; the loss, you're comparing the predictions to the correct targets; the backward pass computes which parameters caused the error; optimizer step adjusts parameters. Repeat millions or billions of times. Checkpointing saves progress so a run can survive interruption. Why distributed training exists: one machine can only do so much - limited memory, limited compute, limited throughput, limited wall-clock speed. As models and datasets get bigger, one machine is not enough, so we split work across many devices. Those devices must stay coordinated. Distributed training is how many machines act like one bigger training system. What does distributed training actually mean? You break a training job into pieces, assign work to multiple workers, each worker computes part of the job, merge or synchronize progress, keep the run moving together. Minimum concepts: multiple workers, clear work assignment, synchronization, aggregation, checkpoint and resume, dropout and recovery. The hard part is keeping the system working when real machines fail, lag, or disappear. Centralized versus distributed training. Centralized training: one controlled cluster, one operator, predictable network, uniform hardware. You know, renting out a cup-... crap-ton of GPUs in one datacenter. Distributed training: many different machines, not all in one datacenter, not equally strong, not all under one operator, must tolerate dropouts and recovery. Distributed training asks whether independent machines can make real progress on the same run. What does the network have to agree on? Shared facts for a distributed run: which run they're on, which window or assignment they're working on, which checkpoint is current, how progress is merged, who contributed valid work, how recovery happens after failure. In open networks, you also need validation, public proof, contribution accounting. Uh, some of the other kind of crypto projects have attached things to blockchains and done stuff with tokens.
Car Gonzalez: Yeah.
Christopher: Um, you don't need those things to solve these problems. Reputation systems, there's other ways of solving this. Bla bla bla. Okay, so Psionic is our ML framework, and a reminder that we are essentially taking all of the Python ecosystem and rewriting it in Rust. So we've ported a bunch of inference code over from llama.cpp, over from legacy local-runtime lane. We've got open models running inference, and we've repeated that process for Prime Intellect, Bittensor, Templar, the top distributed training things. So like a bunch of the other projects we've already solved a lot of the things we just read about. Um, we have their code, we ported it into Psionic. That's what is inside the pylons that are now running on your computer, that's what we're about to be pushing out this training code to in a couple hours. So Psionic is the part of the system that runs the training work. It takes the run instructions, it runs the job on the machine, it saves the checkpoint and the result, it gives the rest of the network a clear record of what happened. That matters because we can check what happened instead of just saying it happened. Okay, so today, what are we doing today? Um, there is a class, an open-source class at Stanford called something like "pre-training a model from scratch." I've got it here somewhere. Here we go. Uh, so they taught this last year and all the coursework is online, and then they're doing right now an updated version of that, they're putting videos and stuff online. They're on, like, second or third class. So language models from scratch. Um, what we're going to be doing over the next however long it takes us, a week or two, we're going to be going through the assignments. To give an example, Assignment 1: get the basic pipeline working. Implement BPE tokenizer, transformer architecture, Adam optimizer, train models on TinyStories and OpenWebText using PyTorch primitives. So as they step you through, you know, Stanford absolute top of the line best possible instruction of how you go from nothing to by the end of the course being familiar with how the big labs are actually running pre-training and post-training pipelines in production, the billion-dollar labs... like, here's the gentle introduction to this, building things up from scratch. We are going to be doing this exact same flow, one assignment maybe every few days, maybe we'll do one episode per assignment, we'll see how fast we get through it all. Again, we're not using PyTorch, we're using Psionic. So we've already gone ahead and ported all of the Assignment number one code into Psionic. Later today hopefully, maybe tomorrow, you're going to see a dashboard that looks a little bit like this. This is the kind of, like, before it runs, we're still fixing some treasury stuff.
Car Gonzalez: So if we're already running pylon, how do we update? Is it still an npm update?
Christopher: You're going to close it out and re-run it. It'll download the most updated binary. I think the updated version you'll need is version zero point one. Um, so by the time you're watching this video, that will be available. If you're running a pylon now, just close it, re-run it, it'll download the new binary. New binary will be giving you little pieces of homework. So we're all going to kind of cheat on our homework. Wouldn't you love to get paid Bitcoin to do your homework? Okay.
Car Gonzalez: I mean, it was great getting paid Bitcoin just to run it, but now...
Christopher: Well, now we're not just paying you for nothing, now we're paying you for... you're going to do our homework. Okay.
Car Gonzalez: Well, there's over a thousand people, so it's going to be really interesting to see how many people move over to this, move the switch now. This is where the rubber meets the road as they say.
Christopher: Yeah. I mean, we're not going to pay you for just being online anymore. You have to install the new version. Okay, so Assignment 1: get basic pipeline learning; Assignment 2: implement Flash Attention in Triton, distributed data parallel plus optimizer sharding; scaling laws... so we've got like a bunch of this stuff in Psionic now, some of it we're going to be kind of like adding as we go live. And then um, doing this all in like a distributed way... where is my...
Car Gonzalez: Your slides?
Christopher: Slides? Yeah.
Car Gonzalez: Slides were just back there.
Christopher: Where? Here? Oh there we go. It's all the same button. Okay, so here's how the system overall works. An admin starts the run. For now, I'm going to be like "do a training run." How long does it take? Okay, we've got this amount of work. Machine picks up the work, Psionic runs the training job, machine sends result back, public page can show what happened. Um, if something is rough or missing, that should be shown too. Okay, so DiLoCo... so DiLoCo is a simple idea for distributed training: let machines do a lot of work locally then sync once in a while instead of talking every step. Basically Google DeepMind a couple years ago introduced DiLoCo, which... do we even know what that stands for? DiLoCo...
Car Gonzalez: And for the people that are asking like, "Oh can I still run this and still get work done?" Like I literally just been on a MacBook Air over there, M1, running pylon, and then it's also running Xcode at the same time. So I think I don't-... we'll see what it looks like today, but so far it's no hiccups.
Christopher: Yeah, so for the crappy devices, if you can't do meaningful gradient descent work then you may just be getting assigned validator work. Basically the goal is for every device to be able to do valid work, but we're going to see exactly what types of machines can do which jobs. We don't know, we're going to find that out through experience. So DeepMind put out a paper in 2024 called DiLoCo, "Distributed Low-Communication Training of Language Models," laying out like the overall architecture. This is the paper that Prime Intellect, Bittensor... everyone's basically doing DiLoCo or DiLoCo-class training runs. Um, and then like we mentioned in the last video, you've got, you know, the current largest decentralized training run, DiLoCo-class training run in the world, a record held by Covenant's Bittensor that just left the Bittensor network. Um, they had seventy participants. Why only seventy? Because they required you do eight-by-B200s, like datacenter-class GPUs. You had to have that in order to participate. Um, we figured out and sort of validated on our own machine that we can do this homework, we can train language models using MacBook M2, MacBook M5, my Nvidia 4080. Because that is what comprises this twenty gigawatts of compute over here. The spare compute that no one's using, it's the gaming computers, it's the Macs that are sitting on everybody's computers that right now the price of this, like what people are paying for this compute is zero. No one's paying for this. So if we come in and say we will pay you a little bit more than zero for this compute, then we have access to all this. Because obviously if you've got just compute sitting there and we say we'll pay you something for it, you're going to pay it to us. Now, we hope to use as much of this compute as we can to train our own models, but we also want to just build the system and the framework and everything that's specialized in this. And then if some other lab, some other company, some other anybody wants to train a model, eventually this will be a completely like self-serve platform, open protocol, you can just kind of like tap into, pay and go. In the beginning it'll probably be like we'll make a deal with someone who wants to... I want to make a... here's ten grand, here's a hundred grand, I'm going to do these different training runs, but all of that money will take a little cut... all of that money's going to flow out to you guys. Um, and so even though you're only getting paid maybe a little bit to start, we're building a market here. So there should be a market for this compute. We're bringing this compute to market.
Car Gonzalez: And and think of all the countries that don't have access to this type of compute, right? Like the people that are in, you know, less privileged communities and stuff like that that could also potentially use and leverage this stuff, especially in like a community setting. Um, it's really bullish, man.
Christopher: So Prime figured out how to do like the joins and the leaves and the checkpoints and the recovery. Templar added public participation, scoring and validator checks, using their token crap. But anyway, the basic lesson is: let machines do useful local work, then sync it later. Bla bla bla. So on our like stats page, you'll be able to see which run is happening, what machines are taking part, what work they've completed, what results they've reported. Uh, right now a lot of people are asking us questions about, like, "What can my device do?" or "How much can I earn?" Like, we don't know, but we're going to be surfacing all of the data and so we're going to be like learning from it. And then we'll have APIs or something that you can have your agents like crawl it and say, you know, "What can my machine expect to earn given that public data?" So we're doing this all in public. We don't have the answers but we know how to get the answers. Uh, why this matters: the point is simple - real machines can do real training work together, that work can be shown publicly, contributors should be paid for useful work. We're starting small, we're kind of like eating our vegetables, building up processes that we control fully from scratch, and then let market forces take over and get you paid for it. Makes sense?
Car Gonzalez: Yeah, it's proof of work. I saw a tweet by Jason earlier, he was asking about AI founders now now actually sharing their work and I'm like, "Yeah, we do that in Bitcoin. It's called proof of work." And OpenAgents is doing it.
Christopher: 100%. Okay, so um, if you're watching this video, join our training run. Go to openagents.com, click this button, paste the instructions to your pylon agent, you'll join, it'll set up your wallet, bla bla bla. Make these numbers go up.
Car Gonzalez: Also jump in the Discord too. There's there's conversations happening in there, so if you have any problems, there's people like myself all the time.
Christopher: Yeah, we'll add a better link to that. But down here bottom right, you can click Discord. And uh, what else to say? We'll also be linking to the stats page from here. Bla bla bla. Okay, see ya soon!