Transcription: OpenAgents Episode 145 - Going Local

Source: https://x.com/OpenAgentsInc/status/1867815868131836103 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 145: Going Local Onyx now runs Llama 3.2 3B locally on your ... Upload date: 20241214 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T15:12:40Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: Sometimes you want to use an API to chat with an AI model. Other times you want to do it locally. So we can now do that in two ways. We can use the latest Llama models, either on the device itself, Onyx, or via the MCP for the larger models. So for example, I'm going to go ahead and load here the new Llama 3.2, a 3 billion parameter model, which downloads directly to the device. Hello, who are you? Hello, I'm an AI chatbot, which means I'm a computer program... blah, blah, blah, blah, blah, blah, blah. 25 tokens per second. Pretty good. That is Llama 3.2, 3 billion. It is a model that runs on your device. What does that mean? It's free. You don't pay anybody. You can use it as much as you want. And it doesn't go anywhere. Your data is right there. That's nice. Now, what if we want to use that in conjunction with a larger model that you can't fit on your phone? Let's see. I've got a different chat interface here. This is hooked up to our MCP server. I'll show you in a second. Actually, I'll show you right now. So you can see here, I previously had Pylon running on my laptop, and I now have it running over here. And you can see the connected client. It says the last message was resource list. That's because it's still loading these folders. But you can see if I go here and then I type chat to say hi, you'll see here that the last message resource list changes to Olama chat. And it says, hello, how are you today? Yeah, because I have Pylon running there. There's the logs of Pylon. And like if I reload this, this is kind of awkward. Reload that. The client goes from zero to one. That is so nice. That is so nice. Okay, so that was actually fast because I was using Llama 3.2 still. But if I go over to here, I'm going to set the default to be Llama 3.3. It'll then use the 70 billion parameter model. This is something that would definitely not fit on the device. But if I say, you know, introduce yourself in two sentences. Here you're going to hear my computer start whirring. As it's running Llama 3.3 70 billion, which is a GPT-4 equivalent, at least that's what they're saying, model running locally. Locally, locally, locally. And this is using the model context protocol that we showed in the last episode using Pylon, which talks to a local Olama that's doing all the model loading and stuff. We're using the MCP protocol to talk directly to our server. And so I had this, and this is taking forever, so, you know, won't use it for that much necessarily. But yeah, what, it'll take 10, 15 seconds, and then you have a GPT-4 quality answer that's all running locally. Now, obviously, we're probably, for a lot of use cases, you're going to kind of want to mix this with API models, but there's just going to be a whole bunch of things where, like, you may not want to go out to pay for an API to get something quicker. If you're okay with it taking longer and then being more private, you can do it like this. Okay, so the kind of idea here is that you've got multiple different ways of consuming models. You can have small models on your device. Super private. Free. Can be quick because they're so small. Not super powerful. Super powerful ones. The most powerful ones, like Cloud Sonnet, you just hit an API. If you have a server that you trust, like your own server, you can connect and do this. Separately, we can kind of connect the same decentralized kind of marketplace piece. Here we use it as a hosted API. You can do the same thing for hosting, like we did that same concept with GPUtopia back in the day. Dusting off that same concept these days, keeping it easy by not recreating all the stuff that Olama has already done, but just using the protocols for kind of network discoverability. The point is that any given, any set of trade-offs that you want to run things locally or kind of in this decentralized way or via an API, all of that is possible via Onyx basically right now. So there's a lot we can do with this. One thing I'm excited to do next is move my day-to-day workflow off of the current OpenAgents interface that uses exclusively Cloud Sonnet to this. I can do some really cool things here. This app is connected to my server and so it's able to just grab the contents. You can make calls and run any kinds of shell commands or whatever I want. That's not something that I would expose to the internet, but I'm just on my private network here. So I can then walk around my house. I've already done this earlier today when I was just walking around my house using this chat interface, chatting with Llama. This is connected to the over web sockets to the Pylon server that I have running on my Linux desktop with a pretty decent NVIDIA GPU in it. And I'm walking around my house talking to this interface and it's sending the message in a web socket over here. Takes a second or two if I'm using Llama 3.2. It comes right back. To be able to have that kind of me walking around my house to also then have it able to read files, suggest edits or make edits to files, had previously done that via the GitHub API, maybe we should do that by doing direct edits to the file system. Now we're getting towards automated coding agents happening and being directed strictly by voice. So that's my weekend project to get my workflow hopefully switched over to that full system. And then our little Jarvis-y Onyx here will be quite powerful, I think, by Monday we'll have a pretty cool demo to show off. I'm gonna connect that back into the Bitcoin payments piece, tighten up some of the kind of initial feature set. And we'll have this ready for you to do some cool stuff with. Aiming for the end of next week. Enjoy your weekend. Stay tuned.