Transcription: OpenAgents Episode 081 - Streaming LLM Response

Source: https://twitter.com/OpenAgentsInc/status/1758659094578340054 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 081: Streaming LLM Response We also stream the GPT-4 response... Upload date: 20240217 Transcription model: mlx-whisper/mlx-community/whisper-tiny Generated at: 2026-06-01T18:45:50Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: So that feels nice to have some status here.

[00:05] Christopher David: But we still are waiting multiple seconds for the LLM in friends, and that's because we're not streaming it.

[00:14] Christopher David: And it'll feel a lot nicer if we're streaming it.

[00:17] Christopher David: The problem with our current implementation is that I at least don't know how to if it is possible.

[00:25] Christopher David: Stream from the Wasm plugin.

[00:29] Christopher David: Wasm plugins, making it HTTP call on returning, we're doing the kind of string and string out.

[00:35] Christopher David: Maybe it's possible for things that require streaming.

[00:43] Christopher David: We may be able to implement that with the host function, but I'm just not going to worry about that.

[00:49] Christopher David: Just want to do the simplest thing that gets streaming working at least for me locally now.

[00:55] Christopher David: So the quickest path to doing that instead of rejuring our plugin configuration, and then just going to put in an override that.

[01:04] Christopher David: If the name of the plugin requested is LLM in friends, I'm just going to like.

[01:10] Christopher David: Implement that locally.

[01:12] Christopher David: We had code, a while back, where we're doing streaming from the together API.

[01:18] Christopher David: We might pull some of that stuff in.

[01:25] Christopher David: Yeah, we might do that.

[01:27] Christopher David: Okay, so let's do this just like as quickly as we can, and then over the weekend I'll kind of go back and re-factor and make things nicer.

[01:36] Christopher David: Now, at some point we deleted our stream controller, but I think that's where that code was.

[01:42] Christopher David: Let's go browse through our older commits here.

[01:46] Christopher David: Do we have in here stream controller.

[01:59] Christopher David: Man, that read stream thing is ugly.

[02:16] Christopher David: But that was a straight, guzzle HTTP calls. That's pretty cool.

[02:26] Christopher David: All right, let's go see where in our code we're going to want to do this override or how we're going to want to do that.

[02:36] Christopher David: So, call this LLM, inferenced stream.

[02:45] Christopher David: Is this our current code base or our old code base?

[02:54] Christopher David: Yeah, okay.

[02:58] Christopher David: Let's think where this should go.

[03:06] Christopher David: Now, we know that we're going to need to stream via the live wire component.

[03:17] Christopher David: But we could probably just pass this exact same function.

[03:27] Christopher David: All right, let's follow this through.

[03:49] Christopher David: We know that one of the step name is going to be run task.

[03:56] Christopher David: Can I do this?

[04:03] Christopher David: So, in run task, in run task, pull steps.

[04:11] Christopher David: Here we go, look.

[04:14] Christopher David: If the step name is LLM inference.

[04:25] Christopher David: Do this with the input.

[04:32] Christopher David: And got a lot of cleaning up of this stuff to do.

[04:54] Christopher David: Yeah, I probably want to do it there. I want to do it in the.

[05:01] Christopher David: In the execution on step executed.

[05:05] Christopher David: And switch to go into step actions. So I think.

[05:18] Christopher David: Oh yeah.

[05:25] Christopher David: Huh.

[05:32] Christopher David: I have this existing inference function, which is intended to work with together.

[05:45] Christopher David: I have a lot of refactoring to do.

[06:06] Christopher David: Plug in input.

[06:16] Christopher David: At this point, it's trying to call.

[06:22] Christopher David: I can't do this.

[06:35] Christopher David: I could have access to this step if I initialize it.

[06:43] Christopher David: Or just being called from step execution.

[06:49] Christopher David: Okay.

[06:59] Christopher David: I'm going to put my little hack here, which is if the step executed step name is LLM inference override with our own.

[07:18] Christopher David: Streaming one.

[07:28] Christopher David: Oh, that's got a little conversation there.

[07:30] Christopher David: I'm not going to worry about conversation for now.

[07:39] Christopher David: So what I want to do first is just make sure that we actually have this.

[07:46] Christopher David: And we can do that.

[07:56] Christopher David: Just do the input.

[08:01] Christopher David: Just make sure that override is overriding.

[08:07] Christopher David: Yeah.

[08:09] Christopher David: Okay. So we have an array of input.

[08:16] Christopher David: So let's do input string equals input input.

[08:24] Christopher David: Let's just make sure that's clean.

[08:28] Christopher David: Okay.

[08:49] Christopher David: Out of curiosity.

[09:02] Christopher David: We have step actions inference here.

[09:05] Christopher David: Should I try to shim this into my old ugly code?

[09:11] Christopher David: It needs inference and conversation.

[09:14] Christopher David: What if I don't pass the conversation?

[09:23] Christopher David: If input context does that exist, do this.

[09:35] Christopher David: And then, oh, it actually does want input input.

[09:41] Christopher David: Last return output last.

[09:47] Christopher David: And this is stream controller, doge.

[09:50] Christopher David: Well, this doesn't exist anymore.

[09:52] Christopher David: I need to remake that.

[09:55] Christopher David: If I want to remake that.

[09:56] Christopher David: And then, is this set up for streaming?

[10:01] Christopher David: Do chat.

[10:06] Christopher David: It is streaming and it's broadcasting.

[10:09] Christopher David: New chat token received.

[10:14] Christopher David: This is where we pass in our token handler thingy.

[10:19] Christopher David: But there's a lot of junk in there.

[10:29] Christopher David: So I try using the together API here.

[10:34] Christopher David: I kind of want to use GPT4, but maybe I'll just try this.

[10:40] Christopher David: And then do I want to put a bunch of.

[10:45] Christopher David: So I don't, I don't really want to use that.

[10:49] Christopher David: So I don't want to use it.

[11:05] Christopher David: Yeah, I guess I'll clutter up step executed.

[11:14] Christopher David: Well, let's try some copyposset here.

[11:20] Christopher David: So client is.

[11:24] Christopher David: Guzzle stuff.

[11:32] Christopher David: URL model.

[11:44] Christopher David: So messages.

[12:10] Christopher David: So messages equals.

[12:24] Christopher David: You are a helpful agent on openagents.com.

[12:30] Christopher David: Respond to the user concitely.

[12:37] Christopher David: And then we'll do.

[12:47] Christopher David: Data equals blah blah blah.

[12:51] Christopher David: Stream tokens true.

[13:06] Christopher David: Do.

[13:16] Christopher David: I still have a working together API key.

[13:32] Christopher David: I don't know if this will work, but for each.

[13:41] Christopher David: Okay, so stream is response get body.

[13:49] Christopher David: Content is that's don't do that.

[13:56] Christopher David: Give me the first.

[13:58] Christopher David: Okay, maybe.

[14:04] Christopher David: This will definitely not work on the first try.

[14:08] Christopher David: Okay, so first of all, let's give her to this.

[14:18] Christopher David: Does what?

[14:27] Christopher David: I think that's a good idea.

[14:37] Christopher David: I think that's a good idea.

[14:45] Christopher David: And my remembering correctly that together or something didn't work locally.

[14:55] Christopher David: Last time I tried this with it together API.

[14:59] Christopher David: Okay, so out of curiosity.

[15:01] Christopher David: Out of curiosity.

[15:02] Christopher David: I would have just like dump my openag API key here.

[15:12] Christopher David: And if I were to say URL.

[15:22] Christopher David: Oops.

[15:24] Christopher David: What happened there?

[15:26] Christopher David: Now I would be fresh.

[15:28] Christopher David: And if I said screw that through that model,

[16:03] Christopher David: reference.

[16:10] Christopher David: Just give me the reference docs, please.

[16:23] Christopher David: Okay.

[16:50] Christopher David: What stream true?

[16:58] Christopher David: Not stream messages true.

[17:05] Christopher David: For together it was stream.

[17:08] Christopher David: Okay, let's try it.

[17:13] Christopher David: Let's see.

[17:23] Christopher David: Model messages.

[17:28] Christopher David: I can't tell that that did anything.

[17:52] Christopher David: Let's make sure that we have a new one.

[17:57] Christopher David: Let's make sure that we get to hear fine.

[18:07] Christopher David: Yep.

[18:14] Christopher David: Yep, that's looking good.

[18:23] Christopher David: Client post URL.

[18:27] Christopher David: JSON data stream true.

[18:32] Christopher David: Okay, do we even get a response here?

[18:51] Christopher David: Okay.

[18:58] Christopher David: It looks like we have a stream.

[19:03] Christopher David: Okay, let's start by deding the response line.

[19:12] Christopher David: Nothing.

[19:34] Christopher David: Okay, so let's try this first.

[19:40] Christopher David: Okay.

[19:59] Christopher David: following its stream but first there's those DD with the body looks like oops

[20:06] Christopher David: if I don't have stream there is that json even the proper thing

[20:19] Christopher David: i'm just not familiar with this raw guisles syntax

[20:26] Christopher David: okay

[20:59] Christopher David: fix this i'm not getting a stream back from open i.e.

[21:30] Christopher David: that is not correct open i does support streaming here are the docs

[21:50] Christopher David: fix that gives response

[22:20] Christopher David: i'll try it

[22:45] Christopher David: so

[22:48] Christopher David: the response we're going to put it in a try

[22:52] Christopher David: and then

[23:22] Christopher David: let's get removed my content definition but let's just try this

[23:30] Christopher David: stream read lie does not like that

[23:46] Christopher David: let's see what it does

[24:00] Christopher David: i don't see anything at all

[24:18] Christopher David: unexpected character

[24:30] Christopher David: i

[24:47] Christopher David: should just try to use the PHP opening i library

[24:51] Christopher David: i don't like one

[25:21] Christopher David: all right this looks easy i guess i'll do this

[25:51] Christopher David: i don't want to but i'll do it just to get a working hopefully

[26:21] Christopher David: okay what does that do

[26:51] Christopher David: all right let's try DDing something to see something

[27:07] Christopher David: looks like a chunk looks like a choice look at token delta roll assisted content blank okay

[27:17] Christopher David: maybe or it responses will blah

[27:29] Christopher David: okay so lm inference

[27:34] Christopher David: stream function

[27:47] Christopher David: input roll user content

[28:03] Christopher David: input stream function response

[28:08] Christopher David: now it's past stream function into lm inference

[28:16] Christopher David: step executed

[28:28] Christopher David: this is going to be ugly

[28:46] Christopher David: and then that gets run from the

[29:02] Christopher David: agent

[29:06] Christopher David: step executed run

[29:12] Christopher David: the agent has access to log function

[29:20] Christopher David: stream function

[29:28] Christopher David: just do it

[29:46] Christopher David: and then when we call run

[29:50] Christopher David: oh i got a pass it from here too

[29:52] Christopher David: they also do but go

[29:56] Christopher David: okay just make it work stream function stream function

[30:02] Christopher David: now in run pass we can get that from chat and we're going to pass it in a stream function

[30:14] Christopher David: and here we're going to say stream function

[30:20] Christopher David: this is going to be like a whole response first of all

[30:24] Christopher David: just to make sure it works

[30:28] Christopher David: a bunch of junk

[30:38] Christopher David: so

[30:56] Christopher David: is it in run task

[31:03] Christopher David: callable

[31:08] Christopher David: up to the passing enough things here

[31:12] Christopher David: task input log yes

[31:16] Christopher David: run task oops agent

[31:22] Christopher David: run

[31:24] Christopher David: stream function

[31:44] Christopher David: and

[31:52] Christopher David: the

[31:58] Christopher David: run

[32:08] Christopher David: and

[32:16] Christopher David: the

[32:26] Christopher David: run

[32:34] Christopher David: and

[32:42] Christopher David: we do

[32:44] Christopher David: passing this into this dot

[32:47] Christopher David: dash lm inference

[32:56] Christopher David: input and stream function right

[33:00] Christopher David: and

[33:10] Christopher David: the

[33:16] Christopher David: stream

[33:22] Christopher David: and

[33:32] Christopher David: the

[33:42] Christopher David: run

[33:52] Christopher David: and

[34:16] Christopher David: the

[34:16] Christopher David: run

[34:26] Christopher David: and

[34:36] Christopher David: the

[34:44] Christopher David: run

[34:59] Christopher David: and

[35:09] Christopher David: the

[35:15] Christopher David: stream

[35:19] Christopher David: and

[35:29] Christopher David: the

[35:36] Christopher David: run

[35:39] Christopher David: and

[35:44] Christopher David: the

[35:50] Christopher David: run

[35:55] Christopher David: and

[36:00] Christopher David: the

[36:05] Christopher David: run

[36:13] Christopher David: and

[36:19] Christopher David: the

[36:23] Christopher David: run

[36:30] Christopher David: and

[36:34] Christopher David: the

[36:38] Christopher David: run

[36:42] Christopher David: and

[36:46] Christopher David: the

[36:57] Christopher David: run

[37:02] Christopher David: and

[37:06] Christopher David: the

[37:11] Christopher David: run

[37:16] Christopher David: and

[37:21] Christopher David: the

[37:28] Christopher David: and

[37:31] Christopher David: the

[37:36] Christopher David: run

[37:40] Christopher David: and

[37:44] Christopher David: the

[37:49] Christopher David: run

[37:54] Christopher David: and

[38:04] Christopher David: the

[38:22] Christopher David: run

[38:26] Christopher David: and

[38:29] Christopher David: the

[38:47] Christopher David: run

[38:52] Christopher David: and

[38:55] Christopher David: the

[38:57] Christopher David: run

[39:01] Christopher David: and

[39:05] Christopher David: the

[39:10] Christopher David: run

[39:15] Christopher David: and

[39:20] Christopher David: the

[39:25] Christopher David: run

[39:29] Christopher David: and

[39:33] Christopher David: the

[39:36] Christopher David: run

[39:40] Christopher David: and

[39:45] Christopher David: the

[39:48] Christopher David: run

[39:50] Christopher David: and

[39:54] Christopher David: the

[39:55] Christopher David: run

[39:58] Christopher David: and

[40:00] Christopher David: the

[40:29] Christopher David: the

[40:31] Christopher David: stream chat chat

[40:36] Christopher David: okay

[40:42] Christopher David: hello

[40:47] Christopher David: I have can I assist you today

[40:54] Christopher David: I'm

[40:58] Christopher David: going

[41:02] Christopher David: to

[41:06] Christopher David: to

[41:13] Christopher David: the

[41:18] Christopher David: I

[42:25] Christopher David: I

[43:05] Christopher David: I

[43:12] Christopher David: can

[43:15] Christopher David: I

[43:25] Christopher David: can

[43:30] Christopher David: I

[43:40] Christopher David: can