Transcription: OpenAgents Episode 040 - Agent Brain Design

Source: https://twitter.com/OpenAgentsInc/status/1742346953210388881 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 040: Agent Brain Design We design and implement our initial a... Upload date: 20240103 Transcription model: mlx-whisper/mlx-community/whisper-tiny Generated at: 2026-06-01T20:24:36Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: Let's see if we can apply that same component driven development approach to something harder.

[00:07] Christopher David: Something requiring back end development.

[00:13] Christopher David: Okay, so we have this.

[00:17] Christopher David: We also have all of the code for each of these steps.

[00:21] Christopher David: More or less implemented, I did that offline in these step actions folder.

[00:31] Christopher David: You have steps for validation, embedding, similarity search, and inference.

[00:37] Christopher David: Each going to code that we've built in previous videos.

[00:42] Christopher David: And the thing that we're missing here that we're right now just kind of hard coding and demo version of is the knowledge base.

[00:53] Christopher David: So instead of just passing input to an LLM, that's all this is doing now, because this isn't connected to anything.

[01:05] Christopher David: But this challenge is going to...

[01:11] Christopher David: I stopped just stopping by just stopping the system problem.

[01:14] Christopher David: But what we wanted to do is consult a knowledge base.

[01:19] Christopher David: And you can see right now that I'm having similarity search, like we actually are generated in the embedding.

[01:28] Christopher David: When I run right now you can run this by going to each chat test and doing PF.

[01:36] Christopher David: Actually I might skip it because it's integration me.

[01:39] Christopher David: We'll move this from there.

[01:42] Christopher David: We're excluding integration tests.

[01:48] Christopher David: So this will run through the steps and I'll show you the logic there.

[01:56] Christopher David: We're looping through all of the steps.

[02:00] Christopher David: And we are passing the output of one as input to the next step.

[02:08] Christopher David: So we are actually going through all of these steps.

[02:14] Christopher David: So we're calling step executed run.

[02:18] Christopher David: If you go to step executed you can look at run.

[02:20] Christopher David: You can see that we're grabbing the category.

[02:23] Christopher David: So it's validation embeddings similarity search or inference right now.

[02:27] Christopher David: We're just assigning that to the one function for each.

[02:32] Christopher David: Just defined on step actions trait.

[02:37] Christopher David: So we have a function for validation.

[02:39] Christopher David: A function for embedding.

[02:40] Christopher David: A function for similarity search and a function for inference.

[02:43] Christopher David: So we are generating the embedding up the query.

[02:46] Christopher David: But right now we're just hard coding what we would expect to get from the similarity search.

[02:53] Christopher David: The last time that we implemented this we did a search through all of the embeddings in the database.

[03:00] Christopher David: We don't want to do that.

[03:02] Christopher David: We need some sort of model here that has embedding associated with it.

[03:09] Christopher David: So this is what we need to build.

[03:14] Christopher David: We're going to be calling this an agent brain more on that in a second.

[03:18] Christopher David: But you can see that we're returning both the user input like what they type into the chat box,

[03:25] Christopher David: as well as the relevant context from the knowledge base based on the input.

[03:33] Christopher David: This is standard retrieval augment the generation.

[03:36] Christopher David: And then in the inference step we are generating a response based on that.

[03:43] Christopher David: So for example I go back to agent chat test and I say.

[03:49] Christopher David: Diant dump the response.

[03:56] Christopher David: This is taking the context that I'm passing in which is open agent is an open platform for AI agents.

[04:03] Christopher David: So you know every person in county above all the benefits is basically the copy from the website of open agent.

[04:09] Christopher David: And the query that I'm passing to it is what is this so just imagine that you type into the chat.

[04:20] Christopher David: What is this.

[04:26] Christopher David: Let's see what it responded.

[04:33] Christopher David: So it responded with we got to kind of clean up some of this stuff but it responded with welcome to open agents open agents is an open platform for AI agents.

[04:52] Christopher David: It is a platform that allows you to create and deploy multiple AI agents to work on your behalf.

[04:56] Christopher David: These agents can be configured blah blah blah.

[05:00] Christopher David: So if you are looking to supercharge your productivity open agents is the platform for you.

[05:04] Christopher David: How many agents will you want working for you?

[05:06] Christopher David: That's cool.

[05:07] Christopher David: Conversational it kind of like mushed up all the context that I pass it and made it you know a logical response to the chat.

[05:16] Christopher David: And just take a look here's the system prompt that we use.

[05:20] Christopher David: You are the conscientious chat about welcoming users to open agents.com a platform for creating AI.

[05:25] Christopher David: And then you are going to limit your responses to it's in the following context.

[05:29] Christopher David: And then we pass it in context.

[05:32] Christopher David: And we just said user is asking what the user asked and then generate the next message.

[05:37] Christopher David: Basic but it works.

[05:40] Christopher David: Okay so now instead of doing the hard coded version.

[05:50] Christopher David: The generating or like retrieving the relevant context from some database.

[06:04] Christopher David: Just take a look at our flow here.

[06:12] Christopher David: So the whole point here is that we are responding to the user chat message after consulting knowledge base.

[06:18] Christopher David: Now for the contiers we are going to pass in the knowledge base of like that basic information about the project.

[06:25] Christopher David: But you could envision that other agents are going to have other knowledge bases.

[06:31] Christopher David: And maybe there's some things that you think should be added to the agent's knowledge base.

[06:38] Christopher David: That's not there.

[06:40] Christopher David: And maybe you suggest an improvement to that.

[06:43] Christopher David: And as the agent author I say hey if someone you know.

[06:47] Christopher David: Suggest an improvement that I accept.

[06:51] Christopher David: Maybe I want to give that person.

[06:53] Christopher David: 1% of any revenue generated by this agent for a week or a year or forever.

[07:01] Christopher David: And that should all be able to be visualized here.

[07:06] Christopher David: So we're going to build in that direction by first.

[07:11] Christopher David: And we're going to do a very simple representation of a knowledge base such that it can show up on our note graph here.

[07:19] Christopher David: So even though we're going to be doing there's a lot of kind of like back in code that needs to be done for this.

[07:25] Christopher David: I'm kind of flipping our usual approach which is to like write the tests for it.

[07:29] Christopher David: And I'm going to try to get as far as we can on.

[07:33] Christopher David: Getting the front end component built first.

[07:36] Christopher David: And because you know there's.

[07:38] Christopher David: If we're building on the back end we have to kind of like think about what the user experience is.

[07:44] Christopher David: But the beautiful thing about.

[07:46] Christopher David: Component driven development is it lets us kind of reason about what that component is starting from the user experience.

[07:53] Christopher David: So.

[07:54] Christopher David: Theoretically that should give us a better result.

[07:58] Christopher David: So let's get our story book going.

[08:00] Christopher David: I think we have it running locally already.

[08:04] Christopher David: Is it here and P.M. runs right back.

[08:11] Christopher David: So let's think about what we want to show up.

[08:18] Christopher David: On our similarity search.

[08:29] Christopher David: Compare input to knowledge base.

[08:39] Christopher David: Let me make a wiki page.

[08:47] Christopher David: I'm losing brains growing here.

[09:00] Christopher David: So nodes like similarity search.

[09:04] Christopher David: We'll require.

[09:07] Christopher David: Reference to a knowledge base.

[09:14] Christopher David: For one or more knowledge bases.

[09:30] Christopher David: We'll want those nodes to have input fields that can be references.

[09:47] Christopher David: To one or more.

[09:49] Christopher David: Knowledge base nodes.

[10:00] Christopher David: What's the appropriate model is if it's brain or if.

[10:15] Christopher David: Because there's going to be maybe like multiple different knowledge bases that we wanted to have.

[10:20] Christopher David: And should each one be called like a patch or a memory module or brain or thought like.

[10:25] Christopher David: And then we want to get to like in the weed that stuff or should start really simple.

[10:33] Christopher David: So we would say one or more knowledge bases.

[10:42] Christopher David: We'll call brains.

[10:56] Christopher David: So you can envision maybe.

[11:05] Christopher David: We want the similarity search node to know about a brain.

[11:10] Christopher David: The brain could conceivably have one or more different modules, but from the perspective of the similarity search node.

[11:17] Christopher David: We want to have it have a brain.

[11:21] Christopher David: So the similarity search node should have an associated brain.

[11:30] Christopher David: Brain model, which itself could have.

[11:37] Christopher David: Multiple memory modules.

[11:49] Christopher David: So we know there's going to be some kind of connection here to a brain model brain node.

[11:58] Christopher David: And so we know that we're going to need a brain node.

[12:08] Christopher David: And the simplest version of this.

[12:11] Christopher David: I just had an array of strings like each string is its own knowledge.

[12:16] Christopher David: You could imagine there being a PDF file in a brain.

[12:21] Christopher David: But that PDF file is probably going to be represented as just a bunch of the strings of the text.

[12:27] Christopher David: But to have a most like minimum viable brain, it seems like just.

[12:33] Christopher David: The brain should have an array of strings.

[12:36] Christopher David: Each of which is going to be vectorized vector embedded separately.

[12:44] Christopher David: So brain node and model.

[12:49] Christopher David: Brain can have.

[13:00] Christopher David: Many strings of.

[13:04] Christopher David: So the brain node should look like.

[13:19] Christopher David: A list of inputs.

[13:35] Christopher David: Should have a plus icon for adding more.

[13:42] Christopher David: Or.

[13:46] Christopher David: Bility to subtract delete.

[13:49] Christopher David: Data.

[13:50] Christopher David: And ideally should have some representation of whether.

[13:56] Christopher David: And embedding is in progress.

[13:59] Christopher David: Or done.

[14:01] Christopher David: Because that might take a few seconds to generate.

[14:12] Christopher David: Okay.

[14:14] Christopher David: So we have I think enough to get started.

[14:19] Christopher David: So the agent node and step node really we need a brain node.

[14:27] Christopher David: So let's do that.

[14:35] Christopher David: So we're going to do the same thing we did in the last video.

[14:39] Christopher David: And.

[14:42] Christopher David: I'm going to skip past some of that setup because we already did it.

[14:48] Christopher David: But I'm going to do the same thing we did in the last video.

[14:51] Christopher David: We're instead of creating an agent node.

[14:53] Christopher David: I'm going to create a brain node.

[14:56] Christopher David: And then we have a new.

[15:00] Christopher David: And then we have a new.

[15:04] Christopher David: And then we have a new.

[15:06] Christopher David: And then we have a new.

[15:08] Christopher David: And then we have a new.

[15:10] Christopher David: And then we have a new.

[15:12] Christopher David: And then we have a new.

[15:14] Christopher David: And then we have a new.

[15:16] Christopher David: And then we have a new.

[15:18] Christopher David: And then we have a new.

[15:20] Christopher David: And then we have a new.

[15:22] Christopher David: And then we have a new.

[15:24] Christopher David: And then we have a new.

[15:25] Christopher David: And then we have a new.

[15:27] Christopher David: And then we have a new.

[15:29] Christopher David: And then we have a new.

[15:31] Christopher David: And then we have a new.

[15:33] Christopher David: And then we have a new.

[15:35] Christopher David: And then we have a new.

[15:37] Christopher David: And then we have a new.

[15:39] Christopher David: And then we have a new.

[15:41] Christopher David: And then we have a new.

[15:43] Christopher David: And then we have a new.

[15:45] Christopher David: And then we have a new.

[15:47] Christopher David: And then we have a new.

[15:49] Christopher David: And then we have a new.

[15:51] Christopher David: And then we have a new.

[15:53] Christopher David: And then we have a new.

[15:55] Christopher David: And then we have a new.

[15:57] Christopher David: And then we have a new.

[15:59] Christopher David: And then we have a new.

[16:01] Christopher David: And then we have a new.

[16:03] Christopher David: And then we have a new.

[16:05] Christopher David: And then we have a new.

[16:07] Christopher David: And then we have a new.

[16:09] Christopher David: And then we have a new.

[16:11] Christopher David: And then we have a new.

[16:13] Christopher David: And then we have a new.

[16:15] Christopher David: And then we have a new.

[16:17] Christopher David: And then we have a new.

[16:19] Christopher David: And then we have a new.

[16:21] Christopher David: And then we have a new.

[16:23] Christopher David: And then we have a new.

[16:25] Christopher David: And then we have a new.

[16:27] Christopher David: And then we have a new.

[16:29] Christopher David: And then we have a new.

[16:31] Christopher David: And then we have a new.

[16:33] Christopher David: And then we have a new.

[16:35] Christopher David: And then we have a new.

[16:37] Christopher David: And then we have a new.

[16:39] Christopher David: And then we have a new.

[16:41] Christopher David: And then we have a new.

[16:43] Christopher David: And then we have a new.

[16:45] Christopher David: And then we have a new.

[16:47] Christopher David: And then we have a new.

[16:49] Christopher David: And then we have a new.

[16:51] Christopher David: And then we have a new.

[16:53] Christopher David: And then we have a new.

[16:55] Christopher David: And then we have a new.

[16:57] Christopher David: And then we have a new.

[16:59] Christopher David: And then we have a new.

[17:01] Christopher David: And then we have a new.

[17:03] Christopher David: And then we have a new.

[17:05] Christopher David: And then we have a new.

[17:07] Christopher David: And then we have a new.

[17:09] Christopher David: And then we have a new.

[17:11] Christopher David: And then we have a new.

[17:13] Christopher David: And then we have a new.

[17:15] Christopher David: And then we have a new.

[17:17] Christopher David: And then we have a new.

[17:19] Christopher David: And then we have a new.

[17:21] Christopher David: And then we have a new.

[17:23] Christopher David: And then we have a new.

[17:25] Christopher David: And then we have a new.

[17:27] Christopher David: And then we have a new.

[17:29] Christopher David: And then we have a new.

[17:31] Christopher David: And then we have a new.

[17:33] Christopher David: And then we have a new.

[17:35] Christopher David: And then we have a new.

[17:37] Christopher David: And then we have a new.

[17:39] Christopher David: And then we have a new.

[17:41] Christopher David: And then we have a new.

[17:43] Christopher David: And then we have a new.

[17:45] Christopher David: And then we have a new.

[17:47] Christopher David: And then we have a new.

[17:49] Christopher David: And then we have a new.

[17:51] Christopher David: And then we have a new.

[17:53] Christopher David: And then we have a new.

[17:55] Christopher David: And then we have a new.

[17:57] Christopher David: And then we have a new.

[17:59] Christopher David: And then we have a new.

[18:01] Christopher David: And then we have a new.

[18:03] Christopher David: And then we have a new.

[18:05] Christopher David: And then we have a new.

[18:07] Christopher David: And then we have a new.

[18:09] Christopher David: And then we have a new.

[18:11] Christopher David: And then we have a new.

[18:13] Christopher David: And then we have a new.

[18:15] Christopher David: And then we have a new.

[18:17] Christopher David: And then we have a new.

[18:19] Christopher David: And then we have a new.

[18:21] Christopher David: And then we have a new.

[18:23] Christopher David: And then we have a new.

[18:25] Christopher David: And then we have a new.

[18:27] Christopher David: And then we have a new.

[18:29] Christopher David: And then we have a new.

[18:31] Christopher David: And then we have a new.

[18:33] Christopher David: And then we have a new.

[18:35] Christopher David: And then we have a new.

[18:37] Christopher David: And then we have a new.

[18:39] Christopher David: And then we have a new.

[18:41] Christopher David: And then we have a new.

[18:43] Christopher David: And then we have a new.

[18:45] Christopher David: And then we have a new.

[18:47] Christopher David: And then we have a new.

[18:49] Christopher David: And then we have a new.

[18:51] Christopher David: And then we have a new.

[18:53] Christopher David: And then we have a new.

[18:55] Christopher David: And then we have a new.

[18:57] Christopher David: And then we have a new.

[18:59] Christopher David: And then we have a new.

[19:01] Christopher David: And then we have a new.

[19:03] Christopher David: And then we have a new.

[19:05] Christopher David: And then we have a new.

[19:07] Christopher David: And then we have a new.

[19:10] Christopher David: And then we have a new.

[19:11] Christopher David: And then we have a new.

[19:13] Christopher David: And then we have a new.

[19:15] Christopher David: And then we have a new.

[19:17] Christopher David: And then we have a new.

[19:19] Christopher David: And then we have a new.

[19:21] Christopher David: And then we have a new.

[19:23] Christopher David: And then we have a new.

[19:25] Christopher David: And then we have a new.

[19:27] Christopher David: And then we have a new.

[19:29] Christopher David: And then we have a new.

[19:31] Christopher David: And then we have a new.

[19:33] Christopher David: And then we have a new.

[19:35] Christopher David: And then we have a new.

[19:37] Christopher David: And then we have a new.

[19:39] Christopher David: And then we have a new.

[19:41] Christopher David: And then we have a new.

[19:43] Christopher David: And then we have a new.

[19:45] Christopher David: And then we have a new.

[19:47] Christopher David: And then we have a new.

[19:49] Christopher David: And then we have a new.

[19:51] Christopher David: And then we have a new.

[19:53] Christopher David: And then we have a new.

[19:55] Christopher David: And then we have a new.

[19:57] Christopher David: And then we have a new.

[19:59] Christopher David: And then we have a new.

[20:01] Christopher David: And then we have a new.

[20:03] Christopher David: And then we have a new.