Transcription: OpenAgents Episode 036 - Agent Modules 101

Source: https://twitter.com/OpenAgentsInc/status/1738000844476371445 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 036: Agent Modules 101 We recap lessons learned from NeurIPS ... Upload date: 20231222 Transcription model: mlx-whisper/mlx-community/whisper-tiny Generated at: 2026-06-01T20:29:45Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: Let's continue. So that's all pushed live now at openaget.com. You log in and then do these little audit runs and see

[00:10] Christopher David: Input and output for each step. We're going to be changing this into a chat interface

[00:14] Christopher David: Very soon, but you're going to least kind of play around with what is

[00:18] Christopher David: there

[00:19] Christopher David: I want to in this video introduce this concept of agent modules and

[00:24] Christopher David: and contextualizes with some of the stuff that I learned at NURPPS 2023-8 which is like the largest

[00:32] Christopher David: I don't know AI researcher conference annually and I learned a whole bunch there

[00:40] Christopher David: That we're now going to be putting into the

[00:46] Christopher David: Codebase

[00:49] Christopher David: So there is a

[00:52] Christopher David: Paper I want to focus on called Voyager

[00:58] Christopher David: The announcement tweet from this from Jim fan who works in video

[01:03] Christopher David: They kind of have shown this

[01:06] Christopher David: agent that is able to

[01:09] Christopher David: Have lifelong learning as in use edit free and it can kind of learn and pick up new skills and use those skills it goes

[01:18] Christopher David: This is a great

[01:22] Christopher David: Kind of discovery of theirs because it as they say unlocks a new paradigm in which the training is

[01:30] Christopher David: Code execution

[01:32] Christopher David: rather than gradient descent so I went specifically to this

[01:40] Christopher David: Alo workshop where the theme was

[01:45] Christopher David: Openedness

[01:47] Christopher David: for agents

[01:49] Christopher David: and a bulk of the focus on

[01:52] Christopher David: agents at least at this NURPPS conference is around

[01:58] Christopher David: Agents that learn through gradient descent are kind of the standard reinforcement learning algorithms

[02:04] Christopher David: There's a whole bunch of different takes on that

[02:08] Christopher David: At least to me as someone who's not an AI researcher which stood out

[02:12] Christopher David: To me the most was as Voyager paper because they are as they say

[02:18] Christopher David: shifting the

[02:21] Christopher David: relevant part of training and agent from the

[02:25] Christopher David: Things that the AI community has previously done

[02:28] Christopher David: and

[02:30] Christopher David: Moving it to the realm of basically prompt engineer

[02:35] Christopher David: interacting with

[02:37] Christopher David: databases

[02:40] Christopher David: APIs

[02:41] Christopher David: Vector databases in API

[02:44] Christopher David: We are pushing no gradient architecture to its lowest

[02:47] Christopher David: So this idea of

[02:49] Christopher David: An agent that has sort of ability for open ended learning and a given environment is exactly what we want to build

[02:56] Christopher David: That's the type of agent that we are trying to build we want to be able to set an agent like this

[03:02] Christopher David: loose an

[03:03] Christopher David: Environment not minecraft although we might do mycraft too

[03:09] Christopher David: But an environment

[03:11] Christopher David: Being get up and this thing that we've been building toward

[03:15] Christopher David: In the past

[03:17] Christopher David: Where we're trying to have a get up agent be able to take relevant actions

[03:22] Christopher David: And learn over time and build up pick up skills and get better over time

[03:27] Christopher David: In any given get every po it's like we were kind of applying the concepts of this Voyager agent

[03:34] Christopher David: to

[03:35] Christopher David: Get up as one environment, but you can imagine all sorts of other environments too

[03:39] Christopher David: And so it's nice to have some of kind of the

[03:45] Christopher David: You know get acquainted with what the state of the art is in

[03:48] Christopher David: AI research

[03:50] Christopher David: And see what we can learn from that in particular in the Voyager paper

[03:56] Christopher David: Which I recommend people read it's at Voyager.minddojo.org

[03:59] Christopher David: They kind of do a one page summary of the algorithm of this Voyager Minecraft agent

[04:14] Christopher David: In Sudoko. It's basically just kind of showing you how

[04:18] Christopher David: It works

[04:20] Christopher David: The algorithm goes and

[04:22] Christopher David: The Minecraft agent of the overall Voyager agent is composed of five different

[04:28] Christopher David: kind of sub

[04:30] Christopher David: modules

[04:31] Christopher David: They even call them like agent

[04:36] Christopher David: But for our purposes, they want to call these modules

[04:40] Christopher David: So the Voyager

[04:42] Christopher David: Agent is composed of environment

[04:45] Christopher David: The environment that uses code as the action space so it's taking action by like

[04:51] Christopher David: Creating code snippets that it can use to manipulate the Minecraft character

[04:58] Christopher David: Critical agent for proposing the next task

[05:03] Christopher David: Action agent to do the actual code generation

[05:07] Christopher David: Critic for self verification

[05:12] Christopher David: Like instantiating another version of the same kind of GBT4 based

[05:16] Christopher David: Egypt to

[05:18] Christopher David: Validate that actions makes sense

[05:20] Christopher David: Retry of the end

[05:22] Christopher David: And then skill manager

[05:24] Christopher David: Skill manager for adding new skills and skill retrieval

[05:30] Christopher David: And then kind of the core algorithm of how it loops through and

[05:34] Christopher David: Some of this will need to be kind of

[05:37] Christopher David: Chalera it a little differently for the

[05:40] Christopher David: Different environment that we have

[05:42] Christopher David: But overall I like that that it's kind of

[05:46] Christopher David: Got these five different sub modules that you can kind of reason about those separately

[05:50] Christopher David: So I've got this

[05:55] Christopher David: Wicked page on the open agents repo

[06:00] Christopher David: I've started called agent modules and I've got those five modules here

[06:06] Christopher David: Plus three more

[06:08] Christopher David: So we got actions and our version of that is writing code or comments

[06:13] Christopher David: Subending port requests

[06:15] Christopher David: Critic being self validation via could be review but also via test

[06:21] Christopher David: So we have these automatic

[06:23] Christopher David: Test the run each time and then it can respond to those. It's kind of a version of critic

[06:29] Christopher David: Crick it in for task planning

[06:31] Christopher David: Environment being interactions with the environment in this case get up API calls and such

[06:37] Christopher David: And then skills

[06:40] Christopher David: Which they are defining and we are defining as a complex learned behaviors

[06:51] Christopher David: So I'm supplementing this with

[06:54] Christopher David: Two additional modules that come from the generative agents paper

[07:07] Christopher David: This also has kind of GPT for based

[07:13] Christopher David: Agents

[07:15] Christopher David: Interactive with the turn

[07:18] Christopher David: And

[07:21] Christopher David: Whereas the Voyager agent relies heavily on the skill library this has a bit more of an emphasis on memories

[07:28] Christopher David: And in the paper they reference the memory module and this is kind of been previously the focus of the

[07:36] Christopher David: Man issue that we've been working on

[07:38] Christopher David: Spec and implement the agent memory manager we've been kind of accepting some

[07:43] Christopher David: Quotes here from the generative agents paper

[07:46] Christopher David: And one thing that this enables is an agent to reflect or kind of

[07:54] Christopher David: Drive higher level insights about things that it absorbs so

[07:58] Christopher David: If it observes

[08:00] Christopher David: People interacting in a certain way having certain conversation and then it thinks about it it can help with planning

[08:07] Christopher David: Every

[08:08] Christopher David: Reflection has its own vector embedding so that might be able to better kind of condense

[08:13] Christopher David: Information that can better fit into a prompt where as being able to fit all the low level raw information would not

[08:19] Christopher David: And so we've kind of recognized that this is probably appropriate some version of this would be appropriate for

[08:24] Christopher David: Having an agent makes sense to get every pope because there's like all of the contexts that we have pretty much

[08:30] Christopher David: ever need for having the agent be able to

[08:35] Christopher David: Productively thin code base is already in the code base

[08:38] Christopher David: And or additional documents like read me is and potentially supplemental material like at some point soon

[08:45] Christopher David: We're gonna transcribe these videos

[08:48] Christopher David: And

[08:49] Christopher David: Put the transcript in here so that the agent can get a better sense of like the different considerations of

[08:55] Christopher David: What we're building and why

[08:59] Christopher David: But

[09:00] Christopher David: There's way more information here than with it in context and

[09:06] Christopher David: If we want the agent to be able to say quickly look up to see

[09:10] Christopher David: You know whether I prefer tabs or spaces or or some kind of particular opinionation that we have instead of kind of

[09:17] Christopher David: Conducting based level analysis of individual code files every time

[09:21] Christopher David: There could just be memories that it has like

[09:24] Christopher David: Chris wants it to be this way

[09:26] Christopher David: But you just imagine they're being all sorts of different architectural decisions that go into this and each should have its own vector embedding for easy retrieval and like you know

[09:35] Christopher David: Somerizing things in the most concise way possible so they could be more easily fed into props

[09:41] Christopher David: So we recognize that this idea of memory stream and reflections

[09:46] Christopher David: Makes sense for

[09:49] Christopher David: What we're doing

[09:51] Christopher David: We did some initial

[09:55] Christopher David: Thinking previously about how that should be reflected in the data models

[10:02] Christopher David: Such

[10:04] Christopher David: There's one more module that we have

[10:06] Christopher David: And so we added memory the storage of the observed events reflection reflecting observations

[10:13] Christopher David: And then one more that I'm calling tools

[10:18] Christopher David: You know you can read for example about

[10:22] Christopher David: I think about kind of like lane chain defines tools as yeah

[10:27] Christopher David: Interfaces that an agent can use to interact with the world so imagine they're being just various integrations

[10:33] Christopher David: Whether it's a search engine or well from out for to do math or calculator or

[10:39] Christopher David: Whatever tool that it would need kind of related to skills, but but skills skills may use tools

[10:44] Christopher David: But he skills being a bit more like composition of different things

[10:49] Christopher David: That it can learn

[10:53] Christopher David: I think that's pretty comprehensive if there's another one that's missing

[10:59] Christopher David: We will probably at some point add payments

[11:04] Christopher David: We can just add it right now

[11:15] Christopher David: You know earning

[11:18] Christopher David: and sending

[11:20] Christopher David: You

[11:26] Christopher David: Not

[11:32] Christopher David: So aside from payments all of these now have modules in the open agents code base

[11:40] Christopher David: And they're all pretty much empty we have self-logger

[11:44] Christopher David: Confident environment and I've been kind of taking the previous code and fully it into environment

[11:53] Christopher David: Just a nicer structure than we had previously been kind of stopping everything into one class

[11:59] Christopher David: And then what we had previously been calling fairy or now I'm gonna call auto dev

[12:07] Christopher David: And this is the first like real agent type

[12:10] Christopher David: We're gonna be focusing on

[12:14] Christopher David: So when we initialize an auto dev agent we're passing it in to get hub repo. It's to get up agent

[12:22] Christopher David: I do not want to be completely tied to get hub over the long term, but

[12:26] Christopher David: For now

[12:29] Christopher David: Good way to start

[12:30] Christopher David: You can imagine that right now like our environment is all get hub specific stuff

[12:34] Christopher David: But you can imagine environment is being an interface that like the get hub environment

[12:38] Christopher David: Implements it in one way or the get lab environment

[12:43] Christopher David: Would implement the same API but in different ways so this is all very

[12:46] Christopher David: Extensible depending on what direction we want to go

[12:49] Christopher David: So when we initialize an auto dev agent it's gonna initialize the action to create it curriculum the environment memory their function skills

[12:58] Christopher David: And so I've begun building this out with these summary and

[13:02] Christopher David: So I'm taking kind of like the the basic

[13:06] Christopher David: Audit flow that I had started up here

[13:09] Christopher David: Audit because we want to kind of start by having an agent build a really sophisticated

[13:17] Christopher David: Understanding of a code base before taking

[13:20] Christopher David: action

[13:23] Christopher David: So for now

[13:26] Christopher David: Just a demonstrate where we're at currently

[13:34] Christopher David: If I run this test

[13:38] Christopher David: So I you know I

[13:40] Christopher David: Tested it initially with our repo. I'm now passing in the llama dot cpp repo as an example

[13:48] Christopher David: I'm initializing it and calling a run

[13:52] Christopher David: So in the constructor so I think if the logger so I think the modules run we're calling which want to observe the environment like generate an observation about the environment in natural language

[14:03] Christopher David: and

[14:06] Christopher David: You can see what we have here

[14:09] Christopher David: Is the get-up repo called llama dot cpp on by gurgon office about the repository with a current issued number name full of ball

[14:16] Christopher David: All right now we're only pulling the most recent issue

[14:19] Christopher David: So this isn't the best

[14:21] Christopher David: summer, but

[14:23] Christopher David: Repository appears to be fairly active blah blah

[14:26] Christopher David: Uh

[14:33] Christopher David: Let's run that one more time and focus on our repo

[14:44] Christopher David: So what this is doing is it's

[14:49] Christopher David: Grabbing raw data from the get-up API of the most recent issue the basic repo details

[14:56] Christopher David: The kind of bolder content of the root level full file that it's passing it to an inference helper

[15:04] Christopher David: That right now just goes to gbt for

[15:06] Christopher David: to summarize the unsharsher data with the prompt of

[15:12] Christopher David: You are a senior developer who excels and summarizing get every repo data into actual insights respond concise

[15:18] Christopher David: We're talking to a junior developer who's getting acquainted with repo

[15:21] Christopher David: So one of the things that we're gonna be doing is is exposing all of these prompts also to the interface

[15:26] Christopher David: to enable other people to kind of try remixing agents and

[15:31] Christopher David: Kind of playing with prompts for different different use cases

[15:45] Christopher David: Okay, so

[15:47] Christopher David: Over the next you know a couple of days a couple of videos will be

[15:51] Christopher David: kind of replacing the basic initial flow that we had here

[15:56] Christopher David: To use these new modules and kind of reorienting this all-stue to use more of the kind of standard

[16:03] Christopher David: Chat UI and have it kind of explain its thoughts as it goes

[16:06] Christopher David: But what people do you ever just kind of get acquainted with these modules here because

[16:12] Christopher David: You can now imagine that any agent that you would want

[16:18] Christopher David: Is gonna be able to be expressed

[16:20] Christopher David: in these eight or nine

[16:24] Christopher David: modules

[16:30] Christopher David: And so when we do things like bounties and assigned bounties

[16:35] Christopher David: We're gonna want people to be kind of

[16:38] Christopher David: Increasingly building out the algorithms for these specific modules

[16:44] Christopher David: If you upgrade the memory module

[16:47] Christopher David: um

[16:48] Christopher David: Payments module we want that reflected in these modules go this

[16:53] Christopher David: files here

[16:56] Christopher David: And then the exciting part which we're gonna start focusing on probably

[17:02] Christopher David: Certainly by the end of the year we want people being able to see

[17:08] Christopher David: What the payments associated with each step are so these two are just get how the API calls so probably no payments associated there

[17:15] Christopher David: This is a GPT4 call but you can imagine that this is instead a mixed-dry call

[17:20] Christopher David: Service by a

[17:23] Christopher David: GPU to a be a computer provider and so the money that gets

[17:28] Christopher David: You know paid from the buyer of this compute to the seller could be represented here

[17:38] Christopher David: Yeah

[17:42] Christopher David: See you soon