Transcription: OpenAgents Episode 020 - Planning a GitHub Agent

Source: https://twitter.com/OpenAgentsInc/status/1725597044981617119 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 020: Planning a GitHub Agent We review our progress and plan ... Upload date: 20231117 Transcription model: mlx-whisper/mlx-community/whisper-tiny Generated at: 2026-06-01T20:41:13Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: All right, now let's plan our agent.

[00:03] Christopher David: So we began with these agent first principles.

[00:07] Christopher David: We took a little detour to get our own retrieval augmented generation,

[00:12] Christopher David: working with our own locally stored embeddings generated by GPU Topia.

[00:18] Christopher David: We built out some of the chat data models, but not yet in the plan and act models

[00:26] Christopher David: that will give this actual agency.

[00:29] Christopher David: But we first want to kind of aim it a little bit more like before we do the actual

[00:36] Christopher David: carrying out of tasks.

[00:38] Christopher David: We kind of need to figure out what the knowledge is actually going to have.

[00:42] Christopher David: So it's really cool that we have the ability not to upload a PDF,

[00:45] Christopher David: turn it into embeddings, and get really good answers.

[00:50] Christopher David: So we've kind of got the knowledge it wasn't trained on.

[00:56] Christopher David: We now have the ability to import knowledge and convert it into a form that an agent is going

[01:02] Christopher David: to be able to take action on.

[01:06] Christopher David: There's a few different ways you could go from here.

[01:08] Christopher David: We could kind of try to build more general purpose things.

[01:12] Christopher David: It can be used by other people.

[01:13] Christopher David: But I really want to laser focus first on one type of agent, one use case.

[01:20] Christopher David: And I was thinking, I think developer focused agents make sense given that the first people

[01:27] Christopher David: that we need to empower our developers because we have developers help build out this platform

[01:33] Christopher David: and ecosystem.

[01:34] Christopher David: And I love the idea of kind of having an agent help with something that someone is not

[01:41] Christopher David: otherwise good at, like smartwatch development, doing some kind of personal agent

[01:46] Christopher David: app on a smartwatch.

[01:47] Christopher David: I think that might be a good project, like number two or number three.

[01:51] Christopher David: But the first thing that I'd love to have is an agent to help build out our actual main product

[01:59] Christopher David: repo here on GitHub.

[02:00] Christopher David: So GitHub agent seems to make sense.

[02:06] Christopher David: So let's do this.

[02:09] Christopher David: Let's review a little bit of work that I've done on this before.

[02:17] Christopher David: Just kind of experimenting in the past with having an agent

[02:23] Christopher David: be able to have intelligent conversations in a GitHub issue.

[02:31] Christopher David: Because I think that'll be a good way to interact with our agents.

[02:35] Christopher David: It's nice to have a chat interface.

[02:37] Christopher David: But for GitHub development, maybe having the agent, I mean, the agent's going to need to interact with

[02:46] Christopher David: third party APIs.

[02:48] Christopher David: There's probably a place where some of the interaction makes sense to have in a chat interface,

[02:53] Christopher David: like we built here.

[02:58] Christopher David: But I also like the idea of being able to have agents use GitHub issues.

[03:11] Christopher David: I'll just make sure that that's working.

[03:14] Christopher David: What are soft bots?

[03:28] Christopher David: It works.

[03:31] Christopher David: I think the longest part of that is passing a bunch of context to our language models,

[03:38] Christopher David: which are currently being served by generally lower end consumer hardware.

[03:43] Christopher David: But that'll speed up over time.

[03:44] Christopher David: But this is all being done in a local and decentralized way.

[03:46] Christopher David: That's really cool.

[03:48] Christopher David: Okay, so let's step through this GitHub issue.

[03:52] Christopher David: I had previously experimented with this idea of a GitHub agent.

[04:01] Christopher David: I called Fairy AI.

[04:05] Christopher David: And this was just a little Python repo.

[04:09] Christopher David: Let's go through this.

[04:11] Christopher David: On learning the information or making observations about the world,

[04:14] Christopher David: Fairy should store that information in a vector database as what we will call a memory.

[04:19] Christopher David: Later, Fairy will query for relevant memories when preparing a response to a question.

[04:23] Christopher David: We're deciding what actions to take next.

[04:26] Christopher David: We need to figure out how to structure the memories along with relevant metadata.

[04:30] Christopher David: For a vector database, we will use Python, their client, Python,

[04:34] Christopher David: client, docs are here.

[04:35] Christopher David: Okay, so this was four months ago.

[04:37] Christopher David: We are not going to use Python.

[04:39] Christopher David: We're going to use our own vectors via PG vector locally via Postgres.

[04:44] Christopher David: But nice to have.

[04:48] Christopher David: This is the generated by GPT4 passing in this and maybe some basic system problem.

[04:55] Christopher David: I propose the following structure for story memories.

[04:58] Christopher David: You need to identify our timestamp vector.

[05:01] Christopher David: This is pretty much what we already have with the vector embedding and metadata.

[05:05] Christopher David: Um, do do do do do.

[05:11] Christopher David: This is a very simplified example, but hopefully it provides a starting point.

[05:14] Christopher David: Managing memories like this might include mechanisms for querying memories based on the metadata

[05:17] Christopher David: and managing the memory index.

[05:19] Christopher David: EG removing old memories.

[05:21] Christopher David: It will need to develop efficiently to perform these operations in the number of memories.

[05:24] Christopher David: Okay.

[05:26] Christopher David: I said, your response seemed directionally accurate, but if you use a correct Python syntax,

[05:30] Christopher David: please consult the following from the documentation.

[05:32] Christopher David: Here's an example of where it had outdated information because GPT4 is trained on older stuff.

[05:37] Christopher David: And like here's the newer stuff and it was able to kind of do this.

[05:41] Christopher David: But here's where us having our agent have access to additional PDFs that we're uploading.

[05:46] Christopher David: We'll come in handy because it's doing information.

[05:49] Christopher David: So this kind of back and forth would not hopefully be needed.

[06:03] Christopher David: Okay.

[06:07] Christopher David: I draw an excerpt here from the generative agents paper, which is pretty relevant to what we're doing.

[06:18] Christopher David: Let me just grab a little bit of Twitter commentary real quick about the generative agents paper.

[06:24] Christopher David: So GPT4, a completely customizable group of AI agents with independent personalities, memories, and directives.

[06:34] Christopher David: We were inspired by the Stanford generative agents paper and decided to make an implementation that anyone could run.

[06:44] Christopher David: And this is kind of puts it in like a...

[06:46] Christopher David: So I don't know... So there's GPT team which has like a little interface like this.

[06:56] Christopher David: And then the original paper like created this like open world,

[07:01] Christopher David: not open world, like a little 2D RPG with different characters learning from each other.

[07:05] Christopher David: But the more interesting part of the paper was they introduced these concepts of

[07:10] Christopher David: long-term memory and retrieval used by agents that it makes sense to adapt.

[07:17] Christopher David: I think.

[07:19] Christopher David: Okay, so let's go through this.

[07:21] Christopher David: We want to adapt the memory stream architecture mentioned in the generative agents paper excerpted below.

[07:27] Christopher David: As a specific information, initially a little center almost exclusively on data received via the

[07:31] Christopher David: GitHub API, reading and writing code, commits issues, et cetera, and web scraping of relevant

[07:36] Christopher David: documentation to provide very enough information to accomplish tasks, memory and retrieval.

[07:41] Christopher David: So a challenge. This is from the generative agents paper. Creating generative agents that

[07:45] Christopher David: can simulate human behavior requires reasoning about a set of experiences that is far larger than

[07:49] Christopher David: would should be described in a prompt. Did they have these typos?

[07:53] Christopher David: As the full memory stream can distract the model and does not even critically fit into the

[07:57] Christopher David: limited context window. Consider an auto-dev agent.

[08:01] Christopher David: I'm guessing I swapped that in.

[08:05] Christopher David: What are the most important? Yeah, I actually. I think I.

[08:11] Christopher David: Adapted this a little bit for our use case. What are the most important lessons you learned?

[08:14] Christopher David: First summarizing all the agents experiences to fit in the limited context window with a language

[08:18] Christopher David: model produces an uninformed number of response. With the agent discussions topics such as

[08:22] Christopher David: observations on particular coding files or its review of a human code is work. Instead of summarizing,

[08:27] Christopher David: the memory stream described below services relevant memories resulting in a more informative and

[08:32] Christopher David: specific response that mentions the agent's lessons learned based on higher level reflections.

[08:38] Christopher David: Approach. The memory stream maintains a comprehensive record of the agent's experience.

[08:44] Christopher David: Let me just skip ahead. We're going to go through this, but just so you can kind of see where we're

[08:49] Christopher David: going with this. Think about as it's talking about like a record of the agent's experience.

[08:56] Christopher David: Things that are like too much information for an agent to be able to stuff into its context window

[09:00] Christopher David: that we still want to be able to reason about. For example, we have here in the open agents repository

[09:07] Christopher David: 121 commits with an annotation and the code diff with the previous version was and the new version.

[09:22] Christopher David: So an agent could learn so much information here. Being able to see like all of how I built

[09:34] Christopher David: all of this is a lot of information that we might want the agent to parse and create higher level

[09:43] Christopher David: reflections and observations from. Just the commit history would be great. Now imagine combining this

[09:51] Christopher David: with whisper transcripts of the last 20 videos where I commented every part of this.

[10:04] Christopher David: There's so much context there. I would love for an agent to be able to say, oh, you know that

[10:10] Christopher David: thing that you did in video for where you had to travel with this integration. Well, it's because

[10:16] Christopher David: you didn't do this or you put this in that wasn't the best. Let's use this. Just being able to

[10:25] Christopher David: smartly navigate the context of what we have built already will give it let it provide sterling

[10:33] Christopher David: or like like glitteringly intelligent responses. And I have to say this GitHub repo, this GitHub conversation.

[10:40] Christopher David: After a number of back and forth, the level of like clarity and intelligence from these responses from

[10:55] Christopher David: ferry are some of the like best written responses I've ever gotten from GPT, any GPT, any agent, any

[11:07] Christopher David: AI, L, and anything ever. And in part is because we were able to like have an intelligent conversation

[11:15] Christopher David: keep accessing that back in. I want to kind of build on this. Okay, let's come back hopefully that's

[11:25] Christopher David: some helpful context. So we're doing a memory stream and to also skip it, we're going to be creating

[11:33] Christopher David: going to be creating like, like, Lairville models to implement this memory stream idea.

[11:39] Christopher David: Approach the memory stream maintains a comprehensive record of the agent's experience.

[11:43] Christopher David: It's a list of memory objects where each object contains a natural language description,

[11:48] Christopher David: a creation timestamp and a most recent access timestamp. The most basic element of the memory stream is

[11:54] Christopher David: an observation which is an event directly perceived by an agent. That'll be important.

[12:01] Christopher David: Common observations include behaviors performed by the agent themselves or behaviors that

[12:05] Christopher David: agents perceive being performed by other agents or non agents, object or people. Okay, yeah,

[12:11] Christopher David: here. For instance, ferry who is the lead developer of Arcade Arcade Labs is our company.

[12:16] Christopher David: Might accrue the following observations over time. Rends a minute of GitHub pull request to

[12:22] Christopher David: fix buggy and introduce a new feature. Eric commented on Rends' pull request.

[12:26] Christopher David: The PR's automated CICD unit test field. Eric and Rends discuss the PR in Slack.

[12:32] Christopher David: One of the issues under discussion had to bug fix listed on Stack overflow,

[12:35] Christopher David: that same issues discussed on Twitter. Our architecture implements a retrieval function

[12:40] Christopher David: that takes the agent's current situation as input and returns a subset of the memory stream

[12:45] Christopher David: to pass on to the language model. There are many possible limitations of a retrieval function

[12:49] Christopher David: depending on what is important that the agent consider when deciding how to act. In our context,

[12:53] Christopher David: we focus on three main components that together produce effective results.

[12:58] Christopher David: Recent C assigns a higher score to memory objects that were recently accessed

[13:02] Christopher David: so that events from a moment ago or this morning are likely to remain in the agent's

[13:06] Christopher David: attentional sphere. In our implementation, we treat Recent C as an exponential decay function over a

[13:11] Christopher David: number of sandbox game hours. That's from the original thing since the memory was last retrieved.

[13:16] Christopher David: Importance distinguishes mundane from core memories by assigning a higher score to those

[13:22] Christopher David: memory objects that the agent believes it'd be important. For instance, a mundane event such

[13:26] Christopher David: as observing an okay message in Slack for one developer to another would yield a low,

[13:30] Christopher David: important score whereas discovery of a catastrophic bug related to an issue the agent was passed

[13:35] Christopher David: to fix with yield in high score. There are again many possible implementations of an important

[13:40] Christopher David: score. We find that directly asking the language models output and integer score as effective.

[13:45] Christopher David: Relevance assigns a higher score to memory objects that are related to the current situation.

[13:50] Christopher David: What is relevant depends on the answer to relevant to what? So we condition relevance on a query

[13:54] Christopher David: memory that sounds like what we already built. If the query for example is that the agent is deciding

[13:58] Christopher David: what files to inspect to address a bug with noster-release subscriptions,

[14:03] Christopher David: memory objects about conversation between human developers should have low relevance,

[14:08] Christopher David: whereas memory objects about how noster-release work and which files in our codebase

[14:12] Christopher David: relate to that should have high relevance. In our implementation we use the language models

[14:16] Christopher David: to generate an embedding vector of the text description of each memory. Then we calculate relevance

[14:22] Christopher David: as the cosine similarity between the memories embedding vector and the query memories embedding vector.

[14:27] Christopher David: We already have that. To calculate the final retrieval score we normalize the

[14:31] Christopher David: ReasonsC relevance and the important scores to the range of 0-1 by min max scaling.

[14:35] Christopher David: The retrieval function scores all memories as a weighted combination of the three elements.

[14:39] Christopher David: The top ranked memories that fit in the language models context window are then included in the

[14:43] Christopher David: product. Let's see what Fairy says to that.

[14:50] Christopher David: The excerpt from the generative agents paper helps to provide context on how we want

[14:55] Christopher David: Fairy's, how we want Fairy to manage its memories. The concept of observations as memories and the

[15:00] Christopher David: importance of ReasonsC importance and relevance in the retrieval of these memories are particularly

[15:05] Christopher David: insightful. The next step would be to implement a similar memory stream architecture for Fairy.

[15:09] Christopher David: Observations in our case will include interactions with the GitHub API and web scraping results.

[15:15] Christopher David: And conceivably making observations based on GitHub commits and just kind of

[15:21] Christopher David: I may maybe they would call those reflections. Each observation will be converted into a vector

[15:25] Christopher David: and added as a memory and the pine cone database. The metadata for these memories might include

[15:30] Christopher David: the source of the observation to type of interaction, the file or GitHub issue in question among

[15:34] Christopher David: other details. We can use the pine cone filtering and query capabilities. We're not going to do that.

[15:40] Christopher David: To handle retrieval based on ReasonsC importance and relevance. ReasonsC can be derived from the

[15:44] Christopher David: timestamp, importance and relevance might require some thought. Perhaps we could use some of the

[15:48] Christopher David: features that we get to have API data and web scraping results. The specifics will depend heavily on

[15:52] Christopher David: the data we deal with and we will likely refine our notions of importance and relevance as we see

[15:56] Christopher David: what it is. Finally, I suggest we create an early MVP of the memory manager then iterate rapidly.

[16:03] Christopher David: Get basic vector insertion and querying working then enhance memory metadata, then boot up the

[16:09] Christopher David: importance and relevance capabilities. I'd love to help with implementing this if you'd like.

[16:14] Christopher David: We can set up a pair programming session to get started. Ferry just invited me to a pair programming

[16:21] Christopher David: session. What did I say? Let's start with an example and think through how we create

[16:27] Christopher David: memories based on this GitHub issue in its comments. Right now I'm having you

[16:32] Christopher David: generating new comments by passing all comments into the prompt we send for chat completion to

[16:37] Christopher David: open AI via the GBT for model. But that won't work indefinitely because the comments will soon

[16:42] Christopher David: exceed the context window of the model and you don't currently have access to memories of relevant

[16:47] Christopher David: information elsewhere in the code or the world that would enable you to provide the best response.

[16:53] Christopher David: One partial solution to this would be summaries. For example, Ferry could create memories of

[16:58] Christopher David: more concise lessons learned from the issue comments as a whole or perhaps segmented like what

[17:02] Christopher David: lessons were learned on each page of comments or in each individual comment. And ideally those summaries

[17:07] Christopher David: would include references to the individual pieces of content so Ferry could drill down and retrieve

[17:12] Christopher David: more details about each particular point if needed. For context, here is the current code that is used

[17:17] Christopher David: to make Ferry write a comment. How should we refactor this to implement our new memory architecture?

[17:29] Christopher David: Here's the prompt. You are Ferry an AI agent specialized in writing an analyzing code.

[17:34] Christopher David: You have been summoned to a word review issue number, the issue is type of a issue body is this

[17:40] Christopher David: comments are this for digital context. The read me is this please just bottom with the comment you'd

[17:44] Christopher David: like to add to the issue write like a senior developer would write don't introduce yourself or use

[17:48] Christopher David: flowery text or a clothing station. Okay. So we're not obviously not continuing with this

[17:56] Christopher David: Python code base where we're doing the same concepts in there. Refactor in your existing comment

[18:02] Christopher David: function to make use of the new memory system will require a few key changes. The goal is to make

[18:06] Christopher David: Ferry store each meaningful piece of information or bundle of details as an individual memory

[18:10] Christopher David: and then allow Ferry to use those memories in general as responses. To start with, let's considering

[18:15] Christopher David: the parsing of the URL and the fetching of issue data. Each of these steps result in a collection of

[18:20] Christopher David: important details for repository issue number issue title etc that will be used later. Rather than

[18:25] Christopher David: just storing them as local variables, we could also store them as individual memories. Each

[18:30] Christopher David: memory will include vector representation of the information it's storing plus potentially some

[18:34] Christopher David: additional metadata. Next, when we're fetching and looping through the comments, rather than

[18:38] Christopher David: just creating stringable comments, we could storage comment as a separate memory. We can add a

[18:42] Christopher David: natural language summary of the comment, the timestamps, and information about the comments

[18:45] Christopher David: authorize part of the memory's metadata. Also, we can use the comments ID as our unique identifier for

[18:51] Christopher David: that memory. Then, when we're creating the prompt to send to OpenAI, rather than inserting

[18:55] Christopher David: all of the raw issue and comment data directly to the prompt, we could instead use our memory database.

[19:00] Christopher David: We can retrieve all memories related to the current issue, rank them based on a

[19:03] Christopher David: recent see importance in relevance, and include the most recent, the most important ones in our prompt.

[19:08] Christopher David: If a memory is included, we can update its last access timestamp. Finally, once we've got

[19:13] Christopher David: a response from OpenAI and comment it on the issue, we can create a new memory that stores what

[19:17] Christopher David: very commented to the information isn't lost next time, very easy to interact with the same issue.

[19:22] Christopher David: Here's a rough idea of how the function might look.

[19:30] Christopher David: Reload it memories, query the memory database for relevant information,

[19:34] Christopher David: and please note this is a simple example and we'll need additional functionality to handle

[19:48] Christopher David: an array based on the recent see importance in relevant scores. Further, we need to implement

[19:52] Christopher David: vectorize, normalize and summarize, query memories, and create prompt methods, which will perform

[19:57] Christopher David: the tasks as indicated by the names. The normalize and such.

[19:59] Christopher David: I'll try to summarize function, for example,

[20:01] Christopher David: would generate a simplified normalized

[20:02] Christopher David: EG lowercase punctuation or move version of a comment text.

[20:14] Christopher David: I asked a bunch of questions.

[20:30] Christopher David: A bunch of important considerations will need to solve at some point.

[20:46] Christopher David: And then we

[20:48] Christopher David: We're going to take up where we left off.

[20:50] Christopher David: In our new open agents open source code base with our soon new domain.

[21:04] Christopher David: Our new domain we brought open agents.com.

[21:08] Christopher David: Starting with an epic GitHub developer agent.

[21:11] Christopher David: Yeah, you're.

[21:15] Christopher David: Okay, so what should we do next?

[21:26] Christopher David: I think I want to create a fresh GitHub issue.

[21:29] Christopher David: And we're going to do all of this in the open on the open agents repo.

[21:34] Christopher David: And I want to kind of start this conversation fresh.

[21:45] Christopher David: So let's take various suggestions.

[22:15] Christopher David: And

[22:27] Christopher David: Let's go back to the

[22:42] Christopher David: implement memory manager.

[23:05] Christopher David: Okay, so part of me was thinking, like, should we provide introductory background information to the project?

[23:11] Christopher David: No, let's get that elsewhere.

[23:14] Christopher David: Let's just say what we want.

[23:20] Christopher David: I don't want to think about this too much.

[23:22] Christopher David: Here's what I want to do.

[23:23] Christopher David: I want to grab this content.

[23:30] Christopher David: We want to implement the memory stream architecture mentioned in the generative agents paper.

[23:38] Christopher David: I want to do that.

[23:48] Christopher David: I want to do that.

[23:53] Christopher David: I want to do that.

[24:03] Christopher David: Well, let's do one of fix here.

[24:23] Christopher David: Let's do one of the things we want to do.

[24:33] Christopher David: Let's do one of the things we want to do.

[24:43] Christopher David: Let's do one of the things we want to do.

[24:53] Christopher David: Let's do one of the things we want to do.

[25:03] Christopher David: Let's do one of the things we want to do.

[25:13] Christopher David: Okay, we will implement the memory stream architecture mentioned in the generative agents paper.

[25:28] Christopher David: Exalted below as for specific information initially it will center almost exclusively on data received via the get up API reading and writing code.

[25:37] Christopher David: Comments issues and web scraping irrelevant documentation provide the agent enough information to accomplish tasks.

[25:51] Christopher David: The agent settings and output should be.

[26:04] Christopher David: We'll be accessible at our open AI agents web application.

[26:22] Christopher David: We'll be using the layer of the application in this repo.

[26:37] Christopher David: Let's figure out.

[26:42] Christopher David: Appropriate data models.

[27:04] Christopher David: Start party integrations.

[27:34] Christopher David: Okay.

[28:00] Christopher David: Okay.

[28:30] Christopher David: Let's do it.

[28:48] Christopher David: Issue number one.

[28:52] Christopher David: This might be the first and last issue that a human even writes here.

[29:02] Christopher David: Certainly no human should be like doing anything other than reviewing and grabbing popcorn.

[29:08] Christopher David: Now I think.

[29:12] Christopher David: Do I have very AI?

[29:22] Christopher David: I think I have very AI is still a collaborator in here.

[29:36] Christopher David: Very AI.

[29:44] Christopher David: Right access.

[29:49] Christopher David: Okay.

[29:53] Christopher David: I think that lets me assign her.

[30:00] Christopher David: I have a sign.

[30:10] Christopher David: Okay.

[30:15] Christopher David: I'm going to add a sign.

[30:25] Christopher David: I'm going to add a sign.

[30:30] Christopher David: Okay.

[30:40] Christopher David: Okay.

[30:50] Christopher David: Have it work.