Transcription: OpenAgents Episode 114 - Planning & Execution

Source: https://x.com/OpenAgentsInc/status/1816689418809319911 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 114: Planning & Execution We upgrade AutoDev's planning ... Upload date: 20240726 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T17:24:15Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: The last video we implemented components 1 through 3 of the Generative Agents paper, memory stream retrieval and reflection. Now we're going to do planning. We did a super basic implementation of planning in episode 110, just kind of seeing, you know, a little planner on the right side. Now, this is cool, and we'll pull this over into our new code base, but that was just making a plan based on this sort of like simple prompt that I had in the chat history. Whereas now, I want it to make a plan based on the available files, so using the memories of files that we created, as well as reflections that are relevant, and also any of the full content of the file, which we are not saving there. We might be saving in the files table. Yeah, so now we're going to come up with a more durable plan based on all of the things that it's learned. And as memories are generated over time, including other categories of memories, for example, there could be memories or observations based on different commits that are made over time and just different patterns and things that it learns. If it's able to build a plan based on its memories, it should get better and better over time. That would be pretty cool. So evolving beyond copy-paste, copy-paste mode is the idea. OK, so we'll do this kind of like we did last video. I didn't really prepare this. We're just going to kind of spitball and build this how I would build it. OK, let's log in. And I think I want to have a... do this as like a tool. Let's take a look at our system prompt. All right, so right now we have the system prompt in... where even is this? All right, we moved the prompts to lib prompts. And we're using system prompt. OK, so this is the current prompt. You are Autodev, blah, blah, blah. You have access to the following tools. OK, let's do this. OK, so I want to say read lib prompts system prompt prev. We want to update that system prompt with a new tool for generating a plan. The plan needs to consult the memories table. And then we'll give it the names of where it'll find the files for that. So docs generative agents. And then to put into the system prompt, like, the planner tool should be used by the agent when it receives a request from the user that demands a multi-step response. Rewrite that system prompt constant accordingly. OK, let's start by seeing what this does. Let's also make sure that I've pushed up the latest code. OK, do you think it'll work? Certainly. I'll read the necessary files, analyze the information, and then suggest updates to the system prompt to include the new planning tool. To include the new planning tool. Now let's view the docs of the generative agents file. Now that we have a good understanding of the current implementation and the desired changes, let's update the system prompt to include the new planning tool. We'll focus on integrating the memory system, reflection system, and planning capabilities into the existing prompt. Here's the updated system prompt. Ooh. All right, what I'm going to do is put this prompt... I'm going to call this main. OK. Hmm. I actually don't want to use that yet because we don't have these tools all built. I don't know if a tool... because, like, we want the generation of the plan to retrieve memories. Man. Yeah, I was thinking that the plan's generation step would retrieve the memories, but I guess it makes sense for the agent to pull those up before generating the plan. OK, let's see what it actually said about this. OK, your workflow should typically follow... wonderful, those backticks missed... backslash. If the task is complex, use the generate plan tool to create a multi-step plan. Wow. This doesn't seem right. Let's try removing these. Out of curiosity, let's just see what happens if I swap that prompt in. Read the readme and make a plan to do the next steps. Now that I've read it, I'll make a plan. Does that fail? OK. It stopped. I'm guessing because it realized that it didn't actually have access to the tool that I told it that it did. OK. So let's undo that. Let's work with the previous prompt to actually write those tools. So I'll say... Let's go back to the previous chat. Let's try it. Thanks, now go look at tools slash index.ts and otherwise browse through the tools directory as needed to write those tools and make the needed edits to index.ts. No, no, no, no, no, no, no. No, you added tool references to the system prompt that are not yet in the tools directory, and you need to write them. Yeah, Sona is really, really good, Sona 3.5. Like, sometimes if it makes a dumb mistake, you can usually just kind of be like, you made a mistake, and it'll self-correct. It's like the retrieval in its own context isn't super accurate, but if you give it a nudge, it'll usually get it. OK. It looks like create patch is already implemented. Ask question. Suggest system improvement. No, this is getting this from somewhere else. No, no, no, no, no. Oh, that's getting it from there. No. I don't remember how to copy out multiple files. There we go. OK. No. We need these tools. OK. Generate plan. Retrieve memories. Yeah. That would be pretty similar to remember insights, but this is the new version. Add memory. Why not? Oops. And reflect on code base. Why not? Let's just try it. Generate plan or retrieve memories. Add memory, reflect on code base. OK, let's do it. Generate plan. Retrieve memories. Add memory, reflect on code base. GPT-4. Yuck. OK. I'll call this initial new tools defs. OK, let's go through these one by one because there's some stuff that's not right, I can see. So generate plan. Generate a multi-step plan for complex tasks. OK, so in this case, we are passing in task and the context. So the expectation is that we're retrieving the context beforehand. We may need to clarify in our system prompt that. But this is cool. We just need to swap out GPT-4 for Do I have? I don't have any LLM generations in tools yet, it looks like. Create patch doesn't, find files doesn't, open PR doesn't, query files by vector doesn't. Record observation at least calls embedding. Where do I generate? Oops. What did I hit? So I don't want the stream in the tool execution. Should I make a util for this? We do not want to use the OpenAI SDK like that. We are using the Vercel AI SDK. And for completions, we are using... What do they even call it? Generate plan, retrieve memories, add memory, reflect on code base. OK, let's go through these one by one because there's some stuff that's not right, I can see. So generate plan. Generate a multi-step plan for complex tasks. OK, so in this case, we are passing in task and the context. So the expectation is that we're retrieving the context beforehand. We may need to clarify in our system prompt that. But this is cool. We just need to swap out GPT-4 for... Do I have? I don't have any LLM generations in tools yet, it looks like. Create patch doesn't, find files doesn't, open PR doesn't, query files by vector doesn't. Record observation at least calls embedding. Where do I generate? Oops. What did I hit? So I don't want the stream in the tool execution. Should I make a util for this? We do not want to use the OpenAI SDK like that. We are using the Vercel AI SDK. And for completions, we are using... What do they even call it? Generate plan, retrieve memories, add memory, reflect on code base. OK, let's go through these one by one because there's some stuff that's not right, I can see. So generate plan. Generate a multi-step plan for complex tasks. OK, so in this case, we are passing in task and the context. So the expectation is that we're retrieving the context beforehand. We may need to clarify in our system prompt that. But this is cool. We just need to swap out GPT-4 for... Do I have... I don't have any LLM generations in tools yet, it looks like. Create patch doesn't, find files doesn't, open PR doesn't, query files by vector doesn't. Record observation at least calls embedding. Where do I generate... Oops. What did I hit? So I don't want the stream in the tool execution. Should I make a util for this? We do not want to use the OpenAI SDK like that. We are using the Vercel AI SDK. And for completions, we are using... What do they even call it? Generate plan, retrieve memories, add memory, reflect on code base. OK, let's go through these one by one because there's some stuff that's not right, I can see. So generate plan. Generate a multi-step plan for complex tasks. OK, so in this case, we are passing in task and the context. So the expectation is that we're retrieving the context beforehand. We may need to clarify in our system prompt that. But this is cool. We just need to swap out GPT-4 for... Do I have... I don't have any LLM generations in tools yet, it looks like. Create patch doesn't, find files doesn't, open PR doesn't, query files by vector doesn't. Record observation at least calls embedding. Where do I generate... Oops. What did I hit? So I don't want the stream in the tool execution. Should I make a util for this? We do not want to use the OpenAI SDK

[19:59] Speaker: Generate text. Is that the correct syntax for chat? Generate text with messages. Okay. That wouldn't be prompt, it would be messages. I appreciate your guidance. Generate embedding for haiku. No. But I will take this. I don't know if that's the right syntax, but we'll try it. And then Claude 3 haiku. No. Okay, so generate plan. Seems right. Retrieve memories. I'm guessing that will need to be workshopped. Add memory.ts and reflect on codebase. Okay. Let's see what happens if we swap in that prompt. So I'll give the same prompt I did last time. Aha! So I think it says no content available while it's working, so I should change that to be something else. Generating plan for task. Successfully generated a multi-step plan for the task. But I don't see it. Did it give it to me in a... Ah, what's that? Hey, there's the plan! Baller! Is it just starting it? Okay, I don't know what it's doing right now. How many... So this will loop through tools up to the... what I specify here, max tool round trips. I need to make like a stop button. LOL. Oh my gosh. Okay, it finally stopped. Let's see what it did. Okay, so when it makes the plan, I want it to show the plan in a sidebar here. Okay? Okay, well it generated a plan. That's freaking cool. Now we need to show it. But first, where did it put the plan? Oh my gosh. Okay, so this is interesting. So tool invocations, it says the task is implement the next steps for Autodev as outlined in the README, focusing on the unfinished items in the buildout order and the next steps section. Yeah, that's a nicer version of what I directed it to do. And then the context is the project is an AI-assisted coding tool called Autodev. The main priority is preparing for a demo video. Promptable requests in 60 seconds. That's outdated. I already did that. Several key features are already implemented, but there are still important tasks to complete. Now what I don't know yet is if that's all of the context that it's using, just this, or if it has access to things earlier in the conversation. If it's earlier in the conversation, then that's easy because I can just make sure that it calls the tools that we'll need for context, like retrieve relevant memories, etc. before that. I can just massage that in the system prompt. So let's read the plan here and see if it needs to be massaged further. Okay, projects. So read and format the plan part of this object in clean markdown code block. So I want to say no. No. Yeah, I don't think it's had any of the relevant context. So now I want to go back to the prompt. I guess I'll swap this back out to use the prompt that was working. So this is our experiment. So basically, the plan that it gave me was just it kind of sort of hallucinating and giving me this kind of general stuff based on the very limited context that we passed in. And now I need more context, like the memories that are relevant to this such that it can give it like specific file names and such. So what I'm going to do now is I'm going to say read lib prompts main.ts. We want to update that system prompt to pass in more context to the planner tool. What did I call it? To the generate plan tool. Right now, the only context being passed in is a brief sentence. But as you can read in doc slash generate pages, readme, we need to pass in relevant memories that we should retrieve beforehand via the other new tools we have. See tools slash index.ts. So when the tool call is made to generate the plan, it will know those relevant memories and reflections to pass into the context. Clarify the system prompt to make sure it does all that. Do you think this will work? Thank you for providing the content. Now let's update the system prompt to incorporate more context for the generate plan tool, including relevant memories and reflections. I'll suggest changes to the existing prompt to achieve this. Here's the updated section for the generate plan tool usage. When using the generate plan tool, use it for complex tools that require multiple steps, blah, blah, blah. Before generating a plan, gather relevant context. Use the retrieve memories tool to fetch relevant memories. Retrieve memories. Use the reflect on codebase tool to generate highlights. Now, I hope this is the right one limitation from previous was it doesn't know the repo ID. I have to put that in codebase overview. Retrieve memories. Did it give? I'm looking for the parameter signature. So query and limit. That's correct. Reflect on codebase. Codebase overview and focus area. I'm actually just going to remove that. The reflection thing doesn't really belong in that. I guess I could put it there. Okay, so for now, let's just go to prompts main when using the generate plan tool. Does the generate plan tool require a repo ID? No. Task and context. Okay. I'll just say call the retrieve memories tool multiple times with different queries until you have sufficient context to answer the question. Optional, you may supplement the memories with insights from traversing the codebase using the view hierarchy and view file tools. Maybe. Oh, I got to escape these. I'm sure there's some baller NeoVim way of doing this. I'm still on NeoVim. Noob. Wow. Should I try it? Oh, wait. And update the workflow steps. Do I have to? Where'd it go? All right. I guess I should do this. Let me see this. Add backslashes before each of the backticks here. Put all in a markdown code block. Remove reference to the generate insights tool. I said remove that. Reflect on code. Okay.

[39:59] Speaker: OK. OK, let's try it. New chat, and I'll use the same prompts. All right, this time I'll say, read the readme and make a plan of the next steps, but don't execute the plan yet. Eager. Certainly, I'll read the readme file and create a plan for the next steps. Now, we want it to generate the build the context first. It did not. OK, so what it's doing here is it's summarizing just a little bit of the readme. Yeah, now it's making the plan based on this context, which is an improvement, but we still need to massage the system prompt some more. Let's see. What if I say, do not call until after collecting context via other tools? Out of curiosity, is that enough to change the response meaningfully? I do not want it to call generate plan right after getting the readme. Use our beautiful memories. No. Let me see if I can guilt it a little bit here. You ignored my instructions to first gather context from memories from the codebase. View file. Let's do that now. OK, failed to retrieve memories because... Oh, we do need to generate embedding in lib llm. I see. OK, well then I'm going to take it from... Actually... We have generate embedding right there. So where was I just retrieve memories? So I should be able to just retrieve it from there. OK, so out of curiosity... So it plowed past that and... Based on the context we've gathered and the generated plan, here's a summary of the next steps. Same limited context. Doesn't seem right. Read the readme and docs slash generativeagents.md. And then generate a plan. Before generating the plan, use retrieve relevant memories of files from via your tools. Query files by vector. Match count, match threshold, query embedding repo ID. OK, so at least it found one relevant file and passed it to generate plan. Let's see if that context looks any better. The repo map feature needs to be implemented to enhance the AI's understanding of the codebase structure and improve its code suggestions. The implementation should follow the components outlined in the repo map document, including the repo map class, file system interface, code parsing, and analysis integration. Yeah, the plan is like thoroughly read and understand it. Although, you know what? Maybe this is good enough because maybe the plan is just a starting point. Well, I guess the proof of whether this is effective or not is if I just say execute the plan and they can do something meaningful. There's always going to be room for tweaking prompts and such and optimizing tool calls, blah, blah, blah. Certainly, I'll start executing the plan. Let's begin with the first few steps. Review and analyze requirements, thoroughly read and understand the repo map document. We've already done this in our previous interactions. Identify all components and functionalities required for the repo map feature. The key components we need to implement are Review the existing AutoDev codebase structure and architecture. To do this, let's use the View hierarchy tool. Identify integration points for the repo map feature. The main integration points for the repo map feature will be lib, components, store, app, set up development environment. Ensure all necessary dependencies are installed. Next.js, React, TypeScript, etc. Let's check package.json. Great, we can confirm that the necessary dependencies are installed. Additionally, we can see that the project is using other relevant libraries like TreeSitter for code parsing, which will be useful for our repo map implementation. Create a new feature branch for repo map implementation. Since we don't have direct access to the repository, we can fix that at some point. We'll assume this step has been completed. In a real development scenario, you would create a new branch with a command like blah blah. Implement core repo map class. Create a new file, repo map, in the appropriate directory. It seems like it doesn't exist yet. Let's create it and implement the basic structure. This implementation provides the basic structure for the repo map class with methods for adding, updating, removing, and querying nodes. It also includes serialization and deserialization methods for persistent repo map. Next steps, we should implement the file system interface to populate the repo map, blah blah blah. Let's start by implementing the file system interface. We'll create a new file in the lib directory. And here's just things where I could give it access to like create patches or, yeah, that's probably the next thing is to like when it wants to create, have it create a patch. It looks like this doesn't exist yet. Let's create it with a basic implementation. This implementation provides methods to scan a directory recursively, read file content to get file statistics. Now that we have both repo map and file system interface implemented, we can create a function to populate the repo map using the file system interface. Let's add this function to the repo map class. Next steps, implement the code parsing analysis components to extract symbols from the files, blah blah. We'll start by implementing a basic code parsing analysis component. TreeSetter, code, blah blah blah. Starting to get a little laggy. Yeah, because what I want, instead of copy pasting these 3 billion things, reconciling diffs, it's like, no. Give me all that as one PR. If this was a complete PR that was submitted to GitHub and it built successfully, like there was no crazy errors, with like a nice comment, then I would happily review this. This is a lot. Like, holy shit. And this might not be 90%, this might not be all the way correct. It might be 80 to 90% correct. If we give it access to the output of a GitHub Vercel build process, which automatically drops the results of that into the comments, we've said that up before. If it reads any failures, it can then self-heal. Okay, so the plan part is not like fully done because I want to be able to like stream it here like our episode 110, but we were able to pass some context in to create the plan and it's executing the plan. So I'm going to call that a win for now. And in the next video, I think we'll do the opening the PR with all these diffs. See you soon.