Transcription: OpenAgents Episode 112 - Using AutoDev

Source: https://x.com/OpenAgentsInc/status/1815818389983564079 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 112: Using AutoDev We review our new AutoDev product and use... Upload date: 20240723 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T17:49:22Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: And we're back. So AutoDev has really evolved into its own product, and it'll be ready for users in a few days. Right now, I thought it would be fun to show you how I use it. The idea is to replace CLOD artifacts. Right now, the sort of state of the art for developer productivity is using CLOD artifacts, which has just been like a massive, massive upgrade for me and others. You know, I've been heavily using ChatGPT for a year, switched from ChatGPT to the OpenAgent's chat interface. But my productivity increase from this interface was blown away by CLOD artifacts. So now it's like, hey, let's create an interface that's leveled me up even further by giving it deep, you know, access to my code bases. So I've got an attached repo here, the AutoDev repo, and I can do things like read a particular file path. And let's just start with like an analysis. And it loops through just different tools that are connected to the GitHub API endpoints. So it can view the file. I need to add the file path up there. But so I've got a directory in this repo with markdown documents about different things to kind of give it a little briefing. So it says, certainly, I'll start by reading the contents of the docs, knowledge MD and analyze related files. Let's begin with viewing the file contents. The workspace component, that's this over here, has a tab called Knowledge. The tab includes React 3Fibre Canvas that displays the knowledge graph. So that's what I'm going to be building in this video. And my approach for this currently is to kind of specify what I want in a markdown document, give it some paths for related files because it can easily kind of pull up files if I give it the path. And we're going to drive out this pane over here. The idea is I want a kind of a visual representation of the knowledge that a code base has access to. Knowledge could be things like a description of a system, a summary of a file, all sorts of different things. But I'd like to see like a visual representation of what that is per code base. So let's set through this a little bit that you've read the contents. Let's analyze the related files, mentioned in this document and gather more information. There's the contents of that, the contents of the actual knowledge component. Now that we have analyzed the related files, let's summarize our findings about the knowledge feature. The knowledge feature is implemented as a tab in the workspace component. The workspace component includes the knowledge tab along with other tabs like patches and coder. The knowledge tab is set at the default. Knowledge component is currently a placeholder. According to the docs slash knowledge MD file, the knowledge tab should include a React 3Fibre canvas that displays the knowledge graph. However, this functionality is not yet implemented in the current version. The knowledge graph is intended to show the nodes and edges representing the knowledge of the code base, specifically insights on the code base. The docs slash knowledge MD file mentions that we should consult docs slash database dot MD for the schema and implementation details for insights, the things that we're going to be representing. We haven't viewed this file yet, but it likely contains important information about how the insights are stored. To fully implement the knowledge feature, we need to add the React 3Fibre canvas to the knowledge component, implement the knowledge graph visualization, fetch and display the insights data as nodes and edges, and then consult this. So my goal for this video is to do all of this and get a basic visualization done. But yeah, I'll say, yes, start by looking at the DB file, DB markdown. You can have typos and stuff like it'll just it'll pick the best tool. And this is using Sonnet 3.5 in the background. So this is just a summary of the database interactions. And I've already got all this set up using the Superbase database files, observations and threats. The observations table has observations that are going to be generated by an LLM. We'll use Sonnet for some of that and maybe like Llama 3.1 for some of that. Schemas, blah, blah, blah. And then each of the observations has vectors so we can do searches, searches over it. OK, thank you for providing the contents of the docs slash database file. This gives us valuable information about the database structure and how it relates to the knowledge feature. Let's summarize the key points. We use Postgres via Superbase with PG vector support for similarity search. There are three main tables relevant to the knowledge feature files, information about files in a repository. So we don't have to keep hitting the GitHub API. We can have our own kind of cache of files with like the SHA of each and like only fetch it on the updates as well to be able to like build intelligence about changes of files between updates and stuff. OK, the observations table is particularly important for the knowledge graph. It has the following key columns. Primary key, the type, whether it's a summary or a reflection or other things. There's some papers I'm excited to implement. What was they called? There's there's been a bunch of papers about like different observations and memories, just types of like reflection. OK, content, the vector, if it references a file or a thread and a parent observation ID. So things can be linked in different ways. The system uses vector representations. We're using the OpenAI vector library for both files and observations, which enables similarity search. Observations can be linked to files, threads or other observations, which will be useful in creating for creating relationships in the knowledge graph. The system supports similarity search across files and observations, which can be used to find related content in the knowledge graph. Based on this information, we can start implementing the knowledge graph feature. Here's a plan to proceed. Update the knowledge component to include a React 3 fiber canvas for rendering the graph. Create a data fetching mechanism to retrieve observations from the database. Implement a graph visualization using the observation data where nodes represent observations, edges represent relationships between observations. Yep. Add interactivity to the graph, allowing users to click on nodes to view observation details. Implement a similarity search feature to find related observations when a node is selected. Let's start by updating the knowledge component to include React 3 fiber canvas. We'll need to install the necessary dependencies first. OK, my patch thing, I must have broken that. So I'll ask it to just put the text here. Wow, it did a bunch of stuff. Oops. My patches thing isn't working. Please just put all files in Markdown blocks here. It says let's start with the updated lib slash insights. So this is cool because usually when I'm doing stuff with like cloud or whatever, I would need to paste it in first. You know, keep copy pasting, make sure it has the most updated version of it and to be able to just kind of dynamically pull. Like I didn't need to paste in the existing lib insights. Quick sanity check to make sure this all looks good. Yeah, it does look good. So we're going to try the old. I'm still copy pasting this out just because the patch thing. Isn't working. But yeah, in a previous tweet I made last week, I had it generate actual patches and then have a button to open a PR and I got that working for one commit at a time. So I got to do some more work to massage that. I'll probably actually use this knowledge graph to build knowledge to help us like rapidly implement that in the next video. Anyway, so OK, so I copy paste that into lib insights. Let's update the knowledge tab. And I guess this is assuming that we have these installed, which we don't. So yarn add react 3 fiber, react 3 dry 3. And we probably need types. These updated files implement the knowledge graph feature as we discussed earlier. The lib insights file now includes the fetch observations function to retrieve observations from the Superbase database. And the knowledge file renders these observations as a 3D graph using 3.js and react 3 fiber. OK, well, I'm sure we missed something, probably. So. Oh, error fetching observations. Relation public observations doesn't exist. So it probably. What is it trying to fetch? Fetch observations. OK, so in this case, it's trying to fetch observations. I don't know why. Maybe because I used insights and observations interchangeably. So I'm just going to say we don't have an observations endpoint. It's insights. I'm just going to give it a little nudge with the right file path. Eventually, it'll be able to better self heal once it has like its own knowledge base of what things it should look at. But for now, we don't have an observations endpoint. Insights. Consult the. app slash API slash insights slash route.ts and use that. Refactor and put code blocks here. Don't make patches. Based on the contents, I can provide you with the relevant code blocks for the insights endpoint. The file contains both post and get methods for handling insights. Here are the refactored code blocks. No, that's not what I wanted. No, I don't want you to refactor that route code unless needed for the knowledge graph. Update the knowledge graph code to use this insights endpoint. I apologize for the misunderstanding. You're right. We should focus on updating the knowledge graph code to use the insights endpoint instead of refactoring the route code. Let's take a look at the current knowledge graph code and update it. Read docs slash knowledge.md for file paths. Cloud 3.5 sonnet seems to do a pretty good job of, you know, you can pass it these massive conversations and it'll do a decent job, I guess, on its own of context management because I think we're pretty much blowing past the content window or the context window. It's probably smartly condensing. I don't know. Got to experiment with that. Definitely a lot of room for improvement, which is like basic prompt engineering context building stuff. It again

[19:59] Speaker: 21164-218 creates insights tables.SQL. And I probably need to give it the repo ID. The repo ID is boom. Do I need, I don't need to thread ID, right? Just repo ID, the repository ID. Let's see if that works. Oh no, I don't want the freaking sequel. I need a stop button. All right, let's just dump this raw SQL and see what happens. I really like Supabase. Success. Oh yeah. Oh yeah. Yes, yes, yes, yes, yes. Not gonna win any design awards and I can't tell what's going on, but we've got six, five observations. Table editor, insights. So, oh, this is cool. Type, this is just like the demo data. Code quality, the types are code quality, performance, security, architecture, code style. Code quality. The code base has a high level of test coverage with 85. Performance. database queries and the user authentication model could be optimized. Input validation has been implemented correctly. The current microservices are blah, blah, blah. Yes. Okay, so one of the things that we will do is generate this analysis by having it go and evaluate different things about our code base. Very cool. Now, we don't have embeddings for those things. We need embeddings for those things. We have the code to generate embeddings, but I think I had previously done some stuff here with... We have a lib insights we're not really using. Oh, I call them observations there. Duh. Okay, so what should I do with these? All right, first let's do a commit. Initial knowledge canvas. Okay, thanks. I see five demo things there, but they are just circles. Let's make them be card components using at components card. Do we have that as a thing? Look that up to see how to use it. Show each card via React 3 dry HTML helper. Show the insight and tag on each card. Did I give it the wrong? Oh, I put at and it doesn't know where to find that. Let's see if I can find it. It does have a tool to traverse. The find files is using the GitHub API. No, no, no, no, no. Yeah, there you go. View the hierarchy. It'll probably be in the components. There it is. Oh, tool use. Oh, tool use. Oh, let's find where the demo circles are implemented. Workspace. I think I've set the max tool iterations to like 15, so it might run out here. Check the knowledge. Yes. Please don't give me a patch. The demo circles aren't there. Oh, I didn't push it live. Dang it. Okay, right now it's keyed off of my. Okay. Sorry, I had not pushed up the most recent version of the knowledge component. So please analyze that again and give me the new code in a markdown doc, markdown code block. Don't use patch. I don't know if me sending a message interrupts it. I apologize for the confusion. Oops. Oh, that's okay. View the current one. Did it stop because it hit 15? You think this is gonna work? Oh man, I gotta add the like nice little bells and whistles that Claude has, the little copy buttons and stuff. It's still a little janky, but man, this is gonna be freaking amazing. Okay. Thank you for providing copy. So this imports the card component. It replaces the node component with a new knowledge card component that uses the card component to display insights, updates the rendering logic, removes the color prop, displays the insight title and color. Lib in, did I put it? Oh, I had the wrong file. Okay, let's try that again. Uh-oh. It would be bad if I broke the current app. Okay. Delete everything, paste everything in. Oh yeah. Oh man. Okay. So... Okay, so first of all, I don't want that selection status going on, so let me do one thing. In the cards, I think if I do select none. Yeah. Oh my gosh. So I have, here's what I have. Here's what I have. Look. I have a React component inserted into a 3D scene that I can position however we want. Having these things have their own embeddings that then have their own sort of space in this scene. Oh my gosh. Okay, so there's a lot of stuff we can do with this. First thing I want to do is, I'll say, I added select none to card in knowledge card. Right now it says insight for each, but I want to see the category, like summary or whatever. Add that. Yeah, I don't know that it's so good at like having the results of past tool use in its context. We gotta like see how the Vercel SDK is using that. LOL. Okay. Consult docs slash knowledge MD for the file path. Make sure to give me code in markdown code blocks, not patches. I should probably just remove that tool. Knowledge card write... Yeah, yeah, yeah. I found it. There is no... I didn't say there was a knowledge card write component, mofo. Okay. Well, insight. That's easy. No, I don't think that... Oh, dang it. It's still using my outdated... Okay, okay, okay, okay. Wrong. Knowledge. These cards. Okay. Sorry, I again forgot to update GitHub remote with the latest changes. Now please consult components slash workspace slash tab slash knowledge.tsx and update it to show the category and put the code here for that entire file in a markdown code block. All right, let's try first copy pasting it. Nothing. Let's take a look at the structure of the object because it may... I asked for category and it might just be hallucinating that that's even there. Type. It's type. Damn it. Whatever. I'll just use the manual there. It does exist there, mofo. Okay. Security, performance, code quality, architecture. Okay. So next I think we will generate some actual insights, kind of summarizing different files and programmatically generating real insights. And that will include vector embedding each insight so we can use that to put related things next to each other and make this a little bit more kind of traversable. But yeah, basic knowledge graph in a 3D canvas. This video is what? 37 minutes with a bunch of narration and stuff of mine. You know, 10, 15 minutes of actual work probably. Cool. See you soon.