Transcription: OpenAgents Episode 122 - Codebase Indexing via GraphRAG

Source: https://x.com/OpenAgentsInc/status/1825094346938401060 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 122: Codebase Indexing via GraphRAG We design a codebase ind... Upload date: 20240818 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T16:59:17Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: Let's try in this video to build the simplest possible index of a codebase. So we're building up to the first relevant codebase for the Sweetbench benchmark. We're starting with PSF requests and we're going to start with the repo of our core product, indexing this. And by simplest possible index, I want to try to get as much of this just in the actual repo itself because we can work with it using our existing repo tools here. And I've kind of done a little bit of this kind of like documenting individual systems in these markdown documents. And I kind of want to have an index in markdown based on this particular advanced-ish form of retrieval-augmented generation from Microsoft research. Thank you, Microsoft. Lol. But this seems like a pretty good algorithm that we want to use for indexing our codebase. So the idea is, given a GitHub issue for this benchmark, we need the agent to be able to pull up any codebase context needed to know what files to edit and how to edit them. In the past, I've seen people build basic coding agents that just like run a cosine similarity search on the sort of like one-line descriptions. So the example I've given in previous videos is this repo PierreBot that I forked from some hackathon a year ago. And what these guys did is they built like a little index or a single JSON file of a one-line summary of all of the files. And then would like embed each of those descriptions. And then based on the query, based on the GitHub issue, they would run a cosine similarity search, find the like top K, top five or 10 most relevant files to the query based on that cosine similarity search of the query against the description of the file. Then they would loop through each of the files and then just ask the model, at that point, they were using DaVinci still, does this file need to be changed in order to address the issue? If yes, give it the surgical code edit. And it was able to make like a reasonably non-horrible PR that solved some issue from some demo that they did. Now they did that a year plus ago using freaking DaVinci. If we did nothing but take that and update it for Sonnet, like that would be at least kind of cool. But that idea of like just doing cosine similarity on the file descriptions, okay, that's one way of doing things. But I think that this idea of graph RAG and graph-based knowledge is probably the more kind of robust algorithm, you know, beating that, better than that PierreBot basic thing. But we're gonna go through this and try to get a super simple approximation of graph RAG built in a Markdown document and see if it can be an improvement at least on what we've been doing so far. What I've been doing so far is kind of feeding different of these docs pages into the context. So I can say things like, if I'm working on the monetization system, for example, I'll say read, you know, the readme and docs systems monetization, and I don't know, like summarize the components in readme you think exist or should exist to implement that doc. So I'm just kind of giving the docs that it can refer to and like quickly load a certain system into context, which has been fine. It's been faster than copy-pasting a bunch of stuff into Claude all the time. But what I would like is some dynamic retrieval of the most relevant systems, subsystems to be retrieved from the codebase, the index that we build of a codebase at query time. There are all sorts of sort of little subsystems worth documenting. Like for example, what happens when a user logs in? What happens when a user signs in for the first time? I want them to receive a welcome email. I might want to vary the welcome email based on when or where or how they signed in, if they connected their GitHub, if I can tell if they're a developer or not. I want to send them a welcome email. Where does that information live about what currently happens and what I want to happen? If there's a GitHub issue that talks about like, here's what I want to add to that process. Ideally, there's an easy way for it to just retrieve information about how that system is implemented currently, if it has been, what are relevant files to it, what are previous commits that have done things. Like it should be able to have all of that connected in one sort of base of knowledge that it can quickly say, okay, oh, you did a commit about that, you know, two days ago, but you did this. Like it should just have all of that information in one sort of graph. And that's kind of, so that's basically the idea behind graph rag. So let's step through this graph rag a little bit and then we'll do a simple implementation of it. So graph rag. Graph rag is a structured hierarchical approach to retrieval augmented generation rag, as opposed to naive semantic search approaches using plain text snippets. That's what I described that they were doing for the peer bot thing. The graph rag process involves extracting a knowledge graph out of raw text, building a community hierarchy, generating summaries for these communities and then leveraging these structures when performing rag based tasks. So graph rag versus baseline rag. Rag is a technique to improve LLM outputs using real world information. Blah, blah, blah. The majority of rag approaches use vector similarity as the search technique, which we call baseline rag. Graph rag uses knowledge graphs to provide substantial improvements in question and answer performance when reasoning about complex information. Do, do, do, do, do. So certain situations, many situations, baseline rag performs very poorly. For example, baseline rag struggles to connect the dots. This happens when answering a question requires traversing disparate pieces of information through their shared attributes in order to provide new synthesized insights. Baseline rag performs poorly when being asked to holistically understand summarized semantic concepts over large data collections or even singular large documents. Yeah. If I want to do what I described, send an email after they log in, I'm going to need like the database schema, the email service, hook into the auth. There's like maybe five-ish files all across the code base that are relevant to that. And I need to be able to reason across those things. Finding files that are relevant and feeding them into context, that's one way of doing things. But we're going to want, I think, kind of granular. Anyway, let's just keep going with this. So Microsoft Research's new approach, GraphRAG uses LLMs to create a knowledge graph based on an input corpus. This graph, along with community summaries and graph machine learning outputs, I don't know what that is, are used to augment prompts at query time. Substantial improvement, blah, blah, blah. So it slices up an input corpus into a series of text units, which act as analyzable units for the rest of the process and provide fine-grained references in our outputs. So this will probably just be files and file paths. Conceivably, we can split up files if we need to. But I think our files are kind of small enough that we can just use the individual files. So extract all entities, relationships, and key claims from the text units using an LLM. So we'll want to take a look at what the system prompts are for how they do this. Perform a hierarchical clustering of the graph using the Lida technique to see this visually. Check out Figure 1. Each circle is an entity with its size representing the degree. I don't know how important this is for our purposes. Generate summaries of each community and its constituents from the bottom up. This aids in holistic understanding. Query. At query time, these structures are used to provide materials for the LLM context window when answering a question. The primary query modes are global search for reasoning about holistic questions about the corpus by leveraging the community summaries. Local search for reasoning about specific entities by fanning out to their neighbors and associated concepts. Okay. So, fine. Let's kind of work with this. Let's work with this. Let's make sure my scrape web page tool is on. Scrape this and summarize in one paragraph. So, now read our readme. We want to implement a basic version of graph rag. Text units for now will just be our individual GitHub files in our repo. Create a new document, docs slash graph rag dot MD that we'll work with. Add initial thoughts about how graph rag can help our project and how we'd implement it. Let's just see what it comes up with. So, let's go look in here. Graph rag is a structured hierarchical blah, blah, blah. By implementing a basic version of graph rag, we can improve the system's ability to understand and reason about the complex relationships within our codebase and user data. By creating a knowledge graph of our codebase, we can help the AI better understand the relationships between different components, functions, and data structures. Improved query responses. Graph rag can provide more accurate and contextually relevant responses to user queries about the codebase features or documentation. Holistic view of user data. By applying graph rag to user threads and messages, we can create a more comprehensive understanding of user interactions and conferences over time. Yeah, holy shit. Yeah. I've heard that ChatGPT and Claude or whatever have implemented some concept of shared memory across conversations. I've never had any powerful experiences with it. I don't know how they do it. They're probably opaque about that. But being able to do graph rag over your previous messages is pretty interesting. Making some of your certain insights from your chats available to other people and getting compensated for that. Even more interesting. So, better thread management. Okay, I don't really care about that. Advance search capabilities. Yeah. Okay, implementation plan. Define text units. Use individual GitHub files in our repo as text units for the codebase. For user data, consider using either individual messages or entire threads as text units. Wow. Entity extraction for

[19:59] Speaker: The document says nothing about the importance, definition, importance, or anything about files, entities, relationships, and communities. Add definitions. Also, add more comments to the schema, maybe each field. Do we have SHA in here? We need to... The schema must reflect that each entry belongs to a user and has a repo, org, name, and branch, which may be different, different content across different branches, and SHA, so things change over time. Edit the schema to reflect Git-like dap. Make sure when updating the doc, you always include the entire file contents. Don't ever truncate with comments like, this part stays the same. Cloud35 Sonnet is remarkable. 35 Opus is going to be insane. So we're moving toward implementing this index in Convex before too long, but I still want to do, like, a super basic version in GitHub and just, like, see something, like a markdown doc with, like, some kind of demo entity extraction or something. Okay, let's see what it says. Added a new section, key concepts. Updated the schema to reflect Git-like data and user ownership. Added user ID to all tables to associate data with specific users. In the code files table, added repo, org, name, branch, and SHA fields to capture Git-like versioning. Updated indexes. Added more detailed comments. So, where is it defined on these things? Files. Individual source code files within a repository. They are the basic units of our codebase and contain the actual code content. Entities. Distinct elements within the code, such as functions, classes, variables, or modules. Entities are the building blocks of our knowledge graph. Relationships. Connections between entities representing how different parts of the code interact or depend on each other. Examples include function calls, inheritance, or imports. Communities. Groups of related entities that form a cohesive unit or serve a common purpose within the codebase. Communities help in understanding the high-level structure and organization of the code. Okay. Before we begin that plan of implementing in Convex, I want you to do a demo. I want an agent to do a demo. I'm going to start a new chat so it starts fresh. This kind of bugs out when it gets too, too long. Before we get to the beginning of the plan, I wanted a new agent to do a demo. Trial run. Indexing slash extracting entities just using this chat plus using our tools of view file, view hierarchy, and saving a demo index in a markdown file called doc slash graph rag test index. I want to see examples of the entities and entities, the entities, relationships, files, communities in a markdown document. Ideally focusing on one particular topic like what happens after a new user signs up. Authentication. First, just write, first just edit doc slash graph rag one more time with instruct with clear instructions so a new agent will be able to read only that doc and do what I just described. I'm excited to see this repo converted into this structure. Because you know that like given something that's this well defined, if you just give this back to Sonnet and you're like, write me an algorithm that can use the GitHub API to traverse all the code and extract entities and put them in this format. It's a pretty smart idea of graph rag. It's like evaluate this document, return some structure of all of the kind of like entities or ideas out of it, out of this content, then use that for retrieval. Okay, it updated something with seven. Oh shit. As a new agent, your task is to perform a demo slash trial run of indexing and extracting entities from the OpenAges code base focusing on the user signup process. Use the view hierarchy tool to explore the project structure and identify relevant files. Use the view file tool to examine the content of the identified files. Extract the document. Holy shit. Okay, let's test it out. So I'm going to say, read the readme and docs graph rag.md. Execute the trial run described. What's gonna happen? I understand the task. I'll now proceed with the demo trial run as described. Let's start by exploring the project structure. Based on the project structure, we can identify some potentially relevant blah, blah, blah. Did I read the readme? I should have some file path in there. This file confirms that Clerk is being used. Now that we have gathered the necessary information, let's create docs slash graph rag test index file with the extracted entities and relationships. Let's take a look. Off convex, off convex user. Okay. Yeah. Hmm. Well, for this to be comprehensive, I would want it to have the front end, although that comes from the Clerk library. Well, this answers the question, which is how do we hook into what happens after the user signs up? It would hook into that right there. Okay. That's good, but I'm not quite sure how this would be relevant in the global or local queries defined in CodeRag. Let me do this. Please first scrape this URL and then tell me about querying. Oops. Oh well, let's see what that says. In the context of our current implementation, we might need to modify... No. Global queries might be used for questions. No, we don't want to modify create or get user itself. We are building an index process as described in that original docs CodeRag.md doc I asked you to read. Please traverse more relevant files using file paths from the readme or other things you discovered and want to traverse. And extend that test doc you made with additional files and entities. I want to better understand the communities and such. Having more info in there will probably help. This title up thing is going crazy over here. I gotta change that. Ah, communities. So this is kind of what I was thinking of, which is like, there's got to be these kind of subtopics that if you ask a question, it's got to be related to one or more sort of subtopics. And I'd love to see like a list of all of those subtopics. Process flow, that's really good. I want to see like process flow. And that's the process flow for the chat system. Okay. Excellent. Thank you. I am now most interested in seeing a comprehensive list of all of the communities in this code base at the same level of granularity or zoom as the four that are in there now. Please loop through any files, folders needed to add another five to 10 communities to that list. Hood panes, definitely. Need them hood panes. Oh yeah. Yes. Yes, yes, yes.

[39:59] Speaker: Okay, this feels really good to see on the page, because this is the kind of stuff that kicks around my head. I'm like, oh, I've got that, and that, and that system. And I'm like, okay, here they all are. Now let me take another look at that schema. So if, like, if those are all the communities, let me visualize what those will look like in our database. Oh, damn it, it removed all this shit. Fuck ass motherfucker. Okay, so the schema of communities, name, description, members, the vector embedding for similarity search. So a vector embedding of what? You removed a ton of content from that doc. Add it back. Oh, can I have two of these going at one time? Watch this. Oh, please, please work. You do that. And you... Thanks. Please do another five to 10 communities or stop when you feel it's comprehensive. Yeah, I want it. I want it. Do I have two chats going right now? Do I have two motherfucking chats going right now? If this works, I do. Do it. Hurry up. Give me back my fucking rest of the content unchanged. Ooh, I'm gonna fight you. Tweak my system prompt. Good. What is this ordering? Has it said that it's comprehensive yet? Payment integration, API management, sharing functionality, system configuration, type definitions, utility functions, font management, server actions. Yeah. Oh, yeah. Um, this may get too long in a second. Thanks, but you removed a lot of content from that file. Add it back in addition to your latest changes. Okay, I'm just gonna try starting a new... Ah, fuck, I gotta fix that. Damn it. That'll terminate those chats. These aren't truly, like, background-id. It relies on the useChat hook, so the component has to be mounted. I think. Yeah, that's probably stuck now. Oh well. Improvements to make. Um... Let's try this. Read, um, readme, docs, graphrag, and docs, graphrag test index. Is that what I call it? Ah, shit. Graphrag test index. I want to know if that list of communities is comprehensive at that zoom level or level of granularity. Please traverse anything needed in the codebase to be confident it is comprehensive or add more communities until it is. When you edit it, preserve all content. Don't say this is unchanged. It is a full file edit. Ah, the beautiful thing about Git is you can always just go back into history and pull it out. It would acknowledge the presence of the test folder. I don't think there's a test folder. Okay, so, um... Oh, it looks like it successfully did add all that content back before I close that window. Okay, um, this is fucking cool. Here's all the components of my app. Okay, so now that I know that it's comprehensive, here's what I want to say. Read these things. Um, pick one or a few communities. Traverse the folders. Traverse the relevant files, folders, and add underneath those communities in sub-bullets as much info as you can. Info that would later be added into other connected database entries, files, entities, etc. Let's focus on the following communities. Authentication and user management, chat system, credit system. What about linking external documentation, explaining something like Clerk? Because I find that I a lot of times have to link in docs because the stuff that it's trained on doesn't have the latest, like, API signatures of the Clerk hooks. And it can't always infer it or it doesn't have the right code and context to properly infer type usage. I would like to just have it be able to consult the documentation. Um, that's beautiful, but you need to edit that markdown doc with all of that and make sure to include the entire previous file contents too. Let's open another window and I'll say read this stuff. Um, if I want to be able to have external documentation associated with a community or whatever, like, for authentication, that uses Clerk, and we should include links to definitive documentation about API signatures, types, etc. Does our current proposed schema in the GraphRag handle that or should we extend it? Let's see if it didn't fuck this up. Oops, I'm on the wrong tree. Yep. Yep, yep, yep. I can start to visualize about how this is going to be easily pulled into context. Okay, entities, files, key points. This is great. This is great. The current proposed schema does not explicitly handle external documentation. However, it can be extended. Here's how we can modify the schema to include external documentation. External documentation. Let me take a look at the GraphRag thing here. I was thinking about getting code entities and code communities. Okay, well, I'll think about docs separately. I think this is an amazing start. So we have both an initial, pretty decent, not comprehensive, but, like, I can start to visualize it, you know? We've got our index consists of 24 communities or subtopics. Pretty definitively comprehensive. And then as examples of drilling down, each of those can have specific details about entities, files, and key points. And then you can do things like, if... You can take a thing like, imagine taking a GitHub issue, asking the LLM to identify which of these 24 communities are relevant. You can rank order them. And then traverse through any... Like, let it see all the list of files. You could even maybe ask the LLM, maybe something simpler, like a Grok Llama 7B, like, given this, which of these related files do you want to inspect? Inspect those, pull up the key points on each, then do surgical code edits. That's basically the pathway to having this be... It's already good. Just looping through files. Like, this is good. This is a dramatic acceleration of my development on top of Qualityfix. We get this GraphRag thing in place. Frickin' huge. So, we will do that. I'll probably do a bunch of that offline. I'll be doing this kind of index first for this codebase. Test that out with some other features that we're building. See how fast that goes. And then once we get happy with it, then we'll start setting up more of the harness for the Sweetbench benchmark and drive toward getting some initial patches and start driving toward a score. See you soon.