Transcription: OpenAgents Episode 121 - SWE-bench Planning

Source: https://x.com/OpenAgentsInc/status/1823896252460704139 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 121: SWE-bench Planning We make a plan to win high score on ... Upload date: 20240815 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T17:04:44Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: All right, let's continue. We've got the 500 repo commit things from SweetBench Verified loaded into our little web app here. You can click on each one and see the info. And let's remind ourselves what these are. So on the Hugging Face link, which is linked from the SweetBench Verified blog, OpenAI. So if you click download SweetBench Verified, that takes you to there. You scroll down and it says an example of a SweetBench datum is as follows. Instance ID, the gold patch, the patch generated by the PR minus the test-related code that resolved the issue. The repo owner name identifier from GitHub, the commit hash of the repo representing the head of the repo before the solution PR is applied. Hints text, comments made on the issue prior to the creation of the solution PR's first commit creation date. The creation date, test file patch, issue, installation version, commit hash to use for environment setup and installation, and then the tests that need to pass or not fail. Okay, so these are the datums. And up here, there's like the link to the datums, the data. And I downloaded that and I asked Claude to give us a script to convert this into JSONL. That script and resulting data file is on the GitHub repo that we used from the last video, swebenchdata.jsonl. And JSONL is what was needed to import it into convex, which we're starting to connect into openagents.com. And this has been pretty cool to work with so far. So I've got all of those things in this database. There's 500 of them exactly. And then just throw up a little web page with a table that dumps all of that onto this page. What I want to evolve this page into, this is the OpenAgents.com local app running. So when this is live in probably a couple of days, we'll have this be at OpenAgents.com slash swebench. And I want this to be kind of publicly visible as a, you know, one-stop shop for seeing where we're at with swebench, what the latest is, stats, maybe some, you could trigger some runs yourself or kind of experiment with the different agents and the different memory. Because one of the cool things is we're going to be building up this sort of like persistent knowledge about these eight or however many code bases are represented in these issues. And that'll be kind of like a good, you know, example of how open agents, coding agents, build knowledge about a repo to be able to solve PRs. We've kind of been doing that in this sort of slow semi-manual way with me working with the OpenAgents code base. You know, I've been wanting to kind of formalize that to make it easier for our users to have that kind of thing work with their code bases. But we're going to be really moving toward this world of, there's going to be agents that are increasingly autonomous. And instead of you needing to keep copy pasting from here and there, it's able to, you know, right now it's kind of like connecting directly to your GitHub repo and pulling in, you know, contacts by just kind of traversing one at a time. But, you know, beyond that, it's how do we build up this sort of like snapshot or index that an agent can like really have a comprehensive picture of everything that's going on in a repo because there's so much data within a given repo. I mean, like an agent can make sense of all that. We need to kind of help it with some certain structured data and, you know, graph rags and all sorts of like different kind of knowledge assist data architectures that we're going to evaluate and, you know, build some versions of that over the next few videos here. But I kind of want to, in this video, kind of strategize about what this all looks like end-to-end. And maybe what we'll do is we'll make a little Excalidraw thing here. So let's make some little widgets. So let's make one for, we'll call it the swebench website. This will be at openagents.com slash swebench. We've also got the openagents swebench repo that will be open source. Our core, like openagents.com is not open source, but we can later open source the swebench relevant pieces of it. OAS swebench repo. There's the openagents.com product. And we're going to be kind of building this with an eye towards building what we need to achieve the benchmark and also making certain of those tools available to users. For example, earlier today, oh, I don't know if it's connected. Yeah, there it is. So earlier today, I've been working on this knowledge system where you can upload a document and then have this knowledge be accessible to one or more of your chats, one or more agents. I think we'll experiment maybe later this month with starting to let you make certain of this knowledge available to other people, have kind of like a pooled collective knowledge that you could get like paid a share of the revenue of people who use your knowledge. All sorts of cool things that we can do there. So knowledge bases, let me go back to my, so there's like the core product. Your all-in-one AI productivity dashboard. Now we're gonna make a little plan here and I'm gonna err on the side of like sharing too much because if and when we hit the benchmark and like when, I want everyone to see how we did it. So speaking of that, the swebench leaderboard is up and I had assumed incorrectly that because verified was just announced in a blog post that there wouldn't be scores, but there are scores because verified is just a subset of the full set. They could kind of like parse all of the ones here, even though the top score on full is 19.75%, you know, extracting the bad ones. So there is an existing leaderboard. The existing score to beat is 38.80% from the Amazon Q developer agent. Do you think we can beat Amazon people? Do you think we can beat Amazon? Okay, so part of the cool thing about this is that instead of just like doing this in a GitHub repo and it's just like me or some team that just does it and puts it up, this little dashboard that we put up at openagents.com slash swebench, it's going to be kind of interactive because part of what's going to drive our success in the benchmarks is the particular algorithm that we use to build and then traverse this sort of like knowledge graph index that we have of any given repo. And that's something that I want to test, refine, iterate, have other people like maybe you help contribute data, ideas, feedback, algorithms, just make it a little bit more open, transparent, deterministic, extensible. And yeah, you know, I am happy to be honest that we are coming from a standpoint of like productizing this. And that's a part of the reason for the transparency is I want everyone to know that we're here, that we know what we're doing, maybe. I guess we'll see how the score goes because I want this to be like, as soon as we have the top coding agent, I want people to be paying us for it. So I argue that we already have the state of the art like coding dashboard because I haven't seen anyone else with anything better. But, you know, there's way much more that Canon should be done in terms of like automated resolution of PRs and things like that. And what I would also like to do is maybe even have a side for openagents.com, maybe just like a separate sub page that's just like, maybe openagents.com slash repo. It's like, just put in a GitHub link or enter GitHub issue link, get analysis slash fix slash like, oh, I can fix it for you. Pay me $100. Okay. All right, so the other thing that I had, let's kind of actually review the chat that I had. So since we left off yesterday, I said, yeah, read this swebench verified page from Hugging Face and take the datum info and add it to this GitHub, or add it to the, yeah, the Markdown doc there, which I was happily trumpeting about that earlier because, yeah, the fact that we can say, read this link and edit the relevant file accordingly and get a perfect GitHub commit is feeling real close to a breakthrough in coding automation. So it took the kind of previous data set structure summary and added all of the relevant info. And then like later on, I would take this same data and feed that into Claude to build up the schema for the convex database. And then what else did we do? I said, thanks. Now please create a Markdown doc called flow.md. We want this to be extremely simple and to the point explaining the flow. Like what OpenAI said in their blog post that for each sample in sweep bench, agents are provided with the original text from the GitHub issue known as the problem statement and are given access to the code base. Given these agents must edit the files in the code base to resolve the issue. The tests are not shown to the agent. So I'm gonna make a flow for like, what are the step-by-steps? It said, certainly, I'll do that. And then I said, edit it to reflect that we wanna save the results. So let me just show you where we're at with that flow. So data ingestion. I did this step one earlier today. That's what's right there. Yeah, next is assign a single sample to an agent. Agent receives the GitHub base commit SHA and the problem statement. Agent accesses the GitHub repo at the specified commit, traverses the code base, storing relevant knowledge. We could literally use our exact tools. We're probably eventually gonna wanna have a sandbox environment where we pull down the repo and do all sorts of more fast operations. But this could be interesting to just try the basic version. Problem analysis. Agent processes the problem statement, identifies specific issues, formulates

[19:59] Speaker: We want maybe the most simple repo. Please pick, please suggest one to three options from this list of repos. You can visit the links of any of them, pending GitHub.com, if you want to read their stuff. I don't know if that'll work, but let's try it. That took a while. Let's start with Flask, as it's known for its simplicity. Pallets, Flask, error. Oh my gosh. Is it a GitHub problem? Yeah, GitHub was like essentially entirely down recently. Is it down now? All systems operational, they say. Okay. Try again. Don't scrape. Just use what you know. And then out of curiosity, I will also go ask that to Claude. Yeah, somebody's going on there. Pallets slash Flask, but there's only one of those repo. One of those, isn't there? Pallets. So PyTest dash dev. So PSF request is one, two, three, four, five, six, seven, eight. And PyTest dash dev has one, two, three, four. That's a lot. Okay. Let's just take a look. I'm going to, I'm trying to pick something that I think would be like, I don't know, easiest. YouTube DL. Let's take a look at PSF requests. Request dot get is always sending content length. I like that. Okay. I think we'll start with PSF requests. It just seems like I would understand it the most. I don't know. All right. So PSF requests. Okay, so we're going to start with this. And what I might want to do starters is instead of showing all of these, because this takes like a few seconds to load, let's change that to only show those from the repo. And then what I want to do is actually edit this via OpenAges. Get some more practice doing that. Let's make sure that we have this SweetBench page pushed up. Okay. And then let's go to OpenAges.com and let's see if things are now working. Okay, OpenAges Inc. V2, Convex. And what file do we need to change here? Let me shrink this down a bit. I've been trying to get good at NeoVim and I love it. But all this type safety stuff, I just don't know how to get NeoVim yet as good as it is out of the box with VS Code. So, all right. So I'll say look at app slash SweetBench slash page.tsx, Convex slash schema.ts, and Convex slash SweetBench.ts. I want that page to only show samples where repo is PSF requests. Edit accordingly. All data CTX DB query SPV data filter QEQ field repo is PSF requests. I'm glad you know the Convex syntax. I do not. Git pull. Yeah. Oh, it did two files. Did it work? Did it work? Did it work? Did it work? Did it work? Did it work? Wrong file. Oh, yeah. It worked. And now we're not doing repo there because it's all the same repo. You know what's a good sign when you spend more time git pulling than git pushing because I'm not doing the coding. I'm just architecting it and someone else is doing the legwork. 838 is my lucky number, so maybe I'll start with this one. Digest quotes. Okay, so let's do this. I want to see this formatted more nicely. Look in the relevant components folder. You know, I think components slash SweetBench. Right now, the modal pop-up thing is not formatted well. I don't want to have to scroll down for the problem statement and it shouldn't be centered like that. And I don't see any line breaks, so fix that. And if you ever see opportunity to split long files into multiple, just do that. Okay? There's not a specific SweetBench component. Oh, no, I put it into the page directly, I guess. But isn't that nice that it can recover and work around my wrongness? First, let's create a new file for the SweetBench modal component. Now let's update the page file to use this new component and improve the organization. The changes have been made to improve the organization and formatting of the SweetBench page. Here's a summary of the modifications. Outside the scrollable area. That's interesting. First of all, does it even work? Yes. That's pretty nice. Pretty nice. So let's imagine that we are an agent or a human who has been tasked with resolving a GitHub issue. What do you do first? You read the problem statement and you're like, you think about what you need to do. First of all, you probably do some triage. You decide between, if you're assigned eight issues, you're probably going to decide which one makes sense for you to focus on first. Wow, some of these are from 2013. Is that Python version 1.1? Oh my gosh. Okay, so first of all, what other information do I want on this table? I don't really need base commit there. In that table, replace base commit with a 30 word, with the first 20-ish words of the problem statement. And force the instance ID and everything to be on a single line, not overflow. Use dot dot dot truncation if too long. Right now, created at is taking up tons of space. I want the problem statement to kind of fill or grow to take up most of the space. Everything else shrunk to its content. Whereas in the past, I'd be like Googling different Tailwind classes and looking at Stack Overflow and just like, make it like this. And then it knows how to figure that out. Oh, real nice. It helps to know the kinds of things to ask for, but man. But you can see how even that is going to be replaced by agents commanding each other. Blah, blah, blah, blah, blah. Git pull. Yeah, that's pretty cool. And actually, I don't want the ID to truncate, but do make it on one line. And then does that work if I shrink this down? Ah, yeah. Is sonnet going to know about Python version 1.1? Yeah. Right now, those are in chronological oldest to newest. Can you reverse that? I think I want to start with the most recent. I'm going to guess that it's like, it's 2022, so it should thoroughly be like in the training data. And I just don't know how much the like 1.1, I don't know anything about that. I'm guessing Python v2 will be a little easier to test. So we'll start from the newest. Okay, I do believe this will be the first one that we focus on. We're going to build like information about this. But man, I'm so excited to like do something like, have kind of an explorer here where like, I would love to be able to see at a glance how all of the other runs performed on this and like be able to analyze me slash agents, analyze what they did wrong or what we can infer about their architecture and why they made certain mistakes. These scores on SweetBench leaderboard. I mean, it's kind of interesting that no one's cracked 50%. Verified. Amazon, 38.8. Top score. We're coming for you. See you soon.