Transcription: OpenAgents Episode 120 - Exploring SWE-bench Verified

Source: https://x.com/OpenAgentsInc/status/1823454256596213969 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 120: Exploring SWE-bench Verified We talk smack about benchm... Upload date: 20240813 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T17:09:07Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: Let's take a look at OpenAI's blog post today. I usually don't pay too much attention to the blog posts that they put out, but this one seems pretty relevant because this SWEBench benchmark is what a bunch of our competitors have used in trumpeting their funding announcements, like, hey, this one just came out a couple days ago. We raised money and we did this SWEBench 30% ahead of these other things. You can't try the product. There's a wait list, blah, blah, blah, but trust us, it's super cool and here's our big score. So people really care about this whole benchmarking of coding agents. We have not really focused on that because while other projects chase these benchmarks, we actually built a product that you can use. And I'm gonna call it state of the art because I haven't seen anything that's better than it. Certainly nothing that's live. So DMS for $10 a credit. New website. So nice. But now that we like have a product that you can use, maybe we should benchmark it. In part because I'm really looking to have reasons to like mock and make fun of of our competitors. And look, if there's, you know, users or investors who are trying to evaluate these projects, if they don't have much to go on by way of like objective analysis of whether something's good or not, maybe there's a, you know, value in seeing what the open agent's score is on some of these benchmarks. So I thought that we would go through this blog post together. And I think that we may just do this over the next few videos. Like, let's just see how OAV2 performs on the new SWEBench. And so we can make fun of people who have lower scores. Okay? So let's read through the blog post. Introducing SWEBench verified. We're releasing a new, we're releasing a human validated subset of SWEBench that more reliably evaluates AI models ability to solve real world software issues. Okay. Preparedness framework, blah, blah, blah. The ability to autonomously complete software engineering tasks is a key component of our medium risk level in the, uh, evaluating these capabilities is challenging due to the complexity of software engineering tasks, the difficulty of accurately assessing generated code, and the challenge of simulating real world development scenarios. Therefore, our approach to preparedness must, okay. Okay. One of the most popular evaluation suites for software engineering is SWEBench, a benchmark for evaluating large language models' abilities to solve real world software issues sourced from GitHub. The benchmark involves giving agents a code repository and issue description and challenging them to generate a patch that resolves the problem described by the issue. Coding agents have made impressive progress on SWEBench with top scoring agents scoring 20% on SWEBench and 43% on SWEBench Lite according to the SWEBench leaderboard as of August 5th. Okay, so there's a leaderboard with like officially reported scores, and then there's people that don't go through that process and they just kind of tweet about it maybe before they submit it. I don't know. Anyway, our testing identified some SWEBench tasks which may be hard or impossible to solve, leading to SWEBench systematically underestimating models' autonomous software engineering capabilities. We've collaborated with the authors of SWEBench to address those issues in a new release of the benchmark that should provide more accurate evaluations. Okay, so we're going to do this because it's probably going to be fun. And I think it's worth us developing the ability to like do evals and testing on our agents. Heck, that might even be a product or service that we could offer to other people who want to benchmark their agents. So I think having our own kind of way of evaluating just how good our agents are, we should probably have the infrastructure to do this. So let's learn about this with an eye toward we're actually going to be implementing it. So background on SWEBench. Each sample in the SWEBench test set is created from a resolved GitHub issue in one of 12 open source Python repositories on GitHub. Each sample has an associated pull request, PR, which includes both the solution code and unit tests to verify code correctness. These unit tests fail before the solution code in the PR is added but pass afterwards and are therefore called fail-to-pass tests. Each sample also has associated pass-to-pass tests, which pass both before and after the PR is merged and are used to check that existing unrelated functionality in the codebase has not been broken by the PR. For each sample in SWEBench, agents are provided with the original text from the GitHub issue, known as the problem statement, and are given access to the codebase. Given these, agents must edit the files in the codebase to resolve the issue. The tests are not shown to the agent. A proposed edit is evaluated by running both the fail-to-pass and pass-to-pass tests. If the fail-to-pass tests pass, this means the edit solves the issue. If the pass-to-pass tests pass, then the edit has not inadvertently broken unrelated sections of the codebase. Both sets of tests are required to pass for the edit to fully resolve the original GitHub issue. Okay. Adapting SWEBench as a preparedness evaluation. Okay, so they basically found three areas of improvement in a test to blah, blah, blah. Difficulty to set up development environments. Okay. Examples. I trust you that. Okay, so SWEBench verified. To address these issues, we launched a human annotation campaign with professional software developers to screen each sample of the SWEBench test set for appropriately scoped unit tests and well-specified issue descriptions. Together with the authors of SWEBench, we are releasing SWEBench verified, a subset of the original test set from SWEBench consisting of 500 samples verified to be non-problematic by our human annotators. This version supersedes the original SWEBench and SWEBench Lite test sets. Additionally, we are releasing our human annotations for all SWEBench test samples. We also collaborated with the SWEBench authors to develop a new evaluation harness for SWEBench, which uses containerized Docker environments to make evaluating on SWEBench easier and more reliable. On SWEBench verified, GPT-4O resolves 33.2% of samples, with the best-performing open-source scaffold agentless doubling its previous score of 16% on SWEBench. I'm just, whenever I hear these numbers, I'm like, what? Only 33%? Like, I mean, this is probably just the bare model in a loop. I don't know. Well, we'll dive into exactly this, but what was the previous high score? Like 20%? We're going to gun for 100%, okay? Our approach. Okay, we worked with developers experienced in Python to manually screen, annotate it. Okay, so this is their data cleaning, blah, blah, blah. Test results specified. I don't care. Dataset, blah, blah, blah. Yeah, seems thorough. I can't evaluate it. I just want the thing. Okay, so with our new SWEBench verified dataset, we tested GPT-4O's performance using several open-source scaffolds that performed well on the original SWEBench leaderboards. We found that GPT-4O's performance on the best-performing scaffold reaches 33.2% on SWEBench verified, more than doubling its score of 16% on the original SWEBench. In general, this validates our initial suspicion that the original SWEBench dataset underestimates agent abilities. Note that the jump from SWEBench Lite to SWEBench verified is not as significant because SWEBench Lite was already filtered in a way that makes it easier than the full dataset, though that process would not fully capture the same issues as the filtering procedure. Okay. Rookie numbers. Performance stratified by difficulty. The increase in performance when evaluating on SWEBench, blah, blah, blah. Discussion and limitations. Okay. Okay, so SWEBench verified is available for download here. The full set of our annotations is here. And our annotation rubric is here. Okay. So cool. So we're going to do this. And I have no real experience doing this aside from a year ago, I submitted an OpenAI eval to get early access to GPT-4. I did a little bit of evaluating harness stuff for the OpenDevin Discord channel a number of months ago. Aside from that, I basically no experience doing this kind of eval stuff, but it'll be fun to see how I can use open agents to establish whatever code we need to run these evals. And I think it'll be super fun for me to do all of this process in public. So I'll do a open source GitHub repo. This core product here is not open source, but the SWEBench stuff that we do, I'll put a link to that. So here's a little me just kind of dropping in an example of the link to the blog post here using the scrape tool, scrape webpage tool, summarizing the key points. And let's just see if I can do something real quick. Let's go make a quick new repo and we'll call it SWEBench verified. Or should we have a cooler name like SWEBench verified easy? I don't know. Okay. Public license. What do I care? Creative Commons. All right. So let's pwn everyone on SWEBench verified. Okay. So what I'm going to do is I'm going to swap that into here. And let's give it view file view hierarchy. I don't need that. Well, maybe I'll keep that in there. Create file. Let's start with this. I've given you access to our new repo. Please... Oops. No, no, no, no. That's not what I meant to do. Okay. I've given... Ah, shit. That's a bug. Please... Let's try this again. I've given you access to our new repo. Please create a readme where you introduce our project, which is that we're going to do whatever necessary to get high scores on SWEBench verified and put your summary and anything else relevant in that doc. Is that going to work? Can I just sit back and let

[19:59] Speaker: to traverse the README and any other files and folders in this repo and create in a Markdown code block in this chat a long Markdown doc, which you will subsequently, after I restore your access to the original repo, add to our README or another Markdown doc. Start with the README and go from there. Explore probably greater than five files and folders. Do you think this will work? Do you think this will work? Certainly. Failed. Sometimes it guesses at file paths. Or maybe it was looking at like an older reference. OK, so it explored the run evaluation file and the inference folder. Let's take a look at the README in the inference folder. So I'm doing all this in a chat right now, right? There's no reason it has to be in a chat. This could occur in a background process that we just like get a window into. Well, would you look at this? So now that we have explored the key components of the Sweepbench repository, I'll create a comprehensive Markdown document summarizing the setup process, usage instructions, and important details for our project. Here's the Markdown content. OK, that's the original stuff. This says clone the Sweepbench repository. We'll be using the datasets library to work with the Sweepbench verified dataset. Now, this is using the harness and stuff, I guess, of the old code base, but one of the things in the blog post said that they have a new harness that uses Docker containers. So this probably is not what we do specifically. And let me actually just ask about that. In the first blog post I gave you, it mentioned new Docker containers for test harness that improve on the original. So I'm not interested in running the old harness, just for thoughts on structure and whatever else relevant to our verified version for background info or whatever. Revise your doc accordingly. And if you want to traverse any other files and folders in this repo while you still can, do that now. I apologize for the confusion. You're right. We should focus on the updated Docker-based test harness for Sweepbench Verified. Let me explore a bit more and then revise the document accordingly. First, let's check if there's any specific documentation about the Docker-based evaluation. Thank you for providing that information. Now let me revise the document to focus on the updated... No, no, no, no, no, no, no, no. This is still accessing the older repo. You said newer Docker something something, but you still only have access to the old repo. I don't think that has the new approach. Anyway, one last chance to look at anything, then I'll switch you over to the new, to our new repo, so you can edit the doc. And there's so much room for improvement here. So like right now, it's just looking at a few files, but you can imagine, like, what if it was able to have perfect context, a perfect index of that whole repo to be able to, like, better pull out relevant insights. But even as relatively limited as it is, it's like, seems to be pretty impressive, the quality of response and stuff. But we'll be able to tell at the end how good it is if it works. Now that you're ready to switch me over to the new repo, I'll be prepared to edit the document with the most up-to-date information about... Okay, so let's go back to OpenAgents Inc. What did I call it? Sweepbench Verified. Main branch. Rewrite file. Okay, you are back in our repo. Create or edit any markdown files in this repo needed for the most thorough possible explanations of anything relevant. Huh. It's creating a project structure of different files for setup instructions, information about the Sweepbench Verified dataset, a guide to running evaluations, strategies for developing high-performing models, results, source. Read through the detailed documentation, set up your development environment, familiarize yourself with the datasets. Okay. It is not licensed under MIT. Oh, it's making me some files. Heyo. Is that true? I don't think... Is that true? Oh yeah, it is. Oh yeah. How true is this stuff? See, like, I don't know. I mean, I know a little bit, but it's largely going to be, we set up the process, we will run it, we keep, you know, running my loud mouth about it. So if we're wrong on anything, people will point it out. I've now created and updated several markdown files to provide comprehensive documentation for our Sweepbench Verified high score project. Updated the readme with an overview of the project and quick start guide. Created setup.md with detailed setup instructions. Created dataset info with information about the Sweepbench Verified dataset. Created evaluation guide with instructions on running evaluations. Created model development with strategies for developing high-performing models and contributing. Yeah. These files should provide a solid foundation for anyone working on or contributing to the project. They cover the essential aspects of understanding the dataset, setting up the development environment, familiarizing yourself with the datasets. Okay. I'm going to call that a good stopping point for our very first video. Thank you to OpenAgents and Sonnet. How long did that take? Half an hour? Not too bad. So I'll do a little bit of work offline and then, yeah, maybe we'll just do one video a day until we get a score. See you soon.