Transcription: OpenAgents Episode 109 - Exploring OpenDevin & CodeAct

Source: https://x.com/OpenAgentsInc/status/1806471593779937581 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 109: Exploring OpenDevin & CodeAct We learn how OpenDevi... Upload date: 20240627 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T17:55:17Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: Okay, the next building block we need is the planner, so AutoDev can make a plan to do what the user wants. For this, we're going to take a page from OpenDevan, which we reviewed in episode 104, one of the best Devin clones. We were able to use it productively, and so their approach is worth studying. Now, they say that their agent, their main agent, the one that we used, implements the CodeAct idea, paper, or tweet. It consolidates LLM agent's actions into a unified code action space, blah, blah, blah. Okay, so we're going to try to just implement our own version of this, and in this video, you'll get to see me kind of from scratch. I haven't even really looked at the paper. We're going to use Claude, use a little bit of reading of it to come up with our own plan to implement this as the planner. And let's see how far we get. Okay, so let's take a look at the announcement tweet first. Large language model agents promise to free us from mundane tasks, but how should they best interact with our world? Introducing CodeAct, an agent, framework, instruction-tuning dataset model. Employs executable Python code to unify the actions of LLM agents. Most existing LLM agents are limited by generating actions in JSON or text formats, constraining them to a narrow action space, e.g., predefined tools, with less flexibility, cannot compose multiple tools together. CodeAct stands out by leveraging existing LLM's pre-training on code data for cost-effective adoption, inherently supporting complex operations through control and data flow, and using extensive software packages for an expanded action space and automated feedback. Blah, blah, blah, blah. Open source LLM agent that interacts with environments by executing interpretable code in cloud. Okay, some datasets, blah, blah, blah. Okay, you know, this is all going to be Python. At least the example code is going to be Python, but we should be able to easily get what we need to do, convert it into Go. Chat interface. Okay, so let's see first if Claude can... Where do I put this? Summarize this paper. Okay, this paper introduces CodeAct, a framework that uses executable Python code as actions for large language model agents. The key points are CodeAct allows LLM agents to use Python code as actions, providing a unified action space. I don't know what that means. Blah, blah, blah. I may be even more interested to see how OpenDevin implements this. What's the difference between these two? Is this one just for the benchmark? I don't care about the benchmark. Okay. Let's try this. Let's study how the project OpenDevin implements this paper. Summarize this. OpenDevin implements the CodeAct concept from the paper in the following ways. Unified code action space. Multi-turn interactions. Jupyter plugin. Should I go and do this in our Golang project? See what we can learn from their docs here. We want to implement the CodeAct paper ourselves in our Golang application. Summarize at a high level how we should do that, borrowing ideas from the OpenDevin project. A CodeAct agent struct that encapsulates the main functionality. Let's try this. Create Golang artifacts for... Create Go artifacts for... Numbers 1 through 4 there. Let's see what it does. Well, it's not making artifacts. Python code. Okay, let's take a spin through the paper. Abstract. Large language model agents capable of performing a broad range of actions such as invoking tools and controlling robots show great potential in tackling real-world challenges. LLM agents are typically prompted to produce actions by generating JSON or text in a predefined format, which is usually limited by constrained action space, e.g., the scope of predefined tools, and restricted flexibility, e.g., inability to compose multiple tools. This work proposes to use executable Python code to consolidate LLM agents' actions into a unified action space, CodeAct. Integrated with a Python interpreter, CodeAct can execute code actions and dynamically revise prior actions or emit new actions upon new observations through multi-turn interactions. Our extensive analysis of 17 LLMs on API bank and a newly curated benchmark shows that CodeAct outperforms widely used alternatives. The encouraging performance of CodeAct motivates us to build an open-source LLM agent that interacts with environments by executing interpretable code in collaboration with users using natural language. To this end, we collect an instruction-tuning dataset, CodeAct Instruct, that consists of 7K multi-turn interactions. We show that it can be used with existing data to improve models in agent-oriented tasks without compromising their general capability. CodeAct agent fine-tuned from Llama2 and Mistral is integrated with Python interpreter and uniquely tailored to perform sophisticated tasks, e.g., model training, using existing libraries and autonomously self-debug. Yeah. Okay, here's examples. LLM agent. Okay, yeah, this is what I was describing. Let's look at this. Instruction. Determine the most cost-effective country to purchase the smartphone model Kodak 1, law. The countries to consider are the USA, Japan, Germany, and India. Your available APIs are lookup rates, country float float, convert, and tax. Estimate final price, lookup phone price, estimate shipping costs. The bad example. LLM agent using text slash JSON as action. Think. I should calculate the phone price in USD for each country, then find the most cost-effective country. Action. Text. Lookup rates. Germany. Lookup phone price. Germany. Convert and tax. Presumably it's doing that for all the countries. And then comparing them. The most cost-effective country to purchase the smartphone model is Japan, with price 904 in USD. Okay, I don't see what's so bad about this. I mean, I guess if that's your only available APIs, then it seems tedious if you're like looping through doing one per country, but I don't know. That seems like it could be structured better. Let's see what it proposes. LLM agent using code as action. Think. I should calculate the phone price in USD for each country, then find the most cost-effective country. Countries is an array. For country in countries. Lookup the rates. Control and data flow of code simplifies complex operations. I get that they're using generated code and more is possible here than just using base APIs. I don't see how there's fewer actions or require, because I think it's still just doing the loop through and calling each thing. Oh, unless you're thinking that the LLM needs to do more stuff here because it's needing to infer more times. I don't know. Like if each of these is its own LLM function call, something. I don't know. I don't know. Okay, success rate. Yeah, folks, we're not going to use Python. We're not going to use Python. We're going to use Golang. Compared to JSON and text with a predefined format, code inherently supports control and data flow, allowing for the storage of intermediate results as variables for reuse and the composition of multiple tools to perform complex logical operations. In figure 1, an LLM using CodeAct can apply the same sequence of tools, e.g., passing one tool's output as input to another tool using the data flow feature, to all inputs through for loops, i.e., control flow feature, with one action, while text or JSON have to take action for every input. Okay. To demonstrate benefit, our first experiment compares CodeAct to baselines on basic tasks involving atomic tool use, i.e., only one tool is used per action, oblating the control and data flow advantage offered by CodeAct.

[19:59] Speaker: CodeAct's performance gains are more prominent on complex tasks. Gear a new benchmark consisting of 82 tasks that typically require multiple calls to multiple tools in multi-turn interactions. Problems in this benchmark often require intricate coordination and composition of multiple tools. In Figure 2, we first introduce a general multi-turn interaction framework for LLM agents real-world usage that considers three roles, agent, user, and environment. We define interaction as the information exchange between the agent and an external entity, user or environment. For each turn of interaction, the agent receives its observation, either from the user or the environment, code execution result. Optionally planning for its action through chain of thought. And emits an action to either user in natural language or the environment. CodeAct employs Python code to consolidate all actions for agent-environment interaction. In CodeAct, each emitted action to the environment is a piece of Python code, and the agent will receive outputs of code execution as observation. CodeAct shows the promise as a strong tool use framework. Example, user, find the sum of the reciprocals of the roots of that. Assistant, to find the sum of the reciprocals of the roots of the quadratic equation, execute, execute. Let's go ask our friend Claude something. For this part, let's say we want to do, to execute Golang code. Users are using this via a Golang server serving a HTMX tailend front-end. How would we do the code execution? Obviously, potential security nightmares here. Do we need to be setting up some additional environment or is there some sort of sandboxing that we can do? How is WASM helpful? Create a safe sandbox environment for executing Golang code. You can use the Go parser, Go token, and Go types packages to parse and type check the code and then use the Go build and OS exec packages to compile and run it in a controlled environment. Tell me options for doing this. WASM mentioned. Could I have it compile X system plugins? Could I have it compile X system plugins? Creating a safe sandbox environment for executing Golang code is crucial for security. Here are several options, ranging from simple to more complex. Basic isolation with OS level restrictions. Use OS level sandboxing tools like Chroot, namespaces, or cgroups on Linux. Limit resources using cgroups. Run the code as a non-preble user. Docker containers. Create a Docker container for each code execution. Apparently, Go is super good at managing Docker, so that might be doable. gVisor. WASM. Compile Go code to WebAssembly and run it in a WASM runtime. Provides good isolation, but limits some Go features. Example using WASM time. Parse the Go code into an abstract syntax tree. OK, let's come back to here. It seems like what we're looking at really is not planner. It's code execution. Is any of this relevant to actual planning or at what point do they generate their plans? Maybe we should step through the OpenDevon codebase. Do I have that here? This is interesting. Here is the system message. Ooh, you are a helpful assistant. You will be provided access as root to a bash shell to complete user-provided tasks. You will be able to execute commands in the bash shell, interact with the file system, install packages, and receive the output of your commands. Do not provide code triple backticks. Blah, blah, blah. So this is interesting. Let's try something. Here is the OpenDevon implementation of CodeAct agent. Speculate about what would be minimally necessary for us to implement, for example, its system message at the top in Golang. Like how we'd set up that bash shell. OK, interesting. What I'm curious now is, do they even have anything specific to planning here? The CodeAct agent is a minimalist agent. The agent works by passing the model a list of action observation pairs and prompting the model to take the next step. Init initializes a new instance. LLM, the LLM to be used. Step performs one step using the CodeAct agent. This includes gathering info on previous steps and prompting the model to take the next step. I mean, can't I just convert this whole thing to Go? State.plan. Hey. So we came into this wanting to do planning. And OpenDevon has that planning piece kind of also wrapped up in this whole CodeAct stuff, which has a lot to do with code execution. I'm going to bet that Claude can help us convert this into Golang and whatever is needed for like the Golang equivalent of the code interpretation. I'm sure we could just do that, even if we need to get creative with Docker containers or whatever. So where else does it say plan? OK, so I see that we're passing in state, and it looks like state is already has plans. We need to find where it defines state. Yeah, the main goal. I'm guessing the main goal is, OK, so the state.plan.main goal. So main goal is probably what the user wants. And then the plan is probably got other stuff. We'll go find where it says that stuff. LM completion. This is pretty cool. Search underscore memory. OK, let's see if we can find where a plan would be. I don't know how to do search in Neovim. I'm still learning. I'm guessing it's in here somewhere. I'm guessing it's in action or LLM or what? Plan. Task initializes a new instance of the task class. Args, the parent task or none if it is a root task. Amazing. This is gold. This is gold. So we have our plan to plan, which is we're going to take the relevant code from OpenDevin, uh, definitely including this plan, probably also including the CodeAct agent. Um, convert it to Go, pull that into our code base and start connecting a UI to that. OK, so I'll do some of that offline and then, um, I'll have something cool to show you in the next video and we'll take it from there. Progress. See you.