Transcription: OpenAgents Episode 005 - Agent First Principles

Source: https://twitter.com/OpenAgentsInc/status/1722274309727752427 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 005: Agent First Principles We think from first principles abo... Upload date: 20231108 Transcription model: gpt-4o-transcribe-diarize Generated at: 2026-06-01T00:54:53Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: So we have a basic Laravel app deployed to openagents.app. That's where our platform will live.

[00:07] Christopher David: Now, what should we put there?

[00:09] Christopher David: There's a lot of people saying and building a whole bunch of stuff around agents,

[00:14] Christopher David: and we are going to start from first principles.

[00:20] Christopher David: Going to try to get down to the very basics and build up from there.

[00:24] Christopher David: We're going to start and kind of ground this in what our users, as they've told us, actually want.

[00:30] Christopher David: How do you want to use AI agents?

[00:33] Christopher David: So what do agents need to be for us to deliver on these things?

[00:42] Christopher David: And let's make some notes here.

[00:46] Christopher David: So for programming.

[00:51] Christopher David: Let's do this.

[00:53] Christopher David: What do you need to do to do programming?

[00:55] Christopher David: Interact with a GitHub repo,

[01:01] Christopher David: maybe interact with a code editor,

[01:03] Christopher David: whether

[01:06] Christopher David: that's desktop or like a web IDE like Replit.

[01:13] Christopher David: For trading and financial analysis,

[01:15] Christopher David: you needed to maybe,

[01:17] Christopher David: I think.

[01:19] Christopher David: This is also around like monitoring price signals in like different websites and things.

[01:24] Christopher David: So you probably need some sort of interaction with either like trading data APIs or if there's just like a signal on a place that you don't have an API for probably like web scraping,

[01:43] Christopher David: maybe keyword research and writing articles.

[01:48] Christopher David: Writing of articles,

[01:49] Christopher David: that's where you'd want generative text,

[01:53] Christopher David: keyword research.

[01:57] Christopher David: You're scraping data from around the web probably,

[02:00] Christopher David: so more scraping.

[02:02] Christopher David: Maintaining my clients'

[02:04] Christopher David: WordPress websites.

[02:11] Christopher David: Interacting with WordPress.

[02:14] Christopher David: WordPress I think has an API it's been years since I've used it but there's probably a bunch of if you want something to kind of manually drive a browser that

[02:31] Christopher David: would probably need to be like browser interactions collecting and optimizing data from model training

[02:49] Christopher David: Probably like working with you think like working with existing data like people giving you files

[03:01] Christopher David: I don't know okay,

[03:03] Christopher David: so let's define our agent like what does our agent need to be able to do

[03:12] Christopher David: Let's kind of extract the the basics out of this

[03:18] Christopher David: So if we're interacting with, let's

[03:24] Christopher David: start with this, like interact with the web generally.

[03:36] Christopher David: And there's a few ways of doing that.

[03:40] Christopher David: There's APIs if a service like gives you a way to directly control it.

[03:45] Christopher David: And then if you don't have APIs,

[03:47] Christopher David: there could be just sort of like controlling

[03:54] Christopher David: a browser.

[03:59] Christopher David: Well, there could be controlling a browser or there could be like a headless scraper, scraping.

[04:10] Christopher David: The agent needs to interact with the web.

[04:23] Christopher David: The agent needs to take directions,

[04:25] Christopher David: right?

[04:27] Christopher David: So we're not going to be expecting the agents to like have their own agency.

[04:34] Christopher David: We're not going to be dealing with like questions of like how intelligent or artificially intelligent is the agent and have it make decisions on like a high level like we want.

[04:51] Christopher David: The agents to be basically extensions of what the human wants.

[04:56] Christopher David: So take directions.

[04:59] Christopher David: You want it to be able to make sort of sub decisions about like what page do I click on.

[05:19] Christopher David: So ideally,

[05:19] Christopher David: it's like you give it a direction and then it makes kind of like the sub-decisions. It breaks it up and does the tasks.

[05:29] Christopher David: Let's see.

[05:38] Christopher David: So if we say interact with the web,

[05:40] Christopher David: take directions from a human,

[05:42] Christopher David: make decisions.

[05:53] Christopher David: make decisions on how to interact so

[06:04] Christopher David: let's try a few different ways of kind of simplifying this because what we want to drive this down toward is like data models like if an agent has tasks then we want agent like tasks to be a data domain

[06:21] Christopher David: data model that will build databases for more on that later let's see so

[06:39] Christopher David: for take directions from human that's

[06:49] Christopher David: probably just chat right

[06:51] Christopher David: because you could give it a direction but it might need to kind of clarify it and to kind of like jump ahead and spoil it a bit like we know that we're going to be borrowing the idea from open ai of having english be the like language that is used to command ai is like natural language so

[07:18] Christopher David: the agent needs to be able to chat

[07:24] Christopher David: The agent needs to be able to take actions.

[07:31] Christopher David: And then on like make decisions on how to interact,

[07:35] Christopher David: like

[07:43] Christopher David: we call this chat.

[07:50] Christopher David: Would we call this think?

[07:58] Christopher David: And would we call this like act?

[08:16] Christopher David: I remember that some people have described this part as plan.

[08:28] Christopher David: I like plan.

[08:34] Christopher David: So the agent needs a chat,

[08:35] Christopher David: plan,

[08:35] Christopher David: and act,

[08:36] Christopher David: right?

[08:46] Christopher David: Chat, plan, act. Chat,

[08:47] Christopher David: plan, act.

[08:56] Christopher David: So like let's say a user chats with agent giving it commands.

[09:07] Christopher David: Agent requests clarification

[09:13] Christopher David: via chat until it has enough to plan.

[09:28] Christopher David: Agent builds a plan,

[09:30] Christopher David: agent

[09:41] Christopher David: executes plan, you know, communicating where

[09:47] Christopher David: there are, you know, successes and failures.

[10:02] Christopher David: All right, let's try to distill this into a nice little document here.

[10:05] Christopher David: So we're going to call this Agent First Principles.

[10:24] Christopher David: So an agent must...

[10:33] Christopher David: take directions from a human or

[10:37] Christopher David: another agent create

[10:49] Christopher David: an agent must take directions

[10:59] Christopher David: a plan

[11:00] Christopher David: to follow the directions,

[11:01] Christopher David: take

[11:06] Christopher David: actions.

[11:50] Christopher David: Yeah, I like specifying human or another agent because you can envision an agent spinning of another agent.

[12:00] Christopher David: I know that's interesting to think about.

[12:06] Christopher David: Create a plan to follow the directions.

[12:08] Christopher David: Take actions.

[12:21] Christopher David: to execute the plan.

[12:39] Christopher David: Web interactions via

[12:49] Christopher David: APIs.

[13:01] Christopher David: Controlling a browser.

[13:22] Christopher David: And then maybe we'll put like eventually you

[13:26] Christopher David: can envision an agent placing phone calls,

[13:29] Christopher David: right?

[13:40] Christopher David: Maybe we'll save that for later.

[13:42] Christopher David: So an agent must chat to take directions from a human or another agent,

[13:46] Christopher David: must plan,

[13:48] Christopher David: create a plan to follow the directions.

[13:52] Christopher David: And this is like converting natural language into a set of steps or tasks.

[13:59] Christopher David: And it must take those actions.

[14:02] Christopher David: And the actions that will have available to it are web interactions via

[14:12] Christopher David: APIs or controlling a browser.

[14:25] Christopher David: So that seems pretty comprehensive. Let's kind of step through what are the things we wanted to do for our users.

[14:34] Christopher David: Interact with the GitHub repo.

[14:37] Christopher David: Yeah,

[14:38] Christopher David: the GitHub API.

[14:40] Christopher David: Interact with the code editor.

[14:56] Christopher David: So code editor if it's a desktop app,

[14:58] Christopher David: it's

[15:04] Christopher David: like actions don't need to be on the web.

[15:12] Christopher David: They could be a desktop app like VS Code and that's probably going to be like out of scope or probably going to want to start with just web.

[15:44] Christopher David: Well, there's desktop like apps like VS Code or also more

[15:52] Christopher David: system level things like I know people are trying to give agents control over an entire computer.

[16:16] Christopher David: So the web is a,

[16:18] Christopher David: what do you call it, like an area

[16:25] Christopher David: that an agent can act in,

[16:26] Christopher David: the desktop being an area the agent can act in,

[16:29] Christopher David: like

[16:41] Christopher David: the telecom system.

[16:46] Christopher David: Being an area that an agent can act in.

[17:18] Christopher David: So for, let me save this, for purposes of our focus,

[17:36] Christopher David: Yeah, so we'll focus first on web APIs.

[17:39] Christopher David: That's going to require the least like, I don't know, browser.

[17:45] Christopher David: Things can be like more hacky if you're working with things that theoretically could be interacted with by machines but aren't really designed for it like APIs are really designed for easy consistent programmatic access so

[18:08] Christopher David: we want to train agents to interact with web APIs

[19:26] Christopher David: So let's see if we can identify what data models are appropriate here.

[19:56] Christopher David: So for chat...

[20:04] Christopher David: Conversation

[20:20] Christopher David: message and probably file because people are going to be uploading files.

[21:01] Christopher David: A

[21:02] @: plan.

[21:08] Christopher David: I know that other protocols break this up into task and step,

[21:14] Christopher David: because I was thinking this could be just called a plan,

[21:17] Christopher David: but

[21:20] Christopher David: I don't think it matters so much. I like task.

[21:26] Christopher David: A task is broken up into multiple steps.

[21:50] Christopher David: Well, I guess the planning is the creation of these and the act is actually going through them. And then for plan,

[21:54] Christopher David: I mean, what does it need to do?

[22:07] Christopher David: It probably needs to do things like interact with a vectorized representation of the file.

[22:22] Christopher David: Is there a relevant data model here?

[22:33] Christopher David: Plan just might be more of a process.

[23:00] Christopher David: Okay, let's see if

[23:05] Christopher David: anything's missing here.

[23:13] Christopher David: Let's say I need to have a programming bot or a programming agent do tasks for me.

[23:23] Christopher David: To chat with it,

[23:25] Christopher David: I'm going to create a conversation in the dashboard.

[23:30] Christopher David: Send it a message.

[23:32] Christopher David: Maybe upload a PDF of a file spec or a markdown doc explaining what I'm trying to do.

[23:59] Christopher David: The agent is going to, based on my messages,

[24:02] Christopher David: think about it, you know, do some combination of retrieval and language model things to create a task.

[24:31] Christopher David: Maybe that's the plan breaks things up into tasks and steps and then the actions generate output.

[25:01] Christopher David: generates a result I've

[25:06] Christopher David: seen someone describe that as an artifact if it like produces some thing at the other end of it

[25:29] Christopher David: Maybe instead of result we'll put output.

[25:59] Christopher David: And then let's just see if we can come up with the relationships between these,

[26:03] Christopher David: and then probably in the next video we'll start turning these into unit tests in our Laravel code base and start building up all of the kind of data implementation of these models in our database and then build interfaces for them.

[26:21] Christopher David: Relationships.

[26:27] Christopher David: Conversation.

[26:31] Christopher David: Has many messages.

[26:41] Christopher David: Conversation has many.

[26:46] Christopher David: Should the file belong to the.

[28:21] Christopher David: We want files to maybe like belong to conversations,

[28:26] Christopher David: but also be something that the user can just have like attached to their account so

[28:36] Christopher David: that it can be reused in multiple conversations.

[28:47] Christopher David: So conversation may have many files.

[29:00] Christopher David: And then there's the user and the agent,

[29:02] Christopher David: right?

[29:23] Christopher David: User

[29:26] Christopher David: Has many agents.

[29:36] Christopher David: User has many agents.

[29:47] Christopher David: User has

[29:50] Christopher David: many conversations.

[29:53] Christopher David: Agent has many conversations.

[30:04] Christopher David: Conversation has many messages.

[30:35] Christopher David: Something like this.

[30:56] Christopher David: For user slash agent, maybe we call those actors,

[30:58] Christopher David: people who act.

[31:11] Christopher David: This is feeling almost comprehensive for our current purposes.

[31:27] Christopher David: An agent must chat, take directions from a human or another agent,

[31:31] Christopher David: plan,

[31:32] Christopher David: create a plan to follow the directions,

[31:34] Christopher David: act,

[31:34] Christopher David: take actions to execute the plan.

[31:36] Christopher David: Actions could be interactions via web,

[31:43] Christopher David: desktop,

[31:44] Christopher David: telecom systems,

[31:46] Christopher David: robotics.

[32:03] Christopher David: We're focusing on the web APIs.

[32:08] Christopher David: Which is like most of the entire internet that we would care about

[32:15] Christopher David: The data models you've got your user and your agent chat

[32:23] Christopher David: conversation message file plan task concept act output artifact a

[32:33] Christopher David: user has many agents

[32:37] Christopher David: User and agent may have many conversations.

[32:42] Christopher David: User has many files.

[32:46] Christopher David: Conversation has many messages.

[32:47] Christopher David: Conversation may have many files.

[32:57] Christopher David: Agent has many tasks.

[33:06] Christopher David: Task has many steps.

[33:10] Christopher David: Step has one output.

[33:14] Christopher David: Step may have many artifacts.

[33:27] Christopher David: Does a task have a output?

[33:49] Christopher David: Okay.

[33:51] Christopher David: Some of this may end up being duplicative depending on what we actually see that we need.

[34:01] Christopher David: Okay,

[34:01] Christopher David: so I think that's about good for this video.

[34:04] Christopher David: We thought about what an agent needs to be and do in order to achieve our goals of helping these users do these things.

[34:16] Christopher David: We've categorized those as, you know, we can do all of these things by interactions with web APIs.

[34:23] Christopher David: So that's what we're going to focus first.

[34:25] Christopher David: But it's nice to think that this same chat plan act could apply equally well to other form factors or whatever,

[34:32] Christopher David: desktop,

[34:33] Christopher David: telecom, robotics,

[34:34] Christopher David: etc.

[34:34] Christopher David: for the future.

[34:35] Christopher David: But web APIs will keep us busy for now.

[34:39] Christopher David: To have our agent interact with Web APIs,

[34:41] Christopher David: the data models that are relevant,

[34:43] Christopher David: like the only things that we really think that we need to do that,

[34:46] Christopher David: and this may even be further simplified,

[34:48] Christopher David: but we have users and agents,

[34:50] Christopher David: chats have conversation messages and files,

[34:53] Christopher David: plans have tasks and steps,

[34:55] Christopher David: act has outputs and artifacts.

[34:57] Christopher David: This all seems to fulfill this for Web APIs to solve our users'

[35:05] Christopher David: needs.

[35:05] Christopher David: with these relationships so next we're going to code this up and code all these kind of data relationships into our Laravel application and produce interface to allow people to start creating conversations and doing this kind of thing over the next few videos see you soon