Transcription: OpenAgents Episode 012 - RAG Planning

Source: https://twitter.com/OpenAgentsInc/status/1723888973213286760 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 012: RAG Planning We review lessons learned from our RAG effo... Upload date: 20231113 Transcription model: gpt-4o-transcribe-diarize Generated at: 2026-05-31T23:57:30Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: So we have basic chat with a dock working.

[00:08] Christopher David: Got a pretty good response back via Veqtara.

[00:13] Christopher David: It like runs a retrieval augmented generation or RAG query over the dock and then uses LLM in this case.

[00:24] Christopher David: Chat GPT, GPT 3.5 Turbo,

[00:26] Christopher David: I believe,

[00:27] Christopher David: to give you a pretty good natural language reply.

[00:33] Christopher David: But had some issues in the last video, both with some things happening that we don't quite, I don't know, understand or like.

[00:44] Christopher David: But largely the corpora thing being limited to 10 per account before needing to talk to sales,

[00:51] Christopher David: that's just not appropriate for what I want to do.

[00:55] Christopher David: But it's a good opportunity to take what we've learned and use that to help us solve retrieval augmented generation from scratch.

[01:06] Christopher David: So we're going to build up RAG from first principles.

[01:10] Christopher David: Later on, if we think that some piece of this would be better served through a third-party integration,

[01:14] Christopher David: we can,

[01:15] Christopher David: but really just want to see what we can build up ourselves.

[01:19] Christopher David: And also don't want to solve for like all use cases at the same time. It seems like a lot of these platforms are trying to build these general purpose platforms where they're solving something like RAG for...

[01:33] Christopher David: multiple different use cases simultaneously and I've just seen generally this theme that the closer you get to the actual use case the better your responses are going to be so in this case I want us to go back to the github repo

[01:58] Christopher David: take another look at our customer segments our customer desired use cases here how people want to use AI agents and I want to start by focusing on programming because if we can solve that one you know being useful for developers first makes sense because we're going to be

[02:27] Christopher David: Like it's developers that are going to be building this project.

[02:30] Christopher David: So anything that helps us speed up is going to help.

[02:36] Christopher David: And that's certainly an awesome kind of customer segment broadly too.

[02:40] Christopher David: I would love to empower developers to have, you know, agents be able to help with programming in various ways.

[02:54] Christopher David: And rather than try to like directly compete or replace or emulate chat

[03:01] Christopher David: GPT, I want to solve something that chat GPT is just really weak at,

[03:08] Christopher David: which is new information. They just updated the knowledge cutoff from like end of 2021 to April 2023,

[03:16] Christopher David: but it's still like if anyone has tried to get.

[03:22] Christopher David: even the relatively newer chat GPT to help them with like the next JS new versions version 13 which has a completely different syntax with the app router which has a completely different syntax in the pages router chat GPT continually gets confused and gives you like suggestions based on the older version of next JS which it knows a lot better than the newer version and so that was kind of a nightmare and as I was

[03:47] Christopher David: building up the initial GPTopia website in XJS. That was a common thing that I ran into.

[03:52] Christopher David: I got slowed down or just could not rely on chat GPT at all.

[03:57] Christopher David: And so for anything that's newer or cutting edge,

[04:02] Christopher David: being able to have an agent,

[04:05] Christopher David: a programming agent assistant that can help.

[04:09] Christopher David: By dumping in new documents with like the latest APIs or pointing it at a URL that has the API documentation,

[04:16] Christopher David: I would love to be able to train an agent to be able to be an expert of some body of information that we give it in PDF form or markdown form or a link to a URL with text like API documentation and to be able for it to be an expert about it, to be able to answer questions about it.

[04:34] Christopher David: And also, once we then,

[04:36] Christopher David: let's say, the logical next thing there for a programming agent would be to connect it to a GitHub repo where the agent can then make specific audits and recommendations based on your code or then actually start updating your code for you.

[04:51] Christopher David: That would be super cool.

[04:53] Christopher David: So we're going to come at it from that angle.

[04:55] Christopher David: We're going to specialize in like reason about and use this as our example.

[05:02] Christopher David: For the documents that we upload to our new agent,

[05:06] Christopher David: there are going to be some kind of documents explaining some piece of information that would not be found in the base kind of chat GPT knowledge base up till April 2023.

[05:21] Christopher David: A great example for this would be the...

[05:27] Christopher David: Swift GPT.

[05:28] Christopher David: This just came out a few days ago.

[05:30] Christopher David: So someone made an example agent.

[05:37] Christopher David: Bro,

[05:37] Christopher David: first of all,

[05:38] Christopher David: this is like hilarious Threadboys posting chat GPT stores about to create millionaires. Make a custom GPT as early and you will be rewarded heavily.

[05:52] Christopher David: This remains completely theoretical,

[05:54] Christopher David: hypothetical as to who's going to make what.

[05:58] Christopher David: Okay, so we have this.

[06:05] Christopher David: Someone has created a GPT or OpenAI's kind of lightweight,

[06:10] Christopher David: watered-down version of what will become full-on agents.

[06:17] Christopher David: And this was cool because I had this thought the other day,

[06:21] Christopher David: which was like, what would be a good project for me to demonstrate?

[06:27] Christopher David: and like use as a case study as we build up this our own rag with this this new document thing I was like well um I would love to build a watch app for my new Apple watch I just got this after seeing all these people show off their like AI pins and stuff I was like well

[06:48] Christopher David: I could probably do what they're doing, but just do it as like a watch app without too much difficulty.

[06:53] Christopher David: I'm an app developer. I've built apps for iOS devices in the past, but via React Native.

[06:59] Christopher David: And so my own personal knowledge of Swift is very limited just from kind of occasionally dipping into some native code and hacking it up, but I don't know Swift,

[07:09] Christopher David: and it would be a huge accomplishment if I could build up an AI agent that was able to know Swift and to either like teach me,

[07:19] Christopher David: guide me, or just straight up like code for me in Swift to be able to build up a GPTOPIA slash.

[07:27] Christopher David: slash open agents uh smartphone you know watch app um you know by me who's never written a a watch app or or any swift app before that would be cool um so that'll help us both build up the agent and figure out what is needed to have an agent do really cool things like create an app given that i have no experience with that um a watch app

[07:56] Christopher David: and also will let us start to reason about like what actually a watch based agent is able to do

[08:06] Christopher David: so that's a little project we're going to take on and so I had this idea of like oh maybe I should have an agent that knows Swift they could teach me Swift well someone else already had that idea and they made a little GPT here and they shared the link to it and I was asking

[08:24] Christopher David: Some questions like so I asked it like can I write an Apple watch app in Swift?

[08:30] Christopher David: Yes, you can do do do do do do do do

[08:33] Christopher David: Can an Apple watch app do what I needed to do?

[08:36] Christopher David: Can it record a user speech upload to a cloud for analysis that paint audio file?

[08:39] Christopher David: Yes

[08:40] Christopher David: Here's some basic stuff for it.

[08:43] Christopher David: Here's some stuff to think about blah blah blah

[08:47] Christopher David: Here's how you create a new

[08:50] Christopher David: blah blah blah and then I was like okay it knows some stuff about Swift what does it know as I mentioned in a recent video you know a number of us discovered that you can download the files from these GPTs that they were trained on or at least get them to tell you things about them so it's like how did you learn this

[09:19] Christopher David: As Swift GPT,

[09:20] Christopher David: my knowledge of Swift programming comes from a combination of the training data provided by OpenAI and specific Swift programming materials that have been uploaded to me.

[09:26] Christopher David: These materials include detailed documents on Swift programming concepts,

[09:31] Christopher David: examples,

[09:31] Christopher David: and best practices.

[09:33] Christopher David: This combination of general blah blah blah.

[09:37] Christopher David: I was like, give me a list of what materials were uploaded to you.

[09:42] Christopher David: So it says,

[09:44] Christopher David: I've access to two specific Swift programming materials that have been uploaded to Enhanced by Knowledge,

[09:48] Christopher David: a document titled swiftlanguage.pdf,

[09:50] Christopher David: and a document called Learn Swift by Examples Beginner Level.

[09:56] Christopher David: I was like,

[09:56] Christopher David: give me download links for both.

[09:59] Christopher David: I was not able to get,

[10:01] Christopher David: if I were to reword it enough like I did previously,

[10:05] Christopher David: you might be able to.

[10:10] Christopher David: I might just try that one more time.

[10:20] Christopher David: It'll depend on what instructions it's been trained on and stuff.

[10:27] Christopher David: But what I was able to do is to have it tell me...

[10:36] Christopher David: First of all, I was able to Google this, and this is just like a straight-up book,

[10:41] Christopher David: this one here,

[10:42] Christopher David: Learn Swift by Examples.

[10:46] Christopher David: So this is one of the PDFs that was uploaded.

[10:49] Christopher David: And then for the other thing,

[10:50] Christopher David: I asked it to give me some direct excerpts from that PDF.

[10:56] Christopher David: And then when I Googled this,

[10:59] Christopher David: it came up with like those things are found in the Swift book.

[11:06] Christopher David: um like word for word so it's obviously i don't know if there's a pdf of this or they converted it into a pdf or if they did some other text format but basically like the text content that this person trained their swift agent on was just these two materials which we have access to um the cool thing about this is because we know what those source materials are

[11:33] Christopher David: We can use those for our own purposes as a starting point for our own Swift agent and we'll be able to benchmark it against this Swift GPT, you know, compare the responses that we get to.

[11:51] Christopher David: what swift gpt gives now um i don't think i don't know i haven't completely seen if these assistants also have access to third-party apis um that's something that we are going to have access to you know this week or next what it will be okay agent you now know swift connected this github repo and like start coding in swift or start analyzing the swift there

[12:16] Christopher David: So I think we will swiftly have more functionality than the GPT equivalent,

[12:24] Christopher David: and that's kind of the idea.

[12:30] Christopher David: So that's a little introduction to what we are about to do, and then in the next video we will start doing building up.

[12:41] Christopher David: From sort of like the first principles of retrieval augmented generation,

[12:46] Christopher David: kind of like we built up the first principles of agents and like what's needed for agentic behavior,

[12:54] Christopher David: you can see that in our wiki.

[13:04] Christopher David: So we just thought from absolute clean sheet of paper,

[13:07] Christopher David: like an agent must chat plan act.

[13:11] Christopher David: What are the data models needed for that?

[13:12] Christopher David: What are the data model relationships needed for that?

[13:15] Christopher David: We have not yet coded up like the plan and act stuff.

[13:18] Christopher David: We have started on the chat stuff.

[13:20] Christopher David: And this is where because we're going to be interacting with files where we want to have good conversations so we can steer the agents. So handling retrieval augmented generation in this kind of general topic.

[13:37] Christopher David: Makes sense.

[13:40] Christopher David: So we're going to dive into RAG and do our own version of that and then build up our programmer agent to learn Swift and then give it access to GitHub and hopefully help us actually code up a really cool watch app.

[14:00] Christopher David: That'll be our example case study.

[14:02] Christopher David: So, see you in the next video.