Transcription: OpenAgents Episode 013 - RAG First Principles

Source: https://twitter.com/OpenAgentsInc/status/1724432749275095365 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 013: RAG First Principles We plan to implement retrieval-augm... Upload date: 20231114 Transcription model: gpt-4o-transcribe-diarize Generated at: 2026-06-01T00:11:49Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: Good morning. In this video,

[00:02] Christopher David: we're going to build RAG from scratch.

[00:07] Christopher David: Retrieval augmented generation.

[00:09] Christopher David: So we kind of did this for the agents,

[00:12] Christopher David: reasoning about what agentic behavior needs to be from scratch so we can build our own very basic data models and implement them all ourselves.

[00:20] Christopher David: And we're going to do the same thing now for the brain of the agent.

[00:26] Christopher David: When it comes time to need to reach for inference, we'll be using the GP Utopia swarm inference,

[00:35] Christopher David: our GPU network with open models for that.

[00:39] Christopher David: But other ways of interacting with a document, if you're new to retrieval augmented generation,

[00:46] Christopher David: you know what, let's just go learn what ChatGPT has to say.

[00:49] Christopher David: What is retrieval augmented generation?

[00:57] Christopher David: And the idea is that we're gonna be doing a new document. Let's actually make it now. We'll call it rag first principles.

[01:06] Christopher David: We build this up completely from scratch so we're not dependent on any third party. And we can build our own understanding of it.

[01:18] Christopher David: Retrieval-augmented generation is a method used in natural language processing that combines the capabilities of pre-trained language models with a retrieval system.

[01:27] Christopher David: Here's how it works.

[01:30] Christopher David: Retrieval component.

[01:31] Christopher David: When the model receives a query,

[01:32] Christopher David: the retrieval component first searches a large data set of documents like Wikipedia or a specialized corpus to find relevant information.

[01:40] Christopher David: This is similar to looking up reference materials before writing an essay.

[01:47] Christopher David: Why this is so slow.

[01:51] Christopher David: Generation component. The retrieved documents are then provided to a generative language model. This model, which has been pre-trained on a vast amount of text, generates a response based on both the input query and the information from the retrieved documents.

[02:05] Christopher David: The key benefit of RAG is its ability to leverage a vast amount of external information,

[02:09] Christopher David: which helps in generating more accurate, informative,

[02:12] Christopher David: and contextually relevant responses,

[02:13] Christopher David: especially for queries where the pre-trained model alone might lack sufficient knowledge.

[02:17] Christopher David: This approach is particularly useful in cases where keeping a large language model continually updated with the latest information is impractical.

[02:24] Christopher David: So the example we mentioned in our last video of the language models whose

[02:30] Christopher David: Training cutoff has been late 2021 or now April 2023.

[02:35] Christopher David: If they don't have the latest specs,

[02:38] Christopher David: documentation,

[02:40] Christopher David: APIs of whatever we're trying to code,

[02:44] Christopher David: a smartwatch app using the latest Swift APIs.

[02:52] Christopher David: If you have outdated information like we had previously been.

[02:57] Christopher David: Try to use ChatGPT to build an XJS website with XJS13 when it keeps giving us like outdated syntax can waste a lot of time or just not know what to tell you.

[03:06] Christopher David: So we're generally needing to enable an agent to have specialized knowledge beyond what it was trained on.

[03:22] Christopher David: So.

[03:23] Christopher David: What can we say about this?

[03:33] Christopher David: Maybe we'll put it in like super basic terms.

[03:38] Christopher David: Put that in one sentence that a seven year old could understand.

[03:59] Christopher David: Retrieval augmented generation is like asking a smart robot to first look up information in a big book and then use that information to answer your question.

[04:07] Christopher David: That's awesome.

[05:30] Christopher David: We need an agent to act

[05:40] Christopher David: on knowledge

[05:44] Christopher David: it wasn't trained on.

[05:47] Christopher David: Yeah.

[05:49] Christopher David: Sort of like simply state what the hell we're doing here.

[06:11] Christopher David: Simplify, simplify, simplify.

[06:33] Christopher David: I like this answer.

[06:35] Christopher David: Retrieval Augmented Generation is a technique where an AI first finds relevant information from a large database,

[06:41] Christopher David: then uses this information to generate an informed response.

[07:36] Christopher David: What's problem and solution?

[07:56] Christopher David: Okay.

[08:02] Christopher David: I'm thinking maybe a section called like how?

[08:08] Christopher David: We are writing a very simple document called

[08:17] Christopher David: rag first principles.

[08:22] Christopher David: Add a section for how with one sentence.

[08:27] Christopher David: I'm thinking like text in text out.

[08:54] Christopher David: No, that is duplicative with the what section.

[09:03] Christopher David: Let's break down the steps needed to implement this.

[09:09] Christopher David: For example, in a Laravel web application.

[09:32] Christopher David: To implement RAG in a Laravel web application, you would typically follow these steps.

[09:38] Christopher David: And while that goes, I'm going to see if we can

[09:46] Christopher David: merge a PR that fixes

[09:51] Christopher David: a bug maybe.

[10:13] Christopher David: Actually, let's just see if this works here.

[10:23] Christopher David: Look how fast that is.

[11:03] Christopher David: Ooh, got the Markdown formatted.

[11:05] Christopher David: They don't even format the Markdown. We format the Markdown.

[11:43] Christopher David: Yeah, we get a better answer here.

[11:57] Christopher David: Let's see if this is something that we want to adapt into an actual build-out order.

[12:04] Christopher David: So to implement RAG in a Laravel web app,

[12:06] Christopher David: number

[12:08] Christopher David: one, integrate a database or knowledge repository.

[12:12] Christopher David: set up a database or connect to an external knowledge repository that the AI can search for relevant information.

[12:18] Christopher David: This could involve storing extensive textual data or linking to an external API.

[12:23] Christopher David: We have a Postgres database with PG Vector support.

[12:33] Christopher David: Choose a language model.

[12:35] Christopher David: We will use open models.

[12:39] Christopher David: From hugging face like

[12:46] Christopher David: Mistral 7b as a base

[12:49] Christopher David: Develop a retrieval mechanism create a retrieval system in Laravel that queries the database or repository based on user input This involves setting up search algorithms or utilizing existing search tool existing search was might know us relevant

[13:04] Christopher David: Documents or data.

[13:07] Christopher David: We will implement a search ourselves probably with vector similarity search with

[13:19] Christopher David: cosine similarity search over vector embeddings.

[13:23] Christopher David: We'll write that down in a second.

[13:25] Christopher David: Store it in the database.

[13:28] Christopher David: Implement the language model.

[13:31] Christopher David: API

[13:41] Christopher David: We will create a

[13:44] Christopher David: an OpenAI compatible API service that works with our existing GPTopia.

[14:00] Christopher David: API.

[14:04] Christopher David: We will create an API's uh Laravel

[14:08] Christopher David: service that works with our existing

[14:12] Christopher David: OpenAI compatible GPT API.

[14:14] Christopher David: Okay.

[14:16] Christopher David: Combine retrieval and generation.

[14:18] Christopher David: Code the logic to combine the retrieved data with the user's query and feed this

[14:43] Christopher David: We will create a Laravel service class for rag that combines retrieval and generation.

[14:55] Christopher David: Optimize your performance and accuracy,

[14:57] Christopher David: blah,

[14:57] Christopher David: blah, blah.

[14:59] Christopher David: UI testing.

[15:01] Christopher David: Okay.

[15:04] Christopher David: All right. Well, this at least helps me visualize how we're going to.

[15:09] Christopher David: implement this.

[15:23] Christopher David: Let's see if we can break this down a

[15:26] Christopher David: little simpler.

[15:37] Christopher David: So I want to get down to the level of the data model like we had in the other document.

[15:44] Christopher David: What is it that we're storing?

[15:47] Christopher David: How does it relate to our existing models?

[15:50] Christopher David: How does it relate to what the user needs to upload? Probably documents and stuff.

[15:57] Christopher David: I get down to the primitives.

[16:25] Speaker B: Wow, that's slow.

[16:37] Christopher David: Here are data models from our broader agent integration that this RAG integration will fit with.

[16:50] Christopher David: Suggests some that will work.

[16:52] Christopher David: Suggest new models needed

[16:56] Christopher David: to implement.

[17:23] Christopher David: Look at that beautiful-ish markdown.

[17:27] Christopher David: Not so fast.

[17:55] Speaker C: That's not very good.

[18:23] Christopher David: I don't know that those should be models.

[18:50] Christopher David: All right, I'm going to make this even simpler.

[18:52] Christopher David: Um,

[18:55] Christopher David: what's needed to enable this? Well,

[18:59] Christopher David: data must be stored in a database

[19:11] Christopher David: such that

[19:15] Christopher David: the agent can

[19:21] Christopher David: Find information relevant to a user query to a query could be a user query could be another agent Korean or prompting it

[19:37] Christopher David: Agents prompting other agents.

[19:42] Christopher David: Oh, this is gonna get fun

[19:49] Christopher David: Data must be stored in a database such that the agent can find information relevant to a query.

[20:06] Christopher David: Blah, this is a bad I don't like this either

[20:42] Christopher David: To jump ahead, I think this will be done as

[20:50] Christopher David: each file will

[20:55] Christopher David: have one

[21:02] Christopher David: embedding.

[21:14] Christopher David: Or more?

[21:30] Christopher David: Should file have chunk that has embedding?

[21:35] Christopher David: Okay.

[22:05] Christopher David: A user query should be converted into

[22:25] Christopher David: a search over documents

[22:28] Christopher David: for appropriate

[22:32] Christopher David: context.

[22:35] Christopher David: I've been pre-trained on terminology,

[22:38] Christopher David: but I do want to break things down into as simple of English as possible.

[22:45] Christopher David: A user query should be converted into a search over documents for appropriate.

[23:12] Christopher David: The user query should be converted to a vector embedding.

[23:29] Christopher David: A cosine similarity search.

[23:35] Christopher David: Similarity search.

[23:37] Christopher David: should be run between that

[23:42] Christopher David: the query embedding and the

[23:50] Christopher David: user's file embeddings.

[24:34] Christopher David: Now we can make these uh numbered steps. I think, does that work with the in-line bullets?

[24:45] Christopher David: No, that's kinda ugly.

[24:56] Christopher David: Store the data properly.

[25:05] Christopher David: convert the user query into an embedding or like breaking this down to the steps that we need to do for each each thing perform the search run the cosine similarity search i think

[25:30] Christopher David: they call this top k but like return the

[25:52] Christopher David: Return the closest Return the most relevant documents.

[26:12] Christopher David: Context and metadata for each

[26:18] Christopher David: are content like the text content.

[26:22] Christopher David: And any metadata like where it is in the document, what document it comes from,

[26:26] Christopher David: um

[26:37] Christopher David: combining

[26:41] Christopher David: resulting text metadata.

[26:53] Christopher David: Into a prompt.

[26:59] Christopher David: Send prompt to LLM to synthesize and answer. Okay.

[27:16] Christopher David: This is a basic flow for how rag works.

[27:21] Christopher David: I think that's somewhat comprehensive.

[27:32] Christopher David: Simplify that. Is anything missing from this?

[27:57] Christopher David: So we store the data such that the information agent can actually find it, find the info.

[28:05] Christopher David: A user query is converted from natural language into a vector embedding.

[28:17] Christopher David: Yeah, converted from natural language.

[28:26] Christopher David: To perform the search, a cosine similarity search should be run between the query embedding and the user's file embeddings.

[28:39] Christopher David: Return the most relevant documents'

[28:41] Christopher David: content metadata.

[28:42] Christopher David: Combine the resulting text and metadata into a prompt.

[28:46] Christopher David: Send the prompt to the LLM to synthesize an answer.

[28:59] Christopher David: Now I'm fairly certain that when we go to what we have, what we're currently relying on for this full flow,

[29:09] Christopher David: this is exactly kind of what it's doing.

[29:19] Christopher David: So if I go ask what is seasteading, and

[29:25] Christopher David: I'm going to ask that of

[29:27] Christopher David: This too.

[29:30] Christopher David: Do we think it's doing the same things?

[29:32] Christopher David: Data must be stored in a database such that the agent can find information relevant to a query.

[29:36] Christopher David: Well,

[29:37] Christopher David: when we uploaded the PDF earlier and it like puts it into a corpus, it does all the processing to do that.

[29:43] Christopher David: A user query should be converted into a search for over documents for perfect context.

[29:47] Christopher David: So I'm fairly certain that when we put this in our natural language.

[29:52] Christopher David: Query is converted to a vector embedding. It's running a vector embedding search over,

[29:55] Christopher David: and then it's running the search,

[29:59] Christopher David: returning

[30:02] Christopher David: the most relevant documents,

[30:04] Christopher David: in this case the different like chunks,

[30:06] Christopher David: content

[30:10] Christopher David: and metadata.

[30:11] Christopher David: So this is the content.

[30:13] Christopher David: This is the metadata.

[30:18] Christopher David: And then it combines the resulting text and metadata into a prompt.

[30:21] Christopher David: And this was my, like one of the problems with like relying on a third party service is that I actually don't know how they're doing this. And this is pretty important information about what they're structuring the prompt,

[30:32] Christopher David: like

[30:36] Christopher David: what the prompt actually says to create the summary.

[30:40] Christopher David: Because maybe I want this to be including other context.

[30:45] Christopher David: or not include these footnotes or whatever but it sends that prompt to the llm i know that in this case also they're using gb3.5 turbo which i don't want to use um but it produces this result of our query um that's pretty cool like i'm fairly certain that that is the process that they're using behind the scenes there and i just want to do our own version of that

[31:27] Christopher David: Okay, let's dig into this cosine similarity search.

[31:31] Christopher David: First of all, what does this say?

[31:35] Christopher David: Your basic description of RAG is quite comprehensive for an initial setup, but there are a few additional considerations and potential refinements.

[31:41] Christopher David: Embedding storage.

[31:44] Christopher David: File versus chunk embeddings.

[31:47] Christopher David: Depending on the granularity of your search,

[31:49] Christopher David: you might prefer storing embeddings at a chunk level rather than a file level. This allows for more precise retrieval of information from larger documents.

[32:01] Christopher David: Because we're not going to be able to pass entire files, like if I've got a 100-page PDF,

[32:06] Christopher David: we don't want to just do an embedding of the entire PDF.

[32:10] Christopher David: We want to chunk it up and do embeddings over the chunks because we're going to have to pass like paragraphs to the LLM.

[32:24] Christopher David: Yeah,

[32:25] Christopher David: let's just make that decision here.

[32:26] Christopher David: File.

[32:30] Christopher David: File has many chunks.

[32:33] Christopher David: This is really what we wanted to get to, is to the level of the data model, because this is something that we can write tests around,

[32:38] Christopher David: same as we did for the agents.

[32:39] Christopher David: File has many chunks.

[32:44] Christopher David: Chunk has one embedding, right?

[33:02] Christopher David: That is something we can implement in Laravel.

[33:04] Christopher David: Okay,

[33:07] Christopher David: to

[33:11] Christopher David: Query conversion.

[33:13] Christopher David: Vector embedding of user query.

[33:14] Christopher David: Ensure that the method for converting user queries into vector embeddings is consistent with the method used for your database embeddings to maintain search accuracy.

[33:23] Christopher David: So we're going to use the standard, in

[33:28] Christopher David: terms of the actual vector embeddings.

[33:39] Christopher David: Um,

[33:39] Christopher David: there's one particular embedding model that we use. It's like the main open source model.

[33:43] Christopher David: I got to remember the exact name of it,

[33:45] Christopher David: but, um,

[34:07] Christopher David: Both file and query will use the same embedding model.

[34:15] Christopher David: Let's just see if I can do it with a quick

[34:18] Christopher David: Google search.

[34:21] Christopher David: Oops, uh Google uh

[34:24] Christopher David: embedding model hugging face.

[34:37] Speaker B: Embedding

[35:07] Christopher David: I think it's this one,

[35:09] Christopher David: BGE.

[35:17] Christopher David: I'll just put that in my notes as a question mark.

[35:24] Christopher David: Alright, a user query.

[35:29] Christopher David: Alright, let's keep going down this.

[35:30] Christopher David: Search process. Efficient search algorithm.

[35:32] Christopher David: The cosine similarity search should be optimized for performance.

[35:36] Christopher David: Um,

[35:40] Christopher David: tell me about cause that's what is that the best standard way to compare vector embeddings or

[35:54] Christopher David: what are other options?

[36:11] Christopher David: Cosine similarity search is a common and effective method for comparing vector embeddings, especially in the context of natural language processing information retrieval.

[36:31] Christopher David: It measures the cosine of the angle between two vectors in a multi-dimensional space.

[36:38] Christopher David: It's pretty cool to think about, and I'm new to these considerations, but like if you're, I think the default open AI embeddings is like 1,536 vectors.

[36:50] Christopher David: So it's like an array of numbers,

[36:53] Christopher David: 1,536 of them. So it's like a 1,536 dimension array.

[36:59] Christopher David: So like if you have coordinates 1, 2, 3,

[37:03] Christopher David: X,

[37:03] Christopher David: Y, Z,

[37:03] Christopher David: like it's in three-dimensional space in a certain place.

[37:07] Christopher David: But if it's got 1536 dimensions,

[37:10] Christopher David: then it's like you can map any language to some space there and they can combine the spaces to see or like run this search to see what's closer to each other in space.

[37:24] Christopher David: And there's people who figured out how to like visualize that in 3D space.

[37:27] Christopher David: It's pretty cool. I don't know so much. I'd like to learn more about this stuff,

[37:31] Christopher David: you know, alternatives to cosine similarity.

[37:35] Christopher David: Yeah, we're not going to do any of that.

[37:39] Christopher David: But use case. Ideal for text comparison as it is less affected by the magnitude of vectors and focuses on the orientation,

[37:44] Christopher David: making it effective for comparing documents of different lengths.

[37:48] Christopher David: It's computationally efficient and scales well with high dimensional data.

[37:51] Christopher David: Okay,

[37:52] Christopher David: so we're definitely going to do that because I don't know enough to choose anything different,

[37:55] Christopher David: and that's what everyone seems to be using.

[37:58] Christopher David: Cosine similarity search.

[37:59] Christopher David: Okay, return the most relevant documents.

[38:03] Christopher David: Implement a system to rank the retrieved documents by relevance and possibly limit the number of documents.

[38:20] Christopher David: How should we limit it?

[38:29] Christopher David: Should we try to stuff as much as possible into the prompts,

[38:34] Christopher David: like fill up the prompts?

[38:39] Christopher David: Should we try to fill up the context window with as much as possible?

[39:04] Christopher David: So in terms of the data models and like how we actually implement this,

[39:17] Christopher David: I think all of this stuff down here

[39:26] Christopher David: is going to be logic in our service class.

[39:31] Christopher David: Like stuff that we're doing with the query,

[39:34] Christopher David: running the search,

[39:37] Christopher David: manipulating the data,

[39:39] Christopher David: returning it,

[39:40] Christopher David: that's going to be just kind of business logic that gets implemented in the service classes that we write.

[39:49] Christopher David: And then for new data models that we'll want tests around,

[39:52] Christopher David: we're going to want to write tests for

[40:00] Christopher David: kind of base it off this file

[40:03] Christopher David: has many chunks chunk has many embedding has one embedding we're going to want to drive out tests for the creation of the embeddings we're going to figure out how we're going to create the embeddings make sure that we're saving and retrieving those properly from the database and

[40:19] Christopher David: then doing some example searches just to make sure that that works so

[40:27] Christopher David: let's do this

[40:29] Christopher David: Make a build out order.

[40:42] Christopher David: Set up new data models.

[40:45] Christopher David: File chunk embedding via TDD.

[41:07] Christopher David: Let's do this. Write unit tests for new data models file chunk embedding.

[41:21] Christopher David: The file is not new. We just need to do the relationships.

[41:40] Christopher David: Right feature test for

[41:45] Christopher David: creating and saving,

[41:47] Christopher David: right feature test for creating, saving,

[41:52] Christopher David: well, creating and retrieving

[41:58] Christopher David: embeddings from database.

[42:19] Christopher David: Okay, and

[42:29] Christopher David: then what else running

[42:37] Christopher David: cosine similarity search

[43:08] Christopher David: We're going to need to um

[43:15] Christopher David: generating a text inference.

[43:33] Christopher David: And then the full

[43:37] Christopher David: write query flow.

[43:35] Christopher David: And there might be some smaller steps in there.

[43:45] Christopher David: So is this logical?

[43:49] Christopher David: We're first setting up the new data models chunk and embedding and the relationships with file.

[43:56] Christopher David: Testing that.

[44:01] Christopher David: Then feature test for creating embeddings and retrieving them from the database.

[44:06] Christopher David: Making sure that our we might need to add like the PG vector extension to Postgres

[44:11] Christopher David: Make sure that it's working fine

[44:14] Christopher David: Actually running a cosine similarity search

[44:25] Christopher David: Let's do this it

[44:28] Christopher David: might be combined but like converting user query into

[44:35] Christopher David: embedding,

[44:37] Christopher David: writing the cosine similarity search,

[44:39] Christopher David: generating a text inference and the full flow.

[44:51] Christopher David: Okay,

[44:53] Christopher David: let's um just review this and see if it feels like we're missing anything or if this feels comprehensive.

[45:01] Christopher David: Why are we doing this?

[45:03] Christopher David: Well, we need an agent to be able to act on knowledge that it wasn't trained on.

[45:15] Christopher David: Specifically,

[45:16] Christopher David: knowledge that we're going to be uploading new docs for.

[45:20] Christopher David: So retrieval augmented generation is a technique where an AI first finds relevant information from a large database.

[45:28] Christopher David: then uses this information to generate an informed response.

[45:33] Christopher David: I'm just going to remove the word large there because it's not needed.

[45:41] Christopher David: How do we do this?

[45:42] Christopher David: We first need to store the data in a database such that the agent can find information relevant to a query.

[45:52] Christopher David: We're doing this like a file has many chunks. A chunk has one embedding.

[45:59] Christopher David: File and query need to use the same embedding model.

[46:02] Christopher David: We'll fill that in with whatever the top hugging face embedding model is.

[46:06] Christopher David: A user query is converted to a search over documents for the appropriate context.

[46:13] Christopher David: Converting from natural language to vector embedding.

[46:15] Christopher David: Perform the search via a cosine similarity search matching the user's query as an embedding to find the embeddings from the chunk that fit that.

[46:26] Christopher David: The closest,

[46:26] Christopher David: the top ten or so, will return that content metadata for each of those chunks.

[46:38] Christopher David: And if needed we can like run an additional LLM call to like help us sort if there's some that are actually not relevant or filter them out.

[46:47] Christopher David: We're going to combine the resulting text and metadata into a prompt.

[46:53] Christopher David: And then send that prompt to the LLM to synthesize an answer.

[46:58] Christopher David: So a good test for this will be as if we can get this flow

[47:04] Christopher David: instead of going out to Victara to do

[47:07] Christopher David: all of this ourselves so that we'll have chat over doc.

[47:19] Christopher David: Having that work all ourselves with our own.

[47:22] Christopher David: embeddings going into our swarm network with no third parties aside from to be able to do chat over docs using no third party integrations just our own swarm network using the our network of GPUs to create the embedding that's the only part that we need actual compute part compute for

[47:48] Christopher David: And then just storing the embeddings and running searches over those.

[47:57] Christopher David: That would be really cool.

[47:59] Christopher David: Then we don't have to worry about consuming API keys or rate limits.

[48:04] Christopher David: We can set them ourselves.

[48:06] Christopher David: Maybe we'll set them to be infinite.

[48:09] Christopher David: Okay, so in the next video we'll focus on setting up these relationships and getting embeddings working.

[48:19] Christopher David: embeddings and embeddings and embeddings and banks.

[48:22] Christopher David: See you soon.