Transcription: OpenAgents Episode 016 - PDF to Embeddings

Source: https://twitter.com/OpenAgentsInc/status/1724801372602950026 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 016: PDF to Embeddings We convert a PDF into vector embedding... Upload date: 20231115 Transcription model: gpt-4o-transcribe-diarize Generated at: 2026-06-01T01:56:17Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: Okay, let's do data models.

[00:03] Christopher David: So how should we do this?

[00:07] Christopher David: I was originally thinking that we would do the unit tests first,

[00:10] Christopher David: but I actually kind of want to go to the highest level test,

[00:14] Christopher David: even above this, and do what's kind of called the happy path test or the main functionality that you're trying to do.

[00:24] Christopher David: you kind of work at that level and then if there's like a sub feature test like hey we need to create or retrieve an embedding then you drop down and you do that feature test if that's missing a unit test for like embedding or something then you drop down to the unit test so our happy path test here is really we're we're trying to upload files the data you know we're in the kind of data preparation phase

[00:46] Christopher David: And the data that we're working with for now is PDFs.

[00:52] Christopher David: We'll probably also want to be able to support HTML so we can have like web scrapers go and collect information.

[01:01] Christopher David: The relevant primitive there is just text, like whether text comes from a PDF or whether text comes from HTML.

[01:08] Christopher David: We're working with text, but we kind of have a bit envisioning that we're going to be starting with files.

[01:14] Christopher David: That's what OpenAI is focused on first,

[01:18] Christopher David: so that's what we'll focus on first.

[01:21] Christopher David: Okay,

[01:24] Christopher David: so we're going to be running our test suite,

[01:27] Christopher David: and I actually want to skip our

[01:36] Christopher David: vector tests.

[01:38] Christopher David: I think it's as simple as doing that.

[01:42] Christopher David: Because they slows down we're probably just gonna straight-up remove them once we get the rest of this built out so now

[01:58] Christopher David: what's taking so long Oh probably creating embedding and

[02:05] Christopher David: querying corpus

[02:11] Christopher David: This is also using Victara. We'll skip that,

[02:20] Christopher David: and then the other potentially

[02:26] Christopher David: long one would be the embeddings,

[02:28] Christopher David: because that's hitting our queen bee, but

[02:32] Christopher David: second and a half, that's alright.

[02:36] Christopher David: Let's do skip Victara.

[02:42] Christopher David: Okay, let's do a new test.

[02:52] Christopher David: So we have upload file test.

[02:57] Christopher David: We are, this kind of is like testing the uploading.

[03:04] Christopher David: The word for like

[03:07] Christopher David: Putting a PDF into a system and doing the necessary conversions to be able to act on it.

[03:13] Christopher David: Everyone seems to call that ingestion.

[03:15] Christopher David: So we're going to call this ingest test.

[03:20] Christopher David: And let's stub out some stuff that has to happen.

[03:29] Christopher David: User uploads a PDF.

[03:35] Christopher David: Actually, let's do this. Given a PDF,

[03:38] Christopher David: oh,

[03:44] Christopher David: Copilot says extract the text from it.

[03:50] Christopher David: We extract

[03:54] Christopher David: extract the text,

[03:57] Christopher David: convert into chunks,

[04:19] Christopher David: Create embeddings for each chunk.

[04:27] Christopher David: Store the embeddings in the database.

[04:34] Christopher David: And then give it a query.

[04:37] Christopher David: Convert the query into an embedding.

[04:39] Christopher David: Find the nearest neighbors.

[04:41] Christopher David: Run the cosine similarity search.

[04:43] Christopher David: Return the results.

[04:48] Christopher David: All right, well this is both ingestion and querying,

[04:54] Christopher David: but

[04:58] Christopher David: that's okay.

[05:07] Christopher David: Okay, don't we have a

[05:16] Christopher David: Demo PDF somewhere here.

[05:29] Christopher David: Let's see what this does.

[05:37] Christopher David: Oh, I guess I should actually write a test.

[05:56] Speaker B: No, I forgot what you call this. Canon just PDF.

[06:30] Christopher David: Okay, so given a PDF,

[06:32] Christopher David: so

[06:36] Christopher David: we have a PDF, now let's extract the text.

[06:57] Christopher David: I'm not really writing tests here, but

[07:02] Christopher David: Alright, so this PDF to text package is what we want to use.

[07:09] Christopher David: That AI with Laravel course that we looked up last time,

[07:13] Christopher David: we did their PG vector tutorial.

[07:16] Christopher David: I got the course and they had some good information in here on different packages to use for doing PDF parsing and stuff in PHP.

[07:24] Christopher David: We always have the option of kind of going out to an external...

[07:28] Christopher David: Worker like we just kind of mocked up this data be thing we might put Python document manipulation stuff there,

[07:34] Christopher David: but um let's see how far we can go with PHP

[07:54] Christopher David: And let me see if

[08:15] Christopher David: Okay, this says you'll need to ha need a PDF to text dependency installed on your system.

[08:22] Christopher David: Do I?

[08:23] Christopher David: No.

[08:28] Christopher David: Brew install poplar. Will that work?

[08:42] Christopher David: And then I'm guessing on.

[08:44] Christopher David: Our production site will need to do popular utils.

[10:20] Christopher David: This Spatia agency has a lot of great packages that I use for different things.

[10:31] Christopher David: Come on, Homebrew.

[10:40] Christopher David: Okay.

[10:41] Christopher David: So there we go.

[10:51] Speaker B: That is where our PDF to text install is.

[11:01] Speaker B: Then we can do this.

[11:18] Speaker B: Okay.

[11:33] Speaker B: services dot PHP.

[11:45] Speaker B: Put that there.

[11:51] Speaker B: And let's try it.

[12:01] Speaker B: I'm curious how long it takes.

[12:18] Speaker B: The tutorials is this WAV document.

[12:19] Speaker B: That's hilarious. Alright so I guess I don't need that.

[12:24] Speaker B: I just needed the path.

[12:30] Speaker B: And let's die and dump the file text,

[12:35] Speaker B: clean that up, and see what happens.

[12:43] Speaker B: Nice.

[12:50] Speaker B: Very nice. Probably want to strip up new lines.

[13:11] Christopher David: Hey, that's great.

[13:14] Christopher David: Configure PDF to text.

[13:20] Christopher David: Text from PDF. That was easy.

[13:34] Christopher David: We could probably turn this into a chat with PDF product really easily.

[13:39] Christopher David: But that's not ambitious enough.

[13:41] Christopher David: So here we go.

[13:54] Christopher David: Let me see what this does.

[13:57] Christopher David: So right now we've got this with a bunch of line breaks,

[13:59] Christopher David: which I think we want to remove.

[14:10] Christopher David: I'm not sure what this does. I didn't pay attention to the class,

[14:13] Christopher David: but I'll just.

[14:21] Christopher David: What does that content line do?

[14:36] Christopher David: Splits the string into an array.

[14:42] Christopher David: Form feed indicates a new character. Oh, sure enough it does.

[14:53] Christopher David: Okay, so it's splitting by page.

[15:09] Christopher David: My content looks like this.

[15:14] Christopher David: For converting a PDF text into vector embeddings for later jag with PDF,

[15:24] Christopher David: do you think I should strip out the new line ands or otherwise

[15:35] Christopher David: reformat that?

[15:51] Christopher David: Okay, okay, tell me how to do it.

[16:06] Christopher David: Yes, please implement those suggestions by adding to this line.

[16:39] Speaker B: Why not remove new lines first?

[17:05] Christopher David: I don't need the explanation.

[17:07] Christopher David: Why can't I stop it? Oh, here we go.

[17:26] Christopher David: Okay, let's see what that does.

[18:09] Christopher David: Fail.

[18:16] Speaker B: Let me try the original thing it gave me, even though it seemed convoluted. Let's try it.

[18:25] Christopher David: There we go.

[18:27] Christopher David: Oh yeah.

[18:46] Christopher David: Okay, so we have a

[18:57] Christopher David: clean-up and converted to chunks.

[19:05] Christopher David: Now um we're currently chunking up by page. Every page of the PDF has its own We're

[19:11] Christopher David: Probably going to want to do that differently,

[19:14] Christopher David: like by character

[19:20] Christopher David: length,

[19:20] Christopher David: but we can tweak that strategy later.

[19:27] Christopher David: Okay,

[19:28] Christopher David: we now want to...

[19:40] Christopher David: Stringable.

[19:46] Christopher David: Now I need this to be an array of strings.

[19:53] Christopher David: So I need to pass it to our embeddings thing.

[20:06] Christopher David: Can convert PDF to KEN and JESS,

[20:09] Christopher David: PDF to uh

[20:15] Christopher David: embeddings

[20:21] Christopher David: chunked embeddings.

[20:24] Christopher David: Chunked.

[20:35] Christopher David: to database as chunked embeddings.

[20:43] Christopher David: Okay.

[20:57] Christopher David: Well, we have our embedding test.

[21:08] Christopher David: Let's see if we can get some embeddings from these.

[21:11] Christopher David: The problem right now, we can't just pass content because this is an array of stringables.

[21:29] Christopher David: This returns

[21:33] Christopher David: an array of stringables.

[21:36] Christopher David: But I need an array of strings.

[22:10] Christopher David: Oh.

[22:15] Christopher David: Yeah, apparently.

[22:17] Christopher David: Okay, we have an array of strings from our

[22:22] Christopher David: chunked up from our ingested PDF.

[22:27] Christopher David: Now, let's create embeddings for each chunk.

[22:58] Christopher David: Oh, no,

[22:58] Christopher David: no, no, no. We did the create the embedding for what is an AI agent.

[23:02] Christopher David: We want the...

[23:06] Christopher David: I'm going to call this chunks so it's clear.

[23:10] Christopher David: And then we're going to pass the chunks.

[23:12] Christopher David: I don't know if...

[23:14] Christopher David: There's no way.

[23:16] Christopher David: We can't handle an entire PDF where the chunks in once.

[23:19] Christopher David: There's got to be a maximum,

[23:20] Christopher David: right?

[23:28] Christopher David: Hang on.

[23:33] Christopher David: It gave us an array of five things.

[23:43] Christopher David: So I don't know what the limits of what clean beat will parse is.

[23:50] Christopher David: If it's more than five pages,

[23:52] Christopher David: if we need to split that up.

[23:53] Christopher David: Um, but we have the embeddings.

[24:03] Christopher David: It might only be a few pages. We'll m

[24:05] Christopher David: figure that out later. Um

[24:11] Christopher David: Okay, let's store the embeddings in the database. And I think we can take that code from here.

[25:03] Christopher David: for each chunks

[25:07] Christopher David: as key

[25:19] Christopher David: result result key embedding metadata

[25:34] Christopher David: chunk maybe and

[25:39] Christopher David: then we want to assert oh

[25:48] Christopher David: my gosh assert

[25:51] Christopher David: that we have the correct number of embeddings based on the length of the chunks well

[26:01] Christopher David: no

[26:02] Christopher David: It should be the length of

[26:21] Christopher David: The amount of chunks.

[26:33] Christopher David: Well, this is going to require that we're getting the correct thing back from a

[26:42] Christopher David: comprehensive result back from Queen Bee, and I'm not sure that we are.

[26:59] Christopher David: Alright,

[26:59] Christopher David: this looks like that we are...

[27:03] Christopher David: Stuck on a bad worker. Let's try it again.

[27:09] Christopher David: Class embedding not found. Okay.

[27:12] Christopher David: Use app models embedding.

[27:23] Christopher David: Holy shit.

[27:27] Christopher David: Holy shit.

[27:39] Christopher David: We can now ingest a PDF to our database as chunked embeddings.

[27:51] Christopher David: We can ingest a PDF to the database as chunked embeddings.

[27:57] Christopher David: What if I say I don't believe you? Show me the embeddings.

[28:03] Christopher David: D-D all embeddings.

[28:04] Christopher David: embeddings

[28:09] Christopher David: Show it to me

[28:17] Christopher David: undefined array key zero

[28:23] Christopher David: Let's print R occasionally we get back a bad result from a

[28:31] Christopher David: Worker.

[28:44] Christopher David: If it takes longer than like five seconds, then we're on a timed out worker.

[28:49] Christopher David: We'll add logic to handle that soon.

[28:56] Christopher David: Heck yes!

[28:59] Christopher David: Oh my goodness.

[29:06] Christopher David: There it is.

[29:07] Christopher David: So our embeddings that are saved to our database,

[29:13] Christopher David: this is saved to the database and then retrieved from the database,

[29:18] Christopher David: ID has the embedding, the 768 dimension PG vector Laravel version.

[29:25] Christopher David: as well as the metadata so now if we do a retrieval based on the embedding we got the text right there and we can compile that into a prompt awesome um yeah

[29:36] Christopher David: queries we did queries in the last video um we'll kind of build out the tests for that uh next but we're we're pretty much ready to start connecting this back to our interface

[29:50] Christopher David: We can ingest a PDF to the database as chunked embeddings.

[29:57] Christopher David: What we're not doing is we're not saving every individual chunk as

[30:03] Christopher David: its own database entry.

[30:08] Christopher David: I had kind of thought that a file would need to have chunks with each having its own embedding.

[30:19] Christopher David: I guess we'll probably just want to attach more.

[30:31] Christopher David: I guess the question of whether or not we need actual models for things like chunk is if it is actually relevant to...

[30:43] Christopher David: something that we're trying to do with that model and this is why it makes more sense to start with the happy path test before dropping down to the feature and unit test that you need because if you do unit tests first you're just kind of guessing at what you think you're going to need but following the conventions of test-driven development you kind of want to start at the higher level and so you're just building the stuff that you actually need

[31:08] Christopher David: As I look at this, I'm thinking that all that we really need now, like we're ready to go the layer above this,

[31:16] Christopher David: which is connecting this to our UI. Because if we can ingest a PDF to database as chunky bennings, and we know from our previous testing that we already are able to run similarity searches on that.

[31:33] Christopher David: I mean, we're well on our way to being able to have a full kind of chat over docs via our UI.

[31:41] Christopher David: So we'll keep driving this out.

[31:42] Christopher David: That's a good stopping point for this video.

[31:44] Christopher David: This is a big accomplishment to be able to do this ourselves locally.

[31:49] Christopher David: We'll continue in the next video.

[31:51] Christopher David: See ya.