Transcription: OpenAgents Episode 019 - Chat with PDF, OpenAgents Edition
June 9, 2026 ยท View on GitHub
Source: https://twitter.com/OpenAgentsInc/status/1725349984952827929 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 019: Chat with PDF, GPUtopia Edition We finish our 'chat with... Upload date: 20231117 Transcription model: mlx-whisper/mlx-community/whisper-tiny Generated at: 2026-06-01T20:41:59Z
Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.
[00:00] Christopher David: Alright, let's finish our goal for today, which is to get chat with PDF working all
[00:11] Christopher David: with our embedding generated by the GPU Tobias warm.
[00:19] Christopher David: And rag all the ourselves.
[00:23] Christopher David: Okay, so what we have now is the upload form.
[00:30] Christopher David: We can throw a small PDF there, large ones aren't working.
[00:35] Christopher David: Yeah, but this 6 page one first fine, 2 page one first fine.
[00:41] Christopher David: Processing.
[00:48] Christopher David: And then we're going to ask you questions.
[00:56] Christopher David: And we will get back a page of content that is the most similar to what we asked.
[01:12] Christopher David: Now we need to pass this to an LOM.
[01:17] Christopher David: And we may also need to change our chunking strategy because right now we're chunking a buy just the page.
[01:27] Christopher David: And we'll probably want that to be smarter.
[01:30] Christopher David: Which will be super easy, but we'll get to that point.
[01:38] Christopher David: So we're going to need to interface with the GPU Tobias chat and point or inference endpoint.
[01:49] Christopher David: So that'll be fun.
[01:51] Christopher David: We'll set up a class for that.
[01:53] Christopher David: And we will probably extend our current cleanby gateway.
[02:02] Christopher David: Class which is basically just anything that has us interface with the query bait.
[02:17] Christopher David: And maybe we'll try that out.
[02:20] Christopher David: Test.
[02:30] Christopher David: We started my computer in half months.
[02:33] Christopher David: Maybe Dave is back up.
[03:00] Christopher David: Let's skip our tests here.
[03:03] Christopher David: Now that our upload file endpoints is actually doing operations on the PDF that require.
[03:12] Christopher David: We actually need to be the F and that's me as a mark.
[03:15] Christopher David: These are failing and better start care enough about them to happen.
[03:20] Christopher David: Pass.
[03:24] Christopher David: Then we'll do that.
[03:26] Christopher David: We're going to read a hard thing to be fine here.
[03:33] Christopher David: I should be found to that to do something else.
[03:41] Christopher David: So the parts PDF is happening in half.
[03:49] Christopher David: Okay.
[03:52] Christopher David: Pass.
[04:01] Christopher David: One thing I want to check real quick.
[04:03] Christopher David: The previous clip base I was working in did I do a test for inference.
[04:14] Christopher David: I'm going to start by copypasting code from stuff that I did a few weeks ago.
[04:39] Christopher David: Here I've got the contract that's implemented by different gateway.
[04:54] Christopher David: And we may use that extra transport.
[05:07] Christopher David: Okay.
[05:08] Christopher David: So here's the expectation.
[05:19] Christopher David: We'll do a feature test for inference.
[05:35] Christopher David: Test can do chat.
[05:39] Christopher David: Okay.
[05:56] Christopher David: So we'll do a quick test.
[06:06] Christopher David: Okay.
[06:14] Christopher David: So it's called undefined method.
[06:19] Christopher David: Defined model.
[06:34] Christopher David: Defined model.
[06:43] Christopher David: Defined model.
[07:23] Christopher David: The master's in struct.
[07:53] Christopher David: Code.
[08:05] Christopher David: Code.
[08:06] Christopher David: This will be good to know for later.
[08:16] Christopher David: I'm going to try continuing to use the same model.
[08:32] Christopher David: So default model.
[08:42] Christopher David: Okay.
[08:54] Christopher David: That's simple.
[09:00] Christopher David: That was the test.
[09:06] Christopher David: And maybe we'll go take a look in our logs in a way to watch it.
[09:18] Christopher David: Try to assign it to.
[10:57] Christopher David: Okay.
[11:37] Christopher David: Okay.
[12:17] Christopher David: Well, what we want to do now is find where we're returning.
[12:32] Christopher David: And instead return a summary.
[12:42] Christopher David: I mean, yeah.
[12:48] Christopher David: Let's see.
[12:58] Christopher David: So it would be in the query controller.
[13:00] Christopher David: And here we're returning the entire results.
[13:06] Christopher David: And in the previous.
[13:13] Christopher David: Now one question is, do I want to do the summary in the query of searcher.
[13:23] Christopher David: Okay.
[13:33] Christopher David: Okay.
[13:58] Christopher David: So we're going to do this as a test.
[14:13] Christopher David: I kind of want to do a method kind of like we had the method in parse document.
[14:22] Christopher David: Kind of like this.
[14:27] Christopher David: And yeah, I think we can put that on the searcher for now.
[14:37] Christopher David: Let's take this out.
[14:42] Christopher David: Let's do this as a.
[14:50] Christopher David: Can summarize.
[14:55] Christopher David: Can summarize.
[15:05] Christopher David: From context.
[15:09] Christopher David: So I'm going to do searcher equals new structure.
[15:19] Christopher David: And then we do searcher.
[15:26] Christopher David: And then how do we verify this?
[15:53] Christopher David: I'm going to do this as a string.
[16:03] Christopher David: I'm going to do this as a string.
[16:11] Christopher David: Okay.
[16:21] Christopher David: And now I want to let's print bars.
[16:26] Christopher David: Okay.
[16:30] Christopher David: So we're going to do a searcher.
[16:44] Christopher David: So we're going to do a searcher.
[16:51] Christopher David: Okay.
[17:01] Christopher David: Okay.
[17:23] Christopher David: So what is the structure of the data that comes in?
[17:51] Christopher David: Results.
[18:01] Christopher David: Okay.
[18:15] Christopher David: Okay.
[18:44] Christopher David: Okay.
[18:54] Christopher David: Okay.
[19:09] Christopher David: Okay.
[19:19] Christopher David: Okay.
[19:29] Christopher David: Okay.
[19:39] Christopher David: Okay.
[19:48] Christopher David: Okay.
[19:57] Christopher David: Okay.
[19:59] Christopher David: I don't know if I can get it.
[20:09] Christopher David: I don't know if I can get it.
[20:11] Christopher David: I don't know if I can get it.
[20:13] Christopher David: I don't know if I can get it.
[20:15] Christopher David: I don't know if I can get it.
[20:17] Christopher David: I don't know if I can get it.
[20:19] Christopher David: I don't know if I can get it.
[20:21] Christopher David: I don't know if I can get it.
[20:23] Christopher David: I don't know if I can get it.
[20:25] Christopher David: I don't know if I can get it.
[20:27] Christopher David: I don't know if I can get it.
[20:43] Christopher David: There we go.
[20:47] Christopher David: Okay, it's an array of objects.
[20:53] Christopher David: Let's just text that and just like that.
[21:05] Christopher David: Jason.
[21:09] Christopher David: With the text.
[21:23] Christopher David: I don't know if I can get it.
[21:27] Christopher David: I don't know if I can get it.
[21:31] Christopher David: I don't know if I can get it.
[21:35] Christopher David: I don't know if I can get it.
[21:39] Christopher David: I don't know if I can get it.
[21:43] Christopher David: I don't know if I can get it.
[21:47] Christopher David: I don't know if I can get it.
[21:51] Christopher David: I'm going to return the summary here.
[22:21] Christopher David: This is the problem.
[22:45] Christopher David: You're a health assistant.
[22:53] Christopher David: Answer the question.
[23:13] Christopher David: Based on answer the user's question.
[23:25] Christopher David: Based on the following context.
[23:31] Christopher David: Now,
[23:35] Christopher David: generate the context.
[23:51] Christopher David: From the data.
[23:53] Christopher David: I'm going to add the data.
[24:01] Christopher David: First, I just want to make sure I have the format as correct.
[24:07] Christopher David: What would you like to suggest, go pilot.
[24:15] Christopher David: That might work.
[24:25] Christopher David: Let's see what this does.
[24:35] Christopher David: This.
[24:47] Christopher David: Where else is the graph?
[24:53] Christopher David: That's JavaScript.
[24:57] Christopher David: I'm going to add the data.
[25:07] Christopher David: I'm going to add the data.
[25:13] Christopher David: I'm going to add the data.
[25:29] Christopher David: I'm going to add the data.
[25:59] Christopher David: I'm going to add the data.
[26:09] Christopher David: I'm going to add the data.
[26:19] Christopher David: I'm going to add the data.
[26:47] Christopher David: I'm going to add the data.
[26:57] Christopher David: I'm going to add the data.
[27:27] Christopher David: I'm going to add the data.
[27:45] Christopher David: I'm going to add the data.
[27:59] Christopher David: I'm going to add the data.
[28:09] Christopher David: I'm going to add the data.
[28:19] Christopher David: I'm going to add the data.
[28:29] Christopher David: I'm going to add the data.
[28:35] Christopher David: I'm going to add the data.
[28:45] Christopher David: I'm going to add the data.
[28:55] Christopher David: I'm going to add the data.
[29:07] Christopher David: I'm going to add the data.
[29:15] Christopher David: I think I need to specify.
[29:25] Christopher David: I'm going to add the data.
[29:35] Christopher David: I'm going to add the data.
[29:45] Christopher David: I'm going to add the data.
[29:55] Christopher David: I can refine the prompt by saying it.
[30:11] Christopher David: I'm going to add the data.
[30:21] Christopher David: I'm going to add the data.
[30:35] Christopher David: I'm going to add the data.
[30:49] Christopher David: I'm going to add the data.
[30:59] Christopher David: I'm going to add the data.
[31:09] Christopher David: I'm going to add the data.
[31:17] Christopher David: I'm going to add the data.
[31:31] Christopher David: I'm going to add the data.
[31:41] Christopher David: I'm going to add the data.
[31:45] Christopher David: I'm going to add the data.
[31:59] Christopher David: I'm going to add the data.
[32:09] Christopher David: I'm going to add the data.
[32:23] Christopher David: I'm going to add the data.
[32:37] Christopher David: I'm going to add the data.
[32:55] Christopher David: I'm going to add the data.
[33:05] Christopher David: I'm going to add the data.
[33:33] Christopher David: I'm going to add the data.
[34:03] Christopher David: I'm going to add the data.
[34:23] Christopher David: I'm going to add the data.
[34:33] Christopher David: I'm going to add the data.
[34:59] Christopher David: I'm going to add the data.
[35:17] Christopher David: I'm going to add the data.
[35:31] Christopher David: I'm going to add the data.
[35:59] Christopher David: I'm going to add the data.
[36:17] Christopher David: I'm going to add the data.
[36:43] Christopher David: I'm going to add the data.
[36:59] Christopher David: I'm going to add the data.
[37:21] Christopher David: I'm going to add the data.
[37:41] Christopher David: I'm going to add the data.
[38:09] Christopher David: I'm going to add the data.
[38:37] Christopher David: I'm going to add the data.
[38:55] Christopher David: I'm going to add the data.
[39:03] Christopher David: I'm going to add the data.
[39:21] Christopher David: I'm going to add the data.
[39:51] Christopher David: I'm going to add the data.
[39:59] Christopher David: Alright, let's go to the parser.
[40:23] Christopher David: So this is chunking a PDF by page.
[40:30] Christopher David: Change it to chunk it by max.
[40:37] Christopher David: That's a appropriate max chunks as.
[40:52] Christopher David: Chunk size 1024 and chunk overlap.
[40:59] Christopher David: I'm not going to do overlap right now.
[41:22] Christopher David: Okay, so previously.
[41:50] Christopher David: Let's reset our local database.
[42:12] Christopher David: So previously that was 18.
[42:18] Christopher David: It was three times the document was six pages with one per page.
[42:24] Christopher David: If we now go do the same thing we're doing, same paper with six pages, let's see how many chunks we get.
[43:07] Christopher David: What else?
[43:37] Christopher David: I'm going to find a rate please zero.
[43:55] Christopher David: Okay, embedding failed.
[44:05] Christopher David: Okay, let's go.
[44:30] Christopher David: Let's try again.
[45:01] Christopher David: Well, here is the massive thing I'm passing into being bedded.
[45:16] Christopher David: But that was just another might not be recent.
[45:27] Christopher David: What happened?
[45:41] Christopher David: Okay, let's try again.
[46:09] Christopher David: Anywhere we'll receive batch alerts to grease.
[46:32] Christopher David: That was a sign to okay whiteatgetalbe.com on his NPDForg's 49th.
[46:44] Christopher David: Thank you, sir.
[46:54] Christopher David: Oh, now I'm a kuna current model, passing bed, something something.
[47:08] Christopher David: Job worker, staff worker.
[47:38] Christopher David: Okay, let's go.
[48:06] Christopher David: And I retrying to quickly because it's more chunks.
[48:16] Christopher David: It takes longer.
[48:34] Christopher David: Okay.
[48:44] Christopher David: Okay.
[49:09] Christopher David: Okay.
[49:37] Christopher David: Okay.
[50:01] Christopher David: Okay.
[50:26] Christopher David: Okay.
[50:41] Christopher David: That works.
[50:56] Christopher David: Okay.
[51:26] Christopher David: Okay.
[51:51] Christopher David: Okay.
[52:16] Christopher David: Okay.
[52:41] Christopher David: Okay.
[53:09] Christopher David: Okay.
[53:22] Christopher David: Okay.
[53:52] Christopher David: Okay.
[54:17] Christopher David: Okay.
[54:47] Christopher David: Okay.
[55:12] Christopher David: Okay.
[55:37] Christopher David: Okay.
[55:53] Christopher David: Okay.
[56:19] Christopher David: Okay.
[56:34] Christopher David: Okay.
[56:59] Christopher David: Okay.
[57:27] Christopher David: Okay.
[57:57] Christopher David: Okay.
[58:17] Christopher David: Okay.
[58:42] Christopher David: Okay.
[59:10] Christopher David: Okay.
[59:34] Christopher David: Okay.
[59:59] Christopher David: Okay.
[59:59] Christopher David: the
[01:00:07] Christopher David: oh
[01:00:11] Christopher David: hell yes
[01:00:17] Christopher David: chad over docks, motherf
[01:00:19] Christopher David: okay, what is a soft spot?
[01:00:21] Christopher David: yeah, we got to fix the speed and the timeout, it's an air handling and stuff
[01:00:29] Christopher David: okay, let's tweak it a bit and see kind of how much content context we can stuff in there
[01:00:45] Christopher David: with a reliability trap, sword choking thing
[01:00:49] Christopher David: right now let's see if we can ramp this up to a certain three chunks
[01:00:55] Christopher David: eight chunks
[01:01:05] Christopher David: but it wasn't in as it's bad it was the embedding those failings
[01:01:15] Christopher David: and we're still only returning them both
[01:01:19] Christopher David: well, which was the lower opportunity to want
[01:01:27] Christopher David: but ideally we just don't
[01:01:29] Christopher David: I dream we just keep all of the chunks
[01:01:43] Christopher David: I'm gonna try
[01:01:51] Christopher David: that worked
[01:01:53] Christopher David: ask us to write a comment
[01:01:55] Christopher David: what is a soft spot
[01:01:59] Christopher David: you know what I can't want to do
[01:02:13] Christopher David: yes, park yes
[01:02:21] Christopher David: that's really nice
[01:02:29] Christopher David: okay, let's test it live
[01:02:43] Christopher David: and let's try the different document
[01:02:59] Christopher David: by the way, if I ask you it's a little different page
[01:03:15] Christopher David: tell me about incomplete
[01:03:17] Christopher David: yeah, how does a soft spot accept
[01:03:21] Christopher David: how does a soft spot
[01:03:25] Christopher David: build with incomplete
[01:03:31] Christopher David: specific
[01:03:35] Christopher David: I don't know if that's what
[01:03:56] Christopher David: this paper is from 1994
[01:03:58] Christopher David: and I haven't read it and now with
[01:04:02] Christopher David: openagents.app, I don't need to
[01:04:07] Christopher David: okay, did that deploy?
[01:04:10] Christopher David: I guess we'll find out
[01:04:13] Christopher David: hey, let's learn about soft bots
[01:04:17] Christopher David: feed me up load
[01:04:19] Christopher David: you better work
[01:04:21] Christopher David: you're doing a live
[01:04:27] Christopher David: what are soft bots
[01:04:29] Christopher David: you know what, I'm not even gonna
[01:04:33] Christopher David: wait for the first reply
[01:04:36] Christopher David: before I make the second one
[01:04:38] Christopher David: I gotta do with them
[01:04:42] Christopher David: who soft bots are artificial intelligence agents
[01:04:45] Christopher David: developed by researchers at the University of Washington
[01:04:48] Christopher David: I'm not even going to do it before
[01:04:50] Christopher David: they're designed to interact with the wide range of internet resources
[01:04:53] Christopher David: using it to shell the platform task
[01:04:55] Christopher David: okay, it works, we'll have a door a second request
[01:04:58] Christopher David: still pending
[01:05:01] Christopher David: still pending
[01:05:03] Christopher David: and might get assigned to a slow worker
[01:05:05] Christopher David: we'll punish them if they'll get it back to us in time
[01:05:07] Christopher David: oh, ho ho
[01:05:11] Christopher David: whoa, look at that long answer
[01:05:13] Christopher David: oh my goodness, that feels good people
[01:05:28] Christopher David: that feels good, okay
[01:05:30] Christopher David: obviously we're not building like a chat over docs product
[01:05:34] Christopher David: we could, we could charge for this
[01:05:37] Christopher David: add some little sidebar with multiple docs and users
[01:05:40] Christopher David: but there's a million of those on cares
[01:05:42] Christopher David: but this is a really important primitive for being able to have an agent
[01:05:47] Christopher David: have some something language models, some script that you're writing
[01:05:51] Christopher David: have access to a data source
[01:05:54] Christopher David: and then be able to make sense of that
[01:05:56] Christopher David: the fun part comes when we start connecting this to third party
[01:05:59] Christopher David: API is like GitHub
[01:06:01] Christopher David: which we're gonna begin planning in the next video
[01:06:07] Christopher David: see ya