Transcription: OpenAgents Episode 069 - URL Scraper Plugin

Source: https://twitter.com/OpenAgentsInc/status/1753237589945905373 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 069: URL Scraper Plugin We write a second plugin that fetches... Upload date: 20240202 Transcription model: mlx-whisper/mlx-community/whisper-tiny Generated at: 2026-06-01T19:08:35Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Christopher David: Alright plugin one is working. Let's do plugin number two. Given a URL we turn its

[00:10] Christopher David: main content sanitized and converted to markdown. This is going to require us to

[00:23] Christopher David: loop through this one for each of these unless we want to make this take an

[00:31] Christopher David: array of URLs. You might be fun to just do this as is and then reason separately

[00:39] Christopher David: about how to loop through because we're going to want there to be some sort of

[00:42] Christopher David: interface for repeating a step. So let's just do this plugin and we do

[00:51] Christopher David: same as we did in the last video. A rust plugin and what I want to do is I'm

[00:56] Christopher David: going to fork this. Maybe not fork just kind of what clone it. I don't want

[01:08] Christopher David: that upstream connection. So let's go to here as you get clone that as plugin URL

[01:21] Christopher David: scraper. Okay let's go into there. Let's set up the repo.

[01:52] Christopher David: Let's go get remote remove origin. Zer. And then that's get push you origin

[02:04] Christopher David: main. Okay. Okay. And then let's pop open that same

[02:22] Christopher David: XDism quick start so we can see the build steps. And then out of curiosity can I

[02:33] Christopher David: just run that right now. Does all the cargo stuff there and then I don't know

[02:50] Christopher David: if I still have access to that same command that I did the other day. By which I mean this morning.

[03:06] Christopher David: So that'll error because it won't be called that's but maybe it'll be called plugin URL

[03:19] Christopher David: scraper. No because that's release. Let's do debug plugin URL scraper doesn't exist

[03:39] Christopher David: because I didn't actually build it or did I cargo build target.

[03:57] Christopher David: Okay plugin URL extractor we need to I'm guessing that's based on the

[04:03] Christopher David: time all file. Forgive my rest new business. Yeah here we go. Plug in URL scraper.

[04:17] Christopher David: And then aside from that it should be the same. So let's build it again.

[04:32] Christopher David: There we go. Okay. So let's get that up speed we'll call it.

[04:42] Christopher David: Rename extractor to scraper. Okay. We are now set up with the same thing that we had before.

[05:02] Christopher David: So let's take a look at our live file.

[05:05] Christopher David: Okay.

[05:15] Christopher David: Okay.

[05:18] Christopher David: If you see a couple more things to do before we say goodbye to it.

[05:23] Christopher David: Okay. So now we are doing plugin number two starting as a fork of the previous one. Let's change this code.

[05:44] Christopher David: To instead to instead to instead do this.

[06:00] Christopher David: To change the plugin functionality to accept the URL and return its main content,

[06:12] Christopher David: and then to the market.

[06:23] Christopher David: Oh, you're not going to give me any of the libraries.

[06:32] Christopher David: These functionality might need to be implemented outside of wasm. Okay. Yeah. Yeah.

[06:41] Christopher David: So that is a question which is what can we do in the context of the ex-tism plugin and where do we need to supplement this with other libraries?

[06:51] Christopher David: Yeah.

[07:02] Christopher David: Hmm.

[07:15] Christopher David: Yeah. This will be something we'll probably want at some point, but our PHP implementation.

[07:25] Christopher David: The PHP SDK does not yet support host functions, but we shouldn't need that for a basic version. Let's see.

[07:38] Christopher David: Replace that some stuff with Rust Standard Lib packages where possible.

[07:47] Christopher David: And if not, suggest packages we can add via cargo, write a completely implemented function.

[08:05] Christopher David: Oh, given the constraints of running web assembly, especially with the direct inability to make network requests from within wasm, I'll guide you

[08:21] Christopher David: through and approach that uses Rust packages.

[08:28] Christopher David: For this demonstration, we'll assume an environment where you can use these packages, keep in mind that for actual fetching and processing.

[08:34] Christopher David: You might need to rely on external systems or adapt the argument.

[08:38] Christopher David: You know, it may be the case that Chad you doesn't actually know enough about ex-tism to give us a good advice here.

[08:44] Christopher David: Or, yeah, maybe we just can't make HTTP requests from inside wasm plugins. I don't know.

[08:52] Christopher David: Oh, what learning?

[08:57] Christopher David: So fetching content normally you use crates like request for asynchronous HTTP requests.

[09:02] Christopher David: However, request and network requests directly aren't supported in wasm compile from Rust to running browser environments due to the lack of network and files to access.

[09:11] Christopher David: You'd need an external service or a server-side component, the fetched URL content.

[09:27] Christopher David: For sanitizing HTML, you can use the ammonia crate, which is a white list based HTML's antenna.

[09:39] Christopher David: Since the direct fetching is not possible in the wasm environment.

[09:57] Christopher David: For a complete solution that includes fetching content from the URL, you would typically implement the fetching logic on a server or backhand service that your wasm module can communicate with, possibly through an API.

[10:14] Christopher David: Okay, so we don't need to be dogmatic if that is the case and we're going to kind of validate and look into that and learn more about exactly what that boundary is.

[10:25] Christopher David: We can do this as just like our piece of functionality that lives in our backhand, but let's just see, let's just see, let's just see, let's just see.

[10:55] Christopher David: Just do one.

[11:26] Christopher David: Okay, this is good.

[11:41] Christopher David: So,

[11:44] Christopher David: Wasm, you need to use the wasm specification to execute web assembly plugins from non-browser host applications.

[12:11] Christopher David: Another example of a limitation is that a wasm function cannot make HTTP requests or write to the console, just play it with.

[12:19] Christopher David: The workaround is to create functions in the host application to perform these operations and expose them to the wasm module for deus.

[12:27] Christopher David: Again, this is not trivial task.

[12:29] Christopher David: Fortunately, we have Xism. Xism provides all the necessary plumbing to overcome the limitations of the wasm specification.

[12:36] Christopher David: Making it easy to develop for instance, a go application that can execute wasm plugins developed and go, write us to leave a JavaScript and other languages of course.

[12:44] Christopher David: Xism also comes with the C.A.L.I. that allows you to test your plugin. So today we will focus on developing web assembly plugins.

[13:06] Christopher David: Did you actually do future?

[13:37] Christopher David: Xism, web assembly plugins and host functions.

[13:49] Christopher David: As I explained in the previous article, wasm programs are limited by default. It's also a security principle.

[13:55] Christopher David: The wasm specification is a set of APIs that will allow the development of web assembly wasm programs to access system resources, if permitted by the host application.

[14:04] Christopher David: The wasm specification is still being written and only a few APIs are available.

[14:08] Christopher David: Therefore, even though it's possible to use a socket or file system API, a wasm programs capabilities in accessing system resources are limited, no terminal display, no HTTP access and so on.

[14:22] Christopher David: We are saved, we have host functions. However, to make our lives easier, the host application can provide additional powers to the guest, the wasm module.

[14:32] Christopher David: We call these host functions. A host function is a function developed within the host applications source code.

[14:40] Christopher David: The host application exposes it, exports it to the wasm module which can then execute it.

[14:47] Christopher David: For example, you can develop a host function to display messages and allow the wasm module to display messages in a terminal during its execution.

[14:57] Christopher David: You should note that when you use host functions, your wasm module will only be executable by your host application.

[15:12] Christopher David: It's a host application, but it's a host application that would be capable of passing in that.

[15:22] Christopher David: The wasm thing that we didn't launch video, that's simple as a regax parse in rust compound to wasm, there's nothing specific, so that could be run anywhere.

[15:32] Christopher David: I'm guessing that if a host function, if you're passing in a fetch, an HTTP fetch that takes in a parameter and returns and does the thing, you could pass in a fetch parameter probably from any different language.

[15:46] Christopher David: I don't know, but that's what I would think. I don't want people to be dependent on the opin agents platform.

[15:52] Christopher David: Even if it's more difficult to run a plugin on a server environment that would have to pass in that same host function, ideally, that would be able to be done by anyone.

[16:05] Christopher David: If you were to spin up your own opin agents instance, because you forker could base or if you write your own.

[16:12] Christopher David: That plugin should ideally still work, but for most people, we would love it if they would use obnages.com for us making that easy to them.

[16:22] Christopher David: So yesterday I explained that the type system in the wasm is the specification for passing function parameters and return values is minimal, only numbers are supported.

[16:30] Christopher David: This implies some acrobatics to develop a host function.

[16:34] Christopher David: Extrism provides ready to use host functions.

[16:38] Christopher David: To help you develop wasm programs without worrying about complexity, the ecstism plugin development kit PDK provides ready to use host functions, including logging HTTP requests and reading and in memory configuration.

[16:52] Christopher David: Creating a new wasm plugin with the ecstism PDK.

[17:00] Christopher David: Guys, I'm trying to learn rust not go.

[17:18] Christopher David: Okay.

[17:20] Christopher David: So, simplest for me would be if we were able to just load the plugins in our existing code base and do the host functions thing there.

[17:32] Christopher David: Their PHP SDK does not currently support host functions. I can request that.

[17:37] Christopher David: In the meantime, it could be fun, like certainly educational, for me to spin up a go API.

[17:48] Christopher David: Do this in go.

[17:52] Christopher David: Bonus points throw that go endpoint behind an L4O2 thingy.

[18:06] Christopher David: What did so we just put out? They just did a really some kind of go thingy today.

[18:17] Christopher David: Maybe a goling L4O2 to consume L4O2.

[18:27] Christopher David: Should we line up some months?

[18:29] Christopher David: Let's go.

[18:31] Christopher David: And it should be the alibi wallet.

[18:35] Christopher David: Use the internet to create the F4O2 client.

[18:41] Christopher David: People like go like, you know.

[18:43] Christopher David: But people like them rust. I'm trying to follow the trends here people.

[19:13] Christopher David: Okay. I think we're going to want to go through this tutorial.

[19:31] Christopher David: And get that working.

[19:39] Christopher David: Okay. So here's the even more caveat. I've spent a little bit of time learning rust.

[19:43] Christopher David: I spent almost no time learning go. So this will be like completely new being it up.

[19:48] Christopher David: But that might be fun for you to watch. Let's see. Okay. Let's keep it going on this.

[19:54] Christopher David: To help you develop walls and programs without worrying about complexity.

[19:57] Christopher David: The ex-tism plugin development kit.

[19:59] Christopher David: provides ready to use host functions, including logging,

[20:02] Christopher David: HTTP requests, and reading, and in-memory configuration.

[20:06] Christopher David: Creating a new Wazom plugin with the access and Pdicate.

[20:09] Christopher David: Start by creating a go.mod file using the command,

[20:13] Christopher David: go mod, and in-radius host functions,

[20:15] Christopher David: and then create a main.go file with the font cover.

[20:21] Christopher David: Pdk.log is a host function provided by the access and CLI,

[20:24] Christopher David: allowing sending messages to the logs.

[20:27] Christopher David: The Pdk.getconfig host function allows reading values

[20:31] Christopher David: from a configuration passed in memory by the host application.

[20:34] Christopher David: In this example, we retrieve a URL for an HTTP request.

[20:37] Christopher David: That's what we need.

[20:38] Christopher David: We again use Pdicate.log, the value associated

[20:41] Christopher David: with the route configuration key in the logs.

[20:43] Christopher David: The Pdk new HTTP request host functions allows making HTTP requests.

[20:49] Christopher David: That's what we want.

[20:52] Christopher David: When you develop your own applications with the access

[20:56] Christopher David: SDK, they will also provide these same host functions,

[21:05] Christopher David: which we can't use from Lairville.

[21:10] Christopher David: It's also possible to develop custom host functions,

[21:12] Christopher David: but that's really good.

[21:15] Christopher David: Tiny Go has built in just OK.

[21:17] Christopher David: Now it's tested.

[21:18] Christopher David: To compile the program, use Tiny Go and the following command,

[21:21] Christopher David: which will produce a host functions, Wazom file.

[21:27] Christopher David: Executing the say hello function of the Wazom plugin.

[21:51] Christopher David: OK.

[21:52] Christopher David: So what we're going to do is we're going to do this tutorial.

[21:56] Christopher David: And then we're going to try to deploy it.

[22:06] Christopher David: All right, I'm going to new Linux desktop policy.

[22:19] Christopher David: All right, you're going to see me do this from scratch.

[22:21] Christopher David: Linux blah blah blah.

[22:23] Christopher David: Do I have a user and local Go?

[22:27] Christopher David: I need to get to see me do it on a new keyboard too.

[22:35] Christopher David: That's not a thing.

[22:46] Christopher David: Download for us.

[22:56] Christopher David: OK, let's just first make sure we don't have a user local Go.

[23:23] Christopher David: And then in downloads, how awkward and painful is this video good?

[23:35] Christopher David: We're going to do this.

[23:41] Christopher David: Prevention denied.

[23:44] Christopher David: Are you doing that with pseudo?

[23:53] Christopher David: OK.

[24:09] Christopher David: Let's do that in our room.

[24:23] Christopher David: All right, we have going install.

[24:37] Christopher David: That wasn't too horrible.

[24:44] Christopher David: All right, we'll call this host function.

[24:52] Christopher David: All right, let's go.

[24:56] Christopher David: This will probably grow into.

[25:06] Christopher David: Let's do a make their plugin URL scraper.

[25:14] Christopher David: Go.

[25:22] Christopher David: Go.

[25:41] Christopher David: Vim mode on the terminal.

[25:52] Christopher David: Go.

[26:12] Christopher David: Tiny Go.

[26:15] Christopher David: I definitely don't have that.

[26:17] Christopher David: Install.

[26:20] Christopher David: Tiny Go.

[26:47] Christopher David: I'm going to do a make-up.

[27:17] Christopher David: I'm going to do a make-up.

[27:37] Christopher David: Looks like they skipped a step there.

[27:39] Christopher David: Titorial.

[27:42] Christopher David: What do I do? Go get.

[27:44] Christopher David: Seems logical.

[27:47] Christopher David: All right, cannot use get on type string constant as PDK HTTP method value in argument to PDK and new

[27:59] Christopher David: HTTP.

[27:59] Christopher David: So I'm going to guess that that's like a version conflict or something.

[28:17] Christopher David: I'm going to do a make-up.

[28:47] Christopher David: I'm going to do a make-up.

[29:17] Christopher David: I'm going to do a make-up.

[29:47] Christopher David: I'm going to do a make-up.

[29:53] Christopher David: I'm going to do a make-up.

[30:01] Christopher David: Some day instead of me copypacing random box in the chat,

[30:05] Christopher David: it'll just like search around with random box.

[30:07] Christopher David: Search around random box and then make a zone decision.

[30:11] Christopher David: See if this go PDK documentation answers how it should be.

[30:19] Christopher David: I'm going to do a make-up.

[30:37] Christopher David: Probably version mismatch.

[30:39] Christopher David: This is like a six-month-old tutorial.

[30:43] Christopher David: I think.

[30:49] Christopher David: And I don't think they say what version of the plugin they use in the PDK.

[30:57] Christopher David: They have a get every bow.

[31:19] Christopher David: Yeah, it's a tutorial where you don't see version numbers and you don't choose what's code is kind of like,

[31:25] Christopher David: I don't know how it's so great.

[31:29] Christopher David: But let's see if we can figure it out.

[31:43] Christopher David: Let's just find the stuff that it's whining about.

[31:47] Christopher David: And so it's new ACs via requests.

[32:09] Christopher David: No, I don't want to search everything.

[32:17] Christopher David: Add it in VO1 up.

[32:35] Christopher David: Based on the documentation you provided, it clarifies the issue you're encountering.

[32:39] Christopher David: The PDK, new ACV request function,

[32:41] Christopher David: expects the first argument we have pet agency method,

[32:43] Christopher David: which is a blah blah blah blah blah blah blah.

[32:45] Christopher David: Before, after that's all I want.

[32:47] Christopher David: That's all I want.

[32:49] Christopher David: Well, not all I want.

[32:51] Christopher David: You need to do it for me.

[32:53] Christopher David: Change my code.

[32:55] Christopher David: Give me a back massage.

[32:57] Christopher David: Get.

[32:59] Christopher David: Where's our get?

[33:01] Christopher David: Get.

[33:03] Christopher David: Some day I will learn to delete things without also copying to the keyboard.

[33:09] Christopher David: Okay.

[33:11] Christopher David: PDK, method get.

[33:13] Christopher David: Anything else I need?

[33:15] Christopher David: I need to do it.

[33:21] Christopher David: Now where's I?

[33:23] Christopher David: You must see what that's.

[33:25] Christopher David: Not there.

[33:29] Christopher David: We're not doing that anymore.

[33:45] Christopher David: Okay.

[33:47] Christopher David: What do I do?

[33:49] Christopher David: Tiny go.

[33:51] Christopher David: Build it, tiny go.

[33:53] Christopher David: Let's take it work.

[33:55] Christopher David: Maybe.

[33:57] Christopher David: What to do?

[33:59] Christopher David: Host dash functions adopt a wasom.

[34:01] Christopher David: I don't know.

[34:03] Christopher David: I don't know.

[34:05] Christopher David: I don't know.

[34:07] Christopher David: I don't know.

[34:09] Christopher David: I don't know.

[34:11] Christopher David: I don't know.

[34:13] Christopher David: Cool.

[34:17] Christopher David: Now what?

[34:19] Christopher David: Executing the say hello function of the wasom plugin.

[34:29] Christopher David: For this we'll use the XSBCLi.

[34:31] Christopher David: I already installed that.

[34:33] Christopher David: To execute the say hello function with the screen creator.

[34:35] Christopher David: And moji hello world emoji.

[34:37] Christopher David: From time to go.

[34:39] Christopher David: Use the following way.

[34:41] Christopher David: I'll allow host star.

[34:45] Christopher David: Config route equals json placeholder.

[34:49] Christopher David: What does that do?

[34:51] Christopher David: I'll look.

[34:53] Christopher David: Hello.

[34:55] Christopher David: This is wasom.

[34:57] Christopher David: Calling.

[34:59] Christopher David: Are you all up?

[35:05] Christopher David: Let's try.

[35:07] Christopher David: Cool.

[35:17] Christopher David: I'm guessing we got that from there.

[35:25] Christopher David: Okay.

[35:29] Christopher David: This does what we thought at least or soon or soon.

[35:35] Christopher David: We can't do without host functions which is make.

[35:39] Christopher David: And it's just a peak call.

[35:43] Christopher David: Let's take a look in the code.

[35:47] Christopher David: Kind of just like look at it.

[35:49] Christopher David: You look at it after you run and see the breaks.

[35:53] Christopher David: So.

[35:55] Christopher David: Export say hello.

[35:57] Christopher David: We're guessing that's a couple of the rust macro.

[36:01] Christopher David: So read function from memory read function argument from memory.

[36:11] Christopher David: Read function argument.

[36:17] Christopher David: From memory.

[36:27] Christopher David: Does it mean config route.

[36:35] Christopher David: So pdk log wave hello.

[36:39] Christopher David: This is wasom.

[36:43] Christopher David: Get the value associated with the route key in the config object.

[36:55] Christopher David: Get info to the logs.

[36:57] Christopher David: Make an HTTP request.

[37:03] Christopher David: Read the result of the request.

[37:13] Christopher David: Parse it.

[37:17] Christopher David: That's kind of clear.

[37:27] Christopher David: Add a curiosity.

[37:35] Christopher David: I go back to our original chat here.

[37:41] Christopher David: Let's use.

[37:51] Christopher David: Using exesom host functions to circumvent the HTTP limitation.

[38:05] Christopher David: We have assembly offers a viable path for fetching content from URLs.

[38:09] Christopher David: Drag us in a while to plug it.

[38:11] Christopher David: Based on the example you can adapt the approach to implement the plugin that fetches the URLs.

[38:15] Christopher David: And then you can convert it to mark down.

[38:17] Christopher David: Since the original example is in go and your plugin is in rust.

[38:21] Christopher David: Let's conceptualize how you can achieve this using rust and exesom host functions.

[39:15] Christopher David: The question is do I want to keep going with the go or try to redo this stuff in rust?

[39:21] Christopher David: I don't trust chat, TPS rust knowledge enough to be able to guide me better even if I'm pacing in docs.

[39:35] Christopher David: I have curiosity.

[39:37] Christopher David: Can I go to this definition?

[39:41] Christopher David: That's so kind of freaking thing there.

[40:29] Christopher David: Well, let's do this.

[40:50] Christopher David: This is a ghost-function tutorial.

[40:57] Christopher David: I'll push up to get up later.

[41:03] Christopher David: So let's change this around now.

[41:31] Christopher David: No, let's not do rust.

[41:34] Christopher David: Let's just get this working in go by extending this.

[42:01] Christopher David: Please change this to take a URL as input.

[42:14] Christopher David: And first, just return the HTML from as a string return from fetching it via get request.

[42:41] Christopher David: Let's do this.

[43:10] Christopher David: Let's do it chatting.

[43:18] Christopher David: Here's your contract.

[43:24] Christopher David: Let's delete all this.

[43:34] Christopher David: Let's try that build command again.

[43:44] Christopher David: See what happens.

[43:59] Christopher David: And sure you correctly know you correctly can be.

[44:29] Christopher David: This should now work.

[44:36] Christopher David: Yeah, I'll be the judge of that.

[44:46] Christopher David: 4 errors down to 2.

[45:11] Christopher David: Because you're making up error handling that doesn't exist.

[46:03] Christopher David: I know you're not giving me the same thing.

[46:20] Christopher David: I think it is, but I'm going to give it the benefit out.

[46:44] Christopher David: You've heard of that.

[46:54] Christopher David: There is no such thing as REST error.

[47:09] Christopher David: That's right.

[47:29] Christopher David: I don't know if that's correct, but this should at least not have the same error.

[47:44] Christopher David: There we go.

[48:00] Christopher David: Now what is not even doing?

[48:10] Christopher David: Now please modify this line for calling that.

[48:18] Christopher David: I'll just give me a thing that I'll do.

[48:32] Christopher David: Grab this URL, please.

[48:43] Christopher David: Do it.

[48:58] Christopher David: Do it.

[49:03] Christopher David: Do it.

[49:30] Christopher David: Do you think it's going to work?

[49:46] Christopher David: Oh yeah.

[49:53] Christopher David: If we fetch a wrong GitHub URL, we don't need to sanitize it.

[50:08] Christopher David: I mean, this is doing what we want.

[50:26] Christopher David: I'm thinking through now is, okay, I've got to plug in.

[50:38] Christopher David: I'm going to put up an API in point online.

[50:52] Christopher David: Maybe we want to put that up in an API in point that we set up and maybe we want to throw it behind an L-4-2 endpoint.

[51:10] Christopher David: I'm going to have to work here.

[51:25] Christopher David: I'm going to put an API endpoint right now.

[51:54] Christopher David: Let's think about that.

[52:34] Christopher David: Instead, we will deploy and set we wrote plug in number two in Go.

[52:52] Christopher David: We'll deploy it separately to a new Go environment, which we may also put behind an L-4-2 endpoint.

[53:17] Christopher David: That via our existing L-4-2 integration.

[53:34] Christopher David: Okay, we have a basic plug-in that does this, given the URL, returns main content.

[53:42] Christopher David: I'm not worried about the sanitizing convertible.

[53:57] Christopher David: I'm going to put it behind an L-4-2 endpoint.

[54:12] Christopher David: Dashboard same as we did in the episode 53-S63.

[54:19] Christopher David: I'm losing track while these videos.

[54:22] Christopher David: See you soon.