Transcription: OpenAgents Episode 146 - Sensemaking: Drones

Source: https://x.com/OpenAgentsInc/status/1868910035281051854 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 146: Sensemaking: Drones What's up with those drones? We as... Upload date: 20241217 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T15:11:00Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: Happy Monday. Welcome to the last week of 12 nights of open agents. We're going to be responding directly to the release of OpenAI today. Search because we really want to search for answers to what's going on with this drone situation. I see it all over my X feed. No one seems to know what's going on. This seems to be where AI assisted sensemaking should help. Like what is the answer? So what we're going to do is look at what OpenAI and perplexity and Grok all have to say about the drone situation. See if we're satisfied by the answers and try by the end of tonight to have some little open agents mini project give better answers, we think, about what these things actually are. Yeah. So I'm going to kind of chop this up. Right now, it's 2.04 p.m. Central time, and we'll kind of skip ahead. But let's start with a little intro here. So I guess OpenAI has already had search working, but they just made it available to free users. Anyway, let's try it. I have been trying it. I installed the Chrome extension, so that replaced my default search. So who is operating the recent drone unidentified drones in New Jersey and other states? All right. Since mid-November, numerous sightings of unidentified drones have been reported across New Jersey and other states, including New York, Pennsylvania, Maryland. These drones are often described as large, with some estimates suggesting diameters up to six feet and have been observed operating during nighttime hours. Despite extensive investigations by federal and state authorities, the operators and origins of these drones remain unknown. The FBI, DHS, and FAA are actively investigating these incidents. A joint statement from DHS and the FBI indicated that many reported sightings might be misidentified manned aircraft operating lawfully. They emphasize that there is currently no evidence suggesting these drones pose a threat to national security or public safety. Okay, that sounds like a fucking lie. And, you know, it'll be fun through this process to like collect the evidence that you might see floating around X or something someone said and like put it into a place where we can actually verify and check these statements from officials. However, some local officials and residents remain concerned. New Jersey Assemblywoman Dawn Fantasia, after a briefing with DHS, reported that the drones operate in a coordinated manner, sometimes flying with their lights off and do not appear to be floating by hobbyists. Additionally, there have been calls for more transparency and action from federal authorities. For instance, former President Donald Trump suggested that the military should intervene to address the drone sites. Despite these concerns, the Pentagon has stated that there is no evidence linking the drones to foreign adversaries. They've also noted that many of the sightings could be misidentified manned aircraft. In summary, while the recent drone sightings in New Jersey and other states have garnered significant attention and concern, the identities of the operators and the intentions remain undetermined. Investigations are ongoing and authorities continue to monitor the situation closely. Okay, so I've had a little bit of conversations with perplexity and such. One of the things that I said, let's do this one first. Give me the three most likely scenarios based on all the available information about the drones, where they're from, who's piloting them, and why. Misidentification of manned aircraft. Many reported drone sightings. Yeah. Several sightings are likely conventional aircraft. Obviously, there's going to be some false positive, but this does not answer the question of what the ones that are legit actually are. Increased drone activity due to regulatory changes. Foreign surveillance operations. Okay. So I also asked ChatGPT-01 Pro, which I'm assuming has search stuff. So I said, who is operating the drones flying around New Jersey and other states in December 2024? I asked this like half an hour ago. As of mid-December, there is no widely confirmed single source responsible for all drones observed flying around New Jersey. It could be a bunch of different... I was like, this is just like what we pulled. I was like, I'm referencing the recent headlines about mysterious drones. Are you aware of those headlines or is that a generic response? What are the main theories about the purpose of these drones that are appearing above sensitive areas, government officials' houses, etc.? I'm aware that in late 2024, a number of media outlets and local communities have reported sightings of unusual unclaimed drone activity, particularly around sensitive areas, critical infrastructure, high-profile officials' residences, and even certain military facilities. While details remain sparse, the combination of sustained presence, apparent coordination, and flight patterns that suggest deliberate reconnaissance have led to intense speculation. Since no apparent... Since no single group or individual has openly claimed responsibility, the purpose remains a subject of debate. The leading theories include foreign state surveillance, industrial or corporate espionage, domestic activism or extremism, criminal surveillance and smuggling, hobbyist pranks or drone tourism, gray zone tactics. Okay, here's a list of theories. Interesting they don't mention government usage. So give me the three most likely scenarios. Foreign state-sponsored surveillance, covert network associated with a foreign government or its intelligence service, like domestic extremists or military groups, organized corporate or criminal espionage. Okay, so I want to assess the likelihood of these different things based on information, because as I'm reading these, I'm just thinking about, well, I saw that one post on X that seems to make certain of these things conflict, and I just don't know what all is going into this response. Can I even look and see? No. Maybe their desktop app, it doesn't have search integrated. Officials downplay, blah, blah, blah. See, if these search things are weighting recent headlines by just what's at the top of their like, what algorithm are they using? Okay, so now this is something that perplexity, I wanted to try perplexity because they bill themselves as the answer engine. You ask them a question, you get an answer. So I started a chat with them 42 minutes ago. I said, who's operating the drones? The identity remains unknown. Despite ongoing investigations, federal and state authorities have not been able to determine who is responsible for these drone flights. Key points, no evidence of a foreign nexus. Don't appear to pose a threat. So what are the main theories? Environmental monitoring, gas leaks or radioactive substances. Foreign surveillance, military or government operations. So this is just like a listing of theories. I said, give me the three most likely scenarios. Now look at this. One of the three posts that Grok used in its analysis is there because it used the phrase likely scenario. So that's just doing a cosine similarity search over what I said. Oh, you want the most likely scenarios? Drones? Here's the most likely scenario. Drones. But no, this is a post from some Anon quote tweeting this TikTok that went viral from John Ferguson that I've separately seen other people in my feed that are more reputable being like, well, that's actually not right. They wouldn't be looking for nuclear because there's this other program that has this. It's like, this is not the best piece of information, but given the unsophisticated search algorithm that you can tell that they're using, it surfaces it as the most relevant, but it's not. Not to the query that I'm asking for. What is the truth? Sorry, Grok, you're not maximally truth seeking in this case. And I'm going to try to build something that's more truth seeking than you in the next 12 hours. Two of these from the same thing. Okay, so that, but this, this guy's thing should be a data point. But I want to be able to do is feed this stuff into a knowledge base, a knowledge graph that's searchable that other people also could add to. Okay. Let's get started. I'm going to do this in a fresh app. I'm going to make a fresh app and I'll just show you guys how I whip up stuff. So let's just do it. What should I call this? Drone sense maker. What the drone? What the drone? What the drone? Public. Okay, I'm going to create a new project. I'm going to call this what the drone. And I'll say we are building a cross-platform Android iOS web app via React Native Expo to help make sense of the recent drone sightings around the US. We want people to be able to search over a publicly visible, editable knowledge graph of recent activity of these drones and relevant responses from authorities. Okay. So here's what I'm going to do. We're going to use my little app factory repo here that I use for whipping up quickly new mobile apps. Npx create expo app latest. I'm going to call this what the drone. Yeah, yeah. I see these people making like a million dollars a month from fucking Cali I like GPT wrapper apps because they just found some like shit experience and they made an AI powered version of it with IAP and like, yeah, yeah. The cost of creating software is approaching zero. The time it takes to create software is approaching zero. Might as well, like create some software that's like simple AI versions of stuff that's out there. Just like generate some fucking revenue. So yeah, we're doing this. Is that done? Yeah. Okay. So just following my little playbook here. Okay. So CD with the drone. Did I make the repo? Yes, I did. This is all open source. You can copy it. I don't care. Get push. And I'll do NPM run reset project. Actually, first I will open it in my editor. I don't know why it opens in the background. And let's do yarn iOS. All right. There's a glorious app.

[19:59] Speaker: And then we also have web browser. So first I want to spec out the app. We're going to add agents Inc with the drone main. We are building it according to the guidelines in app factory repo. Start by reading the README there and the linked setup guide. Okay, let's do it. Read app factory README and setup guide. Summarize here in one paragraph. We are streaming this on X, so good for you to tell the audience what we're doing. Hey X, we're building a cross-platform app using App Factory, which is OpenAgent's toolkit for rapidly building mobile apps. App Factory uses Expo framework to create apps that work on iOS, Android, and web from a single code base. The setup includes TypeScript, Expo router for navigation, and a comprehensive component library. We're using this to build what the drone, an app to help make sense of recent drone sightings across the US by creating a publicly visible editable knowledge graph. The app will let people search and contribute to a database of drone sightings and official responses. We're following App Factory's best practices, which include file-based routing, custom UI components, and platform-specific optimizations. Take a look at what the drone repository, look at all the files, and put in docs slash hierarchy.md of that repo a file folder diagram of everything we have there. I pushed that up, right? Okay, it's just a simple starter app. So when I start a project, I like having everything in one file. If I start a new conversation, I can help the thing get up to speed quicker. Okay, so we have docs, hierarchy, and that's what we've got. Do we want the graph itself to be in 3.js? Do you want to teach it graph RAG? Okay, so let's write a spec. Now write a doc spec.md when you think we have enough detail here for a comprehensive spec and to-do list. Start by asking me any questions needed. Start with this. We're implementing a basic version of graph RAG, having pieces of info represented as nodes with edges between them, and different summaries, tags for categorization and detailed semantic searching. From any given piece of content, like a transcript of a video, we need to store the raw content, links, metadata, time, author, source, etc., as well as a semantic entity extraction, as well as results from a semantic entity extraction according to the graph RAG algorithm. Each entity being, for example, a topic like flight restrictions, which later can be connected with other nodes mentioning the same thing. The core of the app should enable the user to ask questions about the accumulated knowledge and earn some kind of points or reward for adding knowledge that gets added to the graph. Okay, should we store the graph in a traditional graph database? No, let's just use SQL. Let's use local Expo SQLite as much as we can. Like when user first logs in, they get the SQL from some server and can work with it locally and only later maybe send something up to update it. Semantic search using a specific embedding model provider. Or should I recommend one? Ideally something that runs locally on React Native devices. Versioning history of the knowledge graph. Basic version, yes. Any extraction? Predefined ourselves, no. We'll use LLMs to extract that from the first set of content. Should users be able to manually add, edit, extract entities in the first version? No, just read only, but later, yes. Ignore for now. Ignore for now. Ignore for now. Yes, should be zoomable, panable. Maybe in 3.js using Expo 3, etc. Yes, later. Yes, later. Authentication system. Start with no auth. API structure. Rest. Real-time updates. Web socket. When relevant. Okay, that's enough. Write the first spec now. Include a to-do list of implementation steps. Okay, here's what I'm going to do. Where's that thing I posted? I pasted it in here. Create a new doc. Vertical slice. I want basically a demo MVP seeing a graph of the concepts of one node broken out into categories, tags, subnodes, starting with this content. Create a new file. Sources. Some name .md with this information verbatim. Then update the spec to mention the vertical slice in the vertical slice doc. Say details about everything that needs to be done specifically to get the what the drone repo from its current state. You can see what that is in that hierarchy document to the point of being able to interact with initial graph featuring the content from this article. I want to see it on my app in five minutes or less. Go. For setting up the three canvas, I may point it at the Onyx repo to see how that sets up there. That might save us some time. Canvas isn't going to work well on my iOS emulator. I'm using my usual testing device as the camera. Maybe we'll wait on the canvas and just get the data showing. Maybe we'll use something else like SVGs or something. Goal. Create a minimum viable demo showing an interactive graph visualization of concepts extracted from our first source article about DHS and drone powers. View the graph of concepts. Pan and zoom this visualization. Graph view. So what I actually may do is proceed with the 3.js piece, but just pause the video and then resume once I got something worth showing off. But let's just kind of review what it said here. Create essential components. Initial graph data. People. Mayorkas. Hochul. Oh, this is good. Edges. Oh yeah. Oh yeah. Oh yeah. He, he, he. Okay, I'll pause here and... So we're not done yet, but this is a cool milestone. So I've got the llama model 3.2 loaded here. I just ran this like basic analysis just using this kind of very simple graph. Basically, I fed it a bunch of, I fed a bunch of like source docs into here, just transcripts of different like tweets and stuff and like this long article from this Steve Skolache guy with some good analysis. The data quality, I need to put more in. I just want to get this working first. And then we like looped through with this system prompt where just from the like initial dumb implementation that my like coding agent recommended just to get like some basic graph-based analysis in place. It's like the model is you click on a node and then it feeds like the node details and the connection details, the edges into this. And it says, like, what entities are most connected? Are there temporal patterns? Are there geographic patterns? What unusual connections stand out? What might this suggest about drone activities? Provide your insight in this format. Insight, reasoning, confidence, involved nodes. And so I clicked on, I think, the node that I like started the analysis from was Stewart International Airport. And it gave this analysis. The most connected entity appears to be Stewart International Airport with connections to both East Coast and drone sightings. Reasoning, blah, blah, blah, blah, blah. Stewart International Airport may be a key location for drone activity. But this is only with like me putting in the basic, the data around that one node. I want it to consider it holistically. So I'm gonna refactor this now to not focus just on the individual node. Eventually, I would wanna have it like smartly traverse nodes. Like you do the initial lookup to get the most relevant neighborhood and then let it kind of like traverse between connected information. So I'm just gonna refactor this to maybe not use the graph or do something different with this to get to some basic holistic. Like consider all of these things and then kind of like maybe think within it. I don't know. This deserves, you know, a few more days of work to like make it pretty. But I wanted to see if I can just kinda take a quick cludgy ugly code path to getting something that would be hopefully more meaningful than what we got from the billion dollar labs. So keep cranking on this. I'm at a good stopping point here. So I've got this set of insights from having this on-device model, like see the graph, choose a node, look at the connections, think about it, come up with an insight. Do three insights and then like a synthesis. So now these are not perfect. Some of the analysis is a little shitty because I haven't put much data at all in here. But this format of like building blocks, like data points that are reliable. Like number one, military bases are hotspots for drone sightings. Like if we're trying to piece together like an overall thesis, this being like a more or less fact like, yes, military bases are hotspots for drone sightings. What's the reasoning behind that? What are the related nodes? Have that be like a thing that in the future, if you wanna have the agent like think further around that to be able to like pull up that insight and see what the reasoning and what the related nodes are, that's pretty cool. Okay, Mayorkas's involvement in drone-related policies suggests a central role in U.S. drone activities. Hey, he's connected. DHS oversees Secret Service and drone policies and is proposing drone countermeasures for states. High-level officials are deeply involved in drone-related policies.

[39:59] Speaker: High-level government officials, particularly those in the DHS and executive branch, play a central role in shaping U.S. drone policy. So it went from, like, confidence, 90, 80, 90 to this is the synthesis, 95%. Mayorkas understands drone regulations, high-level support for countermeasures, the proximity of military bases raises suspicion about their involvement in drone. So this particular data, we can take with a grain of salt because this is just like kind of very basic node data. But we got the kind of like flow working of having it think. And how I did that was I took the hugging face, scaling test time compute blog post, and fed that into OpenAgents and asked it to give me a basic version of it, minus RL. We're not doing any like weird stuff. We're just kind of like looping through models and prompting it in a particular way. And so we have a simple iterative approach of thinking through nodes and edges. We've got to refine this like system and prop, but it conducts an analysis. Again, need to workshop this, but what are the connected entities? What are the temporal patterns, geographic patterns? What unusual connections stand out? What might this suggest about drone activity? Analyze small subgraphs. So all this is open source on our OpenAgents Inc. slash what the drone repo. And in terms of the format, I love this. The only weakness that I see with this is just like the data that it's looping through is not high quality or well-structured or like I just pasted like four articles and it had like it do the most basic entity extraction. So by kind of cleaning up and strengthening this data pipeline and then having other people maybe contribute to this or be incentivized to like contribute their own data that they think is important, opening this up into kind of like a crowdsourced knowledge base with other people like helping to kind of curate the data could be helpful. So I think that we've made progress. We're not quite to where I like want this to be in terms of like kind of drop in definite superiority to what the billion dollar labs and companies are doing. But I think that we're on to something with this kind of graph rag traversable knowledge base, agent traversing knowledge base. So we're going to kind of continue on this in the next video. All of this is going to be folded into Onyx to kind of do this for any general topics that you want. But I am interested in seeing how much better we can make this if we give it another day of work. So tomorrow we'll kind of do part two of this and flesh out the data pipelines and see if we can get some like really compelling insights once we feed some higher quality data in. Until then, enjoy your weeks. Yeah.