Transcription: OpenAgents Episode 148 - Exploring the Genesis Physics Engine

Source: https://x.com/OpenAgentsInc/status/1869631823182975426 Wiki source: https://raw.githubusercontent.com/wiki/OpenAgentsInc/openagents/Video-Series.md Media title: OpenAgents - Episode 148: Exploring the Genesis Physics Engine We explore today's... Upload date: 20241219 Transcription model: gpt-4o-mini-transcribe Generated at: 2026-06-01T15:09:31Z

Machine-generated transcript. Review speaker labels and wording before using this as quote-grade source material.

[00:00] Speaker: we'll get back to Onyx app stuff tomorrow. I wanted to review this new open source physics engine. This seems like the most exciting open source AI release in a long time. I saw this tweet on my feed. There have been times when I felt this simulator might be too advanced to release, said one of the authors of this new generative physics engine open source project. We're going to watch the video together here in three minutes. A few more introductory tweets. 20 plus labs just worked together to open source Genesis, a physics engine with a VLM, vision language model, I think, agent that turns text prompts into interactive 4D worlds. Think instant physics accurate environments, camera paths, and character animations, all from natural language. Turn natural language prompts into fully simulated environments, tasks, motions, even policies. Robotics and embodied AI just got a serious upgrade. Imagine training AI agents, testing robots, or creating entire virtual worlds at a fraction of the time. 100% Python, fully open source, built for everyone. The barrier to entry is gone. Wow. So we're going to just kind of try to wrap our heads around this. And we'll call this sensemaking because I want to make sense of this to see how it could intersect with open agents in the future. Maybe even intersect with Onyx. Who knows? But let's wrap our heads around this. So we're going to kind of use the open agents pro dashboard in a minute here to kind of ask questions of the readme and the code base and just kind of like get really rapidly up to speed with what Genesis is all about, throw it in some open agents and Onyx docs and kind of like speculate about if there's any intersections between the marketplace that we've been talking about, like the data marketplace we talked about in the last video and the kind of one market idea. Does this intersect with embodied AI robotics, virtual worlds, all this physics fun stuff? I'm looking for an excuse to get into the like fun latest stuff. So let's see. Let's watch this video. Okay. Here's the start of the video. Folks, we're building agents. These are going to be robot brains eventually. We're not going to be a software incorporeal, non-embodied for long. And you're going to want your robot brains to be open source. Yeah. Built on open stuff. Yeah. Yeah. Physics engine fully open source. It's the kind of video that you would say like, all right, this is some like stilt project doing a glossy marketing video to announce their $30 million raise or something. But nope. It's like, Hey, here's the GitHub repo. So tell me about the GitHub repo. So let's do this. We're learning about Genesis. Open the readme and summarize in three paragraphs. So Genesis is a comprehensive physics platform designed for robotics, embodied AI, and physical AI applications. At its core, it combines three main components, a universal physics engine built from scratch, a lightweight and fast robotics simulation platform, and a photo realistic rendering system. The platform also includes a generative data engine that can transform natural language prompts into various types of data. The system boasts impressive capabilities, including unprecedented simulation speeds, 43 million FPS frames per second. What is that? Huh? What? Oh, I guess you would want 43 million frames per second if you're trying to understand, have robots do simulations faster than in real world. Okay. Cross-platform compatibility and support for various physics solvers and material models. It can handle everything from rigid bodies and liquids to gaseous phenomena and deformable objects, while supporting a wide range of robots, including robot arms, legged robots, drones, and soft robots. The platform is designed to be fully differentiable and includes a physics-based tactile sensor simulation module. Genesis is built with three long-term missions in mind, making physics simulations more accessible and lowering barriers to robotics research, unifying various state-of-the-art physics solvers into a single framework for high-fidelity virtual world creation, and minimizing human effort in data collection and generation for robotics applications. The project is open source, Apache 2 licensed, and actively welcomes community contributions to help create a better environment for robotics research and development. Holy shit. Okay, so what they open sourced is part of it. It is what? Here's the post from the main dude. Let's read this. While we read it, let's have this do this. Boom. Here's the post from an author of it. Summarize this and also what you learned from scraping those URLs. Okay. Now let's read this. So, everything you love about generative models now powered by real physics, announcing the Genesis project. After a 24-month large-scale research collaboration involving over 20 research labs, a generative physics engine able to generate 4D dynamical worlds powered by a physics simulation platform designed for general-purpose robotics and physical AI applications. Genesis's physics engine is developed in pure Python while being 10 to 80 times faster than existing GPU-accelerated stacks like Isaac, Gym, and MJX. It delivers a simulation speed 430,000 times faster than in real time and takes only 26 seconds to train a robotic locomotion policy transferable to the real world on a single RTX 4090. The Genesis physics engine and simulation platform is fully open source at this repo. We'll gradually roll out access to our generative framework in the near future. Yeah, this is the kind of thing that makes me think that if they're open sourcing a whole bunch of the underlying code in the same way that MidJourney productized stable diffusion and tweaked it and arguably has done in some ways better as a company than Stability has done, maybe there's an opportunity for open agents to do something interesting here. Genesis implements a unified simulation framework all from scratch, integrating a wide spectrum of state-of-the-art physics solvers, allowing simulation of the whole physical world in a virtual realm with the highest realism. We aim to build a universal data engine that leverages an upper-level generative framework to autonomously create physical worlds together with various modes of data, including environments, camera motions, robotic task proposals, reward functions, robot policies, character motions, fully interactive 3D scenes, open-world articulated assets, and more, aiming towards fully automated data generation for robotics, physical AI, and other applications. Oh my God. And they just dropped it on GitHub. That's how you do it. This is open space AI. This is truly open AI. Open sourcing a ton of research, massive collaboration between 20 labs, and then they're gonna build some products on top of it that they hope to make money from. I hope they succeed. This is amazing. We need more of this. Fucking awesome. But I hope that we can participate as a company and do cool things from open agents. We may have some unique angle on this. Okay, let's see if we got any cool thing here. This scraped the docs for locomotion and that. Here's a summary combining the author's post. Genesis is a groundbreaking physics platform, blah, blah, blah. Technical achievement, generative capabilities. Okay. Cross-platform operation, Linux, macOS, Windows, and various compute backends. CPU, NVIDIA GPU, AMD GPU, Apple Metal. Nice. Okay, cool. So did they say there were like docs? Let's take a look at the docs. API reference. I go straight to the, oh my gosh. Okay, so I want to understand what they have open sourced and speculate about what they might later release as products. Also, what might be possible for us, open agents, to potentially productize or and or combine with our Onyx app. Start by traversing the Genesis repo and this page and any relevant links to tell me more about what they've open sourced. Yeah, this is just Quads on it 3.5 with some tools, GitHub API, and Firecrawl for scraping web pages. It's good for research. Amazing. I don't know if this is familiar with Onyx. Read the Onyx readme. And what? Read about R. And this. Episode 147 transcript. Okay, so what's currently being, please don't. I got to fix that. That's annoying. Come on. Okay, so what's currently open sourced? Core physics engine, scene management system, multiple physics solvers, rigid body dynamics, collision detection, force field simulation, rendering system, rasterizer, ray tracer. Okay, the stuff that you would expect in like a game engine, I guess. Entity management. Yeah, same thing you'd expect in a game engine. File format support. GLB. Nice. Robot specific features. So interestingly, did it scrape anything from Onyx? I don't know why. I don't know what information it's going off of when it, it may just be assuming that there's an agent framework. Okay. So without further context, it says the most interesting opportunity might be combining Genesis's physics simulation capabilities with Onyx's agent framework to create an interactive physics aware AI assistant that can help with 3D design and prototyping, robotics programming, and testing, physical system analysis, interactive learning environments, real-time physics simulations. Based on the documentation and Genesis capabilities, here are specific ways. Physics-based data validation and generation. Use Genesis as a physics validation service in the Onyx data marketplace. Okay. When users submit drone footage or UAP sightings, Genesis could validate physical plausibility of movements. Whoa. Generate physically accurate simulations for comparison. Simulate some of these UAPs. Create synthetic training data for anomaly detection. Offer this as a paid service to the DVM protocol. Simulation as a service integration. Add Genesis simulation capabilities as a tool in Onyx's MCP toolkit. Users could post bounties for custom robot simulations, physical system analysis, synthetic training data generation. Genesis providers could earn Bitcoin by running simulations. Knowledge graph enhancement. Use Genesis's physics engine to add physical properties to Onyx's knowledge graph. Enhance data validation with physics-based constraints. Generate physically accurate visualizations

[19:59] Speaker: details about potential integration and how it would work. I think I also want to connect it to this here, where I give more generic information about OpenAgents. First, read the readme on OpenAgents repo, then help with learn more. Boom, boom, boom. Write in Docs Genesis on the... Should I put it here? Docs. Yeah, I'll put it there on OpenAgents repo. This is amazing. Drone. Fly that pi. Oh my gosh. Hey, should we build a drone army? Drone defense. Look, who... Why has no one captured any of these fucking drones? They're flying around. Fuck that. Let's go capture some. Bounties? Who wants to contribute some bounties? Should we build a drone army? Like an anti-drone defensive situation? I think maybe we should do that. I think maybe we should do that. What do we got? Docs Genesis. Let's see. Let's just see. Let's just see. Genesis integration with OpenAgents slash Onyx. Add a section thinking through us building our own fleet of drones to use as defense against nefarious spy drones. And might people who want to defend against drone incursions want to contract our drone army? Genesis is a state-of-the-art physics simulation platform that could significantly enhance OpenAgents and Onyx's capabilities in physical world understanding, validation, and data generation. This document outlines potential integration points and opportunities. Key Genesis features relevant to OpenAgents. Ultra-fast physics engine, 100% Python implementation, 10 to 80 times faster than existing GPU. We already read that. Universal physics simulation, multiple physics solvers in a unified framework, blah, blah, blah. Read this. Generative framework upcoming. Okay. Integration opportunities. Physics-based data validation service. Integrate Genesis as a validation service in the Onyx data marketplace. Trajectory physics constraints, confidence threshold. Okay. Maybe validate drone flight trajectories, verify object interaction physics, authenticate motion. Okay. Maybe. MTP toolkit. Physics services. Talk to me about the drones. Talk to me about the drones. Fleet simulation and training. Defensive drone fleet system. Here we go. Using Genesis's advanced physics simulation and the Onyx marketplace, we could develop and operate the defensive drone fleet system to protect against unauthorized drone incursions. The system would combine physical drones with simulated training and real-time response optimization. Key components. Fleet simulation and training. Oh my gosh. Train interception. Train fleet responses using Genesis physics. Threat response system. Real-time physics validation of detected threats. Optimal fleet deployment calculation. Automated response pattern generation. Mission success probability estimation. Service model. Drone defense contract. Coverage, area coordinates, altitude range, time window, response requirements, max response time seconds, min success probability allowed, countermeasures, payments, base rate sats per day, success bonus sats, false alarm penalty sats. Oh my god. Oh no. Okay. This is amazing. Let's bring O1 Pro into the mix here. O1 Pro is really powerful. I don't have access to the API, or I guess I don't know if Pro is even accessible via the API, but I'm paying for the $200 a month thing, so I don't want to pay also for the API. So I wrote a script that bundles up a code base and just puts it in a single doc. So I'm going to run this script for two of these repos. Let's start with the code for Genesis. Did I clone this already? Yeah. Okay, so here's what I'm going to do first. If I do this right, I think yarn copy repo should dump the entire contents into a file and then copy that to my clipboard. Analyze what all is in this repo. Okay. Genesis Robotics Platform Overview. Okay, so I'm going to repeat that process for the OpenAgents repo, which right now just has basically a bunch of docs. Now think about how that can integrate with our OpenAgents platform and our Onyx mobile app. So there's the OpenAgents repo. Stuff's going to get real interesting real fast, but look, it has to get real interesting real fast because we've got a bunch of probably hostile fucking drones floating all over the US over our sensitive military bases. Who the fuck are these people or non-people? We don't know. Who the fuck are they? Grab one of them. Fucking reverse engineer it. Okay, so when that finishes its analysis, we will review that while it works on part two, which is seeing how to connect this to OpenAgents. Okay, great. So let's do this. I specifically want at least a few paragraphs of analysis on the defensive drone fleet network. Too long, you fucking guy. Fuck, fuck, fuck. Why? All right, let's first just try copying Onyx. Actually, you know what? I'm going to start with OpenAgents instead. Does that work? There we go. Okay, so analysis number one here. High level overview. This repository contains materials for Genesis, a universal and generative physics engine platform designed for robotics and embodied AI. Okay, top level read me. Yeah, yeah, yeah. So robotic asset files and metadata. Inside assets, there are multiple directories and files related to robotic models and other objects. Notable examples include URDF JSON files for bounding box, meta, mobility, result, panda bullet, probably example models. Other directories provide Mujoco XML descriptions of well-known robot arms. These directories include readme files detailing how the models were created from original robot descriptions, mesh conversions, and geometric modifications. Robotic SIM model conversion documentation. Each robot subdirectory contains a readme explaining the conversion process from URDF to Mujoco, including steps like mesh format, blah, blah, blah, blah, blah. Supporting mesh files, scene files, and configuration details required to integrate the robot models into simulation environments. Integration with external tools and library. JSON schema definitions for geometric primitives and transformations. Template files for various geometric and CAD-like formats. Units. Okay. I don't know how holistic this is or if it's like picking different parts. I just don't know how good that is. So the repository is a comprehensive blend of high-level documentation, robotic model assets, geometric schemas, and tooling infrastructure. Everything needed to support a powerful physics-based simulation ecosystem for robotics research and embodiment. Okay. Integrating a defensive drone fleet network into the OpenAgents platform and the Onyx mobile app. Control my drones. Offers a compelling use case for orchestrating real-time, secure, and autonomous operations. At its core, the OpenAgents ecosystem provides a decentralized marketplace and protocol layer where AI-driven services such as perception modules, situational analysis, path planning algorithms, and threat detection agents can be discovered, combined, and continuously improved. Onyx as a mobile interface serves as the user's command center, enabling field operators, security personnel, or even autonomous controllers to request and supervise advanced drone behaviors with minimal friction. I think we need to do this, guys. I think we need to do this. Within this framework, each drone in the defensive fleet can act as an AI agent that interacts with a diverse set of specialized service providers. For instance, consider that a user initiates a mission directive via the Onyx app, identifying and monitoring any unauthorized aerial activity over the perimeter. The drones, acting as clients, can query the OpenAgents marketplace to find the best available image recognition modules, sensor fusion algorithms, and flight pattern optimizers. Through protocols like MCP, they securely request capabilities such as real-time object classification or route planning that respect local geofencing regulations. NIP-89 could help the drones discover trustworthy services recommended by others in the network, ensuring they select only providers known for accurate threat detection or robust anti-spoofing filters. As the drones carry out their mission, NIP-90 data vending machines can come into play for more competitive tasks. Suppose the drones require an enhanced thermal imaging analysis from multiple providers. They post a public job request and various AI modules respond with their results and pricing structures. By integrating Lightning Network micropayments, Onyx instantly settles these services once the drones confirm which analysis is most useful. This pay-as-you-go model ensures that the drone fleet only compensates providers who deliver tangible mission-critical value and can dynamically adjust their provider relationships based on performance and cost-effectiveness. The benefits to a defensive drone fleet network are numerous. By leveraging the OpenAgents marketplace and Onyx's intuitive interface, the drones gain access to a constantly evolving and improving ecosystem of AI functionalities. There are no lengthy lock-ins with proprietary vendors. Instead, as the security environment changes, new types of aerial threats, updated regulatory conditions, or improved detection capabilities, the drone fleet can seamlessly adapt to discovering and incorporating better modules. The entire value chain remains open, decentralized, and user-driven, ensuring that the best technologies rise to the top and that the operator of the drone fleet retains ultimate authority over data, actions, and spending. In this vision, Onyx and OpenAgents become the backbone of the flexible, secure, and innovative defense drone infrastructure, one that can outpace and outmaneuver the challenges of modern security threats through open protocols, AI marketplaces, and instant micropayments. That's big stuff, guys. It's big because it's so epic. It synthesizes a bunch of different threads. It's clearly, like, a reasonable trajectory that we can go in, and it feels achievable. It feels achievable. No, I mean give me other analysis on topics other than defensive drone stuff. We just, like, have a long conversation about our new defensive drone network here. Do you guys want to orchestrate drone missions? Do you want to orchestrate drone missions? So, in other news, I have declared myself president of a new nation and claimed sovereignty over 45% of the Earth's surface, international waters, on behalf of the Republic of Atlantis. And, like, maybe it's just time we build up our fucking robot air force. By the authority vested in me by myself. Oh, no. We're gonna have some fun, folks. We gotta have fun. We gotta have

[39:59] Speaker: are handled granularly, the warehouse only pays for the exact value received, avoiding large upfront platform fees or inflexible contracts. This is awesome. And so, you should know that it's not like I'm deciding between what should we do as a company? Let me pick one of these things. We're going to do everything. We're going to do fucking everything. Personalized healthcare assistance. In a home healthcare setting, consider a suite of assistive robotic devices helping elderly or differently-abled individuals with daily tasks. By integrating Genesis simulations and Open Agents provided AI modules, these robots can adapt their motion and grip to safely assist patients, respond to changes in mobility, or learn from collective usage data on correct support techniques. The user's onyx app acts as an intuitive interface, voicing commands like, help me stand up and reach the medication on the top shelf. Onyx then seamlessly sources the right balance and stability control algorithm from the marketplace. Since payments flow through micropayments, service providers who develop better fault detection or gentle lift modules are instantly rewarded. The result is a constantly improving healthcare ecosystem that remains flexible and easily upgradable as user needs evolve. Yeah, my bet is that markets are going to outperform any single provider. And we have the technology to connect everything into one kind of mesh. Education and research platforms. Imagine a virtual robotics lab used in universities and research institutions. Students and researchers design experiments, like testing new gripper geometries, simulating different terrain conditions for wheeled robots, or training AI policies for bipedal locomotion. With Genesis at the core for high-fidelity physics, they can turn to Open Agents to find specialized simulation routines, advanced rendering effects, or data analysis tools. Onyx becomes a mobile gateway where a professor can set educational tasks, run a simulation of our new robot's gait under varying friction conditions, and pay small fees to whichever research groups or developers created these simulation add-ons. This open pay-as-you-go model encourages knowledge exchange, letting cutting-edge research tools and scenarios flow seamlessly into teaching environments and vice versa without bureaucracy or lock-in. Okay, I'm not going to read all of the rest of these, but smart agriculture and environmental monitoring, creative arts and entertainment, disaster response and humanitarian aid. From industrial logistics to creative productions, the combination of Genesis, Open Agents, and Onyx fosters an environment of continuous learning, flexible operations, and a user-centric marketplace. Instead of relying on rigid pre-selected vendors, organizations and individuals can fluidly discover and integrate precisely the functionalities they need, pay proportionately for the value delivered, and contribute to a truly open and evolving ecosystem. Folks, I intend to demonstrate that an open source company building on open protocols can become a trillion dollar company. See you soon.