🏆 News

• We release the code of PhysX-Omni, PhysXVerse, and PhysX-Bench 🎉

I. PhysX-Omni

Installation

1. Clone the repo:

git clone --recurse-submodules https://github.com/physx-omni/PhysX-Omni.git
cd PhysX-Omni 

2. Create a new conda environment named physx-anything and install the dependencies:

. ./setup.sh --new-env --basic --xformers --flash-attn --diffoctreerast --spconv --mipgaussian --kaolin --nvdiffrast

Note: The detailed usage of setup.sh can be found at TRELLIS

3. Install the dependencies for Qwen2.5:

pip install transformers==4.50.0
pip install qwen-vl-utils
pip install 'accelerate>=0.26.0'

Note: We release the requirements.txt file. You can install all dependencies by running:

conda create -n physx-omni python=3.10
conda activate physx-omni
pip install -r requirements.txt

Training

1. Download PhysX datasets from PhysXNet, PhysX-Mobility, and PhysXVerse

2. Run the preprocessing script for PhysXVerse.

   cd dataset
   python 1voxel_verse.py
   python 2encode_representation_64_finetune
   python 3generate_data_new_64_finetune_rle.py
   

   Note: Here is a template for you to check the format: template.

   Note: Preprocess the PhysXNet and PhysX-Mobility follows PhysX-Anything

3. Render the conditioning images (25 images per object) based on your requirements.

   For PhysX-Mobility and PhysXVerse, we use dataset_toolkits/render_cond_mobility.py to generate the conditioning images.

   For PhysXNet, please check PhysX-3D/dataset_toolkits/precess.sh

4. Set the path in train configuration

   PHYSXNET = {
       "annotation_path": "xx", #json file path
       "data_path": "xx",  # conditioning image path
   }
   
   PHYSXMOBILITY = {
       "annotation_path": "xx", #json file path
       "data_path": "xx",  # conditioning image path
   }
   
   PHYSXVERSE = {
       "annotation_path": "xx", #json file path
       "data_path": "xx",  # conditioning image path
   }
   

5. Finetune the model

   cd qwen-vl-finetune
   sbatch scripts/train_physx.sh
   

Inference

1. Download the pre-train model from huggingface.

python download.py

2. Run the inference code

python 1vlm_demo.py            # vlm inference
    
python 2infer_geo.py           # decoder inference

python 3jsongen_update.py      # convert to URDF & XML

Note: We use the pretrained TRELLIS decoder in our framework. To obtain finer geometric details and higher-quality structures, it can be directly replaced with the TRELLIS.2 decoder.

II. PhysX-Bench

This repository includes the PhysX-Omni benchmark code under benchmark/.

See benchmark/README.md for the benchmark file structure, asset generation pipeline, VLM evaluation commands, denominator validation, and aggregation workflow.

For environment setup, see benchmark/INSTALL.md.

III. PhysXVerse

For more details about our proposed dataset including dataset structure and annotation, please see this PhysXVerse, PhysX-Mobility and PhysXNet.

IV. Other Tools

We provide convert_objects2scene.py, which converts individual objects into a simulation-ready scene. In addition, we build a simple scene generation pipeline in applications_scene based on existing works.

Acknowledgement

The data and code is based on PartNet-mobility, Qwen, TRELLIS, Depth-Anything, Grounded-Segment-Anything and CAST. We would like to express our sincere thanks to the contributors.

:newspaper_roll: License

Distributed under the S-Lab License. See LICENSE for more information.