Installation

May 26, 2025 ยท View on GitHub

Environment

We use conda to manage the environment. The installation is tested with CUDA 11.8 and NVIDIA A100 (80GB).

conda create -n adaworld python=3.10 -y
conda activate adaworld
pip install -r requirements.txt

Dataset

In this instruction, we provide our automated generation process on Procgen and Gym Retro as examples.

For Gym Retro, we recommend installing an independent conda environment for automated data generation following this guide. After installation, please manually import ROMs for interaction. To obtain all game environments listed in the paper, you may need to search for the missing ROMs by name on the web and import them individually. We also have 12 extra games imported from Stable-Retro to further enrich our data corpus.

Run the following scripts to sample videos using a biased random agent (may take a while). You can customize the number and length of the videos by modifying num_logs and timeout.

mkdir data
python sample_procgen.py
python sample_retro.py
python sample_stableretro.py

The organized data directory should look like:

data/
|--procgen/
|   |--bigfish/
|   |   |--test/
|   |   |   |--00000.mp4
|   |   |   |...
|   |   |--train/
|   |       |--00000.mp4
|   |       |...
|   |--bossfight/
|   |   |--test/
|   |   |   |--00000.mp4
|   |   |   |...
|   |   |--train/
|   |       |--00000.mp4
|   |       |...
|   |...
|--retro/
    |--3NinjasKickBack-Genesis/
    |   |--test/
    |   |   |--00000.mp4
    |   |   |...
    |   |--train/
    |       |--00000.mp4
    |       |...
    |--8Eyes-Nes/
    |   |--test/
    |   |   |--00000.mp4
    |   |   |...
    |   |--train/
    |       |--00000.mp4
    |       |...
    |...

You can also download and compile other datasets we used:

  • Open X-Embodiment
    • We extract as many video sequences as possible regardless of their viewpoint.
    • Please find download_open_x.sh and process_rtx.py to see how we extract raw videos.
  • Ego4D
    • Follow the instruction to obtain the license.
    • We use the full_scale set. You may resize the videos after downloading to save storage and speed up reading.
  • Something-Something V2
  • MiraData
    • We only use the first 8K videos (3D rendered games and city walking tours) in the list.
    • We will progressively upload the video splits we used to Hugging Face.
    • You can download more if you want and have sufficient storage.
    • A modified script can be found at download_miradata_360p.py.

or those not used (but we have tried):

as folders of .mp4/.webm files.

=> Next: [Training]