VerseBench: Benchmark code for Universe-1

Introduction

The benchmark evaluation code for Universe-1.

Benchmark Datasets Download

hf download dorni/Verse-Bench --local-dir ./verse_bench

Model Download

Models	🤗 Hugging Face
VerseBenchModels	VerseBenchModels

download the pretrained evaluation models into ./models

Model Usage

🔧 Dependencies and Installation

pip install -r requirements.txt

We tested the code on Python 3.10, PyTorch 2.6.0, and CUDA 12.4 on Ubuntu 22.04 LTS.

🚀 Inference Scripts

export MODELS_PATH=models
python calculate_metrics.py

Optional arguments:

  --input_dir # Your data for evaluation, the file names must match the names in our Verse-Bench datasets. Each test case must contains both .mp4 and .wav.
                        #(default: ./mini_testset for quick test.)
  --models_path # Path to the models. (default: ./models)