Training and Evaluation Guide

July 18, 2024 · View on GitHub

set dataset parameters before training or evaluation

set dataset parameters in nuplan_extent/planning/script/config/common_cfg/gump.yaml To train on Nuplan dataset:

    dataset: 'nuplan'
    num_past_steps: 12
    past_time_horizon: 6.0 # training

to run simulation on Nuplan Dataset

    dataset: 'nuplan'
    num_past_steps: 2
    past_time_horizon: 1.0 # simulation

To train or eval on Waymo Dataset

    dataset: 'waymo'
    num_past_steps: 2
    past_time_horizon: 1.0

Version 1.0

Training

To train on the Nuplan Dataset, execute the following command:

sh scripts/training/train_nuplan_model.sh

To train on the Waymo Open Dataset, execute the following command:

sh scripts/training/train_wod_model.sh

Version 1.1

Training

To train on the Nuplan Dataset version 1.1, execute the following command:

sh scripts/training/train_nuplan_model_1_1.sh

To train on the Waymo Open Dataset version 1.1, execute the following command:

sh scripts/training/train_wod_model_1_1.sh

Evaluation

Waymo Sim Agents Benchmark

Sure, here's the refined format for fixing the bugs in the Waymo Open Dataset package:

Steps to Fix Bugs in Waymo Open Dataset Package

Install the Waymo Open Dataset Package Make sure you have installed the Waymo Open Dataset package by running:
```
pip install waymo-open-dataset-tf-2-12-0==1.6.4
```

Modify the File Edit the file located at /usr/local/lib/python3.9/dist-packages/waymo_open_dataset/wdl_limited/sim_agents_metrics/metrics.py. Change the code at line 47 from:

config_path = '{pyglib_resource}waymo_open_dataset/wdl_limited/sim_agents_metrics/challenge_2024_config.textproto'.format(pyglib_resource='')

to:

import os
# Get the resource path from an environment variable
pyglib_resource = os.getenv('PYGLIB_RESOURCE_PATH', '')

# Construct the full config path
config_path = f'{pyglib_resource}/waymo_open_dataset/wdl_limited/sim_agents_metrics/challenge_config.textproto'

Set Environment Variable Set the environment variable for the resource path by running:
```
export PYGLIB_RESOURCE_PATH=/usr/local/lib/python3.9/dist-packages
```

By following these steps, you will ensure that the configuration path is correctly set and the package functions properly.

Start Evaluation

To evaluate on the Waymo Sim Agents Benchmark, follow these steps:

Dump the simulated result offline by uncommenting the following lines in nuplan_extent/planning/script/experiments/training/training_wod_gump.yaml:
```
- override /aggregated_metric:
    - sim_agents_metrics 
```
and set your dump path in nuplan_extent/planning/script/config/training/aggregated_metric/sim_agents_metrics.yaml:
```
basepath: ${YOUR_PKL_DUMP_PATH}
```
Modify the configuration in nuplan_extent/planning/script/config/common_cfg/gump.yaml:
```
...
dataset: 'waymo'
num_past_steps: 2
past_time_horizon: 1.0
...
```
Ensure you set the downstream task to 'sim_agents':
```
...
downstream_task: 'sim_agents'
...
```
Running the testing script:
```
sh ./scripts/testing/test_wod_model.sh
```

Running the offline evaluation script with multiprocessing

python ./scripts/testing/offline_sim_agents.py --pkl_dir ${YOUR_PKL_DUMP_PATH} --nprocess 32 --output_dir ${SMOOTHING_OUTPUT_DIR}

Visualizing Sim Agents Results on Waymo Open Dataset

After dumping the result pickles, you can visualize the results by running:

python scripts/visualization/wod_simagents_visualization.py

Checkpoints

We provide several checkpoints for model reproduction. To use a checkpoint, download it and replace the checkpoint path in the bash script:

+checkpoint.ckpt_path=PATH_TO_CHECKPOINT \
+checkpoint.strict=True \

Checkpoint List

We provided the following CKPT:

Model	Version	Dataset	Config	Checkpoint
GPT2-base	v1.0	NuPlan	config	coming soon!
GPT2-Medium	v1.0	NuPlan	config	coming soon!
GPT2-Medium	v1.0	Waymo	config	Google Cloud
GPT2-base	v1.1	NuPlan	config	coming soon!
Llama3-Small	v1.1	NuPlan	config	Google Cloud

TensorBoard Visualization

The visualization is automatically handled by TensorBoard. To visualize your training progress, use the following command:

tensorboard --logdir='${SAVE_DIR}/${EXPERIMENT}'