🚗 Talking Vehicles: Towards Natural Language Communication for Cooperative Autonomous Driving via Self-Play

# 1. [Required] System dependencies
sudo apt-get install mosquitto

# 2. [Required] Create & activate conda env
conda create -n talkingvehicles python=3.8
conda activate talkingvehicles

# 3. [Required] Core Python packages
pip install paho-mqtt==1.6.1
pip install pettingzoo gymnasium omegaconf shapely rich
pip install openai==1.12      # API changed after 0.28
pip install hydra-core --upgrade
pip install transformers==4.38 accelerate

# 4. [Required] PyTorch
# Install CUDA or CPU version as needed; e.g. torch==2.0.1 or 2.2.0 for transformers>=4.40+

# 5. [Optional] Coopernaut-specific
conda install ninja numba
pip install open3d
conda install -c anaconda openblas-devel
conda install typing-extensions
pip install tensorboard
# Then install CPU-only MinkowskiEngine

# 6. [Required] Install CARLA and Scenario Runner
# Follow the installation Section 1 and 2 [Both simulator and leaderboard]
# [https://leaderboard.carla.org/get_started/#11-download-the-carla-leaderboard-package]
#! Important: There is a typo in the Carla installation (egg file) above; please correct it.

# 7. [Optional] Install CARLA extra maps (for highway-merge & highway-exit)
# Download from https://github.com/carla-simulator/carla/releases
cd path/to/carla
# e.g. Import/AdditionalMaps_0.9.14.tar.gz
./ImportAssets.sh

# 8. [Required] Environment paths 
conda env config vars set CARLA_HOME=/path/to/carla
conda env config vars set LEADERBOARD=/path/to/leaderboard
conda env config vars set SCENARIO_RUNNER=/path/to/scenario_runner
conda env config vars set OPENAI_API_KEY=your_openai_key
conda env config vars set PYTHONPATH=/path/to/talking-vehicles/src

If not using ./run.sh as the entry point, then you should launch a CARLA Server first:

./envs/utils/launch_carla.sh 0 1 2000 # Launch one Carla instance on GPU:0 through port 2000

Example of evaluating the distilled agent:

Select scenario, mode, agent_config, policy_config, model, seed, and GPU:

# [percetion_overtake, perception_red_light, perception_left_turn,
# [neogotiation_overtake, negotiation_highway_merge, negotiation_highway_exit]
export SCENARIO=perception_red_light
export MODE=eval               # [train, eval, train-eval, eval-all]
export AGENT_CONFIG=comm       # [silent, comm]
export POLICY_CONFIG=distill   # [untrained, tip, rag, debrief, general, distill, coopernaut]
export MODEL=gpt               # [gpt, llama, reasoning_gpt]
export GPU=0
export SEED=12

python eval/eval.py \
   exp_config=${${SCENARIO}/${ENV_CONFIG}_${POLICY_CONFIG}_${MODEL}} \
   env_config=${${SCENARIO}_${AGENT_CONFIG}} \
   logdir=${results/${SCENARIO}/eval-${SEED}/${AGENT_CONFIG}_${POLICY_CONFIG}_${MODEL}/} \
   num_eval_episodes=30 \
   checkpoint=-1 \
   communication_mode=parallel \
   port=2000 \
   seed=12 \
   record_video=False

Example of running LLM+Debrief learning:

python train/train_centralized_llm.py \
   exp_config=${EXP_CONFIG} \
   env_config=${ENV_CONFIG} \
   logdir=${LOG_DIR} \
   num_train_iterations=${NUM_TRAIN_ITERATIONS} \
   communication_mode=${COMM_MODE}
   port=${PORT} \
   seed=${SEED} \
   record_video=${RECORD_VIDEO}

Example of running a human interaction (or other agent configurations):

python envs/multiagent_env.py \
   --timeout 100 \
   --reloadWorld \
   --scenarioConfig "envs/scenarios/multiagent_scenarios/maneuver_opposite_direction/overtake__human.yaml" \
   --record_video