Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions

December 10, 2025 · View on GitHub

2025/12/10: This work has been published in IEEE Transactions on Intelligent Transportation System. If you find it helpful, please consider citing our work.

Before you start, make sure to have ollama installed and a working environment. If you don’t have ollama installed, check here Ollama. After installing Ollama, run the following command to install llama3:

ollama run llama3

You can choose any other LLM model provided by Ollama—just replace "llama3" with the model you want to use and change llm_agent.py line10 accordingly. Check here for more details. (better with python3.10)

2. Install the dependent package

pip install -r requirements.txt

3. Start training and testing

python train.py
python test.py
python test_with_listener.py

First, run train.py to generate an interaction experience database (you may need to activate line24&25 in memory.py if you do not have a relative embedding model). After that, you can use test.py or test_with_listener.py for testing. The simulation will run at a frequency of 10Hz. In test_with_listener.py, if your device has a microphone, you can act as an HDV and provide human instructions to the CAV. Have fun 😊

Overview 🔍

Repo description 📄

In this repository, you can expect to find the following features: Included:

Code for Actor-Reasoner (including roundabout, merge, intersection)
Video and Data of our experiment

Not included:

Code for Comparison Algorithm

Files description 📂

Main Functions:

train.py: Runs only the Reasoner to train the Actor’s interaction experience database. This must be executed first when running the project for the first time.
test.py: Uses both the Reasoner and Actor. Since most users don’t have access to real vehicles or driving simulators, this function simulates actual operations in a virtual environment with an update frequency of 10Hz.
test_with_listener.py: An upgraded version of test.py. If your device has a microphone, this function allows you to provide human instructions via voice input to observe the CAV’s behavior.

Vehicle Behavior Generation:

llm_agent.py: Takes scene and conflict information as input and generates semantic actions.
idm_controller.py: Converts semantic-level actions generated by the LLM into acceleration.
bayesian_game_agent.py: Decision model for heterogeneous HDVs.
vehicle.py: Defines the vehicle class.

Environment & Tools:

params.py: Contains hyperparameter settings for vehicles and the environment.
tools.py: Includes utility functions such as conflict detection and risk assessment.
prompt_llm.py: Handles prompt design for LLM interactions.
listener.py: Listens to real-time microphone input.
memory.py: Provides tools for managing the interaction experience database.

If you have any questions, feel free to contact Shiyu (fangshiyu@tongji.edu.cn) 📧.

Citation

Our paper has been pre-printed! If you find our work helpful, please consider citing us using the following reference 😊:

@misc{fang2025interactinstructimprovellmdriven,
      title={Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions}, 
      author={Shiyu Fang and Jiaqi Liu and Chengkai Xu and Chen Lv and Peng Hang and Jian Sun},
      year={2025},
      eprint={2503.00502},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2503.00502}, 
}