[ICLR'25] STAMP: Scalable Task And Model-agnostic Collaborative Perception

This repo hosts the official implementation of STAMP: an open heterogeneous multi-agent collaborative perception framework for autonomous driving.

Video

Before CFA	After CFA

Architecture

Our framework supports:

• Heterogeneous Modalities: Each agent can be equipped with sensors of different modalities.

  • LiDAR
  • Camera
  • LiDAR + Camera

• Heterogeneous Model Architectures and Parameters: Each agent can be equipped with different model architectures.

  • Encoder
    • PointPillars (LiDAR)
    • SECOND (LiDAR)
    • Pixor (LiDAR)
    • VoxelNet (LiDAR)
    • PointFormer (LiDAR)
    • Lift-Splat-Shoot [ResNet] (Camera)
    • Lift-Splat-Shoot [EfficientNet] (Camera)
  • Fusion model
    • Window Attention first proposed by V2X-ViT (ECCV 2022)
    • Pyramid Fusion first proposed by HEAL (ICLR 2024)
    • Fused Axial Attention first proposed by CoBevt (PMLR 2023)
    • Cross-Vehicle Aggregation first proposed by V2VNet (ECCV 2022)

• Heterogeneous Downstream Tasks: Each agent can be trained towards various downstream tasks (training objectives).

  • 3D Object Detection
  • BEV Segmentation

• Multiple Datasets:

  • OPV2V
  • OPV2V-H
  • V2V4Real

Future Work

We are committed to expanding our framework's capabilities. Future updates will include support for:

• Additional modalities
• New model architectures
• Diverse downstream tasks
• More datasets

Getting Started

Data Preparation

For data and environment preparation, please refer to the HEAL repository.

Training

To reproduce our results, use the following commands:

3D Object Detection on OPV2V Dataset

bash train_object_detection.sh

3D Object Detection on V2V4Real Dataset

bash train_v2v4real.sh

Task- and Model-Agnostic Setting on OPV2V Dataset

bash task_agnostic.sh

Checkpoints

We are in the process of preparing model checkpoints for release. Please stay tuned for updates.

Acknowledgements

This project builds upon the excellent work of HEAL. We extend our sincere gratitude to their team for their outstanding contributions to the field.

Contributing and Contact

For the purpose of double blind review, we will release the contact information later.

Contact

For any questions or concerns, please open an issue in this repository, and we'll be happy to assist you.