Hierarchical Neural Memory Network
August 4, 2023 · View on GitHub
This repo is a PyTorch implementation of HMNet proposed in our paper: Hierarchical Neural Network for Low Latency Event Processing.
Results and models
The pre-trained weights are released under the Creative Commons BY-SA 4.0 License.
DSEC-Semantic (Semantic Segmentation)
| Model | size | mIoU [%] | latency V100 [ms] | latency V100 x 3 [ms] | weights |
|---|---|---|---|---|---|
| HMNet-B1 | 640 x 440 | 51.2 | 7.0 | - | github |
| HMNet-L1 | 640 x 440 | 55.0 | 10.5 | - | github |
| HMNet-B3 | 640 x 440 | 53.9 | 9.7 | 8.0 | github |
| HMNet-L3 | 640 x 440 | 57.1 | 13.9 | 11.9 | github |
GEN1 (Object Detection)
| Model | size | mAP [%] | latency V100 [ms] | latency V100 x 3 [ms] | weights |
|---|---|---|---|---|---|
| HMNet-B1 | 304 x 240 | 45.5 | 4.6 | - | github |
| HMNet-L1 | 304 x 240 | 47.0 | 5.6 | - | github |
| HMNet-B3 | 304 x 240 | 45.2 | 7.0 | 5.9 | github |
| HMNet-L3 | 304 x 240 | 47.1 | 7.9 | 7.0 | github |
MVSEC day1 (Monocular Depth Estimation)
| Model | size | AbsRel | RMS | RMSElog | latency V100 [ms] | latency V100 x 3 [ms] | weights |
|---|---|---|---|---|---|---|---|
| HMNet-B1 | 346 x 260 | 0.385 | 9.088 | 0.509 | 2.4 | - | github |
| HMNet-L1 | 346 x 260 | 0.310 | 8.383 | 0.393 | 4.1 | - | github |
| HMNet-B3 | 346 x 260 | 0.270 | 7.101 | 0.332 | 5.0 | 4.1 | github |
| HMNet-L3 | 346 x 260 | 0.254 | 6.890 | 0.319 | 6.9 | 5.4 | github |
| HMNet-B3 w/ RGB | 346 x 260 | 0.252 | 6.972 | 0.318 | 5.4 | 4.1 | github |
| HMNet-L3 w/ RGB | 346 x 260 | 0.230 | 6.922 | 0.310 | 7.1 | 5.4 | github |
Requirements
- PyTorch >= 1.12.1
- torch_scatter
- timm
- hdf5plugin
Installation
Create a new conda environment
conda create -n hmnet python=3.7
conda activate hmnet
Install dependencies
pip install -r requirements.txt
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
pip install torch-scatter -f https://data.pyg.org/whl/torch-1.12.1+cu113.html
pip install timm
Experiments
Please see the instructions on each task page.
License
The majority of this project is licensed under BSD 3-clause License. However, some code (psee_evaluator.py, coco_eval.py, det_head_yolox.py) is available under the Apache 2.0 license. The pre-trained weights are released under the Creative Commons BY-SA 4.0 License.
Acknowledgments
This work is based on results obtained from a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO).