👀 Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware 👀

💻 Blog |📜 Paper | 🗂️ Data

Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware
🧑🏻‍🚀 Pietro Bonazzi ¹, Sizhen Bian ¹, Giovanni Lippolis ², Yawei Li¹, Sadique Sheik ², Michele Magno¹

¹ ETH Zurich, Switzerland
² SynSense AG, Switzerland

Quick Comparaison: Ground Truth (GT) vs. Prediction

In the following GIFs, Yellow represents the Ground Truth (GT), and Green represents the Prediction. These images are taken from the validation set.

✉️ Citation ❤️

Leave a star to support our open source initiative!⭐️

@InProceedings{Bonazzi_2024_CVPR,
    author    = {Bonazzi, Pietro and Bian, Sizhen and Lippolis, Giovanni and Li, Yawei and Sheik, Sadique and Magno, Michele},
    title     = {Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2024},
    pages     = {5684-5692}
}

🚀 TL;DR quickstart 🚀

Clone the repo

git clone https://github.com/pbonazzi/retina.git
cd retina

Create the environment

conda create -n retina python=3.8 -y
conda activate retina 
pip install -r requirements.txt 

Downloads

Datasets

Ini-30 Dataset

Click here to download the Ini-30 Dataset.

Verify the structure:

.
├── name
│   ├── annotations.csv
│   └── events.aedat4
├── ...
├── silver.csv

I have benchmarked different methods to aggregate event data and the np.add.at operators was the fastest. The dv.Accumulator in data/datasets/ini_30/ini_30_aeadat_processsor.py is currently the bottleneck causing the dataloader in the Ini-30 Dataset to be slow. You can speed it up by saving/loading the event windows.

3ET Dataset

Follow the instruction here to download the 3ET Dataset

Rename .env.example to .env and change its INI30_DATA_PATH and 3ET_DATA_PATH.

Models

Click here to download a pretrained model.

Training

See the list of arguments in the launch_fire function and the config/defaults.yaml.

python3 -m scripts.train --run_name="retina-ann" 

Deployment

Install dependencies for TFlite

pip install onnx2tf onnx-tf tensorflow onnx_graphsurgeon tf_keras

Quantiazation INT8 : Example for 3et_on_ini30

python3 -m onnxsim output/retina-ann-v6-evs-1000/models/model.onnx output/retina-ann-v6-evs-1000/models/model-simplified.onnx
onnx2tf -i output/retina-ann-v6-evs-1000/models/model.onnx -o output/retina-ann-v6-evs-1000/models/model_tf
python3 -m scripts.quantize