⚡️ Lightning Track ⚡️

🚀 Track with landmark 🚀              🚅 Track with synthesis 🚅

Description

Lightning Track is a monocular face tracker built on FLAME. It provides optimized FLAME parameters and camera parameters, along with the bounding box and landmarks used during optimization.

Our tracker operates at a remarkable speed 🚀, processing 250 frames in approximately 60 seconds under landmark mode and 250 frames in around 200 seconds under synthesis mode.

Installation

Build environment

conda create -n track python=3.9
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install pytorch3d -c pytorch3d
pip3 install mediapipe tqdm rich lmdb einops colored ninja av opencv-python scikit-image onnxruntime-gpu onnx transformers pykalman

conda env create -f environment.yml

If your environment has unknown or unsolvable issues, use the Dockerfile as a final solution.

Build source

Check the build_resources.sh.

Fast start

Track on video:

python track_video.py -v demos/demo.mp4 --synthesis

or track all videos in a directory:

python track_video.py -v demos/ --no_vis

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{
    chu2024gpavatar,
    title={{GPA}vatar: Generalizable and Precise Head Avatar from Image(s)},
    author={Xuangeng Chu and Yu Li and Ailing Zeng and Tianyu Yang and Lijian Lin and Yunfei Liu and Tatsuya Harada},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=hgehGq2bDv}
}