OpenGrpahAU

August 21, 2025 · View on GitHub

We released AU-Canvas, a visualization tool that offers an intuitive UI for facial action unit (FAU) detection and enhanced visualization.

Example running on a RTX 3090 GPU (Avg. FPS>50):

This repo is the OpenGprahAU tool.

demo:

Models were traiend on hybrid dataset of 2,000k images.

This hybrid dataset includes:

The tool can predict action units of 41 categories:

AU1	AU2	AU4	AU5	AU6	AU7	AU9	AU10	AU11	AU12	AU13	AU14	AU15	AU16
Inner brow raiser	Outer brow raiser	Brow lowerer	Upper lid raiser	Cheek raiser	Lid tightener	Nose wrinkler	Upper lip raiser	Nasolabial deepener	Lip corner puller	Sharp lip puller	Dimpler	Lip corner depressor	Lower lip depressor

AU17	AU18	AU19	AU20	AU22	AU23	AU24	AU25	AU26	AU27	AU32	AU38	AU39	-
Chin raiser	Lip pucker	Tongue show	Lip stretcher	Lip funneler	Lip tightener	Lip pressor	Lips part	Jaw drop	Mouth stretch	Lip bite	Nostril dilator	Nostril compressor	-

AUL1	AUL1	AUL2	AUR2	AUL4	AUR4	AUL6	AUR6	AUL10	AUR10	AUL12	AUR12	AUL14	AUR14
Left inner brow raiser	Right inner brow raiser	Left outer brow raiser	Right outer brow raiser	Left brow lowerer	Right brow lowerer	Left cheek raiser	Right cheek raiser	Left upper lip raiser	Right upper lip raiser	Left nasolabial deepener	Right nasolabial deepener	Left dimpler	Right dimpler

We provide tools for prepareing data in tool/. After Downloading raw data files, you can use these tools to process them, aligning with our protocals. We divide the dataset into three independent parts (i.e., train, val, test).

Pretrained models

Hybrid Dataset

Stage1:

arch_type	GoogleDrive link	Average F1-score	Average Acc.
`Ours (MobileNetV3)`	-	-	-
`Ours (ResNet-18)`	link	22.33	92.97
`Ours (ResNet-50)`	link	22.52	92.63
`Ours (Swin-Tiny)`	link	22.66	92.97
`Ours (Swin-Small)`	link	24.49	92.84
`Ours (Swin-Base)`	link	23.53	92.91

Stage2:

arch_type	GoogleDrive link	Average F1-score	Average Acc.
`Ours (MobileNetV3)`	-	-	-
`Ours (ResNet-18)`	link	22.51	93.23
`Ours (ResNet-50)`	link	23.24	93.31
`Ours (Swin-Tiny)`	link	22.74	93.37
`Ours (Swin-Small)`	-	-	-
`Ours (Swin-Base)`	-	-	-

Demo

to detect facial action units in a facial image using our stage1 model, run:

python demo.py --arc resnet50 --stage 1 --exp-name demo --resume checkpoints/OpenGprahAU-ResNet50_first_stage.pth --input demo_imgs/1014.jpg  --draw_text

to detect facial action units in a facial image using our stage2 model, run:

python demo.py --arc resnet50 --stage 2 --exp-name demo --resume checkpoints/OpenGprahAU-ResNet50_second_stage.pth --input demo_imgs/1014.jpg  --draw_text

Training and Testing

to train the first stage of our approach (ResNet-50) on hybrid Dataset, run:

python train_stage1.py --arc resnet50 --exp-name OpenGprahAU-ResNet50_first_stage -b 512 -lr 0.00002

to test the first stage of our approach (SwinT) on hybrid Dataset, run:

python test_stage1.py --arc swin_transformer_tiny --exp-name test_OpenGprahAU-SwinT_first_stage  --resume ./results/OpenGprahAU-SwinT_first_stage/bs_64_seed_0_lr_2e-05/best_model.pth

to train the second stage of our approach (ResNet-50) on hybrid Dataset, run:

python train_stage2.py --arc resnet50 --exp-name OpenGprahAU-ResNet50_second_stage -b 512 -lr 0.00001  --resume checkpoints/OpenGprahAU-ResNet50_first_stage.pth

to test the second stage of our approach (SwinT) on hybrid Dataset, run:

python test_stage2.py --arc swin_transformer_tiny --exp-name test_OpenGprahAU-SwinT_second_stage  --resume ./results/OpenGprahAU-SwinT_second_stage/bs_64_seed_0_lr_1e-05/best_model.pth

🖊️ Citation

If you find this work useful in your research, please cite:

@inproceedings{luo2022learning,
  title     = {Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition},
  author    = {Luo, Cheng and Song, Siyang and Xie, Weicheng and Shen, Linlin and Gunes, Hatice},
  booktitle = {Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, {IJCAI-22}},
  pages     = {1239--1246},
  year      = {2022}
}