README.md

August 3, 2026 · View on GitHub

SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition (ECCV 2024)

Jeonghyeok Do¹ Munchurl Kim^†1

^†Corresponding author

¹Korea Advanced Institute of Science and Technology, South Korea

This repository is the official PyTorch implementation of "SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition". SkateFormer achieves state-of-the-art performance in both skeleton-based action recognition and interaction recognition.

📧 News

Aug 3, 2026: Pretrained models are available on Hugging Face, loadable with from_pretrained :hugs:
Sep 26, 2024: Youtube video about SkateFormer is uploaded :sparkles:
Jul 1, 2024: SkateFormer accepted to ECCV 2024 :tada:
Jun 11, 2024: Codes of SkateFormer (including the training, testing code, and pretrained model) are released :fire:
Mar 19, 2024: This repository is created

Reference

@inproceedings{do2025skateformer,
  title={Skateformer: skeletal-temporal transformer for human action recognition},
  author={Do, Jeonghyeok and Kim, Munchurl},
  booktitle={European Conference on Computer Vision},
  pages={401--420},
  year={2025},
  organization={Springer}
}

Requirements
Data Preparation
Pretrained Model
Hugging Face
Training
Testing
Results
License
Acknowledgement

Requirements

Python >= 3.9.16

PyTorch >= 1.12.1

Platforms: Ubuntu 22.04, CUDA 11.6

We have included a dependency file for our experimental environment. To install all dependencies, create a new Anaconda virtual environment and execute the provided file. Run conda env create -f requirements.yaml.

Run pip install -e torchlight.

Data Preparation

Download datasets

There are 3 datasets to download:

NTU RGB+D
NTU RGB+D 120
NW-UCLA

NTU RGB+D and NTU RGB+D 120

Request dataset from here
Download the skeleton-only datasets:
1. nturgbd_skeletons_s001_to_s017.zip (NTU RGB+D)
2. nturgbd_skeletons_s018_to_s032.zip (NTU RGB+D 120)
3. Extract above files to ./data/nturgbd_raw

NW-UCLA

Download dataset from here
Move all_sqe to ./data/NW-UCLA

Data Processing

Directory Structure

Put downloaded data into the following directory structure:

- data/
  - NW-UCLA/
    - all_sqe
      ... # raw data of NW-UCLA
  - ntu/
  - ntu120/
  - nturgbd_raw/
    - nturgb+d_skeletons/     # from `nturgbd_skeletons_s001_to_s017.zip`
      ...
    - nturgb+d_skeletons120/  # from `nturgbd_skeletons_s018_to_s032.zip`
      ...

Generating Data

Generate NTU RGB+D or NTU RGB+D 120 dataset:

 cd ./data/ntu # or cd ./data/ntu120
 # Get skeleton of each performer
 python get_raw_skes_data.py
 # Remove the bad skeleton 
 python get_raw_denoised_data.py
 # Transform the skeleton to the center of the first frame
 python seq_transformation.py

Pretrained Model

Pre-trained model can be downloaded from here.

pretrained.zip: trained on NTU RGB+D, NTU RGB+D 120, NTU-Inter, NTU-Inter 120 and NW-UCLA.

All 14 pre-trained checkpoints are also hosted on the Hugging Face Hub at JeonghyeokDo/SkateFormer, one sub-directory per dataset / protocol / modality. This does not require downloading pretrained.zip.

pip install git+https://github.com/KAIST-VICLab/SkateFormer.git

import torch
from skateformer import SkateFormer

model = SkateFormer.from_pretrained(
    "JeonghyeokDo/SkateFormer", subfolder="ntu60-xsub-joint"
).eval()

x = torch.randn(1, 3, 64, 24, 2)      # [B, C, T, V, M], joint-partitioned
with torch.no_grad():
    logits = model(x)                 # -> [1, 60]

print(model.id2label[logits.argmax(-1).item()])

skateformer.preprocessing.prepare_input reproduces the evaluation-time path of feeders/ (modality conversion, 64-frame resampling, skeletal partitioning) for raw skeletons:

import numpy as np
from skateformer.preprocessing import prepare_input

raw = np.random.randn(3, 300, 25, 2)  # [C, T, V, M] raw NTU skeleton
x, index_t = prepare_input(raw, valid_frame_num=120, layout="ntu", modality="j")
with torch.no_grad():
    logits = model(x, index_t)

`subfolder`	Dataset	Protocol	Modality	Classes
`ntu60-xsub-joint` / `ntu60-xsub-bone`	NTU RGB+D 60	X-Sub	joint / bone	60
`ntu60-xview-joint` / `ntu60-xview-bone`	NTU RGB+D 60	X-View	joint / bone	60
`ntu120-xsub-joint` / `ntu120-xsub-bone`	NTU RGB+D 120	X-Sub	joint / bone	120
`ntu120-xset-joint` / `ntu120-xset-bone`	NTU RGB+D 120	X-Set	joint / bone	120
`ntu60-inter-xsub-joint`	NTU-Inter	X-Sub	joint	11
`ntu60-inter-xview-joint`	NTU-Inter	X-View	joint	11
`ntu120-inter-xsub-joint`	NTU-Inter 120	X-Sub	joint	26
`ntu120-inter-xset-joint`	NTU-Inter 120	X-Set	joint	26
`nwucla-joint` / `nwucla-bone`	NW-UCLA	official split	joint / bone	10

The skateformer/ package is a packaging layer only: skateformer/modeling_skateformer.py keeps the module and parameter names of model/SkateFormer.py unchanged, so the released .pt files load into it as-is and weights pulled from the Hub load back into the training code in this repository. main.py and everything under model/ are untouched.

Training

# Download code
git clone https://github.com/KAIST-VICLab/SkateFormer
cd SkateFormer

# Train SkateFormer on NTU RGB+D X-Sub60 dataset (joint modality)
python main.py --config ./config/train/ntu_cs/SkateFormer_j.yaml

# Train SkateFormer on NTU RGB+D X-Sub60 dataset (bone modality)
python main.py --config ./config/train/ntu_cs/SkateFormer_b.yaml

# Train SkateFormer on NTU-Inter X-View60 dataset (joint modality)
python main.py --config ./config/train/ntu_cv_inter/SkateFormer_j.yaml

# Train SkateFormer on NTU-Inter 120 X-Set120 dataset (joint modality)
python main.py --config ./config/train/ntu120_cset_inter/SkateFormer_j.yaml 

# Train SkateFormer on NW-UCLA dataset (joint modality)
python main.py --config ./config/train/nw_ucla/SkateFormer_j.yaml

Testing

# Test SkateFormer on NTU RGB+D X-View60 dataset (joint modality)
python main.py --config ./config/test/ntu_cv/SkateFormer_j.yaml

# Test SkateFormer on NTU RGB+D 120 X-Sub120 dataset (joint modality)
python main.py --config ./config/test/ntu120_csub/SkateFormer_j.yaml

# Test SkateFormer on NW-UCLA dataset (bone modality)
python main.py --config ./config/test/nw_ucla/SkateFormer_b.yaml