README.md

Abstract

In this work we propose Sieve & Swap technique, to automatically generate high quality pre-training data for the recipe domain: (i) Sieve: filters irrelevant transcripts and (ii) Swap: acquires high quality text by replacing transcripts with human-written instruction from a text-only recipe dataset. The resulting dataset is three orders of magnitude smaller than current web-scale datasets but enables efficient training of large-scale models. Alongside Sieve & Swap, we propose Procedure Transformer (ProcX), a model for end-to-end step localization and instruction generation for procedural videos. When pre-trained on our curated dataset, this model achieves state-of-the-art performance on YouCook2 and Tasty while using a fraction of the training data.

Dataset

Sieve & Swap : Our curated dataset along with processed features can be downloaded from :hugs: Hugging Face. More details are available in data.md

Raw Pre-Training : HowTo100M, RecipeNLG

Downstream Task : YouCook2, Tasty

Code

This repo contains a minimal PyTorch Lightning + Hydra codebase to finetune ProcX on YouCook2 (YC2), assuming the dataset described in data.md has already been downloaded. It does not include code to reproduce the Sieve & Swap pre-training stage.

Install

conda env create -f environment.yaml
conda activate procx
pip install -r requirements.txt   # if you prefer pip, this alone is enough
pip install -e .

ProcX uses two custom CUDA ops (key-aware deformable attention and 1D grid sampling). Build them from source (requires a CUDA-enabled PyTorch install and CUDA_HOME set):

cd src/cuda_ops/deformable_tx && python setup.py build install && cd -
cd src/cuda_ops/grid_sample1d && python setup.py build install && cd -

Language evaluation (METEOR, PTB tokenization) shells out to Java, so a JRE must be available (java -version), and two jars must be downloaded manually and placed as follows (not redistributed in this repo):

• src/evaluations/pycocoevalcap/meteor/meteor-1.5.jar and src/evaluations/pycocoevalcap/meteor/data/paraphrase-en.gz
• src/evaluations/pycocoevalcap/tokenizer/stanford-corenlp-3.4.1.jar

Data

Download the dataset per data.md and place it under data/ at the repo root (or point paths.root_dir/PROJECT_ROOT elsewhere), so that Hydra's default data_dir (${paths.root_dir}/data/) resolves to it, e.g.:

data/
├── downstream/yc2/data/captiondata/yc2_train.json
├── downstream/yc2/data/captiondata/yc2_val_official_available.json
├── downstream/yc2/features/*.npy
└── tokenizer/{tokenizer.json, tokenizer_config.json, special_tokens_map.json}

Set the PROJECT_ROOT environment variable to the repo root before running anything (used by configs/paths/default.yaml):

export PROJECT_ROOT=$(pwd)

Train / Finetune / Evaluate

Train ProcX from scratch on YC2:

python src/train.py experiment=yc2_deformable_key

Finetune a pretrained checkpoint on YC2, then evaluate the best checkpoint (also available as scripts/run_yc2_finetune.sh <checkpoint_path>):

python src/train.py experiment=yc2_deformable_key_finetune model=lite_procx_ft model.checkpoint_path=<path_to_pretrained.ckpt>
python src/eval.py experiment=yc2_deformable_key_eval_finetune ckpt_path=<path_to_finetuned.ckpt>

:page_facing_up: Citation

If you find this project useful in your research, please consider cite:

@inproceedings{batra2025efficient,
  title={Efficient Pre-training for Localized Instruction Generation of Procedural Videos},
  author={Batra, Anil and Moltisanti, Davide and Sevilla-Lara, Laura and Rohrbach, Marcus and Keller, Frank},
  booktitle={European Conference on Computer Vision},
  pages={347--363},
  year={2025},
  organization={Springer}
}

Licenses

This code is released under the MIT License. The licenses for datasets used in the paper are available at the following links: HowTo100M, YouCook2, and Tasty.

:dizzy: Acknowledgement

Thanks to the open source of the following projects:

PDVC, VidChapter, Lightning-Hydra-Template.