---

📖 Overview

Real-world videos often show routine activities punctuated by memorable, surprising events. However, most Video-LLMs process videos by sampling frames uniformly, likely missing critical moments that define a video's narrative.

We introduce SPIKE, an inference-time framework that quantifies Bayesian Surprise as the belief update triggered by new visual evidence in the video stream, identifying moments where new visual evidence conflicts with prior beliefs.

• 🔍 SPIKE effectively localizes surprise in videos, correlated with humans on positive (FunQA) and negative (Oops!) surprise benchmarks.
• 🤖 SPIKE-RL further improves on SPIKE's ability to detect surprise, leveraging GRPO to refine its belief hypotheses based on a reward signal from the video caption.
• 🎯 Both methods guide query-agnostic surprise-weighted frame sampling, allocating more frames to interesting moments — achieving consistent performance gains on five downstream benchmarks.

---

Installation

1. Set up & Activate a Virtual Environment

virtualenv spike --python=python3.10.13
source spike/bin/activate

2. Install Core Dependencies

cd ..
pip3 install -e ".[dev]"
pip3 install flash_attn --no-build-isolation

3. Install qwen-vl-utils

cd qwen-vl-utils
pip install -e .

4. Install Custom Transformers & Remaining Dependencies

Download the custom transformers from here and run:

unzip transformers-main.zip
cd ./transformers-main
pip install .

pip uninstall transformers -y
pip install trl==0.16.0
pip install torchvision==0.21.0
pip install numpy==1.22.4
pip install sentence_transformers decord peft opencv_python wandb

---

📦 Data & Checkpoints

The dataset and model checkpoints will be made available on HuggingFace soon. Stay tuned!

---

📄 Citation

If you find this work useful, please consider citing us:

@inproceedings{
ravi2026spikerl,
title={{SPIKE}-{RL}: Video-{LLM}s meet Bayesian Surprise},
author={Sahithya Ravi and Aditya Chinchure and Raymond T. Ng and Leonid Sigal and Vered Shwartz},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=QLiXtWEAkq}
}