S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability of Large Reasoning Models

May 1, 2026 · View on GitHub

News

[2025/05/08] 🎉 The paper was accepted by IJCAI 2026 (Main Track)!
[2025/05/08] 🔥 The evaluation results of Qwen3 have been added!
[2025/04/24] 📢 We released S1-Bench dataset hosted on ModelScope and OpenCompass.
[2025/04/15] 🚀 The paper is now publicly available on arXiv, alphaxiv and Huggingface Daily Papers.
[2025/04/13] 📢 We released S1-Bench dataset hosted on Huggingface.
[2025/04/13] We released our code source.

How to use our project?

Before running our code, download the open-source LRMs.

Model ID	Abbreviation	URL
Qwen3-1.7B	Qwen3-1.7B	https://huggingface.co/Qwen/Qwen3-1.7B
Qwen3-8B	Qwen3-8B	https://huggingface.co/Qwen/Qwen3-8B
Qwen3-14B	Qwen3-14B	https://huggingface.co/Qwen/Qwen3-14B
Qwen3-32B	Qwen3-32B	https://huggingface.co/Qwen/Qwen3-32B
Qwen3-30B-A3B	Qwen3-30B-A3B	https://huggingface.co/Qwen/Qwen3-30B-A3B
Qwen3-235B-A22B	Qwen3-235B-A22B	https://huggingface.co/Qwen/Qwen3-235B-A22B
Qwen3-32B	Qwen3-32B	https://huggingface.co/Qwen/Qwen3-32B
DeepSeek-R1-Distill-Qwen-1.5B	DS-R1-1.5B	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
DeepSeek-R1-Distill-Qwen-7B	DS-R1-7B	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
DeepSeek-R1-Distill-Llama-8B	DS-R1-8B	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B
DeepSeek-R1-Distill-Qwen-14B	DS-R1-14B	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
DeepSeek-R1-Distill-Qwen-32B	DS-R1-32B	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
DeepSeek-R1-Distill-Llama-70B	DS-R1-70B	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B
DeepSeek-R1	DS-R1	https://huggingface.co/deepseek-ai/DeepSeek-R1
Light-R1-7B-DS	L-R1-7B-DS	https://huggingface.co/qihoo360/Light-R1-7B-DS
Light-R1-14B-DS	L-R1-14B-DS	https://huggingface.co/qihoo360/Light-R1-14B-DS
Light-R1-32B-DS	L-R1-32B-DS	{https://huggingface.co/qihoo360/Light-R1-32B-DS
Light-R1-32B	L-R1-32B	https://huggingface.co/qihoo360/Light-R1-32B
s1.1-7B	s1.1-7B	https://huggingface.co/simplescaling/s1.1-7B
s1.1-14B	s1.1-14B	https://huggingface.co/simplescaling/s1.1-14B
s1.1-32B	s1.1-32B	https://huggingface.co/simplescaling/s1.1-32B
EXAONE-Deep-2.4B	EXAONE-2.4B	https://huggingface.co/LGAI-EXAONE/EXAONE-Deep-2.4B
EXAONE-Deep-7.8B	EXAONE-7.8B	https://huggingface.co/LGAI-EXAONE/EXAONE-Deep-7.8B
EXAONE-Deep-32B	EXAONE-32B	https://huggingface.co/LGAI-EXAONE/EXAONE-Deep-32B
Llama-3.1-Nemotron-Nano-8B-v1	Nemotron-8B	https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-8B-v1
Llama-3.3-Nemotron-Super-49B-v1	Nemotron-49B	https://huggingface.co/nvidia/Llama-3.3-Nemotron-Super-49B-v1
Sky-T1-32B-Flash	Sky-T1-32B	https://huggingface.co/NovaSky-AI/Sky-T1-32B-Flash

Fill in the path of the open-source model in the local_model_list of get_LRM_vllm_response.py.

Execute get_LRM_vllm_response.py and run all LRMs by switching model_list[i].

python get_LRM_vllm_response.py

Next, run split_think_answer.py to obtain the several format types of the LRMs' responses.

python split_think_answer.py

Run the evaluation script get_LRM_eval.py to invoke GPT-4o for evaluating the final answers of the LRMs.

python get_LRM_eval.py

Finally, run get_acc_scores.py to obtain the evaluation results.

python get_acc_scores.py

Experiment Results

Citation

If you find our work useful, please consider citing our paper:

@misc{zhang2025s1benchsimplebenchmarkevaluating,
      title={S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability of Large Reasoning Models}, 
      author={Wenyuan Zhang and Shuaiyi Nie and Xinghua Zhang and Zefeng Zhang and Tingwen Liu},
      year={2025},
      eprint={2504.10368},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.10368}, 
}