Early-stopping Self-Consistency [ICLR 2024]

This is the repo for ICLR 2024 paper: Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning.

CONTROL SCHEME

step 1. Obtain the sampling result of the first window

For any dataset D, Change the model_type in consistency_{D}.py to a specific model, and change sc_cum to the first sampling window size w. After running, you will get probs{w}.json under {model_type}_result/{D}/.

step 2. Obtain the expected sampling times and impact on performance under the specified sampling window size and maximum sampling times.

Set sc_num as the sampling window size and max_up as the maximum sampling times in control_scheme.py, and directly run it, you will get results like: