awesome-python-benchmarks

Statistical benchmarking of python packages.

Machine-learning benchmarks

• Papers with Code contains many benchmarks in different categories. For instance the ImageNet classification lists papers and methods that have performed well, many of which are in Python.

Python time-series benchmarking

• MCompetitions repository lists winning methods for the M4 and M5 contests. For example the LightGBM approach document can be found there, alongside other winners.

• Time-Series Elo ratings considers methods for autonomous univariate prediction of relatively short sequences (400 lags) and ranks performance on predictions from 1 to 34 steps ahead.

• Papers with code has a couple of benchmarks such as etth1.

Python black-box derivative free benchmarking

• Coco is a platform for comparing continuous optimizers, as explained in the paper.

• BBOB workshop series features ten workshops, most recently the 2019 workshop on black box methods.

• Nevergrad benchmarking suite is discussed in this paper.

• Optimizer Elo ratings rates a hundred approaches to derivative free optimization on an ongoing basis, with methods taken from packages such as NLOPT, Nevergrad, BayesOpt, PySOT, Skopt, Bobyqa, Hebo, Optuna and many others.

R Time-series

• ForecastBenchmark automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains.