Overview
October 14, 2025 · View on GitHub
TorchCP: A Python toolbox for Conformal Prediction in Deep Learning.
Technical Report
·
Documentation
TorchCP is a Python toolbox for conformal prediction research on deep learning models, built on the PyTorch Library with
strong GPU acceleration. In the toolbox, we implement representative methods (including posthoc and training methods)
for many tasks of conformal prediction, including: Classification, Regression, Graph Neural Networks, and LLM. We
for many tasks of conformal prediction, including: Classification, Regression, Graph Neural Networks, and LLM. We
build the basic framework of TorchCP based on AdverTorch. This
codebase is still under construction and maintained by Hongxin Wei's research group
at SUSTech. Comments, issues, contributions, and collaborations are all welcomed!
Updates of New Version (1.2.x)
This release enhances functionality by introducing p-value computation, conformal predictive distributions, and expanding the NORABS score function with additional difficulty estimation methods. Detailed changelog can be found in the Documentation.
Overview
TorchCP has implemented the following methods:
Classification
Regression
Graph
| Year | Title | Venue | Code Link | Implementation |
|---|---|---|---|---|
| 2024 | Similarity-Navigated Conformal Prediction for Graph Neural Networks | NeuIPS'24 | Link | graph.score.snaps |
| 2023 | Distribution Free Prediction Sets for Node Classification | ICML'23 | Link | graph.predictor.naps |
| 2023 | Conformal Prediction Sets for Graph Neural Networks | ICML'23 | Link | graph.score.daps |
| 2023 | Uncertainty Quantification over Graph with Conformalized Graph Neural Networks | NeurIPS'23 | Link | graph.trainer.cfgnn |
Language Models
| Year | Title | Venue | Code Link | Implementation |
|---|---|---|---|---|
| 2023 | Conformal Language Modeling | ICLR'24 | Link | llm.predictor.conformal_llm |
TODO
TorchCP is still under active development. We will add the following features/items down the road:
| Year | Title | Venue | Code |
|---|---|---|---|
| 2023 | Conformal Prediction for Time Series with Modern Hopfield Networks | NeuIPS'23 | Link |
| 2022 | Conformal Prediction Sets with Limited False Positives | ICML'22 | Link |
Installation
TorchCP is developed with Python 3.10 and fully compatible with the latest versions of PyTorch. Users should install PyTorch before proceeding with the TorchCP installation (please refer to the official PyTorch installation guide). Once PyTorch is set up, you can install TorchCP with the command
pip install torchcp
Unit Test
TorchCP achieves 100% unit test coverage. You can use the following command to test the code implementation:
pytest --cov=torchcp tests
Examples
Here, we provide a simple example for a classification task, with LAC score and SplitPredictor.
from torchcp.classification.score import LAC
from torchcp.classification.predictor import SplitPredictor
# Preparing a calibration data and a test data.
cal_dataloader = ...
test_dataloader = ...
# Preparing a pytorch model
model = ...
model.eval()
# Options of score function: LAC, APS, SAPS, RAPS
# Define a conformal prediction algorithm. Optional: SplitPredictor, ClusteredPredictor, ClassConditionalPredictor
# We recommend setting both alpha and device during initialization
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
predictor = SplitPredictor(score_function=LAC(), model=model, alpha=0.1, device = device)
# Calibrating the predictor
# You can also call `calibrate()` again to update the alpha value if needed
#predictor.calibrate(cal_dataloader, alpha=0.1)
predictor.calibrate(cal_dataloader)
#########################################
# Predicting for test instances
########################################
test_instances = ...
predict_sets = predictor.predict(test_instances)
print(predict_sets)
#########################################
# Evaluating the coverage rate and average set size on a given dataset.
########################################
result_dict = predictor.evaluate(test_dataloader)
print(f"Coverage Rate: {result_dict['coverage_rate']:.4f}")
print(f"Average Set Size: {result_dict['average_size']:.4f}")
You may find more tutorials in examples folder.
License
This project is licensed under the LGPL. The terms and conditions can be found in the LICENSE and LICENSE.GPL files.
Citation
If you find our repository useful for your research, please consider citing the following technical report:
@misc{huang2024torchcp,
title={TorchCP: A Python Library for Conformal Prediction},
author={Jianguo Huang and Jianqing Song and Xuanning Zhou and Bingyi Jing and Hongxin Wei},
year={2024},
eprint={2402.12683},
archivePrefix={arXiv},
primaryClass={cs.LG},
}
We welcome you to cite the following works:
@inproceedings{huangconformal,
title={Conformal Prediction for Deep Classifier via Label Ranking},
author={Huang, Jianguo and Xi, HuaJun and Zhang, Linjun and Yao, Huaxiu and Qiu, Yue and Wei, Hongxin},
booktitle={Forty-first International Conference on Machine Learning}
}
@article{xi2024does,
title={Does Confidence Calibration Help Conformal Prediction?},
author={Xi, Huajun and Huang, Jianguo and Feng, Lei and Wei, Hongxin},
journal={TMLR},
year={2024}
}
@inproceedings{
liu2025cadapter,
title={C-Adapter: Adapting Deep Classifiers for Efficient Conformal Prediction Sets},
author={Kangdao Liu and Hao Zeng and Jianguo Huang and Huiping Zhuang and Chi Man VONG and Hongxin Wei},
booktitle={The 28th European Conference on Artificial Intelligence},
year={2025},
}