Awesome Low-Precision Training

May 10, 2025 · View on GitHub

A collection of research papers on low-precision training for foundation models, organized by numerical representation formats. The collection also includes quantization-aware training techniques for large language models.

*🤖 indicates the corresponding paper contains experiments using LLM.

If you find this repository helpful, please consider citing our survey paper:

@article{hao2025low,
  title={Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities},
  author={Hao, Zhiwei and Guo, Jianyuan and Shen, Li and Luo, Yong and Hu, Han and Wang, Guoxia and Yu, Dianhai and Wen, Yonggang and Tao, Dacheng},
  journal={arXiv preprint arXiv:2505.01043},
  year={2025}
}

Fixed-Point and Integer-Based Methods
Floating-Point-Based Methods
Customized Format-Based Methods
Quantization-Aware Training Methods

Fixed-Point and Integer-Based Methods

Fixed-Point

[2024 | arXiv] Trainable Fixed-Point Quantization for Deep Learning Acceleration on FPGAs [📄 paper]
[2021 | arXiv] A Simple and Efficient Stochastic Rounding Method for Training Neural Networks in Low Precision [📄 paper]
[2020 | ICML] Multi-Precision Policy Enforced Training (MuPPET): A precision-switching strategy for quantised fixed-point training of CNNs [📄 paper]
[2019 | ICML] SWALP: Stochastic Weight Averaging in Low-Precision Training [📄 paper] [💻 code]
[2019 | NeurIPS] Backprop with Approximate Activations for Memory-efficient Network Training [📄 paper] [💻 code]
[2019 | ICLR] Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm [📄 paper]
[2018 | ICLR] Mixed Precision Training of Convolutional Neural Networks Using Integer Operations [📄 paper]
[2018 | ICCD] Training Neural Networks with Low Precision Dynamic Fixed-Point [📄 paper]
[2017 | IJCNN] FxpNet: Training a Deep Convolutional Neural Network in Fixed-Point Representation [📄 paper]
[2015 | ICML] Deep Learning with Limited Numerical Precision [📄 paper]
[2015 | ICLR workshop] Training Deep Neural Networks With Low Precision Multiplications [📄 paper]

Integer

General methods

[2025 | TPAMI | 🤖] Latent Weight Quantization for Integerized Training of Deep Neural Networks [📄 paper]
[2025 | FCS] Efficient Deep Neural Network Training via Decreasing Precision With Layer Capacity [📄 paper]
[2025 | arXiv | 🤖] Accurate INT8 Training Through Dynamic Block-Level Fallback [📄 paper]
[2024 | ICML | 🤖] Jetfire: Efficient and Accurate Transformer Pretraining With INT8 Data Flow and Per-Block Quantization [📄 paper]
[2024 | ICML] AMPA: Adaptive Mixed Precision Allocation for Low-Bit Integer Training [📄 paper]
[2024 | ICASSP] Activation Compression of Graph Neural Networks using Block-wise Quantization with Improved Variance Minimization [📄 paper] [💻 code]
[2024 | arXiv | 🤖] Direct Quantized Training of Language Models with Stochastic Rounding [📄 paper] [💻 code]
[2024 | arXiv] Towards Accurate and Efficient Sub-8-Bit Integer Training [📄 paper]
[2024 | arXiv] HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantization [📄 paper]
[2024 | arXiv | 🤖] Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients [📄 paper] [💻 code]
[2023 | NeurIPS] Stable and Low-Precision Training for Large-Scale Vision-Language Models [📄 paper]
[2023 | NeurIPS] Training Transformers With 4-Bit Integers [📄 paper] [💻 code]
[2023 | ICML] Few-bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction [📄 paper] [💻 code]
[2023 | CoLLAs] Hadamard Domain Training with Integers for Class Incremental Quantized Learning [📄 paper] [💻 code]
[2022 | ICML] GACT: Activation Compressed Training for Generic Network Architectures [📄 paper] [💻 code]
[2022 | Neurocomputing] Towards Efficient Full 8-bit Integer DNN Online Training on Resource-limited Devices without Batch Normalization [📄 paper]
[2022 | TPDS] NITI: Training Integer Neural Networks Using Integer-only Arithmetic [📄 paper] [💻 code]
[2021 | ICML] ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training [📄 paper] [💻 code]
[2021 | ICLR] CPT: Efficient Deep Neural Network Training via Cyclic Precision [📄 paper] [💻 code]
[2021 | ICLR] EXACT: Scalable Graph Neural Networks Training via Extreme Activation Compression [📄 paper] [💻 code]
[2021 | AAAI] Distribution Adaptive INT8 Quantization for Training CNNs [📄 paper]
[2021 | Neurocomputing] Training and Inference for Integer-Based Semantic Segmentation Network [📄 paper] [💻 code]
[2021 | CVPR workshop] In-Hindsight Quantization Range Estimation for Quantized Training [📄 paper]
[2020 | NeurIPS] FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training [📄 paper] [💻 code]
[2020 | NeurIPS] A Statistical Framework for Low-bitwidth Training of Deep Neural Networks [📄 paper] [💻 code]
[2020 | CVPR] Towards Unified INT8 Training for Convolutional Neural Network [📄 paper]
[2020 | CVPR] Fixed-Point Back-Propagation Training [📄 paper]
[2020 | Neural Networks] Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers [📄 paper] [💻 code]
[2019 | NeurIPS] Dimension-Free Bounds for Low-Precision Training [📄 paper]
[2018 | NeurIPS] Scalable Methods for 8-bit Training of Neural Networks [📄 paper] [💻 code]
[2018 | ICLR] Training and Inference with Integers in Deep Neural Networks [📄 paper] [💻 code]
[2018 | ECCV] Value-aware Quantization for Training and Inference of Neural Networks [📄 paper]
[2018 | arXiv] Training Deep Neural Network in Limited Precision [📄 paper]
[2018 | arXiv] High-Accuracy Low-Precision Training [📄 paper]
[2017 | ICML] The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning [📄 paper] [💻 code]
[2017 | NeurIPS] Training Quantized Nets: A Deeper Understanding [📄 paper]
[2016 | arXiv] DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients [📄 paper] [💻 code]
[2016 | arXiv] Convolutional Neural Networks using Logarithmic Data Representation [📄 paper]

Optimizer-state-targeted methods

[2024 | NeurIPS | 🤖] MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence [📄 paper] [💻 code]
[2024 | NeurIPS] 4-bit Shampoo for Memory-Efficient Network Training [📄 paper] [💻 code]
[2024 | EMNLP | 🤖] Exploring Quantization for Efficient Pre-Training of Transformer Language Models [📄 paper]
[2024 | arXiv | 🤖] Memory-Efficient 4-bit Preconditioned Stochastic Optimization [📄 paper]
[2023 | NeurIPS | 🤖] Memory Efficient Optimizers with 4-bit States [📄 paper] [💻 code]
[2023 | arXiv | 🤖] QFT: Quantized Full-Parameter Tuning of Llms With Affordable Resources [📄 paper]
[2022 | ICLR | 🤖] 8-bit Optimizers via Block-wise Quantization [📄 paper]

Communication-targeted methods

[2025 | TPAMI | 🤖] LoCo: Low-Bit Communication Adaptor for Large-Scale Model Training [📄 paper] [💻 code]
[2024 | NeurIPS | 🤖] SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training [📄 paper] [💻 code]
[2024 | NSDI | 🤖] THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression [📄 paper] [💻 code]
[2023 | ICML | 🤖] Quantized Distributed Training of Large Models with Convergence Guarantees [📄 paper]
[2023 | arXiv | 🤖] ZeRO++: Extremely Efficient Collective Communication for Giant Model Training [📄 paper]
[2022 | NeurIPS | 🤖] Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees [📄 paper]
[2020 | NeurIPS] Adaptive Gradient Quantization for Data-Parallel SGD [📄 paper] [💻 code]
[2019 | ICML] DOUBLESQUEEZE: Parallel Stochastic Gradient Descent with Double-pass Error-Compensated Compression [📄 paper]
[2019 | NeurIPS] Double Quantization for Communication-Efficient Distributed Optimization [📄 paper]
[2017 | NeurIPS] QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding [📄 paper]
[2015 | arXiv] 8-Bit Approximations for Parallelism in Deep Learning [📄 paper]

Binary

[2024 | arXiv] 1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit [📄 paper] [💻 code]
[2023 | ICLR | 🤖] Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam [📄 paper]
[2023 | NeurIPS] Birder: Communication-Efficient 1-bit Adaptive Optimizer for Practical Distributed DNN Training [📄 paper]
[2023 | TECS] Enabling Binary Neural Network Training on the Edge [📄 paper] [💻 code]
[2022 | HiPC] 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed [📄 paper]
[2022 | DAC] Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression [📄 paper]
[2021 | ICML] 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed [📄 paper]
[2019 | ICML] Error Feedback Fixes SignSGD and other Gradient Compression Schemes [📄 paper] [💻 code]
[2018 | ICML] Quantized Neural Networks: Training Neural Networks With Low Precision Weights and Activations [📄 paper]
[2016 | ECCV] XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks [📄 paper] [💻 code]
[2015 | NeurIPS] Binaryconnect: Training Deep Neural Networks With Binary Weights During Propagations [📄 paper]
[2014 | Interspeech] 1-Bit Stochastic Gradient Descent and its Application to Data-Parallel Distributed Training of Speech DNNs [📄 paper]

Floating-Point-Based Methods

[2025 | ICLR | 🤖] COAT: Compressing Optimizer States and Activation for memory efficient FP8 Training [📄 paper] [💻 code]
[2025 | arXiv | 🤖] Optimizing Large Language Model Training Using FP4 Quantization [📄 paper]
[2025 | arXiv | 🤖] Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models [📄 paper]
[2024 | arXiv | 🤖] DeepSeek-V3 Technical Report [📄 paper]
[2024 | arXiv | 🤖] Scaling FP8 Training to Trillion-Token LLMs [📄 paper]
[2024 | ICML workshop | 🤖] Scalify: Scale Propagation for Efficient Low-Precision LLM Training [📄 paper] [💻 code]
[2024 | DATE] A Stochastic Rounding-Enabled Low-Precision Floating-Point MAC for DNN Training [📄 paper]
[2023 | NeurIPS workshop | 🤖] Training and Inference of Large Language Models Using 8-Bit Floating Point [📄 paper]
[2023 | arXiv | 🤖] FP8-LM: Training FP8 large language models [📄 paper]
[2022 | ACT] Campo: Cost-Aware Performance Optimization for Mixed-Precision Neural Network Training [📄 paper]
[2022 | arXiv] 8-Bit Numerical Formats for Deep Neural Networks [📄 paper] [💻 code]
[2022 | arXiv | 🤖] FP8 Formats for Deep Learning [📄 paper]
[2022 | arXiv] Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training [📄 paper]
[2021 | ICLR] Neural Gradients Are Near-Lognormal: Improved Quantized and Sparse Training [📄 paper]
[2020 | IJCAI] Reducing Underflow in Mixed Precision Training by Gradient Scaling [📄 paper]
[2020 | ICLR] Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks [📄 paper]
[2020 | NeurIPS] Ultra-Low Precision 4-bit Training of Deep Neural Networks [📄 paper]
[2019 | ICLR] Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks [📄 paper]
[2019 | NeurIPS] Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks [📄 paper]
[2019 | arXiv] Mixed Precision Training With 8-bit Floating Point [📄 paper]
[2019 | arXiv] A Study of BFLOAT16 for Deep Learning Training [📄 paper]
[2018 | ICLR] Mixed Precision Training [📄 paper]
[2018 | NeurIPS] Training Deep Neural Networks with 8-bit Floating Point Numbers [📄 paper]
[2018 | NeurIPS] Training DNNs with Hybrid Block Floating Point [📄 paper]

Customized Format-Based Methods

[2025 | arXiv | 🤖] Training LLMs with MXFP4 [📄 paper]
[2025 | arXiv] Oscillation-Reduced MXFP4 Training for Vision Transformers [📄 paper]
[2023 | arXiv | 🤖] Microscaling Data Formats for Deep Learning [📄 paper]
[2022 | TCADICS] Exploring the Potential of Low-bit Training of Convolutional Neural Networks [📄 paper]
[2020 | TC] Evaluations on Deep Neural Networks Training Using Posit Number System [📄 paper]
[2019 | CoNGA] Posits: The Good, the Bad and the Ugly [📄 paper]
[2019 | JETCAS] FloatSD: A New Weight Representation and Associated Update Method for Efficient Convolutional Neural Network Training [📄 paper]
[2017 | NeurIPS] Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks [📄 paper]
[2017 | SFI] Beating Floating Point at its Own Game: Posit Arithmetic [📄 paper]

Quantization-Aware Training Methods

[2025 | arXiv | 🤖] Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models? [📄 paper]
[2025 | arXiv | 🤖] QuEST: Stable Training of LLMs with 1-Bit Weights and Activations [📄 paper] [💻 code]
[2025 | arXiv ] Stabilizing Quantization-Aware Training by Implicit-Regularization on Hessian Matrix [📄 paper]
[2024 | ACL | 🤖] LLM-QAT: Data-Free Quantization Aware Training for Large Language Models [📄 paper] [💻 code]
[2024 | ACL | 🤖] BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation [📄 paper] [💻 code]
[2024 | arXiv | 🤖] EfficientQAT: Efficient Quantization-Aware Training for Large Language Models [📄 paper] [💻 code]
[2024 | arXiv | 🤖] The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits [📄 paper] [💻 code]
[2023 | arXiv | 🤖] BitNet: Scaling 1-bit Transformers for Large Language Models [📄 paper] [💻 code]