AWQ Algorithm Results

September 4, 2026 ยท View on GitHub

This document summarizes an experimental comparison between native AutoRound flows and AWQ-composed flows. The goal is to evaluate whether applying AWQ smoothing before RTN or AutoRound improves accuracy for W4A16, MXFP4, and INT8 (W8A8) schemes.

All table results are based on the AutoRound 0.15.0 release, measured with --format fake, and evaluated on mmlu,gsm8k,piqa,hellaswag,winogrande. These five widely used LLM benchmarks cover complementary error modes: MMLU for factual knowledge and broad understanding, GSM8K for mathematical reasoning, PIQA for physical commonsense, HellaSwag for commonsense sentence completion, and Winogrande for pronoun and coreference resolution. Reporting the average score across this task set reduces sensitivity to single-task variance and provides a more stable view of quantization-induced accuracy changes. Unless noted otherwise, the remaining quantization settings follow the AutoRound defaults.

AWQ Algorithm Characteristics

AWQ is an activation-aware weight smoothing method. It searches for per-channel scaling factors from calibration activations, applies the scaling to reduce quantization-sensitive weight channels, and then hands the transformed model to the terminal quantization algorithm.

AWQ changes the pre-quantization weight distribution, while RTN, AutoRound, and AutoRound2 decide how the final quantized values are produced. In this design, AWQ can be composed with different quantizers without changing their core optimization logic.

From the resource perspective, AWQ does not introduce a persistent RAM or VRAM burden in the quantized model. During calibration, AWQ_RTN stays around the same host RAM level as the other flows and uses much less VRAM than iterative AutoRound flows. When AWQ is followed by AutoRound or AutoRound2, the peak VRAM is mainly determined by the downstream AutoRound optimization rather than by AWQ itself.

The main added cost is calibration runtime. This is a one-time offline cost and is acceptable when AWQ provides measurable accuracy gains, especially for MXFP4 and selected W4A16 cases.

The configurations in the tables map to the following CLI usage:

# Pure RTN baseline
auto-round-rtn --model <model> --scheme <scheme> --format fake \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

# Optimized RTN baseline
auto-round-opt-rtn --model <model> --scheme <scheme> --format fake \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

# Native AutoRound / AutoRound with algorithm extension
auto-round --model <model> --scheme <scheme> --format fake \
  --algorithm auto_round \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

auto-round --model <model> --scheme <scheme> --format fake \
  --enable_alg_ext --algorithm auto_round \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

# AWQ + RTN / AWQ + AutoRound / AWQ + AutoRound with algorithm extension
auto-round --model <model> --scheme <scheme> --format fake \
  --algorithm awq \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

auto-round --model <model> --scheme <scheme> --format fake \
  --algorithm awq,auto_round \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

auto-round --model <model> --scheme <scheme> --format fake \
  --enable_alg_ext --algorithm awq,auto_round \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

In the result tables, AR2 denotes AutoRound with --enable_alg_ext, and AWQ_AR2 denotes AWQ followed by AutoRound with --enable_alg_ext. Accuracy columns are reported as percentages, so a raw score such as 0.7058 is shown as 70.58.

The AWQ rows use the default AWQ smoothing flow. They do not include AWQ weight clipping unless --awq_apply_clip is explicitly enabled.

In the observation tables, AVG Win Rate counts how often the AWQ-composed method improves model-level AVG accuracy for that scheme, Max AVG Delta Gain is the largest absolute AVG improvement in percentage points, and Max AVG Rel. Gain is the largest relative AVG improvement. For AWQ_RTN, the comparison includes both RTN and opt_rtn when both baselines are available.

MXFP4

Model Config GSM8K (%) HellaSwag (%) MMLU (%) PIQA (%) WinoGrande (%) AVG (%) Time cost (s) RAM VRAM
Llama-3.1-8B-I BF16 70.51 59.71 68.37 80.09 73.40 70.42 - - -
RTN 50.11 55.54 57.34 76.01 70.01 61.80 88.00 9.83GB 2.99GB
opt_rtn 52.08 55.82 57.86 76.88 71.11 62.75 100.00 10.02GB 3.84GB
AR 60.73 55.29 61.34 77.53 70.01 64.98 976.75 24.33GB 16.86GB
AR2 61.87 56.59 61.60 77.97 71.11 65.83 1151.81 24.38GB 17.97GB
AWQ_RTN 50.27 55.51 58.55 76.93 70.56 62.36 993.51 23.36GB 7.91GB
AWQ_AR 60.05 56.00 61.96 78.07 72.14 65.64 2042.67 24.35GB 18.80GB
AWQ_AR2 62.02 56.40 61.84 78.62 72.14 66.20 2164.04 24.40GB 17.94GB
Qwen3-8B BF16 87.41 57.16 72.92 76.71 67.72 72.38 - - -
RTN 81.27 52.24 66.13 73.01 63.14 67.16 119.00 7.81GB 2.66GB
opt_rtn 83.55 52.60 66.10 73.50 64.40 68.03 125.00 7.99GB 3.42GB
AR 80.82 52.64 67.98 74.81 66.30 68.51 999.52 22.31GB 15.43GB
AR2 83.70 53.41 68.51 75.57 67.56 69.75 1240.32 22.34GB 16.18GB
AWQ_RTN 82.79 53.54 67.48 75.19 67.17 69.23 1073.95 21.33GB 7.14GB
AWQ_AR 79.91 53.07 68.99 75.35 67.32 68.93 2095.77 22.32GB 16.72GB
AWQ_AR2 83.70 53.89 69.54 75.57 68.19 70.18 2244.87 22.37GB 16.03GB

MXFP4 Observations

ComparisonAVG Win RateMax AVG Delta GainMax AVG Rel. Gain
AWQ_RTN vs RTN / opt_rtn3/4 (75%)+2.07 pts+3.08%
AWQ_AR vs AR2/2 (100%)+0.66 pts+1.02%
AWQ_AR2 vs AR22/2 (100%)+0.43 pts+0.62%
  • MXFP4 remains the clearest case for AWQ composition in these experiments. AWQ improves AVG for both AutoRound and AutoRound2 on both models.
  • AWQ_AR2 gives the highest MXFP4 AVG for both models, reaching 66.20 on Llama and 70.18 on Qwen.
  • AWQ_RTN improves over plain RTN on both models. This suggests AWQ smoothing is useful, while optimized RTN remains a strong low-cost MXFP4 baseline.
  • The main tradeoff is one-time runtime. AWQ_AR and AWQ_AR2 take roughly 1.8x-2.1x the corresponding native AutoRound runtime, while peak RAM/VRAM stays in the same range as the downstream AutoRound optimizer.

W4A16

Model Config GSM8K (%) HellaSwag (%) MMLU (%) PIQA (%) WinoGrande (%) AVG (%) Time cost (s) RAM VRAM
Llama-3.1-8B-I BF16 70.51 59.71 68.37 80.09 73.40 70.42 - - -
RTN 71.80 59.14 65.46 79.43 72.77 69.72 60.00 25.08GB 0.56GB
opt_rtn 67.63 58.99 65.67 78.94 73.95 69.04 162.30 28.31GB 7.90GB
AR 67.40 58.93 66.81 79.87 73.72 69.35 769.27 24.29GB 12.32GB
AR2 69.60 59.13 66.81 79.87 73.09 69.70 918.59 24.04GB 13.26GB
AWQ_RTN 70.58 59.09 65.89 79.87 72.85 69.66 1075.44 24.07GB 7.68GB
AWQ_AR 73.24 59.11 66.97 80.14 72.93 70.48 1813.51 24.07GB 13.32GB
AWQ_AR2 70.13 59.09 67.11 79.87 73.56 69.95 3533.26 24.07GB 13.17GB
Qwen3-8B BF16 87.41 57.16 72.92 76.71 67.72 72.38 - - -
RTN 86.43 56.18 70.84 75.52 67.17 71.23 97.00 23.10GB 0.50GB
opt_rtn 87.26 55.96 71.58 76.33 68.43 71.91 180.17 27.17GB 7.38GB
AR 88.02 56.03 72.13 76.22 67.72 72.02 779.03 21.73GB 11.41GB
AR2 86.96 55.95 72.38 76.71 68.27 72.05 1028.79 22.00GB 12.48GB
AWQ_RTN 86.13 56.04 71.91 75.63 67.56 71.45 1108.32 22.02GB 6.93GB
AWQ_AR 86.73 56.13 71.51 76.22 69.06 71.93 1957.19 21.94GB 12.07GB
AWQ_AR2 86.50 56.07 72.23 76.55 69.14 72.10 3673.00 22.03GB 12.79GB

W4A16 Observations

ComparisonAVG Win RateMax AVG Delta GainMax AVG Rel. Gain
AWQ_RTN vs RTN / opt_rtn2/4 (50%)+0.62 pts+0.90%
AWQ_AR vs AR1/2 (50%)+1.13 pts+1.63%
AWQ_AR2 vs AR22/2 (100%)+0.25 pts+0.36%
  • W4A16 shows model-dependent behavior. The clearest gain is Llama AWQ_AR over AR, with +1.13 average points and +1.63% relative AVG gain.
  • AWQ_RTN is not a strong W4A16 default. It wins 2/4 AVG comparisons against RTN and opt_rtn.
  • AWQ_AR2 improves AVG on both models, but the gain is small compared with the added runtime.

INT8 / W8A8

Model Config GSM8K (%) HellaSwag (%) MMLU (%) PIQA (%) WinoGrande (%) AVG (%) Time cost (s) RAM VRAM
Llama-3.1-8B-I BF16 70.51 59.71 68.37 80.09 73.40 70.42 - - -
RTN 72.25 59.73 68.10 79.71 74.82 70.92 98.00 9.53GB 0.85GB
AR 68.99 59.63 67.87 79.71 74.11 70.06 988.81 23.87GB 13.38GB
AWQ_RTN 72.63 59.54 68.13 79.82 73.72 70.77 1001.96 22.98GB 7.55GB
AWQ_AR 70.05 59.65 68.20 79.87 74.11 70.38 1879.53 23.95GB 14.21GB
Qwen3-8B BF16 87.41 57.16 72.92 76.71 67.72 72.38 - - -
RTN 87.49 56.93 72.53 76.01 63.95 71.38 88.00 7.58GB 0.75GB
AR 88.10 56.74 72.46 76.06 68.11 72.29 905.88 21.83GB 12.37GB
AWQ_RTN 86.58 57.05 72.57 76.77 68.35 72.26 1046.69 20.93GB 6.89GB
AWQ_AR 86.81 56.77 72.58 76.61 68.11 72.18 2277.47 21.92GB 13.30GB

INT8 Observations

ComparisonAVG Win RateMax AVG Delta GainMax AVG Rel. Gain
AWQ_RTN vs RTN1/2 (50%)+0.88 pts+1.23%
AWQ_AR vs AR1/2 (50%)+0.32 pts+0.46%
  • INT8/W8A8 is already a high-accuracy setting in these experiments. Most INT8 recipes are close to the BF16 reference, and some recipes slightly exceed BF16 AVG.
  • For AWQ-composed INT8, the AVG movement is small and model-dependent. AWQ_RTN improves over RTN on Qwen but is slightly below RTN on Llama; AWQ_AR improves over AR on Llama but is slightly below AR on Qwen.
  • Given the strong INT8 baseline and small AWQ deltas, AWQ is better positioned as an optional exploration path for INT8/W8A8 rather than a primary default.

Summary

AWQ is most useful for schemes where quantization is sensitive to activation-aware weight smoothing, such as MXFP4. It can improve scheme accuracy, especially when combined with AutoRound plus --enable_alg_ext.

AWQ does not add persistent memory overhead to the quantized model and is orthogonal to terminal quantization algorithms, so it can be composed with RTN, AutoRound, or AutoRound2 without changing their core quantization logic.