Several reports discuss why quantized large language models (LLMs) may not run faster even after reducing model size. In one account, a 2-bit quantization scheme for a Qwen3 variant cuts memory usage from about 8 GB to roughly 2.6 GB. However, measured token throughput shows only marginal gains versus an FP16 baseline, suggesting that the expected speedup from quantization does not automatically translate into higher performance.
The central claim across sources is that runtime overhead can outweigh quantization benefits. One contributor attributes the slow performance to a large memory copy operation—described as a 778 MB memory copy per generated token—instead of the quantized computation itself. Another outlet emphasizes a similar conclusion: profiling is necessary to identify where time is spent, especially in parts of the pipeline that may be uncompressed or involve large data movement. Together, the accounts frame quantization as reducing storage and bandwidth needs, while performance can still be limited by implementation details such as per-token data transfer and other uncompressed steps.
The sources do not claim quantization is inherently ineffective; rather, they argue that bottlenecks may occur elsewhere and that careful profiling can reveal the true cause of low throughput.
