Quantization

Why cyankiwi serves 4-bit models, and how our proprietary AWQ variant keeps them faithful to the originals.

Why Quantize

An LLM ships its weights in 16-bit floating point (BF16): a 70B-parameter model occupies roughly 140 GB, and every generated token streams all of it through the GPU. Storing the weights at 4 bits cuts that footprint to a quarter, which means smaller GPUs, lower cost, and faster decoding.

The trade is fidelity. A 4-bit grid holds only 16 values, so rounding onto it discards information, and every method differs in which information it keeps. Done carelessly, the model stays fluent but degrades where it matters: reasoning, tool use, non-English text. cyankiwi’s standard is a quantized model that is measurably faithful to the original, and that is what our proprietary AWQ is built to deliver.

The Baseline: AWQ

When reducing LLM weights to low bit-widths, a small number of activation outliers carry a disproportionate amount of the model’s information. Only 0.1% to 1% of weight channels are “salient”, and they can be identified by looking at the activation magnitude.

The original Activation-aware Weight Quantization (AWQ) protects salient channels by scaling up before group-wise 4-bit rounding. Scales are chosen by a one-dimensional grid, and fit against a homogeneous English calibration set. Two limitations follow: the salience map is only as reliable as that calibration data, and the scaling search is decoupled from the clipping ranges that ultimately determine quantization error.

cyankiwi AWQ

cyankiwi AWQ improves on the baseline along two axes: calibration and optimization.

The calibration corpus is heterogeneous and cross-lingual, weighted toward the regimes where modern LLMs are actually stressed under deployment: technical reasoning and tool-using behavior. The diversity of the corpus is deliberate: it draws out the model’s full activation distribution rather than the narrow slice a small monolingual set would expose. Scales are then derived against activation statistics that reflect the model’s true operating range, producing a salience map aligned with how the model is actually used.

cyankiwi treats per-channel scales and quantization ranges as co-dependent variables rather than independent hyperparameters, fitting them together against a reconstruction objective. Scales are no longer chosen in isolation from the clipping that follows, and neither is fixed independently of the rounding it will undergo.

The Results

We benchmarked cyankiwi AWQ 26.05 update against every major 4-bit method on Llama-3 as examples, measuring KL Divergence vs the BF16 baseline on GPQA Diamond responses.

Result: cyankiwi posts the lowest KLD on all three base models. Lower is better.

Llama-3.2-3B-Instruct

Llama-3.1-8B-Instruct

Llama-3.3-70B-Instruct

Acknowledgment

We are grateful to:

• MIT Han Lab for the original AWQ research
• Casper Hansen for the reference AWQ implementation
• The vLLM and Red Hat AI teams for the production tooling we build on