Explaining Quantization

Quantization is one of the fundamental optimization techniques in the field of Machine Learning. It allows for the compression of Large Language Models (LLMs), dramatically reducing memory consumption (VRAMA VRAM (sigla para Video Random Access Memory, ou Memória de Acesso Aleatório de Vídeo) é um tipo especial de memória RAM de altíssima velocidade dedicada exclusivamente ao processamento gráfico e a operações de computação paralela intensa, como o tr... More/RAM) and speeding up response times, with minimal loss in reasoning capability.

In simple terms, quantizing is reducing the numerical precision of a neural network’s weights.

What Does it Mean to Reduce Precision?

During training, LLMs calculate and store their billions of parameters (weights) using high-precision 32-bit floating-point numbers (FP32) or 16-bit floating-point numbers (FP16 / BF16).

• 16-bit Floating-Point (FP16): Stores numbers with extremely detailed decimal places (e.g., 0.0048293712). Each weight occupies 2 bytes of memory.
• 4-bit Integer (INT4): Rounds and maps this continuous range of decimal values to a discrete, limited set of integers (e.g., integers from -8 to 7). Each weight now occupies only 0.5 bytes (4 bits).

The Mathematics of Mapping

The quantization process converts a continuous floating-point value $x \in [\text{min}, \text{max}]$ into a discrete integer value $q$, using two mathematical parameters: a Scale (S) and a Zero-Point (Z).

The most common quantization formula (Uniform Quantization) is:

To dequantize the value during the execution of mathematical operations:

• S (Scale): Defines the step size between numbers.
• Z (Zero-Point): Adjusts the offset so that the true zero value of the original precision maps exactly to the discrete zero of the integer.

Calculation Example

To understand quantization in practice, let’s walk through the mathematical step-by-step process of how a small group of high-precision weights (FP32) is compressed into a low-precision 4-bit format (INT4) and then reconstructed (approximated FP16).

Imagine we extract a small slice of a weight matrix from a layer of a model. The original model uses FP32 (32-bit floating point, occupying 4 bytes per number).

Our 4 original weights (X) are:

Our goal: Compress these values into INT4 (unsigned 4-bit integer), whose possible values vary only across a discrete range from 0 to 15 (a total of $2^4 = 16$ levels).

Step 1: Calculate the Scale (S) and Zero-Point (Z)

To map the continuous range of our real weights $[\text{min}, \text{max}]$ to the discrete range $[q_{\text{min}}, q_{\text{max}}] = [0, 15]$, we first identify the extremes of the original data:

• $\text{min} = -0.85$
• $\text{max} = 0.98$
• $q_{\text{min}} = 0$
• $q_{\text{max}} = 15$

A. Calculating the Scale (S)

The scale determines the distance represented by each integer step:

B. Calculating the Zero-Point ($Z$)

The zero-point shifts the scale so that the real value corresponding to 0.0 is perfectly represented by an integer:

Step 2: Quantization ($X \rightarrow q$)

Now we apply the uniform quantization formula to each of the original weights:

• For $x_1 = -0.85$: $q_1 = \text{round}\left( \frac{-0.85}{0.122} \right) + 7 = \text{round}(-6.967) + 7 = -7 + 7 = \mathbf{0}$
  For $x_2 = 0.12$: $q_2 = \text{round}\left( \frac{0.12}{0.122} \right) + 7 = \text{round}(0.983) + 7 = 1 + 7 = \mathbf{8}$
• For $x_3 = 0.45$: $q_3 = \text{round}\left( \frac{0.45}{0.122} \right) + 7 = \text{round}(3.688) + 7 = 4 + 7 = \mathbf{11}$
• For $x_4 = 0.98$: $q_4 = \text{round}\left( \frac{0.98}{0.122} \right) + 7 = \text{round}(8.032) + 7 = 8 + 7 = \mathbf{15}$

Compressed Result

Instead of saving 4 heavy 32-bit floating-point numbers, the model saves only the sequence of 4-bit integers:

In addition to the vector $q$, the model needs to store block metadata: S = 0.122 and Z = 7.

Memory Gain: Each weight q now occupies only 4 bits (0.5 bytes) instead of 32 bits (4 bytes)—an 8x reduction in weight size.

Step 3: Dequantization During Inference ($q \rightarrow X’$)

When the LLM needs to perform matrix multiplications during tokenTokens são as unidades básicas de informação que uma LLM processa. Eles não são necessariamente palavras inteiras; podem ser sílabas, caracteres ou partes de palavras. A tokenização é o processo de transformar o texto bruto em uma sequência desses to... More generation, it temporarily dequantizes these integers back to floating point using the inverse formula:

$$X’ = S \cdot (q – Z)$$

• $X’_1 = 0.122 \cdot (0 – 7) = 0.122 \cdot (-7) = \mathbf{-0.854}$ (Original: -0.85)
• $X’_2 = 0.122 \cdot (8 – 7) = 0.122 \cdot (1) = \mathbf{0.122}$ (Original: 0.12)
• $X’_3 = 0.122 \cdot (11 – 7) = 0.122 \cdot (4) = \mathbf{0.488}$ (Original: 0.45)
• $X’_4 = 0.122 \cdot (15 – 7) = 0.122 \cdot (8) = \mathbf{0.976}$ (Original: 0.98)

Quantization Error Evaluation

Analyzing the difference between the original and dequantized weights:

Original Value (X)	Reconstructed Value (X′)	Absolute Error (∣X−X′∣)
-0.85	-0.854	0.004
0.12	0.122	0.002
0.45	0.488	0.038
0.98	0.976	0.004

The loss of precision is in the hundredths. Across billions of connections in a neural network, this minor numerical deviation is absorbed by the robustness of the model, while delivering significant gains in VRAMA VRAM (sigla para Video Random Access Memory, ou Memória de Acesso Aleatório de Vídeo) é um tipo especial de memória RAM de altíssima velocidade dedicada exclusivamente ao processamento gráfico e a operações de computação paralela intensa, como o tr... More consumption and execution speed.