The purpose of this document is to present a report on the pros and cons of quantization. In place of that, I will specifically target the definition and applicability of this technique. The report is organized as follows:
- Quantization Definition
- Quantization arithmetic
- Experimental Setup
- Results and discussion
The idea of quantization is to project a high-precision floating point number to a lower-precision one. The target data type can be either an integer or a floating point number. Before we delve into the specifics, let's first understand how a floating point number is represented.
A float number of 32-bit precision is represented by splitting the bits into three parts: sign bit, exponent, and mantissa. The sign bit is 1 bit which indicates whether the number is positive or negative. The exponent is 8 bits that represent the power of 2 with a bias. The mantissa is the remaining 23 bits that represent the fractional part of the number. Now, let us say we want to represent the number 0.375 in 32-bit precision. First, we convert the number to binary.
Notice the number of mantissa bits represent the precision of the number, while the exponent bits are needed to represent a wide range of numbers. Thus by reducing the number of bits in the representation, we can reduce the memory footprint of storing the number. This has a downstream effect on the computational cost of operations. As such, quantization is a technique used to reduce the memory footprint as well as the computational cost of operations. This can be utilized in deep learning models to load bigger models on smaller devices while also speeding up the computation. Now, let us see how quantization works in practice. For the purpose of this article, I will only focus on the transition from FP32 to Int8. Let us see how quantization is achieved and the corresponding arithmetic is performed.
Suppose the source set of numbers lie in the range
Now let us see how quantized matrix multiplication works. Suppose we want to perform the operation
Each number in
Now, let us see how quantization would work. Let's replace each element in the multiplication equation by its quantized value.
The process of quantization can be seen as that of finding a linear mapping from a larger set of discrete numbers to a smaller set. These numbers correspond to the weights or activations in some part of the model. Let us see how we can categorize quantization methods.
The first way is to categorize them based on the data type of the target number. The target number can be either an integer or a floating point number. In addition, we also consider the accumulation data type. The accumulation data type is the data type used to store the intermediate results of the computation.
The second way to categorize quantization techniques is based on the granularity of the quantization. The granularity can be at the level of the tensor, layer, or model itself.
The third way is to categorize based on whether the quantization is done post-training or during training. Within post-training quantization, we can further categorize based on whether the quantization is done statically or dynamically. Static quantization is when the scale factor and zero point are calculated once and used throughout the inference process. A calibration dataset is needed to estimate the appropriate quantization parameters. Dynamic quantization is when the quantization parameters are calculated on the fly. This is useful when the input data distribution changes during inference.
Finally, during training quantization is when the quantization is done during the training process itself. The model is quantized during training so that the optimizer can adjust the weights to account for the quantization error. Next, we will
discuss the effect of quantization on computation.
The idea is to understand the effect of quantization on matrix multiplication. We consider FP32 as the base and Int8 as the target quantization. The accumulation data type is also kept as FP32. The matrix sizes are varied from 10 x 10, 100 x 100, 1000 x 1000, 3000 x 3000. Each matrix is sampled from a normal distribution with mean 0 and standard deviation 1 and a uniform distribution in the range
The matrix multiplication is done using the PyTorch library. The multiplication between the quantized matrices is implemented as an outer product between each column and row of
The first result we find is that the error rate is not affected by the size of the matrix as shown in the figure below. This makes sense since the quantized product is accumulated in an intermediate tensor of higher precision. This saves the accumulation from overflowing and as a result, the degradation is only dependent on the quality of the quantization of each value. Since the values are sampled from the same distribution, the aggregate quantization error will tend towards a normal distribution with increasing narrow range for larger matrices. For smaller matrices, we may see some variation. This effect is visible for the matrices of size 10 where the error rate is smaller or larger than for larger matrices. But beyond that the error is unaffected.
The key result we find is that the errors are correlated with coverage. As we increase the coverage, the error reduces as shown in the figure below. This makes sense since for a smaller coverage, the number lying outside of the range will be clipped. While larger coverage can accommodate more numbers. It is important to note that the large coverage is prone to poorly resolving dense distributions. For example, covering 99.99% of numbers in a normal distribution will poorly resolve the numbers near the peak. And since numbers are densely packed there, this resolution error will affect the downstream computation as a result. In our practice, however, we did not observe this effect.
Another effect of using coverage as the guide to compute the quantization factor is that the scale factor increases for large matrices. This result is a direct consequence of the scale factor being dependent on the range of the numbers in
From the above study, one can draw the conclusion that with an appropriate quantization factor, one can perform quantized arithmetic to reduce memory footprint if each value in the operand matrices is sampled from an IID. This is often the case at the beginning of training a neural network. Later, during the training, however, this may not be the case.