GPU Kernel Engineering · Part 01
Why Using Fewer Bits Can Make AI Run Faster
Follow one FP32 value through a CUDA quantization kernel: load it, scale it, round it, clamp it, store an INT8 code, and reconstruct an approximation.

Quantization is often introduced as a model-compression technique: replace floating-point values with smaller integers and reduce memory consumption. That is true, but it leaves out the part that matters to a CUDA kernel engineer.
What happens between the moment a thread loads an FP32 value and the moment it stores an INT8 byte? Where do scale and zero-point enter the computation? How does rounding work? What happens when the result is outside the INT8 range? And how do we know the GPU implementation is correct?
In this first installment, we will stay with element-wise quantization and dequantization. We will not implement GEMM, Tensor Core instructions, INT4 packing, GPTQ, or AWQ. The objective is to understand the data representation, the thread-level operations, memory access, validation, and a measured baseline.
The experiment
4 → 1
Bytes per value: FP32 to INT8
1
Element handled by each thread in the baseline kernel
2
Kernels: quantization and dequantization
1. Why Quantization Matters: The GPU Memory Wall
A neural network stores weights, activations, and intermediate tensors. Those values must be moved through the GPU memory hierarchy before or during computation. When a workload moves a large amount of data but performs relatively little arithmetic per value, memory traffic can limit performance.
Quantization reduces the number of bits used to represent each value. An FP32 element occupies four bytes; an INT8 element occupies one byte. For a tensor with the same number of elements, the raw data storage can therefore be reduced by a factor of four, excluding scales, zero-points, alignment, and other metadata.
Visual 01 · memory budget
Same parameters, very different traffic
Choose a model size and compare the storage-only footprint. Runtime activations, KV cache, metadata, and workspaces are not included.
4 bytes / value
28 GB
2 bytes / value
14 GB
1 byte / value
7 GB
0.5 bytes / value
3.5 GB
Quantization changes the storage contract. The GPU still needs metadata and conversion work, but fewer bytes travel through the memory hierarchy.
Smaller storage can reduce memory traffic in later operations, but quantization itself also performs work: values must be converted, and metadata must be available. Whether an end-to-end workload becomes faster depends on how those costs interact with the surrounding kernels.
2. FP32 vs INT8: Understanding the Representation
FP32 uses a sign bit, an exponent, and a fraction. The exponent gives floating-point values a wide dynamic range, with spacing between representable values changing with magnitude.
Signed INT8 is an 8-bit two's-complement integer with the range:
-128 <= q <= 127INT8 codes are evenly spaced integers. An integer code alone does not identify the original real value. The quantization scale, and for asymmetric quantization the zero-point, define how the code maps back into the real-valued domain.
int8_t q = 10;
// Same integer code, different reconstructed values:
scale = 0.1 -> real_value = 1.0
scale = 0.01 -> real_value = 0.1
scale = 0.001 -> real_value = 0.01The stored byte is the code. Scale and zero-point are metadata used to interpret that code.
3. What One CUDA Thread Actually Does
In the baseline element-wise kernel, each CUDA thread is responsible for one tensor element. The thread computes its global index from the block index, block size, and thread index. It then checks that the index is valid before accessing memory.
size_t index =
blockIdx.x * blockDim.x + threadIdx.x;
if (index < count) {
float x = input[index];
float transformed = x / scale + zero_point;
int rounded = nearbyintf(transformed);
int q = max(-128, min(127, rounded));
output[index] = static_cast<int8_t>(q);
}This is the essential data path: calculate the index, load the FP32 value, transform it into integer space, round, clamp, and store the INT8 code. The scale and zero-point are supplied to the kernel; this element-wise kernel does not derive them from the tensor.
4. Quantization, Dequantization and Reconstruction
Live data path
Watch one value move through the kernel
Change the input or scale. Each block shows what that thread reads, computes, stores, and reconstructs.
01 · Load
20.00
FP32 input
32 bits total · 4 bytes
02 · Scale
200.00
Integer-space value
One INT8 step represents 0.100 in the original value space.
03 · Round
Fractional information is discarded here.
04 · Store
127
Signed INT8 code
Overflow: 200 → 127
8-bit two’s-complement byte
05 · Rebuild
12.7000
Reconstructed FP32
Did the value survive the round trip?
Shared axis: -30 to +30Clipping changed the value: 200 is outside the signed INT8 range, so the kernel stores 127.
The FP32 byte tiles show storage width, not the individual IEEE-754 bit pattern. The INT8 tiles show the actual signed two’s-complement code produced by this calculation.
For affine quantization, the integer code is computed as:
q = clamp(round(x / scale) + zero_point,
q_min, q_max)Dequantization reconstructs an approximation of the original value:
x_hat = scale * (q - zero_point)Consider a symmetric INT8 example with scale approximately 1/127. The original value 0.4 becomes an integer code of 51. Dequantizing that code gives approximately 0.4016. The difference is quantization error.
If the rounded code falls outside the signed INT8 range, clamping saturates it to -128 or 127. This is different from ordinary rounding: the output is limited by the representable range, and reconstruction error can become much larger.
The first interactive follows one value through the thread's arithmetic. The distribution view shows multiple illustrative weights being assigned to quantization bins and reconstructed. Its error metrics describe this synthetic sample; they are not measurements of neural-network task accuracy.
Visual 02 · quantization levels
From a weight distribution to discrete values
Change the number of uniform quantization bins. Each original weight is mapped to a bin, then reconstructed at that bin's representative value.
Mean absolute reconstruction error
0.0975
Root mean squared error
0.1140
Bin width / quantization step
0.4000
Dequantization and reconstruction
sample values · original → reconstructed-2.150
↓
-2.000
error 0.150
-1.820
↓
-2.000
error 0.180
-1.550
↓
-1.600
error 0.050
-1.220
↓
-1.200
error 0.020
-0.940
↓
-0.800
error 0.140
-0.680
↓
-0.800
error 0.120
-0.390
↓
-0.400
error 0.010
-0.120
↓
0.000
error 0.120
This is a synthetic scalar-quantization demonstration over the fixed range [−3, 3]. The pink values are reconstructed at their bin centers. Lower reconstruction error indicates a closer numerical match, but this toy example does not measure a neural network's task accuracy.
5. Symmetric vs Asymmetric Quantization
Symmetric quantization uses a zero-point of zero. A common signed INT8 scale choice is based on the largest absolute input magnitude:
// Symmetric
scale = max(abs(x)) / 127;
zero_point = 0;
q = clamp(round(x / scale), -128, 127);Asymmetric quantization maps a chosen floating-point interval onto the integer code range. It uses both scale and zero-point:
// Asymmetric signed INT8 example
scale = (x_max - x_min) / 255;
zero_point = round(-128 - x_min / scale);
q = clamp(round(x / scale) + zero_point,
-128, 127);During dequantization, the zero-point is subtracted before multiplying by the scale. Symmetric mapping has simpler metadata and arithmetic. Asymmetric mapping can use the integer code range more effectively when the selected input range is shifted away from zero.
Visual 03 · mapping choice
Symmetric and asymmetric quantization
Metadata with the byte
- scale
- 0.0850
- zero-point
- 0
- real zero code
- 0
Symmetric is simple for many kernels. Asymmetric uses code space more efficiently when the distribution is shifted.
The important CUDA connection is direct: the selected mapping determines which parameters the kernel reads and which arithmetic operations it performs for every element.
7. CPU Reference vs CUDA Kernel Validation
A kernel that completes without an error is not necessarily numerically correct. We validate the GPU output against a CPU reference that uses the same scale, zero-point, rounding rule, and clamping limits.
There are two separate checks. First, compare the quantized integer codes. For deterministic test inputs and matching rounding semantics, the expected mismatch count is zero. Second, compare the dequantized values with the original FP32 inputs. These values are not expected to match exactly because quantization is lossy.
| Validation case | What it checks |
|---|---|
| Positive, negative, and zero values | Basic mapping behavior |
| Rounding boundaries | CPU/GPU rounding agreement |
| Out-of-range values | INT8 saturation and clamping |
| Non-multiple tensor sizes | Final-block bounds checking |
| Dequantized output | Maximum error and mean squared error |
Integer-code equality validates the conversion implementation. Reconstruction metrics quantify the expected information loss. They answer different questions and should not be conflated.
8. Benchmarking on the RTX 3060
After correctness, we measure execution time with CUDA events. The benchmark results below are the measurements recorded for this project. They are specific to the tested GPU, launch configuration, compiler settings, and benchmark procedure; they are not universal performance guarantees.
The table reports element throughput and nominal bandwidth. For these element-wise conversions, nominal bandwidth is calculated from the bytes read and written per element. It is a derived traffic rate, not a direct measurement of physical DRAM traffic.
| Tensor shape | Operation | Time | Gelem/s | Nominal GB/s |
|---|---|---|---|---|
| 256 × 2048 | FP32 → INT8 | 0.011 ms | 49.066 | 245.328 |
| 256 × 2048 | INT8 → FP32 | 0.012 ms | 44.775 | 223.874 |
| 2048 × 2048 | FP32 → INT8 | 0.077 ms | 54.715 | 273.577 |
| 2048 × 2048 | INT8 → FP32 | 0.075 ms | 55.785 | 278.924 |
| 256 × 2048 | Symmetric | 0.011 ms | 48.739 | 243.693 |
| 256 × 2048 | Asymmetric | 0.019 ms | 27.971 | 139.853 |
The 2048 × 2048 results are around 55 Gelem/s for the recorded conversion runs. Small tensors are more sensitive to launch overhead and timing resolution. The symmetric and asymmetric rows also show that additional per-element arithmetic and the particular implementation can affect measured performance.
When comparing kernels, keep the tensor size, warmup, repetition count, synchronization, and timing region consistent. Allocation and host-to-device transfer should be outside the timed region if the goal is to measure kernel execution alone.
Summary
A basic quantization kernel is a sequence of explicit operations: calculate an index, load a float, apply scale and zero-point, round, clamp, and store an integer code. Dequantization reverses the mapping approximately. Correctness depends on matching the numerical rules, while performance depends on launch geometry and memory access as well as arithmetic.
For further reading and a deeper understanding of quantization, including numerical representations, symmetric and asymmetric quantization, and practical applications in large language models, refer to these resources: Quantization from the Ground Up by ngrok and A Visual Guide to Quantization by Maarten Grootendorst.
Explore the implementation
View GPU Quantized Kernels on GitHub↗All interactive elements in this article were developed with the assistance of AI to enhance the learning experience.