← Back to portfolioCUDA · Quantized Kernels

GPU Kernel Engineering · Part 01

Why Using Fewer Bits Can Make AI Run Faster

Follow one FP32 value through a CUDA quantization kernel: load it, scale it, round it, clamp it, store an INT8 code, and reconstruct an approximation.

CUDAFP32INT8Memory coalescing
FP32 values transformed into compact INT8 values by a CUDA quantization kernel
One element-wise conversion: four-byte FP32 input, one-byte INT8 code, and the metadata needed to reconstruct an approximation.

Quantization is often introduced as a model-compression technique: replace floating-point values with smaller integers and reduce memory consumption. That is true, but it leaves out the part that matters to a CUDA kernel engineer.

What happens between the moment a thread loads an FP32 value and the moment it stores an INT8 byte? Where do scale and zero-point enter the computation? How does rounding work? What happens when the result is outside the INT8 range? And how do we know the GPU implementation is correct?

In this first installment, we will stay with element-wise quantization and dequantization. We will not implement GEMM, Tensor Core instructions, INT4 packing, GPTQ, or AWQ. The objective is to understand the data representation, the thread-level operations, memory access, validation, and a measured baseline.

The experiment

4 → 1

Bytes per value: FP32 to INT8

1

Element handled by each thread in the baseline kernel

2

Kernels: quantization and dequantization

1. Why Quantization Matters: The GPU Memory Wall

A neural network stores weights, activations, and intermediate tensors. Those values must be moved through the GPU memory hierarchy before or during computation. When a workload moves a large amount of data but performs relatively little arithmetic per value, memory traffic can limit performance.

Quantization reduces the number of bits used to represent each value. An FP32 element occupies four bytes; an INT8 element occupies one byte. For a tensor with the same number of elements, the raw data storage can therefore be reduced by a factor of four, excluding scales, zero-points, alignment, and other metadata.

Visual 01 · memory budget

Same parameters, very different traffic

Choose a model size and compare the storage-only footprint. Runtime activations, KV cache, metadata, and workspaces are not included.

7B parameters
1B70B
FP32full precision

4 bytes / value

28 GB

FP16half precision

2 bytes / value

14 GB

INT8code + scale

1 byte / value

7 GB

INT4two packed values/byte

0.5 bytes / value

3.5 GB

Quantization changes the storage contract. The GPU still needs metadata and conversion work, but fewer bytes travel through the memory hierarchy.

Smaller storage can reduce memory traffic in later operations, but quantization itself also performs work: values must be converted, and metadata must be available. Whether an end-to-end workload becomes faster depends on how those costs interact with the surrounding kernels.

2. FP32 vs INT8: Understanding the Representation

FP32 uses a sign bit, an exponent, and a fraction. The exponent gives floating-point values a wide dynamic range, with spacing between representable values changing with magnitude.

Signed INT8 is an 8-bit two's-complement integer with the range:

-128 <= q <= 127

INT8 codes are evenly spaced integers. An integer code alone does not identify the original real value. The quantization scale, and for asymmetric quantization the zero-point, define how the code maps back into the real-valued domain.

int8_t q = 10;

// Same integer code, different reconstructed values:
scale = 0.1   ->  real_value = 1.0
scale = 0.01  ->  real_value = 0.1
scale = 0.001 ->  real_value = 0.01

The stored byte is the code. Scale and zero-point are metadata used to interpret that code.

3. What One CUDA Thread Actually Does

In the baseline element-wise kernel, each CUDA thread is responsible for one tensor element. The thread computes its global index from the block index, block size, and thread index. It then checks that the index is valid before accessing memory.

size_t index =
    blockIdx.x * blockDim.x + threadIdx.x;

if (index < count) {
    float x = input[index];

    float transformed = x / scale + zero_point;
    int rounded = nearbyintf(transformed);
    int q = max(-128, min(127, rounded));

    output[index] = static_cast<int8_t>(q);
}

This is the essential data path: calculate the index, load the FP32 value, transform it into integer space, round, clamp, and store the INT8 code. The scale and zero-point are supplied to the kernel; this element-wise kernel does not derive them from the tensor.

4. Quantization, Dequantization and Reconstruction

Live data path

Watch one value move through the kernel

Change the input or scale. Each block shows what that thread reads, computes, stores, and reconstructs.

FP32 → INT8 → FP32

01 · Load

20.00

FP32 input

B08 bits
B18 bits
B28 bits
B38 bits

32 bits total · 4 bytes

02 · Scale

200.00

Integer-space value

20.00 ÷ 0.100

One INT8 step represents 0.100 in the original value space.

03 · Round

Before200.00
↓ nearest integer
After200

Fractional information is discarded here.

04 · Store

127

Signed INT8 code

0
1
1
1
1
1
1
1

Overflow: 200 → 127

8-bit two’s-complement byte

05 · Rebuild

12.7000

Reconstructed FP32

127 × 0.100
B08 bits
B18 bits
B28 bits
B38 bits

Did the value survive the round trip?

Shared axis: -30 to +30
Original FP3220.0000
-300+30
After dequantization12.7000
-300+30
Absolute reconstruction error7.3000

Clipping changed the value: 200 is outside the signed INT8 range, so the kernel stores 127.

The FP32 byte tiles show storage width, not the individual IEEE-754 bit pattern. The INT8 tiles show the actual signed two’s-complement code produced by this calculation.

For affine quantization, the integer code is computed as:

q = clamp(round(x / scale) + zero_point,
          q_min, q_max)

Dequantization reconstructs an approximation of the original value:

x_hat = scale * (q - zero_point)

Consider a symmetric INT8 example with scale approximately 1/127. The original value 0.4 becomes an integer code of 51. Dequantizing that code gives approximately 0.4016. The difference is quantization error.

If the rounded code falls outside the signed INT8 range, clamping saturates it to -128 or 127. This is different from ordinary rounding: the output is limited by the representable range, and reconstruction error can become much larger.

The first interactive follows one value through the thread's arithmetic. The distribution view shows multiple illustrative weights being assigned to quantization bins and reconstructed. Its error metrics describe this synthetic sample; they are not measurements of neural-network task accuracy.

Visual 02 · quantization levels

From a weight distribution to discrete values

Change the number of uniform quantization bins. Each original weight is mapped to a bin, then reconstructed at that bin's representative value.

15 levels
15
4 · coarse32 · fine
Original weight -2.150Original weight -1.820Original weight -1.550Original weight -1.220Original weight -0.940Original weight -0.680Original weight -0.390Original weight -0.120Original weight 0.180Original weight 0.430Original weight 0.720Original weight 1.020Original weight 1.310Original weight 1.620Original weight 1.950Original weight 2.280Dequantized reconstruction -2.000Dequantized reconstruction -2.000Dequantized reconstruction -1.600Dequantized reconstruction -1.200Dequantized reconstruction -0.800Dequantized reconstruction -0.800Dequantized reconstruction -0.400Dequantized reconstruction 0.000Dequantized reconstruction 0.000Dequantized reconstruction 0.400Dequantized reconstruction 0.800Dequantized reconstruction 1.200Dequantized reconstruction 1.200Dequantized reconstruction 1.600Dequantized reconstruction 2.000Dequantized reconstruction 2.400quantized / reconstructed valuesIllustrative Gaussian-shaped weight distribution−30+3
Original weights Quantization bins Quantized / dequantized values Mapping

Mean absolute reconstruction error

0.0975

Root mean squared error

0.1140

Bin width / quantization step

0.4000

Dequantization and reconstruction

sample values · original → reconstructed

-2.150

↓

-2.000

error 0.150

-1.820

↓

-2.000

error 0.180

-1.550

↓

-1.600

error 0.050

-1.220

↓

-1.200

error 0.020

-0.940

↓

-0.800

error 0.140

-0.680

↓

-0.800

error 0.120

-0.390

↓

-0.400

error 0.010

-0.120

↓

0.000

error 0.120

This is a synthetic scalar-quantization demonstration over the fixed range [−3, 3]. The pink values are reconstructed at their bin centers. Lower reconstruction error indicates a closer numerical match, but this toy example does not measure a neural network's task accuracy.

5. Symmetric vs Asymmetric Quantization

Symmetric quantization uses a zero-point of zero. A common signed INT8 scale choice is based on the largest absolute input magnitude:

// Symmetric
scale = max(abs(x)) / 127;
zero_point = 0;

q = clamp(round(x / scale), -128, 127);

Asymmetric quantization maps a chosen floating-point interval onto the integer code range. It uses both scale and zero-point:

// Asymmetric signed INT8 example
scale = (x_max - x_min) / 255;
zero_point = round(-128 - x_min / scale);

q = clamp(round(x / scale) + zero_point,
          -128, 127);

During dequantization, the zero-point is subtracted before multiplying by the scale. Symmetric mapping has simpler metadata and arithmetic. Asymmetric mapping can use the integer code range more effectively when the selected input range is shifted away from zero.

Visual 03 · mapping choice

Symmetric and asymmetric quantization

float range · -10.810.8
Five real values share the line; zero is at the center.
INT8 code · -128127
The same points after the linear map.

Metadata with the byte

scale
0.0850
zero-point
0
real zero code
0

Symmetric is simple for many kernels. Asymmetric uses code space more efficiently when the distribution is shifted.

The important CUDA connection is direct: the selected mapping determines which parameters the kernel reads and which arithmetic operations it performs for every element.

7. CPU Reference vs CUDA Kernel Validation

A kernel that completes without an error is not necessarily numerically correct. We validate the GPU output against a CPU reference that uses the same scale, zero-point, rounding rule, and clamping limits.

There are two separate checks. First, compare the quantized integer codes. For deterministic test inputs and matching rounding semantics, the expected mismatch count is zero. Second, compare the dequantized values with the original FP32 inputs. These values are not expected to match exactly because quantization is lossy.

Validation caseWhat it checks
Positive, negative, and zero valuesBasic mapping behavior
Rounding boundariesCPU/GPU rounding agreement
Out-of-range valuesINT8 saturation and clamping
Non-multiple tensor sizesFinal-block bounds checking
Dequantized outputMaximum error and mean squared error

Integer-code equality validates the conversion implementation. Reconstruction metrics quantify the expected information loss. They answer different questions and should not be conflated.

8. Benchmarking on the RTX 3060

After correctness, we measure execution time with CUDA events. The benchmark results below are the measurements recorded for this project. They are specific to the tested GPU, launch configuration, compiler settings, and benchmark procedure; they are not universal performance guarantees.

The table reports element throughput and nominal bandwidth. For these element-wise conversions, nominal bandwidth is calculated from the bytes read and written per element. It is a derived traffic rate, not a direct measurement of physical DRAM traffic.

Tensor shapeOperationTimeGelem/sNominal GB/s
256 × 2048FP32 → INT80.011 ms49.066245.328
256 × 2048INT8 → FP320.012 ms44.775223.874
2048 × 2048FP32 → INT80.077 ms54.715273.577
2048 × 2048INT8 → FP320.075 ms55.785278.924
256 × 2048Symmetric0.011 ms48.739243.693
256 × 2048Asymmetric0.019 ms27.971139.853

The 2048 × 2048 results are around 55 Gelem/s for the recorded conversion runs. Small tensors are more sensitive to launch overhead and timing resolution. The symmetric and asymmetric rows also show that additional per-element arithmetic and the particular implementation can affect measured performance.

When comparing kernels, keep the tensor size, warmup, repetition count, synchronization, and timing region consistent. Allocation and host-to-device transfer should be outside the timed region if the goal is to measure kernel execution alone.

Summary

A basic quantization kernel is a sequence of explicit operations: calculate an index, load a float, apply scale and zero-point, round, clamp, and store an integer code. Dequantization reverses the mapping approximately. Correctness depends on matching the numerical rules, while performance depends on launch geometry and memory access as well as arithmetic.

For further reading and a deeper understanding of quantization, including numerical representations, symmetric and asymmetric quantization, and practical applications in large language models, refer to these resources: Quantization from the Ground Up by ngrok and A Visual Guide to Quantization by Maarten Grootendorst.

Explore the implementation

View GPU Quantized Kernels on GitHub↗

All interactive elements in this article were developed with the assistance of AI to enhance the learning experience.