GPU kernel deep dive
Why Unused Data Makes GPU Kernels Fly
How adding unused data to the end of every row moved a naive CUDA matrix multiplication from 943.10 us to 650.24 us on an RTX 3060 Laptop GPU.

I recently came across Meta's $100,000 challenge to make Muse Spark run as fast as possible. The idea was simple and extremely tempting: write faster GPU kernels, measure them, and see how far the numbers could move.
I started writing CUDA kernels for the workload and got a result that looked reasonable to me. Then I compared it with other implementations using the same logical matrix shapes and the same datatypes. Mine was slower. Not a little slower either - the gap was large enough that the usual first explanations did not seem sufficient.
That sent me down the rabbit hole. I read kernel-optimization write-ups, checked coalescing rules, studied tiled GEMM implementations, and kept trying to explain where the missing time was going. Eventually I started looking less at the arithmetic and more at the addresses being generated.
The useful clue was that one input matrix was being laid out with a very regular 4096-byte interval between rows. So I tried something deliberately boring: I added unused FP16 values to the end of every row.
Before this test, I had already worked through several more conventional CUDA optimizations: coalesced global-memory access, shared-memory tiling, register blocking, and a vectorized subgroup-style kernel. That last version used a 16 x 16 block to compute a larger output tile and usedhalf2 to group adjacent FP16 memory operations while keeping FP32 accumulation.
Those versions were useful because they improved data reuse. They also changed several things at once. For this experiment I went back to the simple kernel and changed only the physical row stride of A. The math, launch shape, logical dimensions, and datatypes all stayed fixed.
The short version
943.10 us
4096-byte row stride
650.24 us
4128-byte row stride
31.1%
lower median latency
The problem stayed exactly the same
The kernel computes a fixed GEMM. The only variable in the primary experiment is the physical stride of matrix A.
| Matrix | Shape | Type |
|---|---|---|
| A | 256 x 2048 | FP16 |
| B | 2048 x 2048 | FP16 |
| C | 256 x 2048 | FP16 output |
Every product is converted to FP32 before accumulation. There is no quantization, integer datatype, cuBLAS call, WMMA instruction, or cooperative-matrix API in the primary experiment.
float sum = 0.0f;
for (int k = 0; k < 2048; ++k) {
sum += float(A[row * lda + k]) * float(B[k * 2048 + col]);
}
C[row * 2048 + col] = __float2half_rn(sum);With no padding, lda = 2048 and every row occupies 4096 bytes. With 16 padding elements, lda = 2064 and the row occupies 4128 bytes. The kernel still reads only the 2048 valid values.

The padding sweep
I did not want to compare only 0 and 32 padding elements and then build a story around the result. The benchmark sweeps 0, 8, 16, 24, 32, 40, 48, 64, 96, and 128 FP16 elements while keeping the logical work fixed.
| Padding | lda | Stride | Median us | P95 us | TFLOPS |
|---|---|---|---|---|---|
| 0 | 2048 | 4096 B | 943.10 | 1,013.76 | 2.28 |
| 8 | 2056 | 4112 B | 667.65 | 944.13 | 3.22 |
| 16 | 2064 | 4128 B | 650.24 | 657.38 | 3.30 |
| 24 | 2072 | 4144 B | 653.31 | 659.46 | 3.29 |
| 32 | 2080 | 4160 B | 656.38 | 660.48 | 3.27 |
| 40 | 2088 | 4176 B | 650.24 | 661.50 | 3.30 |
| 48 | 2096 | 4192 B | 658.43 | 661.50 | 3.26 |
| 64 | 2112 | 4224 B | 657.41 | 668.35 | 3.27 |
| 96 | 2144 | 4288 B | 658.43 | 661.50 | 3.26 |
| 128 | 2176 | 4352 B | 658.43 | 662.53 | 3.26 |
Padding 16 is the best point in the supplied run, tied closely by padding 40. The important shape is the drop after padding zero and the stable plateau afterward. That makes the result look like a stride-phase effect, not a universal rule that exactly 32 elements are magic.
Why would 16 unused values matter?
The valid payload of every row is still 4096 bytes. Padding does not make the current row more contiguous; it changes where the next row begins.
4096 bytes / 64 bytes = 64 cache-line intervals
4096 bytes / 128 bytes = 32 sector-group intervals
4112 bytes / 64 bytes = 64.25 line intervals
4128 bytes / 64 bytes = 64.5 line intervals
4160 bytes / 64 bytes = 65 line intervalsIf address bits used by a cache set, an L2 slice, or a memory partition repeat under the 4096-byte step, the rows can keep revisiting an unfavorable pattern. A 4112-, 4128-, or 4160-byte stride changes that phase on every row.
The size of the gain is the surprising part. Sixteen FP16 values add only 32 bytes to each row, or 8 KiB across the whole 256-row matrix. That is less than one percent more storage than the valid A payload, yet the median kernel time drops by 31.1%.
If the cache hypothesis is right, this is probably not because the cache suddenly becomes perfectly balanced. A regular stride can make a subset of sets, slices, or memory partitions unusually hot. Warps then wait on the same overloaded path, and a small change in row phase can remove a much larger amount of serialization than the added bytes would suggest.
That explanation is plausible, but it is not proof that cache sets alone produced the entire gain. Sector grouping, replacement, partition routing, and memory stalls may all contribute. I would describe the result as a stride-induced memory-system cliff until counters identify the mechanism.

Cache conflict is a hypothesis, not a conclusion
The access pattern makes the hypothesis plausible. Threads in a warp usually compute neighboring columns, so their loads from Bare contiguous. Loads from A are broadcast-like: threads computing neighboring columns for the same row request the sameA[row, k] value.
The simple kernel does not explicitly stage either operand in shared memory. It repeatedly relies on the cache hierarchy and the compiler's scheduling. That makes row placement and replacement pressure relevant, but it does not identify a specific set or partition.
Other explanations are still possible: sector grouping, split transactions, L1 replacement, L2 slice routing, memory dependency stalls, clock state, or measurement noise. NVIDIA does not document every cache index and partition hash for this GPU generation.
The engineering lesson
Padding is not a universal optimization. It increases the allocation, can hurt another access path, and may become irrelevant after replacing this naive kernel with a tiled or Tensor Core implementation.
The useful lesson is broader: physical layout is a performance parameter, not merely an allocation detail. When a kernel shows an unexplained performance cliff, a small stride sweep can reveal address-dependent behavior that is invisible in the mathematical formulation.
Conclusion
Adding unused FP16 values to each row changed the speed of a fixed CUDA GEMM. The 4096-byte layout was the outlier at 943.10 microseconds. Strides from 4112 through 4352 bytes clustered near 650 to 658 microseconds, with padding 16 and 40 reaching 3.30 effective TFLOPS.
I still would not write “padding fixes the cache” as the final answer. The defensible conclusion is that physical row stride can move a GPU kernel across a performance cliff. The cache-set and partition-camping explanation is a strong lead, and the next step is to validate it with counters rather than intuition.
Reproduce it
The complete CUDA source, CPU reference, padding sweep, Rich terminal runner, CSV, plots, and Nsight Compute commands live in the companion project.
cmake --build build --config Release
python scripts\run_benchmark.py --kernel padded --padding all --warmup 10 --repetitions 100View the experiment repository ↗