01 · The ladder in one picture
Every level trades size for speed
You've now met every rung individually — registers (day 09), shared memory (day 08), global memory (day 07). Put them side by side with the two caches you haven't been introduced to properly, and the whole machine fits in one table (all numbers, RTX 4070, approximate):
REGISTERS · ~1 CYCLE
per-thread · 256 KB per SM
SHARED MEMORY · ~30 CYCLES
per-block · up to 100 KB per SM
L1 CACHE · ~35 CYCLES
per-SM · 128 KB, shared SRAM with shared memory
L2 CACHE · ~200 CYCLES
all 46 SMs · 36 MB
DRAM (GDDR6X) · ~500 CYCLES
the 12 GB · ~504 GB/s when coalesced
AT 2.5 GHZ, 500 CYCLES ≈
200 nanoseconds per DRAM trip
Two names in that table are new. L1 is a small cache attached to each SM — and here's the detail that makes GPUs different: it's the same physical SRAM as shared memory, partitioned at kernel launch. SRAM you don't declare as shared becomes cache. L2 is one big cache shared by all 46 SMs, the last stop before DRAM.
02 · Why bandwidth is the ceiling
The GPU starves long before it runs out of math
The 4070 computes roughly 29 trillion FP32 operations per second. Its DRAM delivers about 504 billion bytes per second. Do the division and something uncomfortable appears: for every byte the memory system delivers, the compute side could burn through ~58 operations. Any kernel that does fewer than ~58 FLOPs per byte of DRAM traffic cannot use the compute it's paying for — it's waiting on the warehouse (day 01) no matter how well the warps are scheduled.
This is why "bandwidth-bound" and "compute-bound" are the two words that organize all kernel performance, and why day 11 is about nothing but drawing that boundary on a chart. The caches exist to move work across it: every byte served from L2 instead of DRAM is a byte that never touched the ~504 GB/s limit.
03 · GPU caches are not CPU caches
Smaller, more transparent, and sharing real estate
If you come from CPUs, recalibrate three instincts. First, GPU caches are small — a CPU core may have more L1 than an entire 4070 SM — because the design philosophy is explicit over implicit: shared memory you schedule beats a cache that guesses, and the GPU trusts you to say what gets staged.
Second, L1 does more jobs at once: cached global loads, spilled registers (day 09), local arrays, and instruction fetch all contend for the same SRAM. Third, read-only data can take a dedicated path — marking pointers const __restrict__ (or using __ldg) tells the compiler the data is read-only for the kernel's lifetime, unlocking a wider, better-latency load route. Cheap to write, occasionally free performance.
04 · Streaming vs reuse
Classify your traffic before you optimize anything
The question that decides which memory tricks apply: how many times is each byte used? Three answers, three playbooks:
USED ONCE — STREAMING
caches can't help. Coalescing (day 07) is everything; vector add is the archetype.
REUSED WITHIN A BLOCK
stage it in shared memory (day 08). Tiled matmul (day 15) is the archetype.
REUSED ACROSS BLOCKS
L2 catches it. 36 MB holds 9.2M floats — sometimes a whole activation tensor.
REUSED BY ONE THREAD
registers (day 09). Keep it live, don't reload it.
That classification is also the diagnostic order. When a kernel underperforms, the first question isn't "which optimization do I apply" — it's "which class of traffic is this?" Day 12 walks a famous kernel — naive matmul — through exactly this interrogation, and it fails the test in an instructive way.
- 01Memorize the ladder, not the marketing. Registers ~1 cycle, shared ~30, L1 ~35, L2 ~200, DRAM ~500. Every optimization in this journey is movement down this ladder.
- 02L1 and shared memory are the same SRAM. What a block doesn't reserve as shared becomes L1 cache. Reserving more shared means less cache — a real, tunable trade.
- 03Bandwidth, not FLOPs, is the usual ceiling. ~29 TFLOPS against ~504 GB/s means ~58 FLOPs per byte of headroom. Most real kernels do far fewer — we'll draw this precisely on day 11.
- 04Classify traffic first: streaming, block-reuse, cross-block, or per-thread. Once → coalescing. Block → shared. Cross-block → L2. Per-thread → registers. The class picks the optimization.
Next: day 11 — the roofline model, the single chart that tells you which ceiling your kernel is actually hitting, with an interactive version to play with.