01 · THE MAGIC NUMBER
Why 32?
Yesterday's hierarchy had one hiding level: inside every block, threads are bundled into warps of 32, and the warp is what the hardware actually schedules. Not a suggestion, not a guideline — the warp is the grain of GPU execution. Threads don't get scheduled; warps do. Registers don't get allocated per thread; they're budgeted per warp. Nothing in GPU programming will make sense at a gut level until the number 32 feels natural.
Why 32 and not 64 or 16? History and arithmetic. The memory system delivers data in 128-byte lines, and 32 threads × 4 bytes = exactly 128 bytes, one transaction. The warp is sized so that a warp of threads reading consecutive numbers fills one memory delivery perfectly. The number is the memory system's own grain, promoted to an execution unit.
02 · THE EXPLORABLE
One instruction, 32 lanes, same clock
This is a warp running a tiny instruction stream. Watch the shape of execution: the scheduler broadcasts one instruction, and all 32 lanes fire it in the same clock, each on its own piece of data. That execution model has a name — SIMT, single instruction, multiple threads.
← the SM's scheduler broadcasts this one instruction to the whole warp
watch: every instruction fires on all 32 lanes in the same clock.
simplified: real warps also stall on memory, mask inactive lanes, and share operands — day 07 and beyond
Notice what you never see: lanes finishing at different times, lanes running different instructions, a lane jumping ahead. Lockstep is the deal the 32 lanes signed. It's what makes a warp cheap to schedule (one program counter per warp, not 32) and it's why GPU code loves uniform work.
03 · WHY THIS NUMBER FOLLOWS YOU
Everything is measured in warps
Once the warp exists, every resource on the chip is priced in warps. Day 02's scheduler runs warps. Registers are budgeted per warp. Occupancy (day 06) is measured in resident warps per SM. Even your block sizes are quietly about warps: a block of 256 threads is 8 warps, and the hardware rounds every block size up to a whole warp — launch 33 threads and the hardware funds 64 lanes' worth of resources while 31 of them sit masked off.
So the first practical rule of GPU programming falls out for free: make your thread counts multiples of 32, because the hardware pays for the rounding whether you use it or not.
04 · THE CATCH
Lockstep has a price
The same deal that makes warps cheap makes them brittle: if the 32 lanes need to take different branches, lockstep can't do both at once. The warp's answer is to run each path with the other lanes switched off, one after the other — and that slowdown has a name, warp divergence. It's the first real performance cliff of the journey, and it's tomorrow's whole subject.
- 01The warp is the grain. 32 threads bundled at compile time; warps are what the SM schedules, stalls, and budgets.
- 02SIMT: one instruction, 32 lanes. All lanes fire the same instruction each clock, each on its own data. One program counter per warp.
- 03128 bytes made 32 threads. The warp matches the memory system's 128B delivery — which is also day 07's story.
- 04Round to 32. Block sizes in whole warps. Launch 33 and the hardware pays for 64 while 31 lanes sit dark.
Next: day 05 breaks the lockstep on purpose. What actually happens when 16 lanes of a warp take one branch and 16 take the other.