The thread hierarchy

01 · THE THREE NOUNS

Grid, block, thread

Every CUDA kernel launch answers three questions at once: how many threads run, how they're grouped, and which piece of the data each one handles. The vocabulary is three nested nouns. The biggest container is the grid. Inside it are blocks. Inside every block are threads. That's the whole hierarchy — and yesterday's anatomy maps onto it one-to-one: threads live on cores, blocks land on SMs, and the grid is the chip.

Here's the mapping for a vector add over one million elements, launched with 256 threads per block:

The dots are a sample, not the census: each block really holds 256 threads. Multiply it out and the grid above stands for 3,907 blocks, a million threads, all created with one line of launch code.

02 · WHO DECIDES WHAT

You choose the shape. The chip places it.

The division of authority is the part beginners get backwards. You decide the block size (how many threads share a block) and the grid size (how many blocks) when you write the launch. The hardware decides everything after that: which SM each block lands on, when it runs, and how its warps interleave with everyone else's. You never pick a core. There is no core zero.

Why the hardware gets to place blocks: because every block is written to be independent. Block 412 doesn't read block 411's output. That independence is what lets the chip scatter 3,907 blocks across 46 SMs in any order, fill every gap, and still be correct. (The day this stops being true — when blocks must cooperate — is day 16's problem, and it costs you.)

Block size is the one knob you'll turn constantly, and it trades two ways. Bigger blocks mean more threads share an SM's resources and the scheduler has more warps to hide latency with. Smaller blocks mean more blocks to scatter, which helps when some SMs finish early. Day 06 turns that trade into actual arithmetic.

03 · EACH LEVEL'S JOB

Who schedules what

Grid: the whole launch. It exists to be divided. One kernel = one grid.

Block: the unit of placement and resource allocation. An SM reserves registers and shared memory for a whole block before running it, and blocks can't share an SM's scratchpad with each other. Independence at this level is what keeps placement trivial.

Thread: the unit of work. Each thread knows its own coordinates (threadIdx, blockIdx) and computes its own piece of the data. Day 13 makes this line of code real.

And one level the vocabulary skips until now: inside every block, threads are bundled into warps of 32 for scheduling. The warp is the hardware's handle on execution, and it deserves its own day, because it's tomorrow's.

Next: day 04 explains the number you'll see everywhere for the rest of your GPU life. 32 threads, one instruction, in lockstep.