Why GPUs are shaped this way

01 · THE THESIS

A CPU is a genius. A GPU is a stadium.

Every claim in this journey hangs on one sentence, so here it is early: a CPU spends its transistors making one stream of work finish as fast as possible; a GPU spends them on having an absurd number of workers, so that waiting for one worker stops mattering. That is the entire difference. Everything else (warps, blocks, shared memory, occupancy) is engineering detail hanging off that choice.

To see why the choice makes sense, give both machines the same boring job: multiply two arrays of numbers, element by element. One million multiplies. No branches, no dependencies: element 412,551 doesn't care what element 412,550 returned. Perfectly parallel work.

02 · THE RACE

Same job, two philosophies

Below is the race, simplified on purpose. The CPU fields 8 workers, each quick. The GPU fields 5,888 workers, each slower, and pays a fixed cost just to start (the launch). Drag the workload from trivial to millions and run it at each size. Watch where the winner flips, and where it flips back.

Two things happened there. At small sizes the GPU loses because starting the stadium costs more than the game. That launch overhead is real, and it's why your operating system, your browser, and your database all run on CPUs. But somewhere around a few thousand elements, the GPU's army finishes before the CPU's specialists have cleared their backlog, and from there the gap only grows. At eight million elements the GPU is done while the CPU is roughly a third of the way through.

Neither machine got smarter. The GPU just changed what the word waiting means.

03 · WHY THE CPU IS A GENIUS

The CPU lies to you about latency

Here's an uncomfortable fact: main memory is brutally slow compared to a CPU core. A core can execute hundreds of instructions in the time it takes to fetch one number from RAM. If the core simply waited for every load, it would idle ~95% of the time, and your 5 GHz processor would feel like a 1998 machine.

So the CPU spends enormous transistor budgets on making sure it never visibly waits: multi-level caches that keep likely data on-chip, branch predictors that guess which way your if goes before it evaluates, out-of-order engines that reorder your instructions to keep every pipeline stage busy, prefetchers that fetch data you haven't asked for yet. The genius of a CPU is that it hides latency for one stream of work. It's a world-class illusionist performing for an audience of one thread.

04 · HOW THE GPU CHEATS BACK

The GPU doesn't hide the wait. It ignores it.

A GPU core has no branch predictor worth mentioning, modest caches, and in-order execution. One GPU thread waiting on memory is genuinely, embarrassingly slow, about 2× slower than one CPU core in raw per-task terms.

The GPU's answer is not to fix that. It's to make it irrelevant: run thousands of threads per chip, and keep a hardware scheduler on every little neighborhood of cores (streaming multiprocessors, covered on day 02). The moment one thread stalls on a memory fetch, the scheduler flips to another thread that's ready to compute. Zero cost, zero ceremony. With enough threads in flight, the memory system is never left idle: the wait for thread #4,097 is hidden by the work of threads #1 through #4,096.

The CPU hides latency with foresight. The GPU hides it with supply. That's why GPU code reviews care about strange things (how many threads you launched, whether memory access arrives in tidy patterns) while CPU code reviews care about entirely different strange things.

05 · THE NUMBERS, ROUGHLY

Stadium seating chart

Ballpark specs, chosen because they're the hardware this journey's numbers come from:

Read the last two rows together: the CPU core is faster, but the GPU has hundreds of times more of them. Throughput is workers × speed, and when the work is parallel, count beats clock every time. (Real figures vary by generation and workload; the shape of the argument doesn't.)

06 · WHEN THE STADIUM LOSES

One suitcase, one bridge

Before you conclude CPUs are obsolete, hand the stadium a job with a dependency: walk one suitcase across a bridge, then another, where each trip needs the previous trip's receipt. A thousand extra walkers don't help; the bottleneck is the order, not the labor.

Serial work with dependencies is CPU country: operating systems, databases, game logic, your build tool. Parallel work without dependencies is GPU country: pixels, matrix math, embeddings, the element-wise guts of every neural network. Most real programs are a mix, which is why they ship on machines with both, and why the interesting engineering question is always which part goes where.

That question is the whole discipline. The next thirty days are just it, asked over and over at higher resolution.

07 · A GLIMPSE AHEAD

The kernel you'll write on day 13

Everything above becomes concrete the moment you see the code. This is vector add, the hello world of GPU programming, and by day 13 every bracket in it will be obvious. For now, read it like a stadium roster:

// runs once per element; the GPU schedules this × N
__global__ void add(float* a, float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}

That i line is the entire trick: instead of looping over a million elements, each thread asks “which element am I?” and does one. The loop didn't disappear; it turned into the hardware's job.

Next: day 02 opens the case and names the parts: streaming multiprocessors, warp schedulers, and the memory hierarchy you'll be negotiating with for the rest of the journey.