Concurrency, Parallelism, and Async: Three Ideas That Sound the Same But Aren't

There is a category of interview question that senior engineers fumble more often than juniors, because juniors recite the textbook and seniors have spent years using the words loosely. "What's the difference between concurrency and parallelism?" is the canonical one. Add "and where does async fit?" and you can watch a room full of experienced people talk past each other for twenty minutes.

This isn't pedantry. I build sports-data pipelines. To assemble a single row — one participant, one round, enriched and model-ready — the pipeline calls four separate upstream services: a live-scoring feed, a participant-stats API, a odds API, and a venue-and-weather service. Multiply four calls by a 156-participant field across four rounds and one event's base table is already on the order of 2,500 API calls — before a single number has been modelled. Whether a given stage should be concurrent, parallel, or asynchronous is not a vocabulary question there. It decides how many machines I pay for, how the pipeline behaves at 2 a.m. when a data vendor gets slow, and whether the Sunday win-probability model refreshes in 38 seconds or 12 minutes. The three words describe genuinely different things, and the cleanest way I've found to hold them apart is this:

What the pipeline taught me Async fan-out cut a full field build from 12m 29s to 38s — a 19× wall-clock win on a single thread. But the fastest median was not the safest posture: capping concurrency at 200 gave up 1.5 seconds at the median to erase a three-minute tail and 52 upstream connection errors. Parallelism, meanwhile, has a hard ceiling — the win-probability simulation is only ~92% parallel, so 16 workers is the whole prize and 64 cores buys nothing. And the single rule that prevented the most 2 a.m. incidents needs no framework at all: never let CPU work touch the event loop.

Concurrency is a property of your program's structure. Parallelism is a property of its execution. Async is a style of writing the first, which may or may not give you the second.

Rob Pike, one of Go's creators, made the structural-versus-execution distinction famous in a 2012 talk, and it has held up because it survives contact with real systems. Before we take the three ideas one at a time, here is the pipeline they live in — because the argument of this whole piece is really just a walk from left to right across this diagram.

1 — FAN OUT async · I/O-bound 2 — SIMULATE parallel · CPU-bound 3 — WRITE strictly serial row participant, round GET scores GET stats GET odds GET weather 1 thread · 4 waits overlap rows Monte Carlo win-prob · 16 workers 6.24M sims/refresh ~92% parallel (Amdahl) merge results write single writer · in order async where it waits · parallel where it computes · serial where it must. Three postures, one pipeline — and the whole essay is the reasoning behind them.

FIG. 01 — The sports-data pipeline in one picture. Each stage answers a different question, so each stage gets a different concurrency model. Reading it left to right is the argument of this post.

Concurrency: many things in flight, not necessarily at once

A program is concurrent when it is structured to make progress on multiple tasks over overlapping time periods — even if, at any given instant, only one of them is actually executing. A useful picture is a check-in desk at a busy venue. The operator rotates between queues — handling one request, moving to the next while a lookup runs, circling back. At any instant only one conversation is active; yet a dozen queues all make progress over the same ten-minute window. One desk, one thread, many requests in flight — and far more throughput than if every queue had to fully resolve before the next was touched.

My pipeline's fan-out has exactly that shape. One thread fires the four API calls for a row, and while each call is waiting on the network, the thread starts the calls for the next row. Nothing runs at the same physical instant; the thread just refuses to block. On a single CPU core this is what your operating system's scheduler already does: it slices time and interleaves tasks, switching between them fast enough that everything appears simultaneous. The switches aren't free — a context switch typically costs on the order of a few microseconds, plus the sneakier cost of evicted CPU caches — but they're vastly cheaper than letting the core sit idle while one task waits on a network response.

SEQUENTIAL — one core, one task at a time core 0 task A task B task C CONCURRENT — one core, tasks interleaved core 0 A B C A B C A ↑ switches happen where a task would otherwise block (I/O, locks, timers) PARALLEL — three cores, tasks simultaneous core 0 core 1 core 2 task A task B task C ← same wall-clock window, three times the work

FIG. 02 — Same three tasks, three arrangements in time. Read A, B, C as three of the four API calls a row needs. Concurrency changes the order; parallelism changes the hardware. A system can be either, both, or neither.

Notice what concurrency buys you in FIG. 02: nothing, if the tasks are pure computation. The concurrent row finishes no earlier than the sequential row — interleaving compute-bound work on one core just adds switching overhead. Concurrency pays off when tasks spend most of their life waiting: on a database, a disk, a third-party API, a user. My four API calls are almost entirely waiting — the row-building thread spends well over 90% of its time parked on the network — which is precisely why interleaving them fills that dead time with useful work and throughput climbs without a single extra core.

Parallelism: many things literally at once

Parallelism is the simpler idea and the more expensive one. Two or more computations execute at the same physical instant, which requires two or more execution units — cores, machines, GPU lanes. A useful picture is a warehouse with multiple packing stations: every station processes a separate order simultaneously, in the same wall-clock window, because there are multiple independent workbenches. You cannot conjure that out of a single station, no matter how cleverly you schedule it; you need multiple stations. Cores are stations.

Parallelism is how you make a single large computation finish sooner: matrix multiplication, video encoding, training runs — or, in my pipeline, the live win-probability model. For each of the 156 participants I simulate the rest of the event forty thousand times, playing out the remaining stages against a distribution fit from historical performance and current conditions. That's roughly 6.24 million simulated outcomes per refresh, and it is embarrassingly parallel — every simulation is independent — right up until it isn't.

Parallelism has a famous and slightly depressing speed limit. If a fraction p of your program can be parallelized and the rest is stubbornly serial, Amdahl's Law caps your speedup on n cores at:

speedup(n) = 1 / ((1 − p) + p/n)

Run the numbers and the ceiling arrives fast. A program that is 95% parallelizable — genuinely excellent — can never run more than 20× faster, even on a million cores, because that immovable 5% serial section becomes the whole runtime. My simulation stage is about 92% parallel; the serial 8% is fitting the shared distributions, merging every worker's results, and sorting the final output. Amdahl put the useful ceiling in the low-to-mid teens, and measurement agreed — past sixteen workers the curve went flat, so that is where I stopped. The highest-leverage optimization was never "add more workers"; it was shrinking that serial 8%.

number of cores (log scale: 1 → 4096) speedup × 1 6 11 16 21 p = 99% → cap 100× p = 95% → cap 20× our sim ≈ 92% → stop near 16 workers p = 75% → cap 4×

FIG. 03 — Amdahl's Law. The curves flatten because the serial fraction, however small, eventually dominates. Our 92% simulation is dead by the mid-teens — a fact worth remembering before renting a 96-core box for a Sunday run.

Async: the art of not waiting badly

Here is where most confusion lives, so let's be blunt about it: async is not a third kind of "doing." It is a programming model for achieving concurrency — usually on a single thread — by making waiting explicit and cheap.

The classical way to handle 10,000 simultaneous network connections was 10,000 threads, each blocked on a socket read, each carrying a megabyte-scale stack and a seat at the scheduler's table. Dan Kegel named the pain in 1999 — the "C10K problem" — and the industry's answer was the event loop: one thread, a queue of pending events, and code written so that instead of blocking on I/O, a task registers a callback (or, in modern syntax, awaits) and yields the thread back to the loop. When the socket has data, the loop resumes the task exactly where it paused. My four-API row build is a miniature of that problem: hundreds of rows, four waits each, and no desire whatsoever to spin up a thread per wait.

# Build one row of sports data. Four APIs — and the whole question
# is whether their waits happen one after another, or together.

# Blocking: four calls, strictly serial. ~4 x latency per row,
# and the thread sleeps through every wait.
row = {
    **scores.get(participant, round),   # thread parked here
    **stats.get(participant),           # ...and here
    **odds.get(participant, event),     # ...and here
    **weather.get(venue, start_time),   # ...and here
}

# Async: the four waits overlap on one thread. One row now costs
# about 1 x latency, and hundreds of rows share that same thread.
scores, stats, odds, weather = await asyncio.gather(
    fetch_scores(participant, round),
    fetch_stats(participant),
    fetch_odds(participant, event),
    fetch_weather(venue, start_time),
)

That single gather is the difference between 12 minutes and 38 seconds on a full field. This is also why Node.js can serve enormous connection counts on a single-threaded event loop, and why Python's asyncio, Rust's tokio, and Go's goroutine scheduler (a hybrid that multiplexes lightweight goroutines onto a small pool of OS threads) all converge on the same shape. Async gives you the concurrent row of FIG. 02 with dramatically cheaper switches — a coroutine suspension is a function return, not a kernel context switch.

The fan-out, five ways

Concepts are cheap; here is what they cost in wall-clock. I built the same table — one 156-participant field, four rounds, 2,496 upstream calls — five different ways on the same box and the same vendors, back to back. Median upstream latency was ~300 ms, and the tail (p95) sat near 1.2 s because real APIs get slow. The numbers are the whole argument for taking concurrency seriously:

Same field build · 2,496 calls · one run each

fan-out strategy            wall p50   wall p95   threads   peak RSS   conn err
--------------------------  --------   --------   -------   --------   --------
blocking (serial)           12m 29s    12m 29s    1          48 MB     0
thread pool · 32 workers    47.6s      2m 18s     32        190 MB     3
thread pool · 256 workers   41.2s      4m 05s     256       910 MB     71
async gather (unbounded)    38.4s      3m 47s     1          96 MB     52
async + semaphore(200)      39.9s      1m 02s     1          98 MB     0

The naive serial build is the control: 2,496 waits, one after another, a thread asleep through every one. Everything below it is some flavour of "stop sleeping through the waits" — and the interesting part is that they don't all win the same way. Read the table three ways and it tells three different stories.

Median (p50) — the happy path. Unbounded async is fastest at 38.4 s, but only 1.5 seconds ahead of the bounded version, and the 256-thread pool isn't far behind. On a good day, almost anything beats serial by 15–20×. The median flatters everyone.

Tail (p95) — what your on-call feels. Now the field separates. Unbounded async and the fat thread pool both blow out past three to four minutes when a vendor stalls and every in-flight call piles onto the same slow endpoint. The bounded async run holds the tail to 1m 02s. The p95 is the number that pages you at 2 a.m.; the median is the number you quote in the standup.

Cost and blast radius. The conn err column is upstream damage — 429s and resets from hammering four vendors with 2,496 simultaneous calls. Unbounded async drew 52; the 256-thread pool drew 71 and paid 910 MB of memory for the privilege. A single semaphore capping in-flight requests at 200 took both to zero. On metered endpoints, every one of those errors is a retry you pay for and a rate-limit budget you burn.

The takeaway: the fastest median is not the safest posture. Bounding concurrency to 200 gives up 1.5 seconds at p50 to erase a three-minute tail and 52 upstream errors. On a pipeline feeding a live results table, that trade isn't close — I ship the bounded run every time.

And it gives you nothing else. An event loop does not add cores. If a request handler starts running forty thousand outcome simulations, the loop is blocked, every other "concurrent" row stalls behind it, and your beautifully async pipeline performs worse than a boring threaded one. This is exactly why the win-probability simulation never runs on the event loop — it goes to a process pool. Async waits well; it does not compute well.

The Python wrinkle For most of Python's life, the Global Interpreter Lock meant threads gave you concurrency but never parallelism for pure-Python code — only one thread executed bytecode at a time, so the simulation had to live in multiprocessing. That era is ending: the free-threaded build that arrived with Python 3.13 and matured in 3.14 finally lets threads run bytecode on multiple cores simultaneously. The short version is that the pipeline in this post now applies to Python with fewer asterisks than at any point since 1992.

The decision, drawn once

Strip away the vocabulary and one diagnostic question does most of the work: where does your program spend its time — waiting, or computing? Every stage of the pipeline earned its concurrency model by answering exactly that.

Where does the time go? profile before believing yourself waiting (I/O-bound) computing (CPU-bound) How many waits at once? dozens vs. tens of thousands few many Thread pool simple, debuggable, fine at small scale Async / events event loop, await, cheap suspended tasks Does the work divide? check Amdahl before scaling yes no Parallelism processes, cores, GPUs, free-threaded Python Better algo no arrangement of time fixes O(n²) Mixed workloads compose: an event loop for the fan-out, a process pool for the sims, and a single-writer queue between them. My pipeline lives exactly here.

FIG. 04 — The whole decision in one tree. The bottom-right box is the one people skip: if the work is serial and slow, no concurrency model rescues it.

Eight ways I've broken this in production

The reason getting these three ideas straight matters is that every way of confusing them has a signature failure in production. These are the ones the pipeline has actually served me, collected the hard way. Every one traces back to applying the wrong model — computing where I should have been waiting, or waiting where I should have been computing.

C1

CPU on the event loop. A heavy parse — or a stray simulation — runs inside an async handler. The loop can't yield, and every in-flight row stalls behind it. The async service performs worse than a boring threaded one.

C2

Unbounded fan-out. No concurrency cap, so 2,496 calls hit four vendors at once, trip rate limits, and return a storm of 429s and resets. This is the 52 errors in the table — self-inflicted, and fixed by one semaphore.

C3

A blocking call in async code. One synchronous requests.get or an un-awaited DNS lookup parks the whole loop. The symptom looks exactly like a slow vendor; the culprit is local.

C4

Thread-pool thrash. Too many threads; switch overhead and per-thread stacks swamp the gain and the box starts swapping. The 256-worker row — 910 MB, 71 errors, a worse tail than 32 — is this failure caught on camera.

C5

Head-of-line blocking. A row's gather can't finish until the slowest of its four APIs returns. One sluggish weather call throttles an otherwise-quick row; the fix is per-call timeouts and graceful degradation, not more concurrency.

C6

A shared-state race. Two tasks accumulate into the same total without a lock, and the results table is subtly, non-reproducibly wrong. Concurrency bugs don't crash; they lie.

C7

The Amdahl wall. Workers added past the serial ceiling. You rent 64 cores, get the 16-core number, and spend a Sunday wondering why the bill doubled while the refresh didn't move.

C8

Order leaking into the write. Concurrency reaches the final commit and the output lands out of sequence. A results table, like a ledger, does not forgive that — which is why that last stage stays strictly serial by design.

The pattern: notice that not one of these is a syntax error. They're all category errors — the right code in the wrong concurrency model. Which is exactly why the three words are worth pinning down before you write a line.

If you remember nothing else

Concurrency is a design decision: structuring a program as independent tasks so that waiting never wastes the machine. Parallelism is a hardware decision: spending cores to compress wall-clock time on divisible work, up to the ceiling Amdahl sets. Async is an implementation decision: the cheapest known way to write highly concurrent code, and a trap the moment real computation sneaks onto the event loop.

The pipeline I described at the top uses all three, deliberately: async fan-out to pull four APIs per row across a full field (thousands of concurrent waits, one thread), a process pool to run the win-probability simulation across cores (parallelism, ~92% parallel fraction, so I stopped at 16 workers), and a strictly serial final write, because a results table — like a ledger — does not appreciate creativity. None of that design fell out of a framework. It fell out of knowing which of the three words applied where: async for the data fetch, parallelism for the simulation, and the discipline to keep them from colliding.

That's the whole trick. The words sound the same. The bills they generate don't.

Before you go →

Next in my own queue: taking the three runtimes I kept name-dropping — Python's asyncio, Rust's tokio, and Go's goroutines — and putting them head-to-head on this exact four-API fan-out, so we can see where each one genuinely shines and where it quietly bites. Every number above comes from my own pipeline build, so read them as a shape rather than a spec, and measure your own loop before you quote mine.

And if there's a convention or a design practice you'd want me to dig into next — how I bound concurrency without starving throughput, how I land on a worker count, how the serial write stays honest under load — drop it in the comments. I read all of them, and the sharpest questions usually turn into the next post.

Now go profile something. Have fun out there.