Your AI model is 2 GB. Its shipping box is 15 GB.

Something is clearly wrong when the wrapping is seven times heavier than the thing inside it. But almost nobody catches it, because the extra weight arrives one reasonable decision at a time and never shows up as a line you could point at.

This post walks through where those gigabytes come from. I’ll explain every term as it appears, so you don’t need a background in this. Let’s start with the three words the rest of it depends on.

Three words, in plain language

• image: A frozen package containing your program plus everything it needs to run — libraries, config, even a small copy of Linux. It’s a file. You build it once and copy it to any machine.
• container: An image that is actually running. The image is the recipe; the container is the meal. One image can start a thousand containers.
• layer: Images are built in slices. Every instruction you write (“install this”, “copy that”) adds a new read-only slice on top of the previous ones. This matters more than it sounds — see below.

That last one has a consequence that catches everyone. Layers only ever add. If step 3 downloads two gigabytes of temporary files and step 4 deletes them, the files look gone when you peek inside the container — but step 3’s slice still physically contains them, and still gets shipped.

It’s like packing a suitcase, then putting a note on top saying “ignore the shoes.” You’re still carrying the shoes. This is why cleanup has to happen in the same step as the download, not a later one.

The one idea behind all of this

Building software and running software need different tools.

To build, you need compilers — programs that translate human-readable source code into instructions a machine can execute. You need header files, build caches, test frameworks. To run, you need almost none of that. The compiler did its job and produced a finished file; it will never be needed again.

You wouldn’t mail someone a bookshelf along with the saw and the workbench. But that is exactly what a normal Dockerfile does, because the machine that builds the image and the machine that runs it are assumed to be the same machine.

On an ordinary web service, this is wasteful. On an AI service it’s wasteful and expensive, because GPUs are billed by the second and the meter starts before your model has loaded. Let’s open a real one up.

What’s actually in there

Here’s a completely normal image for serving a language model. Nobody did anything foolish building it. I’ve split it by one simple test: does the running server ever read this file after it starts?

Four of those red bars have specific causes, and each has a fix that takes one line. Let’s take them one at a time.

Cause 1: you picked the toolbox, not the toolkit

Quick background. A GPU is a chip built to do the same arithmetic on huge amounts of data at once, which is exactly what neural networks do. CUDA is NVIDIA’s system for running your code on that chip. And nvcc is CUDA’s compiler — it turns CUDA source code into something the GPU can execute.

NVIDIA publishes their starting images in three sizes, and the difference matters enormously:

The devel version contains nvcc, plus header files and sample code — everything you need to compile GPU programs. The runtime version contains only the libraries needed to execute programs that are already compiled. That difference is about four gigabytes.

Nearly everyone starts from devel, because it’s the one in the tutorial and it definitely works. But if your server only runs a finished model, it never calls nvcc, and those four gigabytes ride along doing nothing forever.

Cause 2: the same libraries, packed twice

This one is invisible and costs 2.5 GB. To explain it I need to introduce two library names you’ll see everywhere in AI infrastructure.

Two libraries worth knowing

• cuDNN: Short for CUDA Deep Neural Network library. NVIDIA’s hand-optimised collection of the operations neural networks perform constantly — convolutions, normalisation, attention. Their engineers tuned these down to the assembly level for each GPU generation, so they run several times faster than an obvious implementation would. PyTorch doesn’t write these itself; it calls cuDNN underneath. It’s over a gigabyte, because it ships many specialised versions of each operation and picks the fastest one for your data at runtime.
• cuBLAS: The same idea for linear algebra — mostly matrix multiplication. Since a language model is essentially an enormous pile of matrix multiplications, cuBLAS is doing much of the actual work every time your model answers a question.

Here’s the problem. Modern PyTorch installs bring their own private copies of cuDNN and cuBLAS along as dependencies. If you also started from a CUDA image that already includes them, you now have two full copies of the same multi-gigabyte libraries in one box. PyTorch uses its own. The others just sit there.

The same idea applies in reverse if you’re running on ordinary processors instead of GPUs. A plain pip install torch downloads all the GPU libraries whether or not a GPU exists. There’s a separate CPU-only version that skips them and saves you about 2.3 GB:

RUN pip install torch --index-url https://download.pytorch.org/whl/cpu

One flag. Two and a bit gigabytes.

Cause 3: keeping the workshop after the job’s done

Some AI components genuinely have to be compiled when you install them — specialised pieces like flash attention, which speeds up how models handle long inputs. These really do need nvcc and a full compiler setup.

They need it for about four minutes. Then it’s finished, and the output is a small installable file.

So do the compiling in a workspace you throw away. This is called a multi-stage build, and it’s the single highest-value thing in this post. You write two starting points in one recipe file. The first is huge and full of tools; it builds what needs building. The second starts fresh and small, and copies in only the finished result. The first is deleted when the build ends — it never gets published, never reaches a server.

In a recipe file it looks like this. Read the comments rather than the syntax — the shape is the point.

# Box 1: huge, and completely disposable.
FROM nvidia/cuda:12.4.0-devel-ubuntu22.04 AS builder
RUN pip wheel --no-deps -w /wheels flash-attn==2.5.8

# Box 2: the only thing that ever reaches a server.
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir /wheels/*.whl vllm \
    && rm -rf /wheels /root/.cache
COPY server/ /app/
ENTRYPOINT ["python", "-m", "app.serve"]

Notice that the install and the cleanup are joined by && into one step. That’s the suitcase rule from earlier — deleting in a later step wouldn’t remove anything.

Ask of every file: does the running server ever read this? If not, it shouldn’t be in the box.

Cause 4: the model isn’t really part of the program

The biggest one, and the most natural mistake. A model’s weights are simply the numbers the model learned during training — they are the model. A 7-billion-parameter model stored at normal precision works out to roughly 14 GB of numbers.

Copying that file into your image feels obviously right. The model is part of the service, so it goes in the box. But remember that layers are frozen slices, and there’s no such thing as downloading half a slice.

Why any of this matters

Image size on its own is just a number to feel bad about. What it turns into is cold start time — the gap between “we need another server” and “that server can answer questions.”

Services scale up and down with demand. When traffic spikes, a new machine is assigned, and it has to download your image before it can do anything. On GPU hardware the billing clock starts the moment the machine is assigned, not the moment your model is ready. So every second spent downloading is a second of very expensive silicon sitting idle.

The weight shows up in quieter places too: every code change uploads 15 GB from your build system, your storage bill multiplies by every version you keep, and security scanners produce enormous reports listing hundreds of packages your server never even loads.

One trap worth warning you about

You’ll read that Alpine is the smallest Linux base at 7.8 MB, and be tempted. For Python and AI work it usually backfires.

Alpine uses a different version of a fundamental system library than the rest of Linux does. Python packages like NumPy normally arrive pre-compiled, which is why installing them is fast. On Alpine those pre-compiled versions don’t fit, so your machine downloads the raw source and compiles it — which drags in compilers you were trying to avoid, makes the image bigger than the normal small option, and turns a 40-second install into 20 minutes. Use the “slim” variants instead.

How small can it actually get?

Same model, same answers, every time                     Image size
-------------------------------------------------------  ----------
the usual way — everything in one box                    14.70 GB
build in a separate box, ship the runtime version        5.60 GB
+ model moved to cloud storage                           2.10 GB
running on ordinary processors, CPU-only PyTorch         780 MB
PyTorch dropped entirely, converted to a lighter format  310 MB
a compressed model in a small C++ server                 48 MB

You will not get a full GPU serving stack down to 48 MB, and anyone promising that is selling something. But the last two rows are real setups for real workloads. The question they answer is whether your service needs a whole training framework just to run a model that’s already trained. Very often it doesn’t.

What to try first

• Run docker history --no-trunc on your image, or install the free tool dive to browse inside it. Do this before changing anything — the biggest slice is usually a surprise, and it’s usually a forgotten cache.
• Add a .dockerignore file. Without one, Docker uploads your entire working folder — including saved models and your .git history — before the build even begins.
• Change your final starting image from the devel version to the runtime version and see what breaks. Usually nothing does.
• Check for duplicate libraries with pip list | grep nvidia.
• If you’re not using a GPU, install the CPU-only PyTorch.
• Compile things in a throwaway first stage, copy out only the result.
• Always pair an install with its cleanup in the same step, and add --no-cache-dir to pip.
• Move model files out to cloud storage and download them when the server starts.

None of this is difficult, and none of it makes your code uglier. If anything the two-box version is easier to read, because it says out loud which things are tools and which things are the product.

The 14.7 GB image and the 2.1 GB image were built from identical code by equally competent people. The only difference is that someone asked one question: does the running server ever actually open this file?