Inference and Deployment Constraints

A deployed model is judged on more than accuracy. Here are the constraints that come up in almost every project, and the questions behind them:

They pull against each other. A bigger model may be more accurate but slower and hungrier; batching raises throughput but makes each request wait for its batch; quantising saves memory but may cost accuracy. Engineering a feature means choosing a point among these trade-offs, and that needs measurements, which is what this lesson provides.

Figure 7.6.1. Deployment choices trade quality, latency, throughput and memory. The directions shown are engineering considerations to test on the target workload, not measured universal laws.

We'll use the fixed IMDb split from Lesson 7.1 (the function is copied here; the fingerprints confirm the reviews are the same).

Latency is the time from receiving one request to returning its answer: what a single user experiences. Throughput is how many requests the system completes per second: what the operator pays for. They're related but not the same, and improving one can hurt the other.

We load the sentiment model explicitly, as in Lesson 7.4, with a small helper that runs a list of texts and returns their logits.

Now we send 100 validation reviews, one request at a time, and time each one. We report the median, the 95th percentile (p95: 95% of requests were faster than this) and the slowest.

Latency is not a single number. Some requests take noticeably longer than the median, and in a real service the slow ones are what users complain about. That is why services set targets on high percentiles ("p95 under 50 ms") rather than on the average. Here the main cause of the spread is easy to see:

Latency rises in a nearly straight line with the number of tokens in the request, as Lesson 7.3 predicted for a model of this size. The cluster on the right is the 40 of 100 reviews that hit the 256-token limit: truncation also caps the worst-case latency. Lowering max_length is therefore a direct latency lever, paid for in accuracy on long reviews (Lesson 7.3, Section 4).

Serving requests one at a time is simple, but it leaves the hardware under-used. If many requests arrive together, we can batch them (Lesson 7.4):

Batching 32 reviews at a time, sorted by length, processes more reviews per second than handling them one by one. That's the throughput side. The latency side is the catch: in a live service, a request that arrives first has to wait for the batch to fill before it is processed. Real servers use dynamic batching: collect whatever requests arrive within a short window (a few milliseconds), run them together, and return each result. A larger window gives bigger batches and more throughput, at the price of a little extra latency for everyone.

Key terms

• Latency: the time to serve one request.
• Throughput: the number of requests served per second.
• Percentile (p95): the value below which 95% of measurements fall; used to describe the slow end of latency.
• Dynamic batching: grouping requests that arrive close together in time into one batch.

Common mistake: reporting only average latency. A service whose average is fine but whose slowest 5% of requests take ten times longer will still frustrate users. Report percentiles, and look at what makes the slow requests slow.

A GPU (graphics processing unit) has thousands of simple arithmetic units that work in parallel, and a Transformer is mostly large matrix multiplications, which parallelise very well. So GPUs usually run Transformers much faster than CPUs, if they are given enough work at once. That last condition is why batching matters more on a GPU: a single short request leaves most of a GPU idle, while a batch of 32 or 64 keeps it busy. On a CPU, as we just measured, batching gives a smaller gain.

The cell below compares the two if your machine has an NVIDIA GPU; otherwise it says so and skips the timing.

These notes were run on a CPU, so the comparison was skipped. If your machine has an NVIDIA GPU, you'll see both numbers, and you'll typically find the GPU many times faster on the batch. A few practical points either way:

• A GPU has its own memory. The model and every batch must fit in GPU memory, which is often smaller than the computer's main memory. Section 4 shows how to estimate that.
• Moving data costs time. Inputs travel from main memory to the GPU and results travel back, which is why to(device) sits around every batch, and why tiny requests gain little.
• Cost and availability differ. GPUs are expensive to buy or rent, and a small model like ours may serve its traffic perfectly well from cheap CPU machines. Measure before assuming you need one.
• Other accelerators exist. Apple computers have their own GPU, which PyTorch calls mps. That's why Lesson 7.4 asked you to pass device= explicitly, so the same code doesn't silently run on different hardware for different people.

Key terms

• GPU: a processor with many parallel arithmetic units, well suited to the matrix multiplications in neural networks.
• GPU memory: the GPU's own memory, which must hold the model and each batch.

Since Lesson 7.1 we have estimated model size as parameters × bytes per parameter. Let's check that rule against the model's actual tensors in three number formats.

The rule is exact: 66,955,010 parameters take 267.8 MB in float32 (4 bytes each) and 133.9 MB in either 16-bit format (2 bytes each). Halving the bytes halves the memory. Remember that this is the weights only: running the model also needs memory for activations, which grows with batch size and length (Lesson 7.4, Section 6), and generation adds the KV cache (Section 6 below).

Fewer bits means each number is stored less exactly, or within a smaller range. The two 16-bit formats make different trade-offs:

The first three lines show each format's limits:

• float32 can represent numbers up to about 3.4e+38 with a relative precision of about 1.2e-07, roughly seven significant digits.
• float16 keeps reasonable precision (9.8e-04, about three digits) but its range stops at 6.55e+04. A larger number becomes inf, infinity: 70000 is stored as inf.
• bfloat16 keeps float32's full range (70000 is stored as 70144.0, slightly off but not infinite) but has less precision (7.8e-03, about two digits): adding 0.001 to 1 changes nothing, giving 1.000000.

For neural networks, range is often the bigger danger, since an overflow to infinity breaks a computation completely. So bfloat16 has become the common choice for training and running large models on hardware that supports it, such as the Qwen models in Lesson 7.3, which are stored in BF16. Weights themselves cope well with reduced precision, which is what makes the next section possible.

Key terms

• float32, float16, bfloat16: 32- and 16-bit floating-point formats; bfloat16 keeps float32's range with less precision, while float16 keeps more precision with a much smaller range.
• Precision (relative precision): the smallest relative difference a format can represent between two nearby numbers.
• Range: the largest (and smallest) magnitudes a format can store before overflowing to infinity.
• Overflow: a value too large for the format, stored as infinity.

16-bit formats halve memory. Quantisation goes further by storing weights as small integers, usually 8-bit integers (int8), whole numbers from −127 to 127, which take 1 byte each. The trick is a scale: we choose a number s so that the largest weight maps to 127, divide every weight by s, round to the nearest whole number, and store the integers plus s. To use the weights, multiply back by s.

Here it is on four example weights:

The largest magnitude is 1.27, so the scale is 1.27 / 127 = 0.01. Dividing by it gives 80, −30, 5 and −127, all whole numbers that fit in one byte, and multiplying back gives exactly the original weights. (The long decimals in the first line are float32's own rounding of 0.8, 0.3 and so on, from Section 4.2.) Real weights won't divide so neatly, so rounding loses a little. Let's do the same to a real layer, the first feed-forward layer of DistilBERT's first Transformer block, with one scale per output row (per-channel scales), since rows can have very different ranges:

The layer's 3072 × 768 weights shrink from 9.44 MB to 2.36 MB, plus 12.3 kB of scales. The price is rounding error: on average 2.84e-04, which is 0.87% of a typical weight's size. That's small, but errors in thousands of weights add up through six layers, so the real test is the model's output.

The function below applies the same rounding to every nn.Linear layer in a copy of the model: the attention projections, the feed-forward layers and the classifier. To keep the code simple, it puts the rounded-and-rescaled values back as float32 (so the model runs as before), and it counts what int8 storage would take.

Rounding the Linear layers to int8 takes the model from 267.8 MB to 138.8 MB, about half, not a quarter. The reason is in the second line: the embedding tables (23,834,112 numbers, 95.3 MB) aren't Linear layers and stay in float32. This is Lesson 7.1's observation that a third of DistilBERT is embeddings, showing up again in practice.

Size is only good news if the answers survive. We compare the int8 model with the original on the first 300 validation reviews:

The original scores 0.873 on these 300 reviews, and the int8 version 0.877. They agree on 0.997 of the labels, which is all but one review, and no probability moved by more than 0.0804. With 300 reviews, one review is worth about a third of a percentage point, so the tiny accuracy difference is one review changing sides, not a real improvement. Quantisation cost us essentially nothing here.

In practice you don't write quantisation by hand. torchao, PyTorch's quantisation library, does the same int8 weight-only quantisation in one line, replacing each Linear layer's weights with real int8 tensors:

torchao's model saves to 138.9 MB, matching our hand-built estimate, with the same accuracy (0.877) and the same agreement (0.997) with the original. Our by-hand version was the core idea; the library adds efficient storage and computation.

Does a smaller model run faster? Let's time the original, a bfloat16 copy and the torchao int8 model on the same 16 reviews:

On this CPU, neither smaller version is faster. The bfloat16 copy is several times slower than float32, and the int8 model is somewhat slower too. Speed depends on whether the hardware and software have fast routines (kernels) for the number format. This processor and this PyTorch build are heavily optimised for float32 matrix multiplication. Many GPUs and some server CPUs have dedicated fast paths for 16-bit and int8 arithmetic, and there these formats are typically faster as well as smaller. So the reliable benefit of quantisation is memory, while the speed benefit depends on the hardware. Measure on the machine you will deploy to.

Figure 7.6.2. Illustrative symmetric rounding with scale 0.01. The exact reconstruction arithmetic is computed in the figure script; it is separate from the notebook's measured model experiment.

Key terms

• Quantisation: storing numbers with fewer bits, here as 8-bit integers plus a scale.
• int8: 8-bit integers, 1 byte each; we use −127 to 127.
• Scale: the number that converts between the original values and the integers.
• Per-channel quantisation: using a separate scale for each row (output channel) of a weight matrix.
• Weight-only quantisation: quantising the stored weights while computing with higher-precision activations.
• Kernel: a low-level routine that performs an operation, such as matrix multiplication, for a particular hardware and number format.

Common mistake: assuming quantisation always makes a model faster. It always reduces memory, but speed depends on whether your hardware has fast int8 or 16-bit kernels. On this CPU, both smaller formats were slower than float32.

Visual walkthrough: Round, reconstruct, measure error.

1. Choose an illustrative scale. w = 0.806 · scale = 0.01. This scalar example illustrates uniform rounding. The notebook separately measures an actual quantized model.

2. Round to an integer. q = round(0.806 / 0.01) = 81. 81 is inside signed int8 range. A full quantizer must define clipping and how scales are selected.

3. Reconstruct the value. ŵ = 81 × 0.01 = 0.810. The integer and scale together approximate the original value.

4. Measure reconstruction error. ŵ − w = 0.004. Smaller weight error need not imply an equal change in application quality. Measure quality, latency and peak memory on the intended workload.

The app adds a paused animation of this same supplied example.

The cheapest computation is the one you don't do. Sections 2 and 3 covered batching, which does the same work more efficiently. This section covers two kinds of caching, which avoid repeating work at all.

In many applications the same inputs come up again and again: popular products, common questions, a viral review. A response cache stores each answer the first time it is computed and returns it directly for later identical requests. We simulate 1,000 requests drawn from 200 reviews, where, as in real traffic, a few reviews are requested far more often than others:

Only 163 distinct reviews were requested among the 1,000 requests, so with a cache the model runs 163 times and the other 83.7% of requests are answered instantly from memory. How well a cache works depends entirely on how repetitive your traffic is, so measure that on real request logs before building one.

Notice the cache key: it contains the model name and the max_length setting as well as the text. If you update the model or change a setting, old cached answers must not be served as if they came from the new configuration. Forgetting this is a classic source of confusing bugs.

Generation has its own, internal cache. In Lesson 6.6 we saw that a decoder generates one token at a time, and that each new token attends to all the previous ones. Without a cache, every step reruns the model over the whole text so far, recomputing keys and values for tokens that haven't changed. The KV cache stores each token's keys and values once and reuses them, so each step only processes the one new token. Let's time distilgpt2 generating 100 tokens both ways:

With the cache, generation is several times faster, and the text is identical: the cache changes how much work is done, not the result. Counting the token positions the model processes shows why:

For a 10-token prompt and 100 new tokens, generating without the cache processes 5,950 token positions, because every step re-reads everything before it. With the cache it processes 109: the prompt once, then one new position per step. The timing gap was smaller than this count suggests, because each cached step also has to attend over all the stored keys and values, and each step has fixed overheads. The longer the text, the bigger the saving. The KV cache does cost memory, though: it holds keys and values for every layer and every past token, and for long conversations with large models it can rival the weights themselves in size.

Put together, a well-run inference service keeps the hardware busy with useful work. It batches requests so the processor is fully used, sorts or buckets by length so little of that work is padding (Lesson 7.4), caches repeated responses, uses the KV cache for generation, and sizes the model and its number format to the machine's memory. Each lever trades something away (batching adds waiting time, caching uses memory and risks stale answers, quantisation may cost accuracy), so the right mix depends on your traffic and your latency target.

Key terms

• Response cache: stored answers to previous requests, returned directly when the same request repeats.
• Cache key: the information that identifies a cached answer; it must include everything that affects the answer (input, model version, settings).
• Cache hit rate: the share of requests answered from the cache.
• KV cache: stored attention keys and values for past tokens, reused at each generation step.

Lesson 7.3 compared hosted APIs and local models in general terms. With the measurements from this lesson, we can be more concrete about what "running it yourself" involves.

Running locally means you own every constraint in this lesson: you measure latency and throughput, choose the hardware (CPU or GPU) and the number format, decide batching and caching, size the machines for peak traffic, and keep them running. In return you get control: the exact model version, data that never leaves your systems, and costs that are mostly fixed regardless of how many requests you serve.

Using a hosted service means the provider owns most of them: they choose hardware, batching and quantisation, and you pay per request or per token. You give up control over versions and data location, and you add a network round trip to every request's latency.

A useful way to decide is a break-even estimate: work out what a hosted service would charge for your expected monthly requests, and compare it with the monthly cost of machines that can serve the same traffic, using throughput measured as in Section 2. Below the break-even volume, hosted is usually cheaper and simpler; above it, local usually wins, if you have the skills to run it.

Key terms

• Self-hosting: running a model on machines you control.
• Break-even volume: the request volume at which self-hosting and a hosted service cost the same.

A deployed model must give the same answer to the same input, both from one request to the next and from one month to the next, unless you decide to change it. Three things make that possible.

Record exactly what you ran. A short manifest written next to your results lists the model and its exact commit, the settings, the evaluation data's fingerprint, the result, and the software versions:

Everything needed to reproduce the 0.8733 accuracy is here: the repository and the commit 714eb0fa… (Lesson 7.1, Section 7), max_length 256, which reviews were used (with a fingerprint, 52b8a6ed1ef8, to check they're the same), and the Python, PyTorch and Transformers versions. Saved as a JSON file next to a deployed model, a manifest like this answers "which model produced this?" months later.

Check that inference is deterministic. In eval mode, the same input should give exactly the same output (Lesson 7.4):

The same review gives bit-identical logits twice in a row. Processed inside a padded batch of 8, its logits differ by at most 1.91e-06, the tiny floating-point differences that come from doing arithmetic in a different order. That level is harmless for a label, but it means "identical" comparisons in tests should use a tolerance, and it explains why a score can wobble in its last digits between a batched and an unbatched service.

Pin the software too. Library updates can change tokenisation, defaults or numerical details. A requirements.txt with exact versions, like the one that comes with these lessons, together with the pinned model commit, is what makes a result reproducible by someone else.

Key terms

• Manifest: a small file recording the model, its exact version, settings, evaluation data and software versions behind a result.
• Deterministic inference: getting exactly the same output for the same input every time.
• Tolerance: the small difference allowed when comparing floating-point results.

Common mistake: recording only the model's name. A name like distilbert-base-uncased-finetuned-sst-2-english can point to different weights over time; the commit ID, the settings and the library versions are what pin a result down.

Let's reassemble the deployment picture.

A model becomes a feature only when it meets its constraints. Latency is a distribution, not a number. For our model it rose in a straight line with request length, and truncation capped the slow end; services set targets on percentiles such as p95. Throughput rises with batching, at the cost of waiting time; on a GPU, batching matters even more. Memory follows parameters × bytes exactly: 267.8 MB in float32, 133.9 MB in float16 or bfloat16, where bfloat16 keeps the range (70000 stays finite) and float16 keeps the precision.

Quantisation stores weights as int8 plus a scale. By hand and with torchao, it took DistilBERT to 138.8–138.9 MB, not a quarter, because the 95.3 MB of embeddings stayed float32, and the answers agreed with float32 on 0.997 of 300 reviews. On this CPU neither 16-bit nor int8 was faster: memory savings are reliable, while speed depends on the hardware's kernels. Caching avoids work entirely: a response cache answered 83.7% of a skewed request stream without the model, and the KV cache cut generation from 5,950 processed positions to 109, with identical text.

Hosted or local decides who owns these constraints, and a break-even estimate built on measured throughput decides which is cheaper. Reproducibility comes from a manifest (model commit, settings, data fingerprint, versions), deterministic eval-mode inference, and pinned software.

Figure 7.6.3. Source CPU reference measurements and counts. Weight storage is not peak runtime memory, and positions processed with a KV cache are not a direct FLOP or speedup ratio.