A KV cache is the memory where a language model keeps the attention keys and values it has already computed for earlier tokens, so it never has to compute them again. It makes generation fast, but it grows with every token and every user. On long contexts it, not the model's weights, decides how many people one machine can serve.

Cerebras CTO Sean Lie recently walked through exactly that limit in a 9-minute video, Deploying Trillion-parameter Frontier Models, with a companion post by Sarah Chieng. They take Kimi K3, an open-weights model with about 3 trillion parameters, and show how it fits on 59 of Cerebras's wafer-sized chips while serving about 190 long requests at once. I worked through every number and wrote this to explain the ideas underneath them, starting from scratch: the model, the KV cache, SRAM vs DRAM, linear attention and the pipeline. There is a calculator at the end.

All Kimi K3 deployment figures in this post come from Cerebras's video and post. The colours stay the same in every diagram: blue for weights, amber for KV cache, green for activations, violet for linear-attention state.

The short version in six steps

1. Weights: 3 trillion parameters × 4 bits = 1.5 TB. Spread over 93 layers, that is about 16 GB per layer.
2. Placement: a Cerebras wafer holds 44 GB of SRAM. Two linear-attention layers fit on one wafer (32 GB). Each dense layer gets a wafer to itself, because it also needs room for its KV cache.
3. Count: 69 ÷ 2 rounded up, plus 24 = 35 + 24 = 59 wafers.
4. Wiring: the weights never move. Only small per-token activations travel between wafers, over RDMA on Ethernet, at about 2 µs per hop.
5. Pipeline: each wafer is one station on an assembly line, and different requests occupy different stations at the same moment.
6. Concurrency: each dense wafer has 44 − 16 = 28 GB left. One 256k-token request needs 576 bytes × 256,000 ≈ 147 MB of KV cache per layer. 28 GB ÷ 147 MB ≈ 190 requests.

What is a language model, physically?

A large language model is a long list of numbers called parameters or weights, arranged into matrices. "3 trillion parameters" means 3,000,000,000,000 of them. Training chooses their values; inference uses them, unchanged, to predict the next token.

The model never sees letters. A tokenizer splits text into tokens and gives each an ID. Each ID becomes a vector of a few thousand numbers. That vector goes through a stack of layers; each layer reads it, mixes in information from earlier tokens, and passes on an updated vector. The last vector becomes a probability for every possible next token. One is picked, appended to the text, and the whole stack runs again.

Inside one layer

Each layer does two things in order. Attention lets the current token look back at earlier tokens and pull in what matters; in "the cat sat on the mat because it was tired", attention is how "it" picks up information from "cat". This is the only part that needs memory of the past, and that memory is the KV cache.

The feed-forward network then processes each token on its own and holds most of the parameters. Frontier models usually make it a mixture of experts: many smaller networks and a router that sends each token to only a few. The model can own trillions of parameters while doing a fraction of the arithmetic per token, but every expert still has to be stored, because any token may be routed to any of them.

How big is Kimi K3?

Kimi K3 has about 3 trillion parameters in 93 layers: 69 linear-attention layers and 24 dense (full-attention) layers. According to Cerebras, it scores 88.3% on Terminal-Bench 2.1 and 91.2% on BrowseComp. Its size on disk or in memory depends on how many bits each parameter gets. Cerebras assumes 4-bit weights, matching the published Hugging Face checkpoint, and an 8-bit KV cache.

3,000,000,000,000 params × 4 bits ÷ 8 bits per byte = 1.5 TB
1.5 TB ÷ 93 layers                                    ≈ 16 GB per layer

Even at 4 bits the weights are 34 times larger than one wafer. On GPUs the picture is similar. One 8-GPU B200 server has roughly 1.4 to 1.5 TB of HBM in total, which is not enough for the 1.5 TB of weights plus any cache. So Kimi K3's hardware requirement is at least two such servers, or a larger rack, before you serve a single long request.

Why 4 bits is usually fine: weights inside a layer cluster in a narrow range. Quantisation stores one shared scale for each small group of weights (often 32 to 128) plus a 4-bit value per weight, so the rounding error stays small. Moonshot trained Kimi K2 Thinking with quantisation in the loop for this reason.

Prefill vs decode: why memory speed decides

Serving a request has two phases. Prefill reads the whole prompt at once: all prompt tokens go through each layer together, so each weight loaded from memory is reused thousands of times. Prefill is compute-bound. Decode generates the answer one token at a time, and every token needs every layer's weights read again, used for a single token's arithmetic. Decode is memory-bandwidth-bound, and it is the phase users feel.

// rough ceiling for one stream, ignoring compute and batching
tokens/sec ≤ memory bandwidth ÷ bytes of weights read per token

// a dense 70B model at 4-bit reads 35 GB per token
one H100 (HBM, ~3.35 TB/s):  3,350 ÷ 35 ≈ 96 tokens/s at best

GPU servers work around this with batching: read the weights once and use them for 64 or 256 users' tokens together. Throughput rises, but each user still waits for a full weight read every step. Cerebras attacks the other side of the fraction and keeps the weights in much faster memory. I measured the same memory-bandwidth wall on consumer GPUs in CPU, GPU and CUDA fundamentals for engineers.

SRAM vs DRAM: what's the difference?

SRAM (static RAM) is memory built from transistors on the same silicon as the compute cores. It is very fast and close, but each bit takes about six transistors, so it is expensive in chip area. DRAM (dynamic RAM) stores each bit as charge in one tiny capacitor, so it is dense and cheap, but it lives on separate chips. GPUs use HBM, DRAM stacked right beside the processor: fast for DRAM, but every byte still crosses a chip boundary.

	On-chip SRAM	HBM (stacked DRAM)
Where	Same silicon as the cores	Separate chips beside the processor
Capacity	Tens of MB per GPU; 44 GB per Cerebras wafer	80–192 GB per GPU
Bandwidth	Petabytes per second across a wafer	About 3–8 TB/s per GPU
Cost per bit	High (about 6 transistors)	Low (1 transistor + 1 capacitor)
Result	Fast decode, but a big model needs many wafers	Fewer chips hold the model, but decode waits on memory

That is the real question behind the video: is SRAM too small for trillion-parameter models? Not if you use enough chips, connect them well, and lay the model out so the big data never has to move.

What is a KV cache, exactly?

In attention, each token produces three vectors in every layer. The query (Q) says what the token is looking for. The key (K) says what the token contains, for matching. The value (V) is what the token hands over if it is picked. To produce token 7, its query is compared with the keys of tokens 1 to 6, the scores become weights, and the output is a weighted blend of their values.

Old tokens' keys and values never change, so the server stores them instead of recomputing them at every step. That store is the KV cache.

How big is the KV cache?

The formula is in the figure: bytes per token per layer, times tokens, times attention layers, times requests. Cerebras gives Kimi K3's figure as 576 bytes per token per layer at 8 bits, which is unusually small. 576 = 512 + 64, exactly the layout of Multi-head Latent Attention (MLA), which DeepSeek introduced and Kimi K2 also uses. Instead of storing a full key and value for each of many attention heads, MLA stores one compressed 512-number vector per token, plus a 64-number piece that carries position.

KV per request, per dense layer   = 576 B × 256,000 tokens ≈ 147 MB
KV per request, all 24 dense layers = 147 MB × 24          ≈ 3.5 GB

The attention design matters a lot. Here is how many 256k-token requests fit in the 28 GB a dense wafer has free, with the same 8-bit storage. The first two rows are hypothetical configurations for comparison, not Kimi K3's.

A cache 28 times smaller means 28 times more simultaneous users per wafer, with no other change.

Why don't linear-attention layers need a KV cache?

Linear attention keeps one fixed-size matrix per request, called the state, instead of a list of every past token. Each new token is folded into the state, and reading is a multiplication of the query against it. The state is the same size at token 10 and at token 256,000.

The cost is precision. A fixed-size state is a lossy summary, so linear layers are worse at exact recall, such as quoting a line from 200,000 tokens back. That is why Kimi K3 is a hybrid: 69 cheap linear layers, with a full-attention layer every few layers (24 in all) that can still look back exactly. Moonshot's earlier Kimi Linear model used about three linear layers per full one, and 69:24 is close to that ratio.

How do 93 layers fit on 59 wafers?

Every layer is about 16 GB of weights. What else a wafer needs depends on the layer type.

69 linear layers ÷ 2 per wafer, rounded up = 35 wafers
24 dense layers  ÷ 1 per wafer              = 24 wafers
                                       total = 59 wafers

linear wafer, 2 layersdense wafer, 1 layer + KV cache

22222222222222222222222222222222221DDDDDDDDDDDDDDDDDDDDDDDD

Why move activations instead of weights?

An activation is the vector one layer hands to the next: the token's current representation, one number per hidden dimension. Kimi K2's hidden size is 7,168; at 16 bits that is about 14 KB per token. The wafer receiving it holds 32 GB of weights. Moving the activation is roughly two million times less data than moving the weights, so the weights stay put and only activations travel.

hops per token  = 58 (between 59 wafers)
network time    = 58 × 2 µs ≈ 0.12 ms
target speed    = 1,000–2,000 tokens/s → 0.5–1 ms per token
network share   ≈ 12–23% of the per-token budget

The network matters but does not dominate. The 14 KB figure is my estimate from Kimi K2's published hidden size; Cerebras only says that activations are what move. I hit the opposite situation in running a 397B model across 20 GPUs through a relay, where each hop cost milliseconds and the network was most of the time per token.

Doesn't a pipeline leave most wafers idle?

Only if one request is in flight. The pipeline is kept full: while wafer 3 works on request A, wafer 2 works on B and wafer 1 on C. Here are four stages and five requests over seven time steps:

From t4 onward every wafer is busy with a different request. In decode, a request leaving the last stage with a new token re-enters stage 1 for the next one, so the line stays full. Each stage can also take a micro-batch of several requests at once, which gives Cerebras's rule of thumb:

requests in flight ≈ number of stages × micro-batch size

Smaller micro-batches move each request through faster, which is better for speed per user. Larger micro-batches serve more tokens per weight read, which is better for total throughput.

How many requests can it serve at once?

Memory sets the ceiling, not compute. Every request in flight needs its own KV cache on every dense wafer.

space per dense wafer = 44 GB − 16 GB (weights) = 28 GB
cache per request     = 576 B × 256,000 tokens    ≈ 147 MB
concurrent requests   = 28 GB ÷ 147 MB            ≈ 190

• 256k is an assumed average. A 32k-token chat needs one eighth of the cache, so about 1,500 would fit by memory alone.
• This is not a GPU batch. The 190 requests do not start and stop together; they stream through the pipeline, and a finished one is replaced immediately.
• Concurrency is not the efficiency metric. Cerebras argues for tokens per second per megawatt, measured at the same per-user speed. A GPU cluster can hold more users at lower speed; the fair question is what each delivers at, say, 1,000 tokens per second per user.

How do GPUs serve a model this size?

A GPU deployment splits the 1.5 TB several ways at once. Tensor parallelism slices every layer's matrices across 8 or 72 GPUs joined by NVLink, which exchange partial results after each layer. Expert parallelism gives different GPUs different experts. Pipeline parallelism puts groups of layers on different nodes, the same idea Cerebras uses, usually with far fewer stages.

	GPU cluster	Cerebras wafers
Main split	Each layer sliced across many GPUs	Whole layers per wafer, chained
Weights read from	HBM, every step	On-wafer SRAM
What crosses the network	Partial results after every layer	One activation per hop
How it gets efficient	Large batches share each weight read	Full pipeline, fast memory
Strength	Throughput and capacity per chip	Speed per user

GPU serving engines such as vLLM and SGLang add a few more techniques. Continuous batching lets new requests join a running batch at any step. Paged KV cache (PagedAttention) stores the cache in fixed-size blocks so long and short requests share memory without fragmentation. Prefill/decode disaggregation runs the two phases on separate GPU pools. Speculative decoding lets a small model draft several tokens for the big one to check in a single pass.

How is training different from inference?

Inference runs the model forward. Training runs forward on known text, measures the error (the loss), runs backward through every layer to get each weight's gradient, and lets an optimiser such as Adam update every weight, over trillions of tokens. It must also keep every layer's activations from the forward pass until the backward pass uses them.

A common mixed-precision recipe needs about 16 bytes per parameter: 2 for the weight, 2 for its gradient, 4 for a 32-bit master copy and 8 for Adam's two running averages. For 3 trillion parameters that is about 48 TB before activations, 32 times the 1.5 TB needed to serve it at 4 bits. This is a standard rule of thumb, not a figure from Cerebras.

	Training	Inference (decode)
Passes	Forward, backward, update	Forward only
Tokens per step	Millions, in parallel	One per request
Bottleneck	Compute and chip-to-chip bandwidth	Memory bandwidth and latency
Memory per parameter	About 16 bytes plus activations	About 0.5 bytes plus KV cache

What changes at 10T and 100T parameters?

The video ends on 10-trillion-parameter models and beyond. The method stays the same: work out memory for weights and per-request state, spread it across wafers, chain them. The arithmetic below is mine, at 4-bit weights.

Model	Weights at 4-bit	Wafers for weights alone	With a similar layout
3T (Kimi K3)	1.5 TB	at least 35	59 (Cerebras)
10T	5 TB	at least 114	about 190–200
100T	50 TB	at least 1,137	about 2,000

Two things change. At 10T with 93 layers, one layer would be about 54 GB, larger than a wafer, so layers must be split or made thinner and more numerous. And a longer pipeline means more hops: at 200 hops, the network alone costs 0.4 ms per token, so per-hop latency becomes a headline number.

KV cache and wafer calculator

Change any input to see how wafer count and concurrency move. The defaults reproduce Cerebras's Kimi K3 figures. It uses their simple rules: whole layers per wafer, one dense layer per wafer, KV cache only on dense wafers. It shows 189 rather than 190 because it keeps the layer size at 16.1 GB instead of rounding to 16.

Glossary

Parameter / weight

One learned number in the model, fixed during inference.

Activation

The vector a layer passes to the next for one token. Small, temporary, different for every token.

Q, K, V

Query, key and value. A token's query is matched against earlier keys, and the matching values are blended.

KV cache

Stored keys and values of earlier tokens, kept so they are not recomputed. Grows with context length and with the number of requests.

MLA

Multi-head Latent Attention: stores one compressed vector per token instead of full per-head keys and values.

Linear attention

Attention that keeps a fixed-size running state instead of a growing cache.

Prefill / decode

Reading the prompt in parallel / generating output one token at a time.

SRAM / DRAM / HBM

On-chip fast memory / dense off-chip memory / DRAM stacked beside a GPU.

Wafer-scale engine

Cerebras's processor: a whole silicon wafer used as one chip, with 44 GB of SRAM among its cores.

RDMA

Remote direct memory access: writing into another machine's memory without CPU copies.

Micro-batch

A small group of requests one pipeline stage processes together.

If you are new to how language models work at all, start with how LLMs work, from tokens to transformers to memory. If the hardware side is new to you, start with how CPUs, GPUs and memory bandwidth work, then the hands-on running half a language model on a CPU and a GPU, or the interactive course on how LLMs work.

Sources

• Cerebras, Cerebras CTO Sean Lie: Deploying Trillion-parameter Frontier Models (video, 9:12).
• Sarah Chieng, Cerebras, How we deploy Trillion-parameter frontier models on Cerebras, 22 October 2026. Source of every Kimi K3 deployment figure here: 1.5 TB, 16 GB per layer, 59 wafers, 576 bytes, 147 MB, 28 GB, about 190 requests, about 2 µs per hop.
• GPU memory figures (H100, B200), the MLA, GQA and linear-attention comparisons, the activation size, training memory and the 10T/100T table are general background or my own arithmetic, labelled as such where they appear. I drafted this post with help from Claude and checked every figure against the sources above.