A language model writes an answer one token at a time. A token is a small piece of text. To choose the next one, the model needs information from the words that came before it.
The KV cache holds that information. It is a set of working notes stored for the current chat. A longer chat creates more notes, so it needs more memory.
This became news on 10 September 2026, when DeepSeek released V4.1-Flash. DeepSeek says the new model uses about one-quarter of what it calls the global KV-cache memory, and one-eighth of the long-lived storage, compared with V4-Flash.
Think of it as a box of index cards
This AI-generated illustration turns the KV cache into a box of index cards. The cards are a teaching analogy. A real cache holds rows of numbers called keys and values.
Here is what happens:
- The model reads the prompt.
- Each attention layer, one of the model's stages for looking back at earlier tokens, makes a key and a value for every token.
- The model saves those rows of numbers in the KV cache.
- When it writes the next token, it reuses the saved rows and adds a new pair.
Reuse saves time because the model does not have to rebuild the same rows of numbers at every step. The trade-off is memory. Each active chat has its own growing cache. The AI chips also have to load the saved data while writing the answer.
This affects both space and speed. Memory space helps decide how many chats a server can hold at once. Memory bandwidth tells us how quickly the chips can fetch their saved notes. Wasted or copied cache space can also reduce the number of chats served together.
Two numbers with different jobs
Large AI models often come with two size numbers: a total parameter count for the whole model and an active parameter count for the part working on one token. In a mixture-of-experts model, only some parts work on each token. KV-cache size describes the attention notes saved for each chat. These numbers describe separate costs of running the model.
DeepSeek describes V4.1-Flash as a main model it calls the backbone, with 552 billion parameters. It also has a separate memory system called Engram, with 196 billion parameters. DeepSeek says about 8 billion parameters are active while reading the prompt and about 16 billion are active while writing the answer. Those active counts do not tell us the KV-cache size. The attention design, chat length, number of active chats, and number format all affect that bill.
Why the cache grows
The full engineering formula has several parts. For a first look, we can group them into one simple idea:
cache memory = size of one token's saved notes × number of tokens × number of chats
More tokens make each chat's cache longer. More chats create more separate caches. A model with more attention layers also saves more information for each token. Storing each number in fewer bits can shrink every saved note.
The hand-drawn example below uses the published settings of a model called Llama 3.1 8B. It also assumes that its cache uses bfloat16 numbers, which take two bytes each. Under that assumption, one token needs 128 KiB across all layers. One KiB holds 1,024 bytes, slightly more than a kilobyte. A chat with 8,192 tokens needs 1 GiB, or about one gigabyte, for the KV cache alone.
How DeepSeek says it made the cache smaller
DeepSeek calls its new layout a Causal Encoder-Decoder. The encoder is the half that reads the prompt. The decoder is the half that writes the answer. V4.1-Flash has 20 layers in each half.
DeepSeek reports four main changes:
- The encoder builds the attention notes that DeepSeek calls the global cache. The decoder will use them, so most prompt tokens can skip a full trip through the decoder.
- A method called CSA2 lets different layers reuse the same global notes.
- The cache stores numbers in FP4, a 4-bit format. Four bits take half a byte.
- Short-lived notes for nearby tokens go into the server's main memory. The system can rebuild a small missing section when it needs it.
DeepSeek reports that the final global cache uses 890 bytes per token. At that rate, one million tokens need 0.89 GB of global cache before other serving costs are added.
The chart below turns DeepSeek's two comparisons into an easy scale. V4-Flash is set to 100. On that scale, one-quarter becomes 25 and one-eighth becomes 12.5. HBM is high-bandwidth memory packaged close to the AI chips. DeepSeek's long-lived global cache sits on an SSD storage drive. The chart labels it "Persistent SSD." The nearby-token notes in the server's main memory are short-lived.
Show the numbers
| V4-Flash | V4.1-Flash | |
|---|---|---|
| Global HBM | 100.0 | 25.0 |
| Persistent SSD | 100.0 | 12.5 |
What outside checks tell us
As of 13 September 2026, the available independent sources had not measured the 890-byte cache or the one-quarter and one-eighth claims.
Artificial Analysis offers a smaller outside check. It compared V4.1-Flash with the April 2026 V4-Pro. On 13 September 2026, its quality index scored them 40 and 31. It measured output speeds of about 214 and 76 tokens per second. This comparison supports part of DeepSeek's speed and quality story. It does not test the cache figures.
DeepSeek's own benchmark table gives a mixed result. V4.1-Flash leads some tests, while V4-Pro leads others. DeepSeek also warns that its cache-saving methods can make mistakes in cases the company has not tested.
If other teams reproduce the memory results, a server could keep more long chats ready at once and save less data in long-lived storage. The real savings will change with chat length, the number of active users, the hardware, and the serving software.