A language model is really just a very long list of numbers, called weights, that decide how it answers. Llama 3.1, a well-known open model, has about eight billion of them. Each number is normally written down with sixteen bits of computer memory, so the whole list takes up sixteen billion bits, roughly 16 GB as a file. Quantisation writes those same numbers down less precisely, so they take less room. Round every one of Llama 3.1's weights to four bits instead of sixteen, and the file shrinks to about 4.9 GB. No weight is deleted. Each one is simply written more roughly.
Why it matters
That shrinking is what lets a model run on an ordinary computer instead of a data centre. Every one of a model's weights has to sit in memory at once while it works. A laptop with 16 GB of memory, a common amount, cannot fit a 16 GB model and still run anything else. A 4.9 GB model fits easily, with room to spare. That is the whole trick behind local AI: chatting with a model that lives entirely on your own machine, with no message sent anywhere and nothing to pay per word.
A smaller file is also a faster one. To produce each new word, a model has to read through all of its weights, and that reading usually takes longer than the arithmetic that follows. Fewer bytes to read means a shorter wait. In a 2023 test on a MacBook, a 4-bit version of a similar-sized model produced a word in 55 milliseconds; the full-size version took 127.
The saving does cost something: a small amount of quality. How much depends on how far the rounding goes.
How it works in outline
Here is the idea with one weight. Say a weight is stored as 0.72. Instead of keeping all those digits, quantisation rounds it to the nearest mark on a much simpler ruler, one with sixteen marks spaced 0.1 apart. 0.72 lands nearest the seventh mark, so that mark's number, 7, is all that gets stored. A whole group of nearby weights shares the same ruler, so its spacing, 0.1, is written down once for the whole group rather than once per weight. To use the weight again, the model just multiplies the mark by the spacing: 7 times 0.1 is 0.7, close to the original 0.72 but not exact. The missing 0.02 is the cost of rounding.
A model's answer comes from billions of these weights acting together, so rounding any one of them barely shows. Researchers have also found ways to protect the handful of weights that matter most, leaving those closer to their original value while rounding the rest more freely, which keeps quality high even when most of the model has been rounded down a lot.
The table below shows what that rounding actually costs, measured on an 8-billion-weight Llama 3 model.
| Rounding level | File size | Compared with the original |
|---|---|---|
| 8 bits | 7.96 GB | As good as the original |
| 4 bits | 4.58 GB | Almost as good |
| 3 bits | 3.74 GB | A little worse |
| 2 bits | 2.96 GB | Clearly worse |
Where you meet it
In the names of downloadable models: a file called q4_K_M is one rounded to about four bits, and it is what Ollama gives you by default when you ask for a model by name, at roughly a third the size of the full version. In the apps that run models on your own computer: Ollama, LM Studio, and anything built on llama.cpp, a project whose original goal, in 2023, was simply to run a well-known model on a MacBook. And on your phone: Apple said in 2025 that it now rounds its on-device assistant down to about two bits per weight, with only a small effect on quality, which is a large part of how a useful assistant fits on an iPhone at all.