Imatrix vs Static Quants: 20% Less Drift, Same File Size

The static Q4_K_M I made tonight says in its own header that it was built with an importance matrix. It was not. I had quantized bartowski's Q8_0 of Llama 3.1 8B into Q4_K_M twice, once with the imatrix file bartowski published and once without, and llama-quantize copied the source's metadata into both outputs. The source carries bartowski's four quantize.imatrix.* keys, so the static file inherited them. The only record of the difference was the quantize log: one run printed loaded 224 importance matrix entries, the other printed nothing about a matrix.

Those keys are how you are supposed to tell imatrix vs static quants apart, because the two files are otherwise twins: the same size to within 32 bytes, the same 292 tensors in the same encodings. What differs is how close their output stays to the model they came from, and that I could measure. Against Q8_0 on WikiText-2, the imatrix Q4_K_M drifted 20% less than the static one, and Q3_K_M showed the same gap. Reading the headers of files people actually download turned up one more thing: the Llama 3.1 that ollama run llama3.1 pulls is a static quant.

What an importance matrix changes inside a Q4_K_M file

A K-quant stores weights in small blocks that share a scale, and quantizing a block means choosing the scale and the rounded values that lose the least. "Least" needs a weighting, and that weighting is the only thing the matrix touches. In llama.cpp's ggml-quants.c, a Q4_K block quantized without a matrix weights each value by its own magnitude, av_x + fabsf(x), so big weights are rounded more carefully than small ones. With a matrix the weight becomes qw[l] * sqrtf(sigma2 + x*x), where qw was recorded by running real text through the model and measuring how strongly each input channel fires, so a small weight that meets large activations gets protected.

The matrix is the diagonal of those activation statistics, one number per input column of each weight tensor, which is how its author, ikawrakow, described it in pull request #4861, merged in January 2024. bartowski's matrix for Llama 3.1 8B is a 4,988,170-byte file with 224 entries, computed on 125 chunks of text. It does not change the file format. Both of my Q4_K_M files hold 193 tensors in Q4_K and 33 in Q6_K at 4.894 bits per weight, the same layout I counted in bartowski's file, so they run through the same kernels. One type ignores the matrix entirely: quantize_q8_0 opens with (void)quant_weights; // not used, so an imatrix Q8_0 and a static Q8_0 are the same file.

The test: one source file, one flag

To isolate the matrix I needed two files that differ in nothing else. F16 Llama 3.1 8B is 16 GB and does not fit on this 16 GB Mac mini, so both arms start from bartowski's Q8_0, the reference file from my Q4_K_M vs Q8_0 test earlier tonight. llama-quantize only accepts that with --allow-requantize, and warns that it "can severely reduce quality compared to quantizing from 16bit or 32bit". The warning applies to both arms equally, so it does not tilt the comparison. I made Q4_K_M and Q3_K_M, each with and without the matrix:

llama-quantize --allow-requantize --imatrix Meta-Llama-3.1-8B-Instruct.imatrix \
    Meta-Llama-3.1-8B-Instruct-Q8_0.gguf rq-Q4_K_M-imat.gguf Q4_K_M
llama-quantize --allow-requantize \
    Meta-Llama-3.1-8B-Instruct-Q8_0.gguf rq-Q4_K_M-noimat.gguf Q4_K_M

The matrix made quantizing slower, 68 seconds against 48 for Q4_K_M and 47 against 27 for Q3_K_M. It changed each output's size by 32 bytes, all of it in the header, where the imatrix run stored a shorter file path. Then I scored every file against Q8_0 with the KL-divergence method in llama.cpp's perplexity documentation, the one I used for the KV cache test: 16 chunks of 2,048 tokens from the WikiText-2 test set, and 16 chunks of llama.cpp's own ggml-quants.c, to see whether the effect survives outside encyclopedia prose.

One check came first. A matrix built on the test text would flatter itself, and llama.cpp's own comparison does exactly that: its "WT 10m" matrix was computed on Wikitext and scored on Wikitext. bartowski's matrix records its source as calibration_datav3.txt, published in a gist in May 2024 and byte-identical today. It is 42,538 words, mostly English prose, math exercises and children's stories, with about a dozen lines of code. It shares no 12-word sequence with the WikiText-2 test file, so the wiki numbers below measure a matrix scoring text it never saw.

Results: the static files drift about 20% further

Scored against Q8_0 on WikiText-2, the imatrix Q4_K_M had a mean KL divergence of 0.0173 and the static one 0.0216, so the matrix removed 20% of the drift. Q3_K_M moved by the same share, 0.0572 against 0.0714. The tail moved more than the average. On the worst 0.1% of tokens the static Q4_K_M reached 0.584 and the imatrix file 0.416, 29% lower, and those rare tokens are where a quantized model drops a wrong word into an otherwise good answer.

The first five columns are WikiText-2; on the C file every number is smaller, because code is easy to predict (Q8_0's own perplexity there was 1.53, against 6.38 on WikiText). The matrix still helped, 17% at Q4_K_M (0.0068 against 0.0082) and 22% at Q3_K_M (0.0200 against 0.0256), so the effect held on text the calibration file barely covers. Perplexity alone would have hidden the Q4_K_M difference on code: +0.71% against +0.83%, with error bars of about ±0.1% on each.

Three things keep this in proportion. A static Q4_K_M is still far closer to Q8_0 than an imatrix Q3_K_M: the matrix does not buy a size class. llama.cpp's own table for Llama 3 8B, scored against F16 with a matrix built from Wikitext, found 10% less divergence at Q4_K_M and 17% at Q3_K_M; my Q4_K_M effect is twice theirs, and my setup differs from theirs in three ways at once (the reference, the source file, the model), so I cannot say which one explains it. And the requantized imatrix file landing on bartowski's F32-made file, 0.0173 against 0.0174, looks like requantizing cost nothing, but the reference is Q8_0 itself, which favors any file made from it, so I would not read it that way.

How to tell whether a GGUF is imatrix or static

When you pass --imatrix, quantize.cpp writes four header keys: quantize.imatrix.file, quantize.imatrix.dataset, quantize.imatrix.entries_count and quantize.imatrix.chunks_count. When you do not, it writes nothing. The catch is one step earlier. llama-quant.cpp starts every output header with gguf_set_kv(ctx_out.get(), ml.metadata), a copy of every key in the source file, and the four keys only overwrite that copy. Quantize from a 16-bit source and the keys are honest, because the source has none. Requantize an imatrix file without a matrix, as I did, and the source's keys come along. bartowski's Q8_0 carries them even though Q8_0 never uses a matrix, so my static Q4_K_M names a file on bartowski's build machine, /models_out/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct.imatrix, that my run never loaded.

For published files the keys are good evidence, since the big uploaders quantize from 16-bit weights. To read them without downloading gigabytes I wrote a short Python parser that streams a GGUF over HTTP and stops where the metadata ends, 7.8 to 10.9 MB into these files. Here is what five files from four common sources say:

The Ollama row is the one I did not expect. ollama run llama3.1 pulls a 4,920,738,944-byte Q4_K_M whose header holds 29 keys, none of them about a matrix. By this test the default Llama 3.1 in Ollama is a static quant, 288 bytes smaller than bartowski's imatrix file of the same type, the one I have run since my first Llama 3.1 8B post. Ollama builds its library files itself and does not publish the command, so the header is all I can go on. Keys can be inherited by accident, as mine were, but they do not vanish unless something removes them.

What people get wrong about imatrix vs static quants

The naming invites confusion. mradermacher is explicit: the i1 repository holds "weighted/imatrix quants" and "static quants are available at" a separate repo, which is where the "i1 gguf vs gguf" searches come from. bartowski's card says "All quants made using imatrix option" once, near the top, and after that a file is just Q4_K_M. On Hacker News, which I read through the Algolia API because I cannot access Reddit, the misreadings are specific.

• Under a Qwen3.8 27B quant benchmark, one commenter wrote that "they tested Q4_K_M which should be just the standard K-quant without any imatrix calibration" (comment). The benchmark used Unsloth files replaced on 19 August 2026, so I cannot read the tested one, but the current UD-Q4_K_M there records a 1,251-chunk matrix.
• Another explained that "Q4_K (4-bit K quant) uses an imatrix" (comment). Q4_K is a block format. It uses a matrix only if whoever ran the quantizer passed one, and my static file is a Q4_K_M made without.
• A third wrote that "The I-prefix stands for Imatrix smoothing" (comment). IQ types are a separate family of block formats. What ties them to the matrix is a rule in llama-quant.cpp: it refuses to make IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S or IQ3_XXS tensors without one, while "as a general rule, the k-type quantizations don't require imatrix data."

The accurate version sat one comment up in the second thread: "All the Q quants from big quant providers are importance-weighted (imatrix) nowadays," and "if one needs/wants static quants, ignoring IQ_ isn't enough" (comment). My header survey agrees about the uploaders and adds the exception people are most likely to be running, the Ollama default.

Which one to download

An imatrix file whenever one exists, at any size below Q8_0. On this test it cost nothing at run time, the same bytes and the same kernels, and removed a fifth of the drift at Q4_K_M and Q3_K_M. The argument for static quants is that a matrix is tuned to its calibration text, so a use far from that text inherits someone else's bias. On the C source the matrix still removed 17% and 22% of the drift, so I have no measurement where the static file wins. At Q8_0 the question does not arise, because the quantizer never reads the matrix.

If you run Ollama's default, you are on the static file. Ollama can run a Hugging Face GGUF directly, and Hugging Face's instructions give the form ollama run hf.co/{username}/{repository}:{quantization}, so ollama run hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M should fetch the imatrix file of the same size; I read that from the documentation and did not run it. And if you make your own quants from someone else's Q8_0, check the header afterwards. Without --imatrix your file will claim a matrix it never used, and llama.cpp's gguf_new_metadata.py script has a --remove-metadata option for deleting the four keys.

FAQ

What is the difference between imatrix and static quants?

They use the same file formats, such as Q4_K_M, and come out the same size. The difference is how each block of weights is rounded. A static quant weights the rounding by each weight's own magnitude; an imatrix quant also weights it by how strongly real text activates each input channel, measured beforehand on a calibration file. On Llama 3.1 8B made from the same Q8_0 file, the imatrix Q4_K_M drifted 20% less from Q8_0 than the static one on WikiText-2 (mean KL divergence 0.0173 against 0.0216), and the imatrix Q3_K_M drifted 20% less than its static twin.

How can I tell if a GGUF file was made with an imatrix?

Read its header metadata. llama-quantize writes four keys when it is given an importance matrix: quantize.imatrix.file, quantize.imatrix.dataset, quantize.imatrix.entries_count and quantize.imatrix.chunks_count. A file quantized from 16-bit weights without a matrix has none of them. The exception is requantizing: llama.cpp copies every key from the source file, so a static quant made from an imatrix Q8_0 inherits keys it never used. bartowski's files and mradermacher's i1 files carry the keys; mradermacher's static repo and Ollama's default llama3.1 do not.

Does an imatrix quant run slower or use more memory?

No. The matrix changes which values each block is rounded to, not the block format. Llama 3.1 8B Q4_K_M made with and without an imatrix had the same 292 tensors in the same encodings at 4.894 bits per weight, and the two files differed by 32 bytes of header text, so the same kernels run either way. Making the file takes longer: 68 seconds against 48 for Q4_K_M on a base M4 Mac mini. Q8_0 is the one type where the matrix does nothing, because its quantizer ignores it.

Sources and verification: every divergence, perplexity, size and timing figure is my own reading on this Mac16,10 (base M4, 10-core GPU, 16 GB, macOS 26.4.1) on 2026-09-15 between 21:04 and 21:42 KST, using Homebrew llama.cpp 0.4.0 build 10809. The source is bartowski's Llama 3.1 8B Instruct Q8_0 (8,540,775,840 bytes, checked against the SHA-256 Hugging Face publishes), requantized with and without the .imatrix file bartowski published. Quality runs used llama-perplexity with --kl-divergence-base and Q8_0 as the reference, 16 chunks of 2,048 tokens from the WikiText-2 test file and from ggml-quants.c at commit bec4772f6. Because the reference is Q8_0 and the requantized files were made from it, these numbers compare the two arms with each other; they cannot say what requantizing costs against the original weights. Header keys were read with my own streaming GGUF parser from Hugging Face and from Ollama's registry, where the blob is the model layer of the llama3.1:latest manifest. Code quotes are from llama.cpp master fetched the same evening, the official comparison is the Llama 3 8B table in the perplexity README, and Hacker News comments were read through the Algolia API; I could not access Reddit. Not tested: F16 as the source or the reference, matrices other than bartowski's, IQ types, speed (the tensor layouts are identical), and any task score.