Four bits tell me how much room a code takes. They do not tell me what number it represents. That is the question I kept coming back to while preparing the quantization video for Antshiv Robotics. MXFP4, NVFP4 and Q4_K all put two element codes in a byte. But if I hand the same byte to three decoders, I have not handed them the same weights.

My way into this comes from building C-Kernel-Engine. A small checkpoint is useful only when the runtime can interpret its bytes, select the right kernel, and produce an answer I can check. I like to shrink that problem until I can do the arithmetic on paper. Then I open the code.

This is the hands-on reading companion to the video. The diagrams have larger labels, the arithmetic stays on the page, and the controls let you change one thing at a time. My earlier article, Why 4-Bit Quantization Is King, covers the wider memory and format story. Here I want to follow a value all the way from its four bits to its contribution to a dot product.

1. Start with 0101. What number is it?

If I read 0101 as an unsigned integer, it is five. That is the interpretation used for the Q4_K element code, before applying its gain and offset. But E2M1 reads those same bits as three. Nothing about the ink on the page changed. The decoding rule changed.

E2M1 is the four-bit floating-point element format used by MXFP4 and NVFP4. Its name describes two exponent bits and one stored fraction bit; the fourth bit is the sign. For a normal value, the rule is:

value = (−1)^sign × 2^(exponent − 1) × (1 + fraction / 2)
0101  = sign 0 | exponent 10 | fraction 1
      = (+1) × 2^(2 − 1) × (1 + 1/2)
      = 3

The implicit leading one is part of the normal-value rule. It is not a fifth stored bit. When the exponent field is zero, the normal formula is not the right formula: E2M1 includes zero and the subnormal magnitude 0.5. Its magnitude table is small enough to keep beside the decoder: 0, 0.5, 1, 1.5, 2, 3, 4, 6. Both signs are encoded, including signed zero.

Try code 7, then code 8. The first is positive six; the second is negative zero. Treating the raw nibble as a signed two's-complement INT4 value would give a completely different answer. That is why “four-bit support” is not a sufficiently precise kernel interface.

The gaps in this codebook are also worth noticing. Near zero, neighboring positive values are half a unit apart. At the upper end, four and six are two units apart. This is a tiny floating-point ruler, not a uniform integer ruler. The OCP MX specification defines the element and shared-scale formats; the examples here make their arithmetic visible.

2. The scale changes the range as well as the spacing

Let c be the decoded E2M1 value and s a positive scale. Reconstruction is ŵ = s × c. At scale 0.5, the positive representable weights are 0, 0.25, 0.5, 0.75, 1, 1.5, 2 and 3. At scale 0.125, those same codes reach only 0.75.

This gives me a useful question to ask before touching a slider: will a smaller scale help every weight in the group? Take 0.35 and 1.7. At scale 0.5 they reconstruct to 0.25 and 1.5. At scale 0.125, 0.35 reconstructs to 0.375, which is closer. But 1.7 is outside the available range and clips to 0.75. One error improved; the other became much worse.

A smaller ruler does not manufacture additional codes. It brings the existing marks closer together and shortens the distance they can reach. Choosing a scale is a compromise over the values that share it.

Watch the output, not just the weights

For the video, I used this small row and input vector, with bias set to zero:

w = [−1, 0.3, 0.9, 1.7]
x = [ 2,   1,  −1, 0.5]
y = w · x = −2 + 0.3 − 0.9 + 0.85 = −1.75

At scale 0.5, the reconstructed weights are [−1, 0.25, 1, 1.5]. Their dot product is −2, a signed output error of −0.25. At scale 0.125, they become [−0.75, 0.25, 0.75, 0.75], and the output is −1.625. Its error is +0.125—even though clipping has made some individual weight errors larger.

That is not a recommendation to clip weights. It is a reminder that output error depends on the input. For this one row, Δy = Σ(ŵᵢ − wᵢ)xᵢ. Different inputs can amplify or cancel different terms. A single attractive output cannot establish model quality. I would check representative inputs and the task the model is meant to perform.

Lab convention: this widget picks the nearest scaled E2M1 value and chooses the lowest unsigned code on an exact tie. It uses power-of-two scale choices to keep the arithmetic legible. It is not an implementation of a complete MXFP4 or NVFP4 quantization/calibration recipe.

3. MXFP4: one power-of-two ruler for 32 values

For an MXFP4 scaling block, 32 E2M1 element codes share one E8M0 scale byte. E8M0 has eight exponent bits and no stored fraction or sign bit. For finite encodings 0 through 254, decode the scale as 2^(byte − 127). Encoding 255 is reserved for NaN; finite E8M0 has no zero scale.

Open the byte 0x7E. It is 126 as an unsigned integer, so the exponent is 126 − 127 = −1. The scale is 0.5. If an element nibble decodes to E2M1 value 3, its reconstructed weight is 3 × 0.5 = 1.5. All 32 values in this block use that same scale.

Thirty-two four-bit codes occupy sixteen payload bytes. Add the one scale byte and the block uses seventeen bytes: 17 × 8 / 32 = 4.25 bits per weight. This counts the format's element and scale storage, not padding, an entire model file, or runtime memory.

The ownership view is more important than making the drawing look like a particular GPU's physical layout. A real kernel may consume scales in a tiled or swizzled arrangement. The logical group and the kernel's required memory layout are related contracts that I check separately.

4. NVFP4: keep the codebook, change the scales

NVFP4 retains the E2M1 element codebook. The reconstruction scheme described in NVIDIA Transformer Engine's NVFP4 documentation uses an E4M3 scale per 16-element block and an FP32 scale shared across the tensor. To compare with 32 MXFP4 values, picture two NVFP4 groups, each with its own local ruler.

E4M3 has a sign bit, four exponent bits and three stored fraction bits. For a positive normal scale with exponent field 5 and fraction field 2, the arithmetic is:

local scale = 2^(5 − 7) × (1 + 2/8)
            = 0.25 × 1.25 = 0.3125
weight      = E2M1(code) × local scale × tensor scale
            = 3 × 0.3125 × 1 = 0.9375

This normal-value example does not specify every special or subnormal E4M3 encoding. It shows why a scale with fraction bits offers choices between powers of two. The tensor scale extends the usable scaling range. In this article, “scale” always means the multiplier used for reconstruction; an API may instead expose a reciprocal quantization scale, so names alone are not enough.

The smaller group gives neighboring regions more freedom to use different local scales. That can reduce a particular source of error, but it costs more metadata and does not make accuracy automatic. Calibration, the values being quantized, the model and the execution path still matter.

The simple storage calculation is (16 × 4 + 8) / 16 = 4.5 bits per weight for the codes and local scale. With one FP32 scale for a tensor of N weights, add 32/N. Treat that as a representation calculation for this scheme, not a promise about every checkpoint's exact size.

5. Q4_K: integer codes with a gain and an offset

Now change the decoding rule. Q4_K stores unsigned four-bit integers, 0 through 15. For subblock j and a code qᵢ, reconstruction is:

ŵᵢ = d × sⱼ × qᵢ − dmin × mⱼ
       └ gain ┘       └ offset ┘

The standard 256-weight block we are inspecting has eight subblocks of 32 weights. Each subblock has a six-bit scale parameter sⱼ and a six-bit minimum parameter mⱼ. The two superblock factors d and dmin are stored as FP16. The name “minimum” does not mean I should add a raw signed minimum from memory: the actual decoder subtracts the product dmin × mⱼ.

Here is the full byte budget, following the GGML block definition and reference decoder:

This is why looking only at a nibble is not enough. To recover one weight I need its code, the relevant subblock metadata, the superblock factors and its logical output position.

Follow 0xA5 into lanes 0 and 32

Take the first payload byte in the presentation fixture: 0xA5, or binary 1010 0101. Its low nibble is 5 and its high nibble is 10. Set d=0.125 and dmin=0.0625. The first two scale/minimum pairs are (2,4) and (3,8).

For the low nibble, the gain is 0.125 × 2 = 0.25 and the offset is 0.0625 × 4 = 0.25. The weight is 0.25 × 5 − 0.25 = 1. For the high nibble, the gain is 0.375 and the offset is 0.5, giving 0.375 × 10 − 0.5 = 3.25.

Those outputs go to lanes 0 and 32, not lanes 0 and 1. The nibble split is easy to write correctly while putting the result in the wrong place. In each 64-weight chunk, 32 payload bytes hold the low codes for one subblock and high codes for the next. For chunk c and byte position l:

payload index = 32c + l       where c = 0..3, l = 0..31
low output    = 64c + l       uses subblock 2c
high output   = 64c + 32 + l  uses subblock 2c + 1

Try byte zero in the lab. Both codes are zero, but the reconstructed weights are −0.25 and −0.5 because the offsets remain. Then try 255: both codes are 15, but the two gains and offsets still produce different weights. Shared storage does not imply shared metadata.

6. The metadata has a packing problem of its own

Eight scales and eight minima at six bits each make 96 bits, or twelve bytes. They are not sixteen ordinary byte fields. The first four pairs keep their low six bits in separate bytes; the later pairs combine four-bit fields with upper bits stored in earlier bytes. This packing is where a plausible-looking implementation can quietly diverge.

Using B for the twelve metadata bytes, the reference indexing can be written as:

For j = 0..3:
  s[j] = B[j]   & 63
  m[j] = B[j+4] & 63
For j = 4..7:
  s[j] = (B[j+4] & 15) | ((B[j-4] >> 6) << 4)
  m[j] = (B[j+4] >> 4) | ((B[j]   >> 6) << 4)

In the presentation fixture, B[8]=33 and B[0]=130. For j=4, the scale is (33 & 15) | ((130 >> 6) << 4) = 1 | 32 = 33. If I had just read B[4] as that scale, I would be reading the wrong information.

The fixture deliberately varies the later scales and minima. A test where every block uses the same scale can hide an indexing bug: the decoder may retrieve the wrong field and still get the expected number. That is why I want asymmetric test data before I optimize.

7. Q4_K_M is a recipe label, not another nibble codebook

A model contains many tensors. Q4_K describes the block format we just opened. Q4_K_M describes a model quantization recipe that can select different storage types for different tensors. The exact selection depends on the converter, its version and the model architecture.

I would not infer the whole tensor inventory from the filename. I would inspect it. Which tensors are Q4_K? Which are higher precision? Does the loader understand all of them? What happens if it encounters an unsupported type? A clear rejection is much easier to debug than silently treating unfamiliar bytes as a familiar layout.

8. The file is only the beginning of the runtime contract

There are several different meanings of “supports four-bit.” A tool might write a four-bit file. A loader might parse that file and expand every tensor to FP32. A CPU kernel might unpack a small tile into registers as it computes. A GPU kernel might use a native low-precision instruction with a specific scale layout. Those implementations have different memory footprints and performance characteristics.

In CKE, the part I want to keep inspectable is the path from a declared tensor type to its decoder and compute kernel. The dequantization kernels provide a concrete place to read the Q4_K reconstruction. The presentation's animation follows that scalar contract. This article does not establish a native MXFP4 or NVFP4 execution path in CKE.

An optimized kernel does not necessarily materialize the full dequantized matrix. It can reconstruct a tile, consume it, then move on. Some implementations can exploit algebra or integer dot products to avoid an obvious scalar sequence altogether. The visible steps are the reference semantics; the optimized implementation must preserve their effect within its stated numerical tolerance.

That is also why I separate prefill and decode. Processing many tokens gives a different reuse pattern from generating one token at a time. The same packed weights can pass through different kernels and encounter different bottlenecks.

9. Smaller storage does not guarantee a proportional speedup

For an illustrative dense model with seven billion weights, BF16 payload alone is 14 GB, using decimal gigabytes. Ideal four-bit payload alone is 3.5 GB. If every weight used the 4.5-bit Q4_K block accounting above, it would be 3.9375 GB before model-file overhead and any higher-precision tensors.

That is useful capacity arithmetic. It is not a token-per-second prediction. Runtime memory includes activations, scratch space, the KV cache and whatever copies the loader creates. Latency includes more than reading weights. The benefit depends on saved memory traffic, decoding overhead, supported instructions, batching, reuse and the workload.

On a robotics system I care about the complete path: sensor input, preprocessing, model execution, the logic that consumes the result and the action deadline. Saving model memory may make room for another component even when the model's speedup is modest. Conversely, a fast isolated kernel cannot compensate for a slow or unreliable surrounding pipeline. The 3D drawings in the presentation are teaching tools, not evidence that the rendering GPU supports a native FP4 inference instruction.

10. The checks I want before trusting an optimized decoder

1. Decode individual codes. Cover all sixteen E2M1 nibbles, both signed zeros, and integer nibble extraction separately.
2. Check metadata independently. Test all eight Q4_K scale/minimum pairs, especially j=4..7 where fields are split across bytes.
3. Check lane placement. Use different values for the two nibbles and different metadata for their subblocks. Check all 256 destinations.
4. Compare reconstructed weights. A simple scalar implementation is useful here, but agreement between two copies of the same bug is not independent validation.
5. Compare a dot product. Use inputs with varied signs and magnitudes. Inspect signed output error as well as maximum weight error.
6. Test the actual optimized path. Include block boundaries, lengths the API accepts, alignment assumptions and explicit rejection of unsupported cases.
7. Measure the real workload. Record the model, tensor types, hardware, build, thread settings, prompt shape and whether the run is prefill or decode.

For the wider reasoning behind these checks, see What Numerical Parity Actually Requires. Dequantization reconstructs the numbers represented by a stored block. It cannot recover the information already lost when the original weights were quantized.

My way of remembering the three formats

I keep three questions in mind: what does the code mean, who owns its scale, and where does the result go?

• MXFP4: read an E2M1 code and use the E8M0 scale shared by its 32-value group.
• NVFP4: read an E2M1 code and apply its 16-value local scale and the tensor scale in the scheme discussed here.
• Q4_K: read an unsigned code, recover its subblock metadata, apply the gain and subtract the offset, then place the result in the correct lane.

That gives me a smaller problem than “understand quantization.” I can open one byte, calculate two outputs and write a test. Then I can make the block bigger, trace the kernel and measure it. If you want to follow the mathematics underneath that multiply, the wx+b training loop is the next connection. If you want the memory and format overview, return to the original four-bit article.

The interactive examples are deliberately small and deterministic. They are not benchmark results, a model-quality evaluation, or a full implementation of a vendor quantization recipe.