This is the bridge to the kernel shapes. Prefill exposes many activation rows at once: approximately B × T rows for a padded batch, or the equivalent packed-token rows for a packed batch. A projection can reuse weight tiles across those rows, which creates a GEMM-shaped workload. A private, single-request decode step exposes one new row, which creates a GEMV-shaped workload. A high-concurrency server can batch new rows from several active requests and recover some GEMM-like reuse during decode, but that is not the same contract as one user running one model locally.

The prerequisite notes explain each boundary independently: tokenization turns text into model IDs; matrix Wx+b connects scalar linear algebra to GEMV, GEMM, and transformer projections; attention explains how queries use keys and values; and the KV-cache article explains why incremental decode can reuse prompt state instead of recomputing it.

C-Kernel-Engine, abbreviated CKE, is a compiler and runtime project that represents an AI model as a circuit of mathematical kernels. Model weights, their sidecar metadata, and a circuit definition are lowered through intermediate representations into explicit memory plans, C calls, and a compiled runtime. This article continues the CKE kernel-registry article: the registry explains how CKE identifies an exact provider; here we examine why one model needs different providers and execution plans for prefill and decode.

Why should a user care? Prefill largely determines time to first token: how long the model appears to think before the answer begins. Decode determines inter-token latency: how quickly the answer continues once it starts. A runtime can improve one number while leaving the other almost unchanged. During recent CKE work, one production-shape Q4 prefill provider moved from roughly 1.64 times llama.cpp's time to 1.035 times on one measured i7-14700T. That did not make the complete request 1.6 times faster. It removed one constraint and exposed the next ones in attention, Q6 scheduling, runtime dispatch, and sustained package power.

The underlying reason is shape. During prefill, a projection can reuse one block of weights across many prompt rows and behave like matrix-matrix multiplication. During low-batch decode, the same weights are applied to one new row and behave like matrix-vector multiplication. Full attention changes shape too: prefill relates prompt tokens to one another, while decode relates one new query to the existing cache. CKE therefore cannot optimize “the model” once. It must preserve one mathematical circuit while compiling phase-specific execution plans.

1. The Request Changes Shape

Suppose the hidden width is K, a projection produces N outputs, and the prompt contains T tokens. Ignoring storage layout for a moment, prefill evaluates:

This is GEMM: general matrix-matrix multiplication. Each block of weights can be loaded, unpacked, or repacked once and reused across several activation rows. A useful prefill tile might process four or eight token rows and eight output columns together. That reuse is why replacing prefill with T independent GEMV calls can be mathematically correct and still unnecessarily slow.

After prefill, one decode step has one current hidden row:

This is GEMV: general matrix-vector multiplication. At low concurrency, there is no large token dimension over which to reuse every streamed weight. The runtime often becomes limited by how quickly quantized weights and KV-cache rows can move through memory and cache, not by the processor's peak multiply-add count. Parallelism usually divides independent output rows among workers. If a serving system batches many independent decode requests, a matrix-shaped opportunity can reappear, but that is a different workload contract from one private request producing one token.

2. The Attention Complexity Needs A Precise Statement

It is common to write “prefill is O(T^2) and decode is O(T).” That is only a description of the full-attention component under fixed head dimensions; it is not the complexity of the entire transformer layer. The linear projections and MLP remain linear in token count for fixed model dimensions.

For full causal attention with prompt length T and head dimension d:

At decode step s, one new query reads the s cached key/value rows:

The KV cache removes the need to recompute the old keys and values. It does not make the old context disappear: the new query still has to attend across the retained cache unless the model uses a bounded-window, recurrent, compressed-memory, or other specialized attention contract. Over many generated tokens, those repeated cache reads remain real work.

3. What CKE Actually Exposes

The v8 runtime has separate public dispatch surfaces. ck_parallel_prefill_v8.h describes thread-pool-parallel GEMM wrappers with the shared call form gemm_nt_*(A, B, bias, C, M, N, K). It includes Q4_K/Q8_K and Q6_K/Q8_K paths used by quantized projections. ck_parallel_decode_v8.h describes GEMV dispatch where y[M] = W[M,K] @ x[K] and output rows are partitioned across the threadpool.

// Prefill: many activation rows.
gemm_nt_q4_k_q8_k_pairwise_split_min_parallel_dispatch(
    activations, weights, bias, output, T, N, K);

// Decode: one activation row.
gemv_q4_k_q8_k_repacked_parallel_dispatch(
    output, weights, activation_q8, N, K);

These signatures expose only part of the contract. The corresponding Q4 prefill kernel map records packed-weight ownership, split-min reduction, persistent repacking scratch, AVX2 and AVX-VNNI providers, shape gates, threading capacity, exact llama.cpp comparison, and fallback behavior. The Q4 decode map separately records canonical Q8_K activation quantization, repacked matrix-vector arithmetic, runtime ISA dispatch, and bit-exact production evidence.

This separation matters because “Q4_K weights and Q8_K activations” still does not uniquely identify the arithmetic. Prefill can pack four activation rows using nearest-even behavior and reuse an interleaved eight-output weight layout. Decode can quantize one canonical row and traverse a repacked GEMV layout. The output shape may agree while byte packing, integer traversal, minimum-term accumulation, FP32 reduction order, and rounding differ.

4. One Circuit, Two Lowered Plans

A circuit describes the model semantics once: normalization, projections, rotary positions, attention, residuals, MLP operations, cache policy, and generation policy. The Qwen3-VL circuit, for example, declares an FP16 decoder KV cache, mixed visual/text prefill, a persistent cache, and incremental decode after prefill. The compiler then resolves phase-appropriate kernel maps rather than duplicating the architecture by hand.

1. The tokenizer and, for a vision model, the vision bridge produce decoder input rows.
2. Prefill lowers those rows through GEMM-oriented projections and prefill attention.
3. Each layer appends its K/V rows into persistent cache storage.
4. The final prompt row produces first-token logits.
5. Decode consumes one selected token, executes GEMV-oriented projections and one-query attention, appends one cache row, and produces the next logits.
6. The decode plan repeats without reconstructing the prompt circuit or reallocating the entire cache.

The kernel registry is the bridge between the shared circuit and these distinct plans. It lets IR1 request execution.phase=prefill or execution.phase=decode, resolve a provider whose numerical and storage contracts match, and preserve that provider through memory planning and generated C. A later code generator should not rediscover the phase from a model name or silently replace GEMM with repeated GEMV.

5. Why “Just Use GEMM” Was Not Enough

CKE's recent history is a useful record of what CPU optimization actually requires. The sequence below is not a claim that every model became faster by the same percentage. It is a bounded progression on specific Qwen3-VL and quantized prefill shapes, primarily measured on one i7-14700T. The important lesson is how many independent contracts had to become correct before a serious llama.cpp comparison was possible.

The order matters. Packing had to remove repeated work before wider tiles could expose their real benefit. Exact wider tiles then made instruction cost and output layout visible. Once the arithmetic provider approached llama.cpp, thread topology and runtime ISA visibility became meaningful constraints. After the projections improved, serial visual attention became large enough to dominate the profile. Each fix removed one bottleneck and made the next bottleneck measurable; Section 8 reconnects this history to the Theory of Constraints.

6. What One FP32 ULP Means

ULP means unit in the last place: the distance between adjacent representable floating-point values near a particular number. A one-ULP difference can be extremely small and still matter when the acceptance contract is bit-exact. It can also become amplified by later normalization, softmax, residual accumulation, and repeated layers.

The Q=63 case showed why source-level similarity is insufficient. A compiler may contract a multiply and add into one fused operation in one build but round them separately in another. Writing an explicit fmaf stated the intended arithmetic rather than hoping compiler flags reproduced it. CKE did not increase the tolerance until the gate passed. It corrected the operation that defined the reference boundary.

This is also why an optimized GEMM cannot automatically replace a collection of exact GEMVs. Different tiling can change integer traversal, horizontal reductions, FP32 accumulator grouping, bias timing, and rounding. CKE promotes a provider only when its actual production shape and dispatch ABI satisfy the intended oracle, not merely when a tiny leaf function appears close.

7. What A Real llama.cpp Comparison Requires

CKE is not claiming a general victory over llama.cpp. The recent work shows why such a claim is difficult. llama.cpp has mature quantized layouts, architecture-specific kernels, repacking, scheduling, cache management, and years of measurement across many machines. A specialized engine may legitimately outperform a general engine on one model, one approved weight file, one platform, or one constrained workload. Hard-coding those assumptions can remove abstraction and branching costs. But “the model runs” does not establish a comparable result.

A serious performance comparison needs at least:

• the same checkpoint and quantization bytes, or a clearly explained quality-equivalent alternative;
• the same prompt, context, generated-token count, sampling policy, stop policy, and cache precision;
• separate prefill and decode timing rather than one blended number;
• the same thread, affinity, power, and thermal conditions;
• exact compiler, ISA, library, and executable provenance;
• quality or token-parity evidence, not only fluent output;
• repeated alternating runs so warm cache or package power does not favor one runtime;
• a distinction between isolated kernel speed and end-to-end model speed.

Prompts can generate a plausible loop. They do not generate trustworthy performance by magic. The missing work is usually in layouts, reductions, cache policy, dispatch, measurement, and the thousands of cases where “almost equivalent” is not equivalent.

8. The Theory Of Constraints View

A fast projection does not make a fast multimodal request if attention remains serial. More cores do not help if the provider reloads the same packed weights. AVX-VNNI does not help if generated code cannot discover the linked engine's capability. Every logical thread is not automatically useful if SMT causes contention or exceeds the package's sustained power window. The bottleneck moves as each constraint is removed.

The same reasoning will matter on larger Xeon and EPYC systems. More memory channels, cache, cores, AVX-512, VNNI, AMX, and accelerator blocks expand the available machine, but they do not choose the right tile or reduction. A workload reaches useful hardware throughput only when data layout, reuse, worker placement, numerical contract, and sustained system behavior agree.

9. Measure The Two Phases Separately

A single “tokens per second” number can conceal the result. CKE should report at least:

CKE's CKE Throughput Unit extends this idea toward topology-aware measurement. The purpose is not to invent a flattering universal score. It is to record useful work together with memory, communication, topology, and execution context so that a distributed or heterogeneous result remains interpretable.

10. Current Evidence Boundary

CKE now has strong evidence for several exact AVX2 and AVX-VNNI Q4/Q6 prefill providers and for the hardened Qwen3-VL AVX2 parity path. That does not mean every ISA, compiler, model family, prefill shape, and decode path is equally mature. AVX-512, BF16/AMX, additional quantized formats, larger systems, and end-to-end comparative runs remain separate promotion surfaces. The exact boundaries should continue to be recorded in kernel maps, tests, nightly reports, and commit evidence rather than compressed into “CPU support complete.”

The practical result of this work is not a viral benchmark yet. It is a reusable method: identify the phase, resolve the exact provider, measure the production route, compare every value when possible, preserve provenance, remove the current bottleneck, and run the gates again. That is how CKE can eventually make credible claims on laptops, Xeon workstations, servers, embedded ARM systems, and distributed CPU nodes without pretending those machines are interchangeable.

Related Notes And Source Artifacts

• CKE Kernel Registry: How Exact C Functions And Numerical Contracts Connect
• Gemma4-VL In CKE: How Images Become Decoder Tokens
• Nemotron Kernels From Inference To Backpropagation
• CKE Prefill Performance Roadmap
• CKE v8 kernel maps
• llama.cpp source repository

To inspect or contribute to CKE, use the contributor path or join the CKE Discord channel. Profiling adapters, reproducible baselines, numerical fixtures, kernel work, and compiler/runtime work are all valid entry points; the hardest multimodal parity problem is not the entrance exam.