For anyone new to the project, C-Kernel-Engine (CKE) is an open-source CPU runtime and kernel compiler for transformer-based language, vision, and audio models. CKE represents a model as mathematical circuits, resolves those circuits through explicit kernel contracts, and generates C runtimes that we compare with PyTorch and llama.cpp.

The previous article, How CKE X-Ray Found Qwen3.6's First Bad Circuit, followed a wrong token back to a grouped recurrent-topology mismatch. This article continues from there. The focus is no longer only whether the circuit is correct. It is how we used the same attribution system to make the tested Qwen3.6 prefill path about 2.6 times faster while keeping strict output and recurrent-state gates.

What Model And CPU Did We Actually Measure?

This was a single-node CPU experiment, not a distributed run. The model artifact was Qwen3.6-27B Q4_K_M GGUF: a 27-billion-parameter-class model with mixed Q4_K_M quantized weights. The host processor identified itself as Intel Xeon Gold 6542Y. Intel lists this as a 5th Gen Xeon Scalable processor with 24 physical cores, 2.9 GHz base frequency, 60 MB cache, AVX-512, and Intel AMX.

The recorded benchmark did not have unrestricted use of all 24 cores. It ran inside a 14-CPU cgroup using an ICX native AVX-512/VNNI build and a templated 29-token prompt. This distinction matters when someone tries to reproduce the result or compare it with a different Xeon configuration.

We Did Not Start By Making The Wrong Circuit Faster

Qwen3.6 uses a hybrid decoder containing recurrent DeltaNet layers and full-attention layers. During bring-up, CKE had several independent problems: grouped Q/K topology, quantized projection routes, recurrent storage behavior, prefill scheduling, and provider arithmetic. Some short runs still produced coherent text, which was not enough evidence that these contracts were correct.

This matters for optimization because a faster implementation can hide a mistake more deeply. If a new thread schedule changes reduction order, state ownership, token order, storage dtype, or the selected provider, a lower runtime is not a clean performance result. We may have timed a different computation.

CKE therefore established scoped production oracles first. The tested recurrent operators, quantized providers, and attention paths were compared against a pinned llama.cpp commit. The generated runtime recorded which provider ran, with which shapes, ISA, thread count, model, and artifact hashes. Only after those boundaries existed did we start promoting faster schedules.

Recurrent Does Not Mean The Entire Layer Must Be Serial

A recurrent state update for head \(h\) and token \(t\) can be written abstractly as:

The state at token \(t\) depends on the state from token \(t-1\). We cannot process token 20 for one head before token 19 and still claim to execute the same recurrence. But Qwen3.6 has multiple value heads, and those heads own disjoint state and output slices. Head 0 does not need head 17's state before it can advance its own sequence.

The useful schedule is therefore two-dimensional. Keep token order serial inside each assigned head, while distributing independent head ranges across persistent CPU workers. Each worker owns several heads and walks those heads from the first prompt token to the last. We remove unnecessary cross-head waiting without changing the arithmetic order inside a head.

worker 0: heads  0- 9, token 0 -> token 1 -> ... -> token T
worker 1: heads 10-19, token 0 -> token 1 -> ... -> token T
worker 2: heads 20-29, token 0 -> token 1 -> ... -> token T
worker 3: heads 30-39, token 0 -> token 1 -> ... -> token T
worker 4: heads 40-47, token 0 -> token 1 -> ... -> token T

This is the core idea behind the later prefill and decode changes. It is not a general claim that any recurrent model can be parallelized arbitrarily. The safe boundary came from the actual state ownership and tensor topology of this circuit.

X-Ray Had To Measure Performance, Not Only Numerical Drift

CKE X-Ray originally focused on first divergence: identify the last passing semantic checkpoint and the first failing one. The Qwen3.6 work extended that idea into performance attribution. A generated call can now be connected to its operation, layer, selected kernel, shape, and measured time. Reference layer profiles can be persisted with hashes and observer provenance instead of existing as temporary debugging files.

The measurements showed that the slow path was not one vague "DeltaNet problem." Prompt time was spread across repeated Q4 packing, narrow Q4 providers, Q8 recurrent projections, Q6 MLP-down projections, thread-pool dispatch overhead, and serial recurrent execution. Each item needed a different repair and a different regression gate.

The Optimization Was A Sequence, Not One Magical Kernel

Five recent pull requests form the clearest performance story. Their measurements are not one continuous benchmark series: prompt lengths, hosts, shapes, and enabled providers differ. I am listing them together to show the engineering progression, not to multiply all their speedups.

Reusing Q6 Work Across Prompt Rows

During prefill, several token rows multiply against the same weight matrix. The established Q6 provider treated every row independently, which meant unpacking the same Q6 weight block again for each row. The M4 provider unpacks that block once and reuses it across as many as four Q8_K activation rows.

Reuse does not mean combining the rows into one reduction. Each output row keeps an independent integer accumulator, block-scale FMA order, and llama.cpp-compatible horizontal FP32 reduction tree. The shared work is weight decoding, not numerical accumulation. That distinction is why the provider can be faster while remaining byte-exact in the production oracle.

Selection is deliberately narrow: short prefill with \(4 \le M \le 63\), large Q6 MLP-down shapes, and the measured recurrent-QKV shape. Decode, strict-parity mode, forced-reference mode, and longer prompts remain on established providers. Boundary tests at \(M=63\) and \(M=64\) prevent a shape-rule change from silently widening this route.

Packing Q4 Once And Respecting The Tail

Quantized matrix multiplication includes work beyond the visible dot products. A Q4_K provider may need to reorganize packed blocks into the layout expected by its vectorized microkernel. If that preparation happens during every prompt, model-invariant work is charged repeatedly to inference.

The optimized path prepares eligible x16 Q4 weights during model loading and caches them by weight identity. The measured prompt can then use the prepared layout immediately. This route is still experimental and requires AVX-512/VNNI plus explicit batched-prefill configuration. AVX2 and unrelated model shapes keep their existing providers.

Prompt lengths also rarely divide perfectly by a tile size. For a 23-row projection, twenty rows may use the matrix path while three rows remain. CKE batches those eligible residual rows into one thread-pool dispatch but evaluates each with the already certified exact GEMV function. Applying this schedule indiscriminately was slower for a wider gate/up shape, so the kernel map limits it to the measured \(N=6144\) recurrent-gate projection.

The Decode Kernel Improved 4.84x, Not The Whole Model

Decode profiling attributed roughly 36% of measured token time to a serial grouped DeltaNet core. PR #275 exposed a head-range primitive and dispatched independent decode heads through CKE's persistent thread pool. At the production geometry of 48 heads, 16 compact Q/K groups, and state dimension 128, the isolated core measured:

This is a kernel-level result. Projections, MLPs, normalization, memory movement, and other model work still contribute to each generated token. It would be incorrect to say Qwen3.6 decode became 4.84 times faster end to end. The result says one previously serial core became up to 4.84 times faster in the measured thread matrix, and X-Ray tells us how much of total token time it owned.

The Combined Batched-Prefill Result

On the recorded single Xeon Gold 6542Y node, with a 14-CPU cgroup, an ICX native AVX-512/VNNI build, the Qwen3.6-27B Q4_K_M GGUF artifact, and a templated 29-token prompt, the batched baseline took 6682.61 ms at 4.34 tokens per second. Assigning complete recurrent-head ranges to persistent workers reduced that to 4014.20 ms at 7.22 tokens per second. Adding the measured Q4 x16 routes reduced it again to 2588.18 ms at 11.20 tokens per second.

Repeated final runs ranged from 2552.64 to 2601.74 ms, or 11.15 to 11.36 tokens per second. That is approximately 2.6 times the baseline throughput and 61.3% less prompt time. Same-host llama.cpp measured approximately 20.7 tokens per second, leaving CKE about 1.85 times behind. This is meaningful progress, but it is not parity with llama.cpp performance.

What guarded the result?

• The production-shaped 29 x 48 x 128 CKE fixture required byte-identical serial and parallel output and recurrent state.
• The pinned llama.cpp DeltaNet oracle passed at the recorded thread counts.
• The Q4_K x Q8_K production matrix covered every promoted x16 shape.
• AVX2, AVX-VNNI, and AVX-512/VNNI compilation remained part of the provider matrix.
• The complete numerical-contract suite and shared v7/v8 regression gates passed.
• No tolerance was widened to admit the faster schedule.

Why The Faster Path Is Still Opt-In

Short trajectory tests remained stable: the recorded eight-token prompt trajectory matched, and the three measured greedy tokens after both the 353-token and 1K-token prefixes matched. But top-1 agreement is not the complete numerical story. At the 1K first step, the recorded logit cosine was about 0.866, with RMSE about 2.014 and maximum absolute difference about 19.67.

That long-context accumulation existed before the scheduling change, and the exact serial versus parallel fixtures show that the new head schedule did not create it. It still blocks default promotion. CKE therefore requires CK_V8_FORCE_BATCHED_PREFILL=1 for this graph while the long-context evaluation order is investigated.

There are other boundaries too. The x16 path is hardware-specific and experimental. The pinned llama.cpp oracle remains necessary because newer llama.cpp revisions changed the gated-DeltaNet API. The decode result still needs a complete end-to-end rerun to convert the isolated kernel gain into a measured model-level improvement. Q6 and Q8 projections remain visible optimization targets.

What I Take From This Work

The useful lesson is not simply that more threads make a model faster. Threads only help when the model and memory layout expose independent ownership. For Qwen3.6 DeltaNet, token order is a real dependency, but head state is independently owned. That gave us a safe axis for parallelism.

The second lesson is that numerical evidence and performance evidence need to meet at the same generated call. X-Ray identified the expensive operation. The circuit and kernel map explained its topology and selected provider. Primitive oracles constrained the arithmetic. Performance gates proved that the measured schedule was faster on its intended shape. Long-context tests then stopped us from presenting a promising opt-in path as fully promoted production behavior.

CKE is still slower than llama.cpp on this measured Qwen3.6 prefill. That is exactly why this work is useful. The remaining gap is no longer one undifferentiated complaint that CKE is slow. It is a sequence of measured providers, schedules, and memory costs that can be changed one at a time without losing track of the mathematics.

Code, Pull Requests, And Related Reading

• C-Kernel-Engine source repository
• CKE X-Ray visualizer runbook
• Exact batched-prefill engineering note
• PR #266: recurrent Q4 prefill optimization
• PR #268: Q6 unpack reuse across short rows
• PR #273: recurrent projection optimization
• PR #275: exact parallel DeltaNet decode
• PR #277: exact batched-prefill acceleration
• How CKE X-Ray Found Qwen3.6's First Bad Circuit
• How DeltaNet State And Full Attention Share One Decoder
• Prefill Vs Decode In CKE