Yesterday I wrote about the small CPU AI lab I have started building for C-Kernel-Engine. The important next question was obvious: what can CKE actually distribute across the Ryzen and the Intel P3?

The first answer landed in PR #377. CKE ran a real Qwen3.5-35B-A3B Q4_K_M prefill across both machines over the existing 1 GbE connection. But it divided the token axis, not the weights. That helped establish the transport and evidence path, but it could not make the machines' memory additive or help ordinary one-token decode.

PR #382 is the next step. It first split a real routed MoE layer by weight, then extended the experiment to one complete Qwen3.5 transformer block containing full attention and MoE. This is much closer to the larger Distributed Zip research design: partition each graph region along an axis that lets both nodes contribute to the same token or prompt, and synchronize only where the mathematics requires it.

What The First Prototype Split

At an MoE layer, prefill presents a hidden-state matrix with one row per token:

X has shape [T, H]

T = prefill token rows
H = hidden-state width

The coordinator chooses a local percentage and cuts the matrix along T. In the measured run it used a 78/22 Ryzen/P3 split:

Ryzen: X[0 : T_local, :]
P3:    X[T_local : T, :]

both machines map the complete model weights

The coordinator sends the P3 complete hidden rows, the selected top-k expert indices, and their routing weights. The P3 computes the routed and shared SwiGLU branches for those rows using its own full BUMP weight artifact. It returns one completed output matrix for its assigned rows. The Ryzen computes the other rows locally while the remote work is running, then places the returned rows into their original positions.

This is legitimate compute aggregation for prefill. Token rows are independent through this MoE region, so two machines can process different rows at the same time. It also gives CKE a real transport protocol, deterministic token fixture, strict message dimensions, per-node timing, byte accounting, output hashes, and a place to test overlap.

It does not increase model capacity. Both machines need the weights. It does not divide one token's matrix-vector operation across CPUs. It does not make the P3's DRAM additive to the Ryzen's DRAM for this model. The correct description is token-row parallel prefill for selected MoE providers.

Why It Does Nothing For Ordinary Decode

Autoregressive decode usually gives the model one new token at a time:

T = 1
X has shape [1, H]

There is no useful 78/22 split of one row. The research preload says this mechanically: when rows <= 1, it calls the normal local provider and bypasses the Zip coordinator. That is the correct fallback. Sending part of a single row through a protocol designed for complete rows would change the operation rather than distribute it.

This is why a strong prefill result would not, by itself, answer the main decode question. Prefill has a token axis with abundant independent work. Decode needs another axis.

What PR #382 Now Splits

The implementation is inspectable in the pinned weight-sharded MoE harness and the complete attention-plus-MoE block harness. These links are pinned to merged commit 341e2b1e3, so this article continues to point at the measured implementation even as CKE changes.

For one decode row, a projection is approximately:

y = xW

x: [1, H]
W: [H, F]
y: [1, F]

If CKE divides W by output columns, each node owns a different [H, F_i] shard and produces a different slice of y. If it divides W by input rows, each node produces a partial y and the nodes must reduce those partial sums. The choice decides how much data moves, whether the next kernel can remain sharded, and where rounding order may change.

PR #382 tests the second form directly. Qwen3.5-35B has 256 routed experts and selects eight per token. For every expert, CKE divides the intermediate dimension 256/256. Each rank owns half of the Q4_K gate and up work and the matching half of the Q5_K down input. Both ranks execute gate, up, SwiGLU and down locally, then add their hidden-state partials. The shared expert is divided the same way.

The complete-block harness also partitions attention heads and matching output-projection columns. Qwen3.5 layer 3 has 16 query heads and two KV heads in this fixture. Each rank receives eight query heads and one KV head. The two attention partials are reduced before the attention residual. The synchronized router then selects the same top-eight experts on both ISAs, the two MoE partials are reduced, and the block completes its second residual.

rank 0 attention partial + rank 1 attention partial
+ residual
+ synchronized router decision
+ rank 0 MoE partial + rank 1 MoE partial
+ residual
= one reconstructed transformer block

At 4,096 rows, each hidden-state collective carries 32 MiB. The current coordinator path sends the attention partial, broadcasts the combined attention state, sends the MoE partial, and broadcasts the combined MoE state: four sequential 32 MiB transfers over 1 GbE. Router synchronization adds only 256 KiB. This is a deliberately plain transport baseline, not the final RDMA design described by CKE's compute, memory and network constraint model.

The First Result Also Needed A Correction

PR #377 initially reported approximately a 10 percent improvement at 512 tokens with the 78/22 split. At 4096 tokens, the experiment moved roughly 282 MiB in each direction over 1 GbE and retained the same top-1 token. A same-ISA two-process control reproduced the single-Ryzen full-vocabulary logit hash bit-for-bit.

Then PR #378 found that the generated runtime had selected a newer prepared MoE provider while the preload intercepted only an older compact provider boundary. In other words, an apparently distributed run could bypass the transport path CKE thought it was measuring. The fix made the preload follow the provider selected by the kernel map.

The later correction in PR #380 retained the same top-1 token, 16/16 top-token overlap, and approximately 0.997875 full-logit cosine against the fresh single-node result. But the corrected heterogeneous computation was slower and was not promoted as a distributed speedup.

That correction is more useful than preserving the attractive number. It shows why provider identity, generated ABI, transport interception, and the exact execution axis must be evidence. The first run proved that the machines could communicate and compose an output. The hardened run clarified that this token-row schedule over 1 GbE had not yet earned a performance claim.

The Weight-Sharded Block Works, But The Ryzen Still Wins

The complete PR #382 measurement uses 4,096 rows through one real layer-3 Qwen3.5 full-attention-plus-MoE block. The numbers are useful because they separate three different questions: does the block reconstruct correctly, does dividing the compute reduce the critical compute path, and does this particular pair of machines win after communication?

The present two-node wall time is about 3.4× slower than one Ryzen. The P3 owns half of every shard, so the critical path waits for the slower machine. Four sequential 32 MiB TCP transfers over 1 GbE then add more time. ZIP therefore beats the P3 alone, but it does not beat the Ryzen. The matched-Ryzen figure is a compute projection from measured half-shards, not a measured two-Ryzen network result.

The numerical reconstruction is the more important result at this stage. On the same ISA, attention reconstructed with maximum absolute error 9.54e-7. The complete layer reconstructed with maximum absolute error 1.03e-3 and cosine similarity 0.999999999984. The heterogeneous hosts agreed on input, routing and final-output hashes after synchronizing the router decision. Cross-ISA execution is not bit-exact, so the stagewise comparisons remain necessary. This follows the same evidence discipline described in CKE X-Ray and the earlier article on numerical parity, ISA drift and reduction order.

There is still no matched isolated llama.cpp block measurement. CKE cannot claim a llama.cpp win from this experiment. The defensible statement is narrower: the weight-sharded architecture reconstructs a real Qwen3.5 block accurately and shows measured compute scaling that could become useful on matched nodes with a better collective transport.

A Separate Decode Result Shows Why Axis Choice Matters

PR #380 also improved Qwen3.5 decode, but this was inside one Ryzen node, not through distributed Zip. One token selected eight independent expert routes. CKE assigned those routes to persistent workers, gave each route disjoint scratch, and merged outputs in the original selected-slot order.

The routed provider improved from 38.09 to 11.27 ms per token, or 3.38x. Complete generated-model decode improved from 12.53 to a median 19.02 tokens per second across three runs. The matched llama.cpp result was 18.4 tokens per second on that fixture. The optimized and baseline CKE trajectories retained an identical SHA-256.

This result is not evidence that distributed decode works. It demonstrates the axis lesson: with one token row, the useful independent work was the eight selected expert routes, not the token dimension.

What CKE Should Test Next

1. Move the harness into generated-model lowering. PR #382 validates one block, but the full generated circuit does not yet carry rank ownership and collective boundaries through every layer.
2. Run decode with weight shards. The new partition is compatible with one-token work in a way token-row sharding was not; it still needs a measured generated decode path.
3. Keep intermediate state sharded longer. Two reductions per block are mathematically clear, but broadcasting every combined state may still be avoidable across compatible adjacent operations.
4. Replace sequential TCP carefully. First measure concurrent send/receive, then 10/25/100 GbE and RDMA. Faster networking should serve a sound graph schedule rather than conceal unnecessary transfers.
5. Repeat on matched Ryzen nodes. A second Ryzen can test whether the measured ~1.79× compute projection survives real collectives without the P3 controlling the critical path.
6. Add an isolated llama.cpp block boundary. Compare equivalent work before making any runtime-performance claim.
7. Report numerical trajectory with timing. X-Ray should identify the first layer and operation where changed reduction order moves logits during complete distributed decode.

Where Zip Stands Today

CKE has crossed two different lines. PR #377 established a real two-host token-row prefill path. PR #382 then split real Qwen3.5 attention and MoE weights so both machines contribute partial results to the same complete transformer block. The second experiment is the architecture CKE needs for distributed decode and additive model memory.

It is not fast enough on this heterogeneous pair. One Ryzen remains 3.4× faster than Ryzen plus P3 over 1 GbE, and the matched-node estimate has not yet been measured as a two-host wall time. The result is still consequential: the block reconstructs accurately, the compute halves scale, and the remaining work is now visible in the transport, node matching, generated lowering and synchronization schedule rather than hidden behind the wrong partition axis.