A multimodal model is not magic. The image becomes patches, patches become embeddings, and embeddings become a prefix the decoder can consume.
A vision-language model is not magic. The decoder does not consume pixels. The image is resized, divided into patches, embedded by a vision encoder, projected into the decoder dimension, and inserted into the token stream as vectors the text model can attend to.
The point is not to say support exists. The point is to show what must be true before support is meaningful.
What This Post Is Really About
C-Kernel-Engine is a lean transformer and AI-kernel engine built around explicit C kernels, generated execution paths, and measurable parity against reference behavior. The project is not trying to hide the model behind a black box. It is trying to make the path from model architecture to runtime artifact understandable: model circuit, GraphIR, lowering, kernel registry, memory plan, generated C, and run report. If you are new to the project, start with What Is the C-Kernel-Engine?, then read Templates Are Circuit Maps. The earlier Gemma4 attention article explains the text decoder's full, sliding, owned-KV, and shared-KV paths; this post follows the separate vision path into that decoder.
This post is part of the v8 hardening arc. v8 is focused on making inference, vision, model-family support, and runtime evidence more concrete. That does not mean CKE is inference-only. It means this lane is currently pressure-testing the pieces that make model execution inspectable before broader training and distributed work can be trusted.
| Reader question | Short answer | CKE artifact |
|---|---|---|
| What is CKE? | A lean transformer runtime and compiler path built around deterministic C kernels. | Circuits, IR, registry, generated C |
| Where are we in v8? | Hardening inference, vision, model-family bring-up, parity, and execution artifacts. | v8 runbook and vision encoder contract |
| Why should I care? | Because architecture support is only real when the path is inspectable and measurable. | Diagrams, tables, plots, parity checks |
Gemma4-VL Vision Bridge
That bridge is fragile. Pixel layout, patch order, 2D position IDs, spatial RoPE, merge policy, projector weights, and special tokens must agree. One mismatch can make the model run while quietly destroying the image signal.
$$ \text{image} \rightarrow \text{patches} \rightarrow \text{vision encoder} \rightarrow \text{projector} \rightarrow \text{decoder prefix} $$
How Gemma4 Vision Differs From Qwen3-VL
Gemma4 and Qwen3-VL both use a bidirectional vision transformer to turn an image grid into vectors for a text decoder. That common description is not enough to share one circuit. Their patch frontends, normalization, projection layout, position contracts, feed-forward blocks, spatial merge operations, projector structures, and decoder insertion policies differ.
| Contract surface | Gemma4 vision | Qwen3-VL vision | Why CKE must care |
|---|---|---|---|
| Patch frontend | One F16 patch projection | Dual patch projections, reordered sum, bias, and tiled position handling | The first encoder tensor already has different producers, storage precision, and spatial ordering. |
| Normalization | RMSNorm, Q/K normalization, post-attention norm, and post-FFN norm | LayerNorm around attention and MLP boundaries | Normalization equations, weights, epsilon, and rounding points are model-family contracts. |
| Q/K/V projection | Separate Q, K, and V projections without QKV bias | Biased packed QKV projection followed by an explicit split | Generated calls, weight names, layouts, and scratch buffers cannot be substituted by operation name alone. |
| Spatial position | Learned XY embeddings plus Gemma4V XY NeoX RoPE | Learned 2D embeddings, explicit rank-four position IDs, and four-section 2D M-RoPE | A generic “vision RoPE” kernel would silently rotate the wrong coordinate sections. |
| Vision MLP | Gate/up projections followed by GeGLU and down projection | Up projection, GELU, and down projection | The body circuit has different producers, activation math, and weight topology. |
| Spatial reduction | Average each merge-size by merge-size tile while preserving embedding width | Reorder/concatenate a tiled region into a wider projector input | Average pooling and feature concatenation produce different tensor dimensions and values. |
| DeepStack | No DeepStack side branches in the current circuit | Selected encoder layers are independently merged, normalized, projected, and concatenated with the main projector output | Qwen3-VL carries intermediate visual features into the decoder-width bridge; Gemma4 uses one final encoder path. |
| Projector | Scale, unit-RMS normalization, then one F16 linear projection | Two-layer projector with GELU plus concatenated DeepStack slices | The bridge is a numerical and dimensional contract, not simply “convert vision width to text width.” |
| Decoder bridge | Linear visual positions and a non-causal visual chunk between <|image> and <image|> | 2D M-RoPE positions and a causal mixed prefix between Qwen vision markers | Position progression, masking, KV state, and prompt tokenization differ after the encoder finishes. |
Why Qwen3-VL Has DeepStack And Gemma4 Does Not
In the current CKE Qwen3-VL circuit, configured encoder layers feed a fixed deepstack branch. Each selected hidden state is spatially merged, normalized, passed through its own two-layer projection, and collected as a feature slice. The footer concatenates those slices with the main projector output. This gives the decoder bridge access to visual representations from more than the final encoder boundary.
The current Gemma4 circuit has no equivalent branch. Its footer takes the final encoder stream, applies spatial average pooling, performs projector preparation, and executes a single linear projection. CKE should not add DeepStack merely because another vision family uses it; the circuit must follow the model's actual weights and architecture contract.
The Spatial Merge Is A Different Algorithm
Gemma4 reduces a dense patch grid by averaging each spatial tile. For merge size M, every output vector is the channel-wise mean of M² neighboring patch vectors:
Token count falls by M², but embedding width stays constant. Qwen3-VL's tiled merge instead rearranges neighboring patch features into the projector input and therefore changes the feature width. Calling both operations “spatial merge” is convenient at the model-description layer; their executable kernel contracts are different.
These details are visible in the current Gemma4 vision circuit and Qwen3-VL vision circuit. The comparison describes CKE's current implemented contracts; it does not claim complete Gemma4 semantic parity or optimized vision performance.
How A Patch Coordinate Moves Through RoPE
The word rotation can make RoPE sound as if the image patch itself moves around the grid. It does not. Patch order and tensor shape stay fixed. RoPE changes the orientation of two-dimensional pairs inside each patch token's query and key vectors.
Suppose one attention head has shape [T,D]. Row t contains the head vector for one patch token. If the patch grid has width W, Gemma4 recovers its coordinate directly from the flattened row:
RoPE then selects pairs of dimensions inside q[t] and k[t]. For a pair (z_0,z_1), it applies an ordinary two-dimensional rotation:
The pair keeps the same two storage locations. Its length is preserved, but its direction changes. Different pair indices use different frequencies, so one pair may rotate slowly across the image while another rotates rapidly. Because queries and keys use the same coordinate and frequency rule, their dot product becomes sensitive to relative spatial displacement.
Gemma4: Two Contiguous Axis Halves
Let the rotary width be R. Gemma4 divides those R dimensions into two contiguous spans. Dimensions 0 through R/2-1 form the X span. Dimensions R/2 through R-1 form the Y span. Each span uses NeoX split-half pairing: the first quarter is paired with the second quarter for X, and the third quarter is paired with the fourth quarter for Y.
For pair index i, the angles are conceptually:
where b is the vision RoPE frequency base. CKE's Gemma4V kernel defaults that base to 100 when the model does not provide a valid value. The same operation is applied independently to every Q head and K head.
A Small Gemma4 Pair Example
Take a patch at (x=2,y=1). Suppose an X-controlled pair contains (z_0,z_1)=(1,0) and its frequency for this pair is 1. Its angle is 2 radians, so the rotated pair becomes approximately:
A Y-controlled pair at the same frequency uses angle 1 instead. The patch has one location, but different embedding pairs encode its horizontal and vertical coordinates separately.
[4,T] must stay aligned with the reordered patch rows.Qwen3-VL: Explicit Position Streams And Sections
Qwen3-VL does not recover coordinates inside the RoPE kernel from t and grid_w. CKE first generates an integer position tensor with shape [4,T]:
The rows are [Y,X,Y,X]. Positions are emitted in the same merged-tile traversal used to reorder the patch tensor. For a 2×2 merge tile, the local order is (0,0), (1,0), (0,1), (1,1). If the position tensor remained in ordinary row-major order while activations used tile order, every later rotation would be attached to the wrong patch.
The M-RoPE call also carries four section widths. These widths state which coordinate stream controls each band of rotary pairs. Within one pair, dimension i is paired with dimension i+n_dims:
where s(i) selects the position stream for pair i and ωᵢ is its frequency. The current Qwen3-VL manifest uses duplicated Y/X streams and explicit section metadata compatible with the reference M-RoPE interface. Exact effective widths must come from the converted model manifest rather than being guessed in codegen.
The Most Important Difference
Gemma4's kernel owns a fixed geometric rule: flatten the grid, derive x and y, then apply X to one contiguous rotary half and Y to the other. Qwen3-VL separates geometry from rotation: another kernel emits explicit coordinate streams in merged-tile order, and section metadata tells M-RoPE how those streams map onto pair bands.
| Question | Gemma4 XY RoPE | Qwen3-VL M-RoPE |
|---|---|---|
| Where do coordinates come from? | Derived from token row and grid width inside the kernel. | Read from an explicit [4,T] position tensor. |
| How are axes assigned? | Fixed contiguous X half and Y half. | Model-provided section widths select position streams for pair bands. |
| How are dimensions paired? | NeoX split-half pairing inside each axis span. | Dimension i pairs with i+n_dims. |
| Does tensor shape change? | No; Q and K remain [heads,tokens,head_dim]. | No; Q and K remain [heads,tokens,head_dim]. |
| Primary alignment risk | Wrong grid width, rotary width, or frequency base. | Position stream order or section metadata disagreeing with reordered patch rows. |
The corresponding CKE implementations are the Gemma4V XY and vision M-RoPE kernels, with the Qwen position tensor generated by vision_position_ids_2d_merge. These are executable numerical contracts, not interchangeable illustrations of the same generic RoPE operation.
The CKE Surface
The useful way to read this topic is as an interface boundary. The model family describes what should happen. The compiler artifacts describe how that intent flows. The registry and memory planner decide which concrete kernel and buffer layout make sense. The generated C is the final executable statement of those choices.
| Stage | CKE kernel idea | Failure mode |
|---|---|---|
| Patch extraction | im2patch / spatial merge | Wrong pixel layout |
| 2D positions | vision position IDs | Wrong geometry |
| Vision RoPE | Gemma4V XY RoPE | Wrong spatial rotation |
| Projector | vision projector prep | Bad decoder prefix |
Concrete Example
The CKE mental model should be explicit.
RGB image
-> resize / normalize
-> patch grid
-> vision encoder blocks
-> projector to decoder width
-> image-prefix embeddings
-> text decoder prefillThe reason this belongs in CKE is that multimodal support is a runtime problem, not only a model-loading problem. The engine needs a circuit for the vision side, a bridge into the decoder, and parity checks that isolate whether failure came from image preprocessing, vision hidden states, projector output, or text-side attention.
What Evidence Should Look Like
For CKE, the important habit is to connect claims to artifacts. A post should not only say that a model path exists. It should point to the specific circuit, lowering decision, kernel contract, memory behavior, or parity signal that makes the claim credible.
| Bridge stage | Common failure | Useful check |
|---|---|---|
| Preprocess | Wrong normalization or channel order | Known image tensor checksum |
| Patch grid | Wrong spatial order | Patch index visualization |
| Vision encoder | Layer drift | Hidden-state parity |
| Projector | Bad decoder prefix | Projected embedding diff |
What Can Go Wrong
The most dangerous failures are the quiet ones. A loud failure is easy: the generated file does not compile, a symbol is missing, or a smoke test crashes. A quiet failure is worse because the system appears to work while the math has drifted. The model may still emit fluent text, the graph may still look reasonable, and the demo may still be convincing. That is why CKE needs artifact-level checks instead of demo-level confidence.
The common pattern is a boundary mismatch. A template may describe the wrong block order. GraphIR may connect a tensor to the wrong consumer. LoweredIR may select the wrong dtype path. The registry may pick a kernel whose layout constraints are not satisfied. The memory planner may reuse a buffer whose lifetime has not actually ended. Codegen may preserve the shape but emit a pointer expression that indexes the wrong stride. Each mistake is small in isolation; together they are exactly why runtime engineering needs discipline.
Practical Takeaway
The practical takeaway is to read CKE artifacts like a kernel engineer, not like a framework user. Do not stop at the model name. Ask what exact circuit was compiled. Ask which kernels were selected. Ask which buffers are persistent and which are scratch. Ask where the first parity diff appears. Ask whether the claimed performance path is compute-bound, bandwidth-bound, cache-bound, or synchronization-bound. Those questions turn the project from a black-box runtime into a systems laboratory.
That is also the content strategy for this series. Each post should give readers one more inspection tool: how to read a template, how to reason about lowering, how to understand a registry contract, how to separate prefill from decode, how to verify family-specific behavior, and how to treat memory as a designed surface. The goal is not to make the engine sound finished. The goal is to make the engineering legible.
CKE Engineering Note
The v8 story should be honest: vision hardening is about making every boundary observable. Pixels to patches, patches to embeddings, embeddings to decoder prefix.
How To Read This As A Kernel Engineer
A kernel engineer should read this as a constraint map. What is the data shape? Which buffers live across calls? Which operation is allowed to reorder memory? Which part is numerically sensitive? Which part is bandwidth-bound? Which part is compute-bound? If those questions are not visible in the artifact chain, the engine is still too magical.
The broader CKE philosophy is that CPU AI is not a fallback path. It is a serious systems surface. The machine has memory hierarchy, vector units, cache lines, branch behavior, thread placement, and network boundaries. Good software should expose those constraints and then make them useful.
Why This Belongs In The Series
This topic expands the public explanation of CKE beyond generic transformer concepts. It helps readers understand the real engineering surface: how model families become circuits, how circuits become kernels, how buffers become pointer expressions, and how correctness is measured before performance claims matter.
Related Reading And Source Evidence
- What Is the C-Kernel-Engine? introduces the compiler, runtime, and generated-C architecture.
- Gemma4 In CKE: Four Attention Paths, Shared KV, And Sliding Windows explains the decoder-side architecture that receives the visual prefix.
- CKE v8 Vision Encoder documents the current vision execution contract and validation lane.
- CKE v8 Numerical Contracts explains why layout, dtype, rounding, and reduction order are part of correctness.
- CKE Model And Kernel Matrix tracks model-family and kernel coverage without treating every path as equally mature.
- Gemma4 vision circuit and Qwen3-VL vision circuit expose the concrete architectural differences discussed above.
- Vision RoPE kernels and vision preprocessing and position kernels show the executable C contracts.
Gemma4-VL support in CKE is still being developed. CKE is preparing compatibility by making the Gemma4 vision circuit, dimensions, conversion path, image-marker policy, XY RoPE, spatial pooling, projector bridge, generated C, and validation boundaries explicit.
This article documents the intended and currently implemented engineering surface. It does not claim complete reference parity, support for every Gemma4-VL checkpoint or image shape, optimized CPU performance, long-generation stability, or production readiness. Those claims require successful conversion, leaf-kernel tests, bridge smoke tests, intermediate-tensor comparisons, end-to-end semantic tests, and measured profiling results.
As that evidence closes, this note will be updated with exact model hashes, hardware, commands, commits, parity results, known limitations, and performance measurements.