It is easy to summarize Gemma4 as “a model with sliding-window attention.” That description is incomplete. The current C-Kernel-Engine v8 circuit has to represent several interacting decisions: which layers see the full prefix, which see only a local window, which layers produce new keys and values, which reuse KV from another layer, and which RoPE and normalization policy applies to each path.

This matters because a model can load, compile, and emit fluent text while still implementing the wrong architecture. Meaningful support requires the circuit, dimensions, operation order, selected kernels, generated C, and numerical comparisons to agree. If CKE is new to you, first read What Is C-Kernel-Engine? and Templates Are Circuit Maps.

The Four Attention Paths

For a token position t, full causal attention may read every valid key from position 0 through t. Sliding attention restricts that set to the most recent W positions. KV ownership is a separate issue. An owned-KV layer computes Q, K, and V. A shared-KV layer computes only Q and consumes K and V produced by an earlier designated layer.

\(\displaystyle \mathcal{I}_{\mathrm{full}}(t)=\{0,\ldots,t\},\qquad \mathcal{I}_{\mathrm{slide}}(t)=\{\max(0,t-W+1),\ldots,t\}\)

For either index set, attention still has the familiar form:

\(\displaystyle \operatorname{Attn}(q_t,K,V)= \sum_{j\in\mathcal{I}(t)}\operatorname{softmax}_j\!\left( \frac{q_t k_j^{\mathsf T}}{\sqrt{d_h}}\right)v_j\)

The distinction is explicit in the v8 Gemma4 circuit. It is not inferred later from an accidental tensor shape. The converter constructs a per-layer attention plan, and the circuit expands each kind into a different operation list.

Owned KV Versus Shared KV

In an owned-KV path, CKE projects the normalized hidden state into Q, K, and V. It applies the family’s Q/K normalization contract, normalizes V where required, performs split-layout RoPE on Q and K, and writes the resulting K/V state. The attention kernel then reads that state using either a full or sliding visibility policy.

A shared-KV path must not quietly run the same block and discard its K/V result. It is structurally different: Q projection, Q normalization, Q-only RoPE, then attention against the owner layer’s K/V buffers. That distinction affects tensor dependencies, memory lifetimes, generated function arguments, and the kernels that are legal to select.

owned KV layer:
  h_norm -> Q projection -> Q norm --+-> RoPE(Q,K) -> attention
         -> K projection -> K norm --+
         -> V projection -> V norm ------> KV storage

shared KV layer:
  h_norm -> Q projection -> Q norm -> RoPE(Q) -> attention
                                              ^
                         K,V from owner layer-+

Sliding Attention Is A Memory Policy

Sliding attention does not replace attention math. It changes which rows of the KV history are eligible. During decode, a window of size W bounds the historical KV read per sliding layer. During prefill, the mask and kernel must enforce the same local causal range for every query position. Full-attention layers remain the model’s periodic global-memory path.

This is why “Gemma4 uses sliding attention” should not be interpreted as “Gemma4 eliminated full attention.” Hybrid architectures trade some photographic access to the complete token history for lower memory traffic and lower attention work, then retain full-attention layers to refresh global information. The exact layer plan is model metadata, not a universal alternating pattern that a runtime should guess.

RoPE Is Also Per Layer

CKE’s Gemma4 contract uses split-layout RoPE and a per_layer_direct parameter mode. This is more than selecting a label such as full or swa. Gemma4 metadata can provide a full-attention rotary dimension and a separate sliding-attention rotary dimension, along with distinct full and SWA frequency bases. CKE materializes layer_rotary_dim and layer_rope_kind for every layer; GraphIR then resolves the effective full or SWA theta into that layer’s concrete rope_freq_base.

If a head has width d but a layer declares rotary width d_r, only the configured rotary channels participate in the position-dependent rotations. Changing d_r changes how much of each query/key head carries explicit positional phase; changing the frequency base changes the angle schedule across those channel pairs. Full and sliding layers can therefore encode position with different geometry even before their attention masks differ.

Owned-KV layers rotate Q and K together. Shared-KV layers rotate only their newly produced Q because their K has already been prepared by the owner path. This is why CKE needs both rope_qk and a Gemma4-specific Q-only RoPE path.

How The Template Carries RoPE Across Layers

The architecture information moves through CKE in stages rather than being rediscovered inside a kernel:

The implementation boundaries are directly inspectable: the GGUF converter builds the explicit layer plan, the Safetensors converter reads full/sliding RoPE parameter groups, and build_ir_v8.py resolves each layer’s frequency base and rotary dimension into concrete operation parameters.

This separation is deliberate. The RoPE kernel should not know that it is executing “layer 17 of Gemma4.” It should receive explicit pointers, head dimensions, rotary width, position offset, and frequency base. Architecture policy belongs in metadata, the circuit, and IR; the kernel performs the resolved math.

A wrong RoPE layout is particularly deceptive. Shapes remain valid, compilation succeeds, and the model may still produce language. The geometric relationship between Q and K is nevertheless wrong, so attention scores and logits drift. CKE therefore treats RoPE as an explicit operation and kernel contract rather than a generic helper hidden inside attention.

Critical Gemma4 Kernel Maps

• rope_forward_qk_split_direct.json: owned-KV Q/K rotation with explicit rotary_dim and per-layer freq_base.
• rope_forward_q_gemma4.json: Q-only rotation used by Gemma4 shared-KV paths.
• attention_forward_causal_head_major_gqa_flash_strided_sliding_gemma4.json: owned-KV sliding prefill attention.
• attention_forward_causal_head_major_shared_kv_sliding_gemma4.json: shared-KV sliding prefill attention.
• gemma4_final_logit_softcap_forward.json: the final tanh logit-softcap contract.

The Rest Of The Gemma4 Block Still Matters

The attention dispatch is only the center of the circuit. The current v8 template also describes the surrounding numerical order:

1. RMS-normalize the block input.
2. Execute one of the four attention paths.
3. Apply the post-attention normalization and residual update.
4. Apply the feed-forward normalization.
5. Compute gate and up projections, combine them with GEGLU, and apply the down projection.
6. Apply post-FFN normalization and the second residual update.
7. Apply the configured per-layer embedding contribution.

The footer then performs final RMSNorm, tied-output projection, logits, and Gemma’s final logit softcap. For cap c, the limiting operation is:

\(\displaystyle z' = c\tanh\!\left(\frac{z}{c}\right)\)

That softcap is not sampling decoration. It changes the numerical logits presented to token selection and therefore belongs in the model contract.

How CKE Represents The Architecture

The current scaffold tests check the architecture plan itself: layer classification, explicit plans read from metadata, all four path kinds, Q-only shared-KV operation lists, per-layer windows and RoPE kinds, V normalization behavior, and per-layer embeddings. Those tests establish structural intent. They do not, by themselves, prove complete end-to-end numerical parity or peak CPU performance.

What Is Still Being Hardened

This post documents the current architecture contract, not a declaration that every Gemma4 variant is finished. The remaining engineering standard is stricter: conversion must preserve model metadata; generated IR must expose the expected owner and consumer relationships; prefill and decode must agree with a reference at intermediate boundaries; mixed-quant kernels must respect their numerical contracts; and profiling must show where CPU time and memory traffic actually go.

That is the larger point of CKE’s circuit approach. Adding model support should not mean growing an opaque pile of model-name conditionals. A model family should become an inspectable circuit, a concrete per-layer plan, explicit kernel contracts, and generated C that can be measured.

Related Reading

• Templates Are Circuit Maps: How CKE Describes A Model Family
• The CKE v8 IR and code-generation pipeline
• CKE v8 Numerical Contracts
• CKE v8 Inference And Training Runbook
• Quantized Weight Formats Used By CKE
• Gemma4 v8 circuit source