A decision model has a different job from a conversational model. Sometimes we do not need a paragraph. We need to choose an action from a small set: read a record, ask for clarification, or send a case for review. The interesting part is how we train that choice, where the gradients go, and what survives when we export the model.

I wanted to follow that path all the way through C-Kernel-Engine: start with a small Clef head, describe its computation, generate the native training runtime, run AdamW, export the weights and load the head again. Then do the same from a Jupyter notebook, where we can inspect every intermediate result.

This article uses an actual 40-update FP32 run. It also includes a useful detour: the first run found a validation-loss reporting bug. We fixed the runtime, regenerated the code and checked the results again. That is part of building a runtime too.

New to CKE or Clef? Start here

C-Kernel-Engine (CKE) is the runtime project behind this experiment. It describes model computations, connects them to native C kernels and generates the code that executes them. Here we use that machinery to train and reload a small decision head.

For the architecture first, read From LM Heads to Decision Heads: How Clef-Style Models Work and Train. That article explains the move from predicting a next token to scoring allowed choices. This walkthrough follows the next step: preparing the tensors, updating the head and checking its predictions.

What we will build

• A small head with two evidence-routing layers and four joint field layers.
• A pinned package containing weights, supplied features, question spans, candidate spans and labels.
• A generated forward/backward graph and native AdamW updates.
• An exported head that produces the same scores after reload.
• A CLI and notebook workflow we can inspect before connecting an application UI.

The features in this example are synthetic. We are testing the training machinery, not claiming that a tiny model has learned to understand support requests.

1. What is the Clef head actually doing?

Think of the backbone as a machine that turns a context into a sequence of feature vectors. The head receives those vectors along with a structured set of questions and allowed answers. It returns a score for each answer.

For example, a real application might ask “What should happen next?” with choices read and clarify. Another question might describe the handling mode. These fields can be related, so the head reasons over them jointly rather than treating every choice as an unrelated output neuron.

Cloudflare describes Clef as a prefill-based decision model using a joint schema head. Its published training approach includes components beyond the head-only experiment here. Our run starts from supplied features: it does not execute or fine-tune Qwen. Cloudflare’s Clef introduction and the Clef-Flash model card provide the original model context.

Follow the tensors through the head

1. Normalize and project the memory. Hidden states become the memory that attention can read.
2. Construct option representations. Combine context pooled from an option span, lexical information from embedding rows and the corresponding question representation.
3. Route evidence. Two evidence layers refine the option representations using attention over the available context.
4. Build field representations. Pool the options for each question, then combine that summary with the question, global context and question-type information.
5. Reason across fields. Four field layers mix these representations and attend to the memory.
6. Score each candidate. Combine prior similarity, joint similarity and a learned residual scoring branch.

At the kernel level, this means linear projections, LayerNorm, attention, exact-erf GELU, span means, indexed gathers, weighted pooling, normalization and additions. The head is a composition of operations, not one extra matrix multiplication attached to the end.

2. How CKE designs and stitches the runtime

In CKE, the “template” for this work is a typed computation graph. Each operation declares the tensors it consumes, the shape it produces, its kernel binding and the weights it needs. Those declarations let the runtime generator assemble an execution schedule.

If you want to follow that assembly process more closely, Templates Are Circuit Maps explains the model description, and v8 IR Pipeline Codegen follows the intermediate representation into generated C. Here we extend that view to the backward pass and optimizer updates.

The v8 recipe currently reuses parts of the v7 training code generator. Those filenames describe shared implementation, not a second model running inside the head.

The generated output includes train.c, a compiled train.so, backward_ir.json, layout.json and runtime identity metadata. Python loads records and calls the compiled entry points. It does not perform this run’s backward pass or parameter updates. The notebook verifies that distinction by blocking PyTorch and Transformers imports in the training child process.

PyTorch is used to construct the small deterministic fixture weights. That is separate from the native training execution. This is a teaching fixture from the pinned repository, not a claim that fixture helpers are a stable public SDK.

3. What decisions are we training?

There are two training records and one validation record. Each contains six token positions with eight feature channels. The head width is eight, with two attention heads and a feed-forward width of sixteen. This keeps compilation and inspection manageable while retaining the multi-layer head structure.

The action field has two candidates: [read, clarify]. The mode field has one: [normal]. The latter checks grouped bookkeeping but gives the model no meaningful choice. Its probability is one by construction.

The hidden vectors are seeded random numbers. Giving a candidate the name “clarify” does not imbue the vectors with language understanding. A real dataset needs features captured from an actual backbone and aligned to the context and schema that produced them.

One record at the tensor boundary

# Frozen token features: 6 positions × 8 channels
hidden:             float32[6, 8]

# Token IDs and half-open [start, end) spans
input_ids:          int32[6]
question_spans:     int32[2, 2]
option_spans:       int32[3, 2]
global_span:        int32[1, 2]

# Two questions; three candidate options in total
option_counts:     int32[2]    # [2, 1]
question_type_ids: int32[2]
labels:            int32[2]    # [1, 0] → clarify, normal

Labels are local indices within each question’s candidates. Label 1 means the second action choice; it does not mean candidate 1 in a flattened list shared by every field. The package also pins model, backbone and tokenizer identities, initial weights, record files and their hashes. The loader checks shapes, bounds and finite values before native execution.

4. The loss, gradients and AdamW update

Let the head produce scores z for a question. Softmax turns those scores into probabilities within that question. Cross-entropy penalizes low probability on the labelled answer. Average over the questions to obtain the record loss.

p[q, j] = exp(z[q, j]) / sum_k exp(z[q, k])
L = -(1 / Q) * sum_q log(p[q, label[q]])

dL/dz[q, j] = (p[q, j] - 1[j == label[q]]) / Q

# Here Q = 2. The one-option mode field contributes zero loss.
L_record = -log(p_action[target]) / 2

That logit gradient flows backward through the scoring branches, field layers and evidence routing into the trainable head parameters. The supplied hidden features and output embedding table are frozen. Frozen does not mean irrelevant: their values affect the computation, but the optimizer does not change them.

AdamW keeps first and second moment estimates for each trainable parameter. It uses those estimates to scale the update and applies decoupled weight decay. This recipe uses FP32, learning rate 0.001, epsilon 0.0001, beta values 0.9 and 0.999, and weight decay 0.01. It updates once per training record; twenty passes over two records produce forty updates.

Muon is an interesting comparison for matrix parameters, but it is not a supported optimizer in this registered recipe. Selecting a different optimizer requires its own parameter grouping, implementation and numerical checks. For the background, see Muon: from wx + b to matrix updates.

5. Run it from the command line

Use a separate checkout so the generated runtime and artifacts belong to a known source revision. The commands below pin the corrected implementation used for this article. They assume a Linux development environment with Git, a C compiler, Make and Python virtual environments.

git clone https://github.com/C-Kernel-Engine/C-Kernel-Engine.git cke-clef-demo
cd cke-clef-demo
git checkout 63105d9a86efd013dc58e4360e928fc656bb3a97
python3 -m venv .venv
. .venv/bin/activate
python -m pip install numpy torch
make -j2
export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

Create the synthetic package using the fixture included at this revision. Use a fresh directory each time; keep previous results intact.

from pathlib import Path
import json
from tests.test_v8_clef_training_recipe import fixture

folder = Path("demo/package")
folder.mkdir(parents=True, exist_ok=False)
path, digest, _ = fixture(folder)
Path("demo/identity.json").write_text(json.dumps({
    "path": str(path.resolve()), "sha256": digest
}, indent=2))
print("Package:", path)
print("SHA-256:", digest)

python make_demo_package.py
PACKAGE_SHA=$(python -c 'import json; print(json.load(open("demo/identity.json"))["sha256"])')
python version/v8/decision/train_clef_head.py \
  --package demo/package/package.json \
  --package-sha256 "$PACKAGE_SHA" \
  --run-dir demo/trained \
  --json-out demo/result.json \
  --epochs 20 --lr 0.001

The recipe validates the package, generates the runtime, trains on records marked train, evaluates held-out records and exports the head. A successful result includes checks for native execution, frozen embedding preservation, checkpoint reload and inference export.

Checkpoint reload checks that serialized weights and optimizer state can be read back. Export parity checks that the generated inference head reproduces the training runtime’s logits. Neither check tells us whether the model is useful on new real-world examples.

6. What happened in our run?

The two action targets became more likely. The first training record initially preferred the wrong action; after training it selected its label, clarify.

The training trace alternates between two different records. Comparing the first point directly to the last mixes both parameter changes and different inputs. The chart separates the records so we can see each trajectory.

For the single held-out record, mean grouped cross-entropy decreased from 0.22789864 to 0.17800277. One held-out synthetic record is a useful execution check, not an estimate of deployment accuracy or calibration.

The original CLI run took about 7.1 seconds on P3, including code generation and export. Treat that as a measurement of this tiny fixture on this machine, not a throughput benchmark for a released-size head.

We trained on these examples. Now ask the head to decide.

Let’s make the result concrete. The training set contains record-0 → clarify and record-1 → read. Each input is a different frozen 6 × 8 feature matrix, together with token IDs and question/option spans. During training, the labels tell the loss which answer to reward. We repeat those two records for twenty epochs.

Record-2 is held out. It never contributes an optimizer update. After training, we pass its features through the head, rank the two action scores, and compare the selected action with its stored label. The label is used to evaluate the answer, not to choose it.

The clearest change is record-0: the untrained head preferred read, but training moved it to clarify. Record-1 already selected read and became more confident in that choice. The held-out record already selected clarify; its probability increased, but its selected action did not change.

What does the reloaded model actually return?

We exported the trained head, loaded it again and requested record-0-q0, the action question for the first record. This is the relevant excerpt from the actual response:

{
  "scores": [
    {
      "candidate_id": "read",
      "logit": -0.847121000289917
    },
    {
      "candidate_id": "clarify",
      "logit": 0.16207897663116455
    }
  ],
  "score_semantics": "raw_logits",
  "decision": {
    "status": "ranked",
    "reason": null,
    "candidate_id": "clarify"
  },
  "executed": false
}

The head returns raw logits: approximately −0.8471 for read and 0.1621 for clarify. Softmax over these two values gives 26.71% and 73.29%. The higher score selects clarify. These are model probabilities, not a calibrated guarantee that the answer is correct.

executed: false matters: the model ranked an action; it did not send a message or read a database. In a real application, a separate policy layer would check permissions and decide whether to act, abstain or ask for review.

There is no natural-language request hidden behind these examples. The input vectors are synthetic, so we cannot honestly translate record-0 into a sentence such as “the customer forgot their account number.” That would require a real labelled text dataset and features from a pinned backbone. What this run demonstrates is the complete mechanical path: labelled features → native updates → exported scores → a selected candidate.

A longer experiment: can it learn—or just memorize?

After the three-record walkthrough, I expanded the test to 16 training records and 8 held-out records. Every record has a different seeded random feature matrix. Action labels alternate between read and clarify, giving balanced classes, with no meaningful relationship between those labels and the random features. The single-option mode field remains unchanged.

I ran three experiments from identical initial weights and the same data split: 2, 10 and 40 epochs, or 32, 160 and 640 native AdamW updates. Learning rate stayed at 0.001. These are separate runs from the same initialization, not a checkpoint-resume benchmark.

The training fit improved dramatically. The held-out decisions did not improve beyond the 50% always-read baseline in the two longer runs. Held-out loss increased between 160 and 640 updates, even as training loss approached zero. With these independent random features, there is no useful underlying task rule to discover.

This is a small, controlled example of fitting the training set without learning a useful general rule. It does not establish how Clef performs on real text. Eight held-out records also make accuracy coarse: one changed answer moves the result by 12.5 percentage points.

All three runs passed native execution, frozen-embedding preservation, checkpoint reload and exported-head score checks. I also recalculated held-out grouped cross-entropy from the returned logits; it matched CKE’s report. Each CLI run took roughly 7.2–7.4 seconds including code generation and export on P3. These are bounded tests, not overnight runs or production performance benchmarks.

For a real application, this is why we need labelled examples with a meaningful feature–label relationship, a held-out evaluation set, and a stopping rule based on validation results. Simply increasing the number of updates cannot supply the information missing from the dataset.

Scaling up: 256 records and 7,680 native updates

The next CKE revision, PR #1111, expanded the fixed-geometry dataset limits and added periodic validation and checkpoints. I repeated the experiment at commit 7325ab9fe97a249f34ea97e396ac5f326a882e88 with 192 training records and 64 held-out records, each containing 96 token positions and eight channels.

The features are still seeded random values and the action labels are balanced, independent of the features. The two evidence layers and four field layers are unchanged. This run tests a larger dataset and execution schedule; it does not introduce Qwen or a real language task.

The pattern is clearer now: training accuracy increases, while held-out loss worsens. At 7,680 updates the head classifies 182 of 192 training records correctly, but only 25 of 64 held-out records. Always choosing read would score 32 of 64 on this balanced held-out set. These observed results do not establish that the model is systematically worse than chance; they show that this run did not learn a useful general rule from random features.

Did the new reporting and checkpoint path work?

• All three runs passed native execution, frozen-embedding preservation, checkpoint reload and generated-head export checks.
• The long run recorded validation at steps 0, 1,000, 2,000, 3,000, 4,000, 5,000, 6,000 and 7,000, plus final metrics at 7,680.
• The retained checkpoints were 6,000, 7,000 and the final 7,680—two periodic checkpoints plus the final one.
• Independent accuracy calculations matched CKE’s per-question report. Confusion-matrix totals matched the split sizes, and held-out cross-entropy matched the logits.
• The longest run used about 109 MiB peak process RSS. Its 14.7-second CLI time includes code generation and export; this is tiny-head timing on P3, not a released-model benchmark.

The action confusion matrix below comes from the longest run. Rows are expected labels; columns are predicted labels. The one-option mode question is excluded because it cannot make a meaningful classification error.

The recipe now admits up to 4,096 records, 512 positions, 1,000 epochs and 100,000 total updates, subject to data and workspace limits. These are admission ceilings, not tested performance claims for every possible geometry. Our largest run here used 256 records and 7,680 updates.

We can now train larger fixed-shape datasets. The remaining real-text work is a checked multi-record feature converter and support for mixed lengths or appropriate shared-optimizer shape buckets. Those are different problems from increasing the update budget.

7. Use Jupyter to inspect the same computation

A notebook does not require a separate training engine. It can create the package, call the same CLI with subprocess, read JSON artifacts and plot the result. That keeps the executable path identical while adding a convenient inspection surface.

The companion notebook checks the source revision, builds the fixture, captures pre-training logits, runs forty native updates, calculates grouped probabilities, reloads the exported head and independently verifies validation loss. It was executed with a real Jupyter kernel on P3.

# In the same virtual environment as the CLI example:
python -m pip install jupyterlab ipykernel
python -m ipykernel install --user --name cke-demo
jupyter lab

Choose that environment’s kernel. Run this cell after the CLI package has been created, using a new output directory.

from pathlib import Path
import json, subprocess, sys
root = Path.cwd()
identity = json.loads((root / "demo/identity.json").read_text())
subprocess.run([
    sys.executable, "version/v8/decision/train_clef_head.py",
    "--package", identity["path"],
    "--package-sha256", identity["sha256"],
    "--run-dir", "demo/notebook-trained",
    "--json-out", "demo/notebook-result.json",
    "--epochs", "20", "--lr", "0.001"
], check=True, timeout=600)
report = json.loads(Path("demo/notebook-result.json").read_text())
print(report["status"], report["completed_steps"])
print(report["metrics"])

Then calculate held-out loss yourself. This small independent check caught the original bug. Use stable log-sum-exp and normalize each question separately.

import numpy as np

def grouped_cross_entropy(logits, counts, labels):
    losses, offset = [], 0
    for count, label in zip(counts, labels):
        z = np.asarray(logits[offset:offset + int(count)], dtype=np.float64)
        shifted = z - z.max()
        losses.append(np.log(np.exp(shifted).sum()) - shifted[int(label)])
        offset += int(count)
    assert offset == len(logits)
    return float(np.mean(losses))

# From this fixture's manifest/record; do not assume these for another dataset.
held_out = next(r for r in report["records"] if r["id"] == "record-2")
independent = grouped_cross_entropy(held_out["logits"], [2, 1], [1, 0])
assert np.isclose(independent, report["metrics"]["validation_loss_after"],
                  rtol=1e-5, atol=1e-7)
print("Held-out loss:", independent)

Load the exported head and score a fixture case

from server.decision.native import NativeHead
from pathlib import Path
head = NativeHead(
    Path(report["artifacts"]["inference_manifest"]),
    report["artifacts"]["inference_package_sha256"]
)
try:
    status, reply = head.evaluate(head.request("record-0-q0"))
    assert status == 200 and reply["ok"]
    print(reply["result"]["scores"])
finally:
    head.close()

This scores an exported supplied-feature case. It is not a text API: passing a new sentence here does not automatically tokenize it or run Qwen. The actual backbone capture path must provide compatible features and schema metadata.

8. Move from the fixture to a real task

Start with one bounded decision that has a clear label. For a request-routing system, that might be read-only lookup, clarification or human review. Define what each choice means before collecting examples. If annotators disagree, record that ambiguity rather than assuming the model can resolve an unclear policy.

1. Define the schema. Stable question and candidate IDs; an explicit way to handle uncertainty.
2. Collect labelled contexts. Keep source, permissions, task definition and label provenance.
3. Split before feature capture. Separate related cases, customers or time periods so near-duplicates do not leak across splits.
4. Pin the backbone and tokenizer. Record their revisions and the prompt/schema template. Changing them can change the feature distribution.
5. Capture and validate features. Align token spans to the exact sequence, hash arrays and inspect samples. Do not reuse this toy geometry blindly.
6. Train within a declared budget. Keep checkpoints, configuration and source identities together.
7. Evaluate decisions. Measure per-question accuracy, error costs, calibration and abstention on enough held-out examples.
8. Reload and compare. Verify serving scores agree with the exported artifact before applying an application policy.

A model choosing “read” does not authorize a database read. The application still checks the user, tenant, resource and permitted action. Probabilities describe model outputs; they do not replace access control.

Four real-text applications to try next

The same head-training pattern can support different bounded choices. The table below describes planned experiments, not results from the synthetic run above. Each needs text converted into compatible backbone features before CKE can train on it.

For each application, we want to show the actual held-out text, the available choices, the model’s scores, its selected answer and the expected label. We should show mistakes as well as correct predictions.

Can we use datasets from an Unsloth workflow?

We can reuse appropriate underlying datasets; the preparation changes. Unsloth’s dataset guide covers structured and conversational training data. CKE’s supplied-feature head recipe expects arrays and schema metadata, not an instruction/chat dataset dropped directly into the training command.

Our conversion path is: labelled text → question and candidate schema → pinned tokenizer/backbone → aligned feature package → native head training. We must preserve splits and provenance through that conversion. Dataset labels and permissions also need to fit the intended application.

How big is the model in this walkthrough?

The trained toy head has 5,596 parameters, plus a frozen 128-value output embedding table. Its input is a synthetic 6 × 8 feature matrix. No Qwen backbone ran in this demonstration. The published Clef-Flash model is based on Qwen3.5-9B. Its scale and capabilities must not be confused with this small runtime exercise.

The original walkthrough revision allowed 32 records and 64 token positions. The later scaling experiment above verifies the expanded recipe with 256 records and 96 positions. The original reference feature-capture workflow used a development-only PyTorch backbone path. Later PR #1112 added a host-orchestrated native backbone/head path, discussed in the closing update. These synthetic experiments do not establish variable-length training or real-text application quality.

9. Where Antsand AI Studio fits

Antsand is the software factory we build and use internally to manage our websites, content and application workflows, including ShivasNotes; the Factory method explains how those reusable pieces fit together. We may open it to selected customers or users later, but today it serves our own work. I mention AI Studio here because it gives us an interface for organizing CKE experiments and reviewing their configuration and results as the integration develops.

The CLI and notebook are useful because we can see the computation directly. A shared UI adds a different layer: who owns the experiment, which package was selected, what configuration was reviewed, and where the results belong.

Antsand currently supports group-owned experiment drafts with shared UI/API validation and revision checks. Clef drafts record the package ID and hash, FP32 head-only AdamW configuration, epochs and learning rate. Saving a draft does not launch training.

CKE’s local jobs v2 contract supplies discovery, prepare, submit, observe and artifact retrieval. It preserves Antsand’s opaque revision token, binds plans to worker identity and capabilities, and supports idempotent submission. The authenticated remote connection still needs to be implemented before the Studio can submit these jobs to our CKE compute server. We call that machine P3 internally; it runs the training experiments described here, while Antsand provides the application interface.

That gives us a practical sequence: get the numerical work correct, make the CLI reproducible, inspect it in a notebook, then connect the same job contract to the UI. We can add the interface without hiding what the runtime is doing.

Keep exploring

Choose the part you want to follow next. For the Clef architecture, start with the first article.

Sources and reproducibility

The original 40-update walkthrough pins CKE to 63105d9a86efd013dc58e4360e928fc656bb3a97. The measurements come from our small synthetic-feature run; they are not Cloudflare model benchmarks.

• Cloudflare: introducing Clef decision models.
• Clef-Flash model card.
• Pinned native training recipe.
• Package validation and data contract.
• Training graph construction.
• Evaluation-loss fix and regression evidence.
• Versioned local job lifecycle.

From the first MVP to a cleaner training path

1. Where we started

The first review, at 1224b611d on 14 November 2026, found a working native training head with some temporary wiring around it. This was the original picture: the CKE design we wanted, what the MVP actually did, and the gaps we planned to close.

First-review snapshot: “Today” and “cleanup” in this diagram describe that earlier revision. Some of those items are now fixed, as the next diagram shows.

The next figure shows which gaps were addressed by PR #1114 (c000bdcc3).

2. From Clef-specific glue to reusable compiler code

The first iteration got Clef training working, but it needed quite a bit of code written specifically for Clef. The head already used generated C; the shortcut was the extra adapter that manually translated its inference circuit into a training graph. That helped us get a working experiment quickly, but repeating that adapter for every new model would make CKE harder to maintain.

The intended CKE path is simpler: the circuit template describes the computation and its connections; kernel maps describe the implementations; the shared compiler uses those descriptions, together with gradient rules, to generate inference or backpropagation code. Think of the template as the wiring diagram and the kernel maps as the parts catalogue. The compiler should assemble supported operations without needing a special case for each model.

At c000bdcc3, we have moved closer to that design:

• Share the translation. #1114 moves the large Clef-specific training translator into shared operation-level code, with reuse tested on a small non-Clef circuit. Supported operations still need explicit kernel bindings and backward rules.
• Separate data from weights. Feature batches now enter through their own checked input interface. Changing a batch no longer means copying the full parameter array out and back in.
• Build and check through shared machinery. #1117, included through #1114, uses shared build code and declared kernel dependencies to package the runtime. New graph checks also verify which operation produces a value and which consumes it.

3. What comes next

The decision head was already there; these changes improve how CKE builds and connects it. The remaining job is to describe the backbone-to-head connection inside the generated graph, rather than passing features between them in host code, and resolve the score differences against the reference. For now, training updates the head only. We still need real held-out application results too.

That is the direction: describe more of the model in circuits and maps, keep the compiler reusable, and check the complete result. The PR list above also covers the related TTS and visualizer cleanup. A separate article can walk through those compiler changes with a small circuit, its kernel maps and the generated code side by side.

What we checked

I reran six focused test files: 23 tests and 48 subtests passed, including small native training and relocated-runtime checks. I did not rerun the full 9B model. The separate released-head relocation report records one update matching earlier checked gradients and weights; it is not a fresh independent or connected-model check. The earlier plots in this article remain results from their original commits.