I have been writing about how quickly C-Kernel-Engine is adding model families. Qwen3.8 had already produced useful output in CKE, so I treated that support as progress we had earned. Then issue #454 showed a real Qwen3.8 path producing gibberish. A compiler/provider-selection defect had reached a user, and our tests had not stopped it.

This was not merely a new user having difficulty. They exposed a CKE failure. For the person trying CKE, our circuit architecture and numerical tests do not make those failed attempts less frustrating. We asked for their time and compute. The software did not deliver the experience our guidance led them to expect.

I cannot measure how much trust this cost, and this thread is not evidence that every Qwen artifact was affected. The exact commit that introduced the defect has also not been established. But this is still a regression in the practical sense that matters: a capability we had presented as working contained a broken execution path, and our regression gates did not detect it before a user did. I want to explain what broke, what we repaired, and what CKE needs to do differently as the number of supported models grows.

Three Problems, One Bad First Experience

The reporter was new to CKE, running Debian on a Ryzen 9 5900HX with 32 GB of RAM. The small Gemma example ran, so they moved to Qwen3.8-27B. The issue then exposed three distinct problems.

The follow-up report supplied the memory error, command and output. That detail made the issue actionable. The user should not have had to discover this sequence for us, but their report gave us a concrete case to preserve.

Did The New Flash Commit Cause It?

The issue discussion mentioned the recent Qwen circuit work in Flash-Next PR #456. It is reasonable to investigate a large recent change. It is not enough to establish causation.

The diagnosis record says the same current GGUF also produced gibberish on the parent of #456. Reverting that PR alone therefore did not fix this artifact. The exact introducing commit has not been established.

So I am not going to turn this into a claim that Flash support broke dense Qwen. The defensible statement is that dense Qwen was broken for this artifact while we were expanding the Qwen family. Our existing validation and onboarding process did not protect this user's path.

The Wrong Provider For The Actual Weight Storage

The numerical investigation bypassed the tokenizer and compared the model internally. Layer-zero normalization, QKV projection, gate projection and convolution matched the reference exactly. The first reported failure was at the alpha/beta projections, where NaNs appeared.

The selected numerical contract expected FP32 weights. The actual alpha/beta weights in this file were stored as Q8_0. The selector had relied on shape and QKV-related facts without requiring the storage facts for the weights that this provider would actually consume.

Wrong decision:
    shape / surrounding configuration
        -> choose FP32 scalar projection provider
        -> hand it packed Q8_0 weights

Required decision:
    actual manifest entries and weight storage
        -> select a compatible provider
        -> validate the binding before generating C
        -> reject incompatible packed-to-float inputs

Q8_0 is not an array of ordinary FP32 values. A provider that expects floating-point weights cannot simply read packed quantized storage as if the two representations were interchangeable. The mathematical operation may have the same name, but its storage contract is different.

This is why a filename ending in Q4_K_M is not a complete description of every tensor in that file. The actual manifest matters. A model-family label is even less specific. The kernel maps and provider-selection documentation should be read as an interface contract, not merely a catalogue of function names.

What The Repair Changed

Merged PR #457 derives uniform layer-weight storage facts from manifest entries. The scalar FP32 contract now requires FP32 alpha and beta storage; otherwise selection follows the normal dtype-aware path. Lowering also rejects packed quantized weights bound to providers expecting floating-point storage.

The patch adds coverage for FP32, Q8_0, BF16 and mixed-layer cases. No new numerical kernel was needed for this repair. The problem was choosing a provider whose requirements did not match the artifact.

There was a separate metadata problem. The current file identified itself through general.name but lacked general.basename. That caused it to miss the dense model's intended bounded-memory defaults. The repair recognizes that identity case and preserves its declared chat variant. PR #458 then added linked dense and Flash quickstarts so readers do not have to infer the intended artifact from a family name.

The unsupported UD tensor format remains a separate support boundary. Fixing the standard Q4_K_M path does not make every similarly named dynamic-quant artifact supported.

The Fix Has Evidence, But The Scope Matters

The repair report records coherent recipe generation and 128 out of 128 full-vocabulary logit comparisons matching llama.cpp bit-exactly on Ryzen. Those comparisons used a one-token prefix, 1024 capacity and 16 threads. This is substantially stronger than looking at a plausible sentence and declaring success.

It is still not a 128K-input certification, a 32 GB memory-fit guarantee, or verification of every Qwen3.8 quantization. The report pins the tested GGUF revision and hash. That specificity should travel with the claim.

The updated 131072-capacity memory plan reports a prefill arena of about 28.6 GiB and a decode arena of about 25.9 GiB. Those are separate planned arenas, not measured total process memory. Prepared weights and other overhead are additional. We must not look at either number and tell a user that the complete model will comfortably fit in 32 GB.

As of this review, #457 is merged, while issue #454 remains open and the visible discussion does not yet contain a successful retest from the original reporter. The PR's draft-era text also records broader validation limitations. A merged repair and maintainer-side evidence are not the same thing as closing the loop with the person who experienced the failure.

What CKE Must Do Before Calling A Path Supported

Publish exact, reproducible starting points. The quickstart should identify the repository, filename, revision, quantization coverage and tested command. A new user should not need to guess whether the closest search result is compatible. Unsupported tensor types should be reported clearly, including which tensor failed and why.

Check the memory plan before allocating. A default example should begin with a conservative tested context, explain capacity versus actual consumed input, and show memory assumptions. Larger-context examples belong behind a visible resource estimate, not an unexplained allocation failure.

Keep dense and Flash regression lanes distinct. PR #457 adds a dedicated dense-Qwen contract target to nightly and demo-readiness checks. That is useful precisely because a passing new-family lane should not stand in for a previously working dense path.

Require real-model evidence for shared compiler changes. Metadata tests and selector tests are cheap and necessary. They cannot prove that a complete 27B model generates correctly. High-memory nodes should run a small, pinned acceptance set after changes to shared lowering, storage contracts or provider selection. Record the exact source revision, model bytes, token inputs and numerical comparisons.

Exercise both clean and cached workflows. A known-good cached runtime does not prove that the current compiler regenerates it correctly. A fresh conversion does not prove that an upgrade invalidates incompatible caches correctly. Both are user-facing paths. Retests should use an isolated run directory and explicit conversion/compilation controls rather than casually asking users to delete all their caches.

Make the gates block the claim. If an affected real-model lane has not run, the release notes should say so. A missing oracle or unavailable runner is an evidence gap, not a pass. A new model-support announcement should not outrun certification of shared paths. A small known-good release or commit recommendation gives users a recovery route while main continues to move.

This is the same class of lesson CKE just learned from Whisper. In I Used CKE To Transcribe My Video. It Found A Bug In My Compiler, a previously working audio path regressed because lowering connected a declared graph input through the wrong slot. Lightweight checks passed; a pinned five-minute recording exposed the failure. Qwen and Whisper broke for different technical reasons, but they point to the same process defect: demonstrations were allowed to prove a capability once without immediately becoming permanent end-to-end regression fixtures.

The plan is therefore cumulative. Every meaningful success should leave behind a reproducible artifact, command, expected numerical boundary and named machine lane. Fast pull-request tests protect contracts. Scheduled high-memory tests protect real model execution. Numerical oracles protect correctness. User-facing smoke tests protect the documented workflow. None replaces the others.

Keep the resolution in the public issue. Acknowledge the failure, link the fix, give one tested retest command with resource limits, and invite confirmation. Discord can help, but the answer should not require joining another platform. The person who reported the problem should be able to see what changed without reading every PR.

Trust Is Part Of The Engineering Work

For me, the lesson is not to stop adding models. It is that the validation obligation grows with the portfolio. Reusable compiler code gives CKE leverage, but a mistaken shared assumption can affect more than the model I happen to be testing that day.

The hardware I am investing in needs to support that obligation too. Some of its time should protect existing model paths, not only bring up the next family or produce the next benchmark. The whole point of asking people to contribute is that their work should build on something they can inspect and reproduce.

I am grateful the reporter stayed long enough to provide the failures. But patience is not a dependency I should design into CKE. The way to earn confidence back is a working command, bounded expectations, reproducible evidence and a regression test that keeps running after the issue stops being interesting.

Read The Investigation

• Issue #454: the original report and follow-up attempts
• PR #457: weight-storage selection, memory defaults and regression coverage
• PR #458: linked dense and Flash quickstarts
• CKE v8 runbook
• Kernel maps and provider contracts
• Related note: Whisper optimization and the evidence behind it

Related Notes

• I Used CKE To Transcribe My Video. It Found A Bug In My Compiler. covers the parallel Whisper regression and real-audio testing.
• CKE Added Qwen3.8 In About An Hour records the earlier bring-up success that must be protected as the compiler evolves.
• One Ryzen, 128K Context And A 27B Model shows an earlier working workload, not a guarantee for every artifact or machine.
• A Green Checkmark Should Name Its Machine explains the evidence standard behind meaningful regression reports.