Speech recognition is mostly signal processing wearing a transformer hat.

C-Kernel-Engine (CKE) runs Whisper Tiny, Base, and Small end to end in generated FP32 C — no PyTorch at inference. Because every stage is a named kernel with a parity oracle, the whole path from a WAV file to a transcript is unusually easy to read. This post walks it kernel by kernel, using CKE’s Audio Kernels Deep Dive as the map, so you can explain to anyone how sound becomes words.

An audio transformer is not one model. It is a short assembly line of fixed math that turns sound into a tensor, followed by two transformers that turn that tensor into text. If you have read my earlier walkthrough of why audio is not a VLM patch, this is the level below it: the actual kernels, the exact shapes, and the one number — 1500 — that quietly runs the entire system.

The Whole Path, At A Glance

Here is the entire journey on one page. Read it left to right: fixed signal processing on the left produces 1500 tokens, the encoder contextualizes them into a memory, and the decoder writes the transcript one token at a time.

A three-lane diagram of the audio pipeline. Lane 1, the frontend (fixed math, no weights): WAV bytes, resample to 16 kHz, pad or truncate to 30 seconds, STFT to power spectrum, 80 mel filters plus log, a 2x Conv1D stem, producing a 1500 by 384 token matrix. Lane 2, the encoder (understand the sound): add learned positions, 4 bidirectional blocks where each frame sees all 30 seconds via full-square self-attention with no mask and no RoPE, producing a frozen 1500 by 384 audio memory. Lane 3, the decoder (write the transcript): start tokens for language and task, causal self-attention, cross-attention into the memory, MLP plus LayerNorm, tied logits producing the next token, then append and repeat until the end-of-text token, in an autoregressive loop. A footer notes the key number 1500: 30 s times 16 kHz equals 480,000 samples, STFT hop 160 gives 3000 frames, a stride-2 conv gives 1500 tokens.
The whole system. Everything on the left is deterministic math with no learned weights; the two transformers on the right are where the model’s knowledge lives. The number 1500 ties all three lanes together.

The Frontend: Sound Becomes A Tensor

Most speech stacks hand this part to a library and forget about it. CKE does the opposite: the frontend is the numerical contract, so it is broken into eight explicit kernels, each with one job and one PyTorch oracle. Seven of the eight have no learned weights at all — they are pure, auditable signal processing.

A table of the eight frontend kernels. 1 WAV decode (audio_wav_decode): RIFF walk to mono float32 with typed rejects and mean downmix, bytes to samples. 2 Resample (audio_resample): linear or windowed-sinc to exactly 16 kHz. 3 Pad or truncate (audio_pad_or_truncate): zero-pad short, cut long, to a fixed 480,000 samples. 4 STFT power (audio_stft_power): reflect-pad, periodic Hann, 20x20 mixed-radix FFT, to a 201 by 3000 grid. 5 Mel filters (audio_mel_filters_slaney): 80 triangles on the Slaney scale, area-normalized, built once, 201 to 80 bins. 6 Log-mel (audio_log_mel): log10, a global minus-80 dB clamp, and an (x+4)/4 rescale, to 80 by 3000. 7 Conv1D stem (audio_conv1d_channel_major): two 3-tap convs plus GELU with stride 2 halving time, to 384 by 1500. 8 Token transpose (audio_transpose): channel-major to token-major, one row per token, then add positions, to 1500 by 384.
Eight kernels, WAV to tokens. Only the Conv1D stem is learned; the rest is fixed math. Each stage is pinned to PyTorch so a faster implementation can never silently change the numbers.

Walking the chain in plain terms:

  • Decode and resample. A 60-line RIFF parser turns PCM16 bytes into mono float32, downmixing stereo by mean so a stereo file and its mono version land at the same scale. Whisper wants 16 kHz, so anything else is resampled — either a fast linear path or a band-limited windowed-sinc path, and the run report records which one was used.
  • Fix the time extent. Everything downstream assumes exactly 480,000 samples (30 seconds). Short clips are zero-padded; long clips are truncated, not chunked. That single policy is why the frame counts downstream are constant.
  • STFT → power spectrum. This is where nearly all the FLOPs and nearly all the parity risk live. Whisper’s geometry is fixed: n_fft=400, hop=160, a periodic Hann window, reflect padding. CKE computes it as a 20×20 mixed-radix FFT — about 6× fewer multiply-adds than a direct DFT — and matches torch.stft to a relative max of ~2e‑5.
  • Mel filters and log. The 201 power bins are projected through 80 triangular filters spaced on the Slaney mel scale — narrow at low pitch where your ear resolves detail, wide up high. Then log10, a global −80 dB clamp (relative to the loudest frame of the whole clip, so quiet stays quiet), and an (x+4)/4 rescale into roughly [−1, 1].
  • The one learned stage. Two 3-tap 1D convolutions with GELU between them lift 80 channels to 384 and — crucially — the second conv runs at stride 2, halving 3000 time frames to 1500. That single stride is a 4× attention saving, and it is the origin of the 1500 that appears everywhere after.
  • The layout seam. An eight-line transpose flips channel-major [384×1500] into token-major [1500×384] — one row per token — and the learned positional table is added. Now it is literally the sequence the encoder reads.

What The Encoder Actually Sees

It helps to picture the tensor that comes out of the frontend, because attention never sees a waveform. It sees an image: 80 pitch bands stacked over 3000 time slices, log-compressed so a whisper and a shout share one frame of reference.

A two-panel diagram. Panel 1: a raw waveform, 480,000 amplitude samples over 30 seconds at 16 kHz. An arrow leads to Panel 2: an 80-by-3000 log-mel spectrogram heatmap, showing low pitch at the bottom and high pitch at the top across time, with warm colors for energy. Below are three cards. Mel not Hz: 80 bands on the Slaney mel scale, dense at low pitch where the ear resolves detail, wide up high. Global log clamp: log10 then a floor 80 dB below the loudest frame of the whole clip, so quiet frames stay quiet. Conv stem trims it: two 3-tap convs, the second at stride 2, halve 3000 to 1500 and lift 80 to 384 feature channels. A footer notes that Whisper large-v3 uses 128 mel bands instead of 80.
The log-mel spectrogram is the real input to the model — a perceptually-scaled picture of sound. The conv stem then compresses this [80×3000] image into the [1500×384] token sequence.

The Encoder: Understand Once

The encoder’s only job is to contextualize. Four identical pre-LayerNorm blocks turn the raw frame sequence into a memory where each frame’s vector has absorbed meaning from the whole 30 seconds. The attention here differs from the decoder in three ways, all declared in the circuit contract:

  • Bidirectional, not causal. The score grid is the full 1500×1500 square. The frame under the word “country” borrows from frames before and after it — the future is allowed, because we already have the whole clip.
  • No RoPE. Position came from the absolute learned table added back at the transpose; the encoder does not rotate anything.
  • Ephemeral K/V. Keys and values are computed per layer, used once, and freed. Caching belongs to the decoder, not here.

The payoff is a stable, immutable memory for the segment. That immutability is what lets the decoder project keys and values from it exactly once and cache them — the single most important optimization on the write side.

The Decoder: Write One Token At A Time

Now the transcript appears. The decoder is autoregressive: it writes left to right, and each block does three things in order — look back at its own words (causal self-attention), look out at the audio memory (cross-attention), then think (MLP). A final weight-tied projection turns the last hidden vector into logits over the vocabulary, and the highest-scoring token is the next word.

A two-panel comparison. Left, the encoder: bidirectional self-attention over the full 1500 by 1500 square with no mask, shown as a full grid; every frame attends to every frame; no causal mask and no RoPE; K/V ephemeral; output is the frozen audio memory. Right, the decoder: causal plus cross attention. A causal triangle shows self-attention over only past tokens. An arrow points to a cross-attention box where Q is the 1 current token and K,V are the 1500 memory rows, cached once per segment, described as the 79x decode win. Below, a ladder shows how a transcript appears: start with special tokens for start-of-transcript, language en, and transcribe; produce logits, take argmax, get the word And; append and run the block again to get the next word; continue until the end-of-text token. A footer explains the heavy encoder runs once per 30 seconds while each new decoder token is a cheap Q=1 lookup.
Two transformers, two jobs. The encoder reads everything at once; the decoder writes one token at a time, retrieving from the frozen memory through cross-attention. Caching the memory’s K/V once is what makes decoding cheap.

The loop is worth spelling out because it is where “transcription” literally happens. Whisper begins not with audio but with a short prompt of special tokens — <|startoftranscript|>, a language tag like <|en|>, and a task tag like <|transcribe|> (or <|translate|>). The decoder runs one block pass, cross-attends into the audio memory, emits logits, and the argmax is the first real word. That word is appended and the pass repeats until the model emits <|endoftext|>. The next two sections show exactly how CKE wires that up — first the full stitched circuit, then the token-by-token loop over time.

The Full Stitched Circuit: One Block, Stamped N Times

Everything so far has been the pieces. Here is how CKE actually wires them into one executable circuit — the exact op chain straight from the audio circuit JSONs, from WAV bytes all the way to a token. This is what “stitching kernels” means in practice: the frontend chain, the conv stem, one encoder block, the frozen memory, one decoder block, and the output head, each op a named kernel with its own contract.

The full stitched audio circuit as one vertical flow. A depth legend shows N (encoder = decoder blocks): tiny d=384 L=4, medium d=1024 L=24, large d=1280 L=32. FRONTEND (fixed signal processing, no weights): wav_decode, resample, pad/trunc, stft_tables, stft, mel_filters, log_mel, feat_window, producing WAV to [80x3000]. Arrow down to CONV STEM + LAYOUT (the only learned pre-attention layers): conv1d s1, gelu, conv1d s2, gelu, transpose, plus pos_emb, producing [1500 x d]. Arrow down to ENCODER BLOCK (bidirectional self-attention, no mask, no RoPE), stamped x4/24/32: self-attention plus residual row (LN, q_proj, k_proj, v_proj, attn over full 1500x1500, out_proj, residual add) and feed-forward plus residual row (LN, mlp_up d to 4d, gelu, mlp_down 4d to d, residual add). Arrow down to final_norm producing AUDIO MEMORY [1500 x d], frozen for the 30 s segment, read by every decoder step. Arrow down to DECODER BLOCK (causal self-attn plus cross-attn into the memory), stamped x4/24/32, with three sub-rows: 1 causal self-attention (LN, q_proj, k_proj, v_proj, causal attn, out_proj, residual), 2 cross-attention with Q from token and K/V from audio memory (LN, cross_q, cross_k from mem, cross_v from mem, cross_attn, cross_out, residual), 3 feed-forward (LN, mlp_up, gelu, mlp_down, residual). A dashed arrow runs from the audio memory into the cross-attention K/V. Arrow down to the output head: final_norm, weight_tying, logits over vocab, argmax to token. Footer: one block definition stamped N times; tiny stacks 4+4, large stacks 32+32; kernels, wiring, and shapes are identical, only d_model, head count, and N grow.
The complete circuit CKE generates. Read top to bottom. The two big blocks are definitions — the generator stamps each one N times (N = 4 for tiny, 24 for medium, 32 for large). The dashed blue arrow is the only cross-block wire: every decoder step reads the frozen encoder memory.

A few things worth noticing in the wiring. The encoder block is symmetric and simple: normalize, project Q/K/V, attend over the full square, project out, add the residual, then a normalized MLP with its own residual. The decoder block is the same skeleton with a second attention stage bolted in the middle — the cross-attention, whose queries come from the token stream but whose keys and values come from the audio memory (that is the cross_k ←mem / cross_v ←mem wire). Both blocks end in the identical MLP. Nothing in this diagram is Whisper-specific code: q_proj, attn, gelu, and layernorm are the same library kernels the text and vision models use. That is the whole point of the kernel-map approach — a new model family is mostly a new arrangement, not new kernels.

And this is exactly why the tiny-to-large question has such a clean answer. The generator does not switch algorithms; it stamps the same encoder and decoder block definitions a different number of times, at a different width. Tiny is 4 + 4 blocks at width 384; large is 32 + 32 at width 1280. The stitched circuit above is every Whisper size — you just change N and d.

Decoding Token By Token

The stitched circuit runs the encoder once, then hands its frozen memory to the decoder, which loops. This is where the transcript physically appears — one token per pass — and it is worth seeing the loop laid out in time, because it is also where all the CPU efficiency comes from.

A timeline of autoregressive decoding. At the top, a frozen bar: AUDIO MEMORY [1500 x d], encoded once, with cross-attention K/V projected once and cached. Below, six steps, each showing a tag, the token produced, and a growing self-attention KV cache glyph. Start: prompt tokens (start-of-transcript, en, transcribe) seed the loop, cache size 1. Step 1: produces the token And, Q=1 self-attention over 3, cross-attention to memory, cache grows to 2. Step 2: produces so, Q=1 self-attention over 4, KV cache reused, cache grows to 3. Step 3: produces my, Q=1 self-attention over 5, cross-attention to memory, cache 4. Then dots: fellow Americans, each step one cheap Q=1 lookup, cache 5. Stop: model emits end-of-text, loop ends, cache 6. Dashed arrows from each generating step point up to the frozen memory, labelled cross-attn every step. Footer: why it is fast on a CPU, the 1500-row memory is encoded once and its cross-attn K/V cached once, every new token is a single Q=1 attention against cache plus memory, the 79x decode win.
The autoregressive loop over time. Whisper starts from prompt tokens, then each pass emits one token and appends it. The self-attention KV cache grows by one per step, while the cross-attention keys and values — projected from the frozen memory — are computed once and reused forever.

Read it as a loop with two kinds of memory. The audio memory at the top never changes for the segment; the decoder’s cross-attention projects it into keys and values exactly once and caches them. The self-attention KV cache grows by one row each step, holding the tokens written so far. So step t is cheap: the query is a single vector (Q=1), it attends over the small, growing token cache and over the fixed 1500-row audio memory, produces logits, and the argmax is token t. Append, repeat, until the model emits <|endoftext|>. Because the expensive encoder ran once and the memory’s K/V are cached, each additional token costs almost nothing — the roughly 79× decode speedup CKE measures is a direct consequence of this structure, and it is what makes real-time transcription on a CPU realistic.

Tiny, Base, Small, Large: What Actually Changes

Here is the part that surprises people. The entire frontend I just described — 16 kHz, the 30-second window, 480,000 samples, the STFT geometry, the 80 mel bins, the stride-2 stem, the 1500 tokens — is identical in every Whisper size. “Larger” does not add new machinery. It stacks more, and wider, transformer blocks.

A table of the Whisper family with a banner listing what is identical across all sizes: 16 kHz, 30 s window, 480,000 samples, STFT n_fft 400 and hop 160, 80 mel bins, stride-2 conv stem, and 1500 encoder tokens. Then rows: tiny, 39M params, 4 layers, d_model 384, 6 heads, certified in CKE FP32. base, 74M, 6 layers, 512, 8 heads, certified in CKE FP32. small, 244M, 12 layers, 768, 12 heads, certified in CKE FP32. medium, 769M, 24 layers, 1024, 16 heads, same recipe more blocks. large, 1550M, 32 layers, 1280, 20 heads, v3 uses 128 mel bins. A footer explains large does not add new machinery, just about 8x the layers and 3x the width, giving a bigger memory for hard audio: heavy accents, overlapping speech, noise, and 99 languages.
What scales and what stays. From tiny to large, only the transformer depth, width, and head count grow. CKE currently certifies tiny, base, and small in FP32 against the token-exact Hugging Face trajectory.

So what does “large” actually buy? Not a different algorithm — more capacity for the same one. Roughly 8× the layers and 3× the width give the model a bigger internal memory to disambiguate hard audio: heavy accents, overlapping speakers, background noise, and Whisper’s ~99 languages. Tiny transcribes clean English narration astonishingly well for 39M parameters; large earns its FLOPs precisely where tiny starts to guess. (One genuine machinery change: large-v3 widens the picture itself, using 128 mel bins instead of 80.) The practical lesson, and the reason CKE cares, is that you should size to your audio — a clean podcast does not need a 1.5B-parameter model, and on a CPU that difference is the difference between real-time and not.

Why CKE Builds It This Way

Every stage above is a named C function with a PyTorch oracle, not hidden preprocessing. That is a deliberate discipline, and it is the same one behind the rest of C-Kernel-Engine. Splitting the STFT, the mel filterbank, and the log-mel into separate parity-gated kernels means a faster FFT can be swapped in without ever changing the numbers the model was validated against. Making the layout transpose its own kernel means a silent channel-vs-token bug — the kind that produces plausible-but-wrong output — simply cannot hide. And because the GELU, the attention, and the positional-embedding kernels are inherited from the text and vision lanes, the audio encoder proves almost nothing new; it composes pieces whose oracles already exist.

This is what “runs on a CPU” really requires: not a port, but a pipeline where every number is accounted for. Whisper Tiny, Base, and Small already match Hugging Face token-for-token on the public JFK fixture in generated FP32 — the encoder alone is ~11.15 s of a ~12.22 s end-to-end run on the reference machine, which is exactly why it carries the deepest parity evidence. Audio is not a bolt-on. It is signal processing you can read, feeding two small transformers you can certify, on hardware you already own.

Further reading: the full kernel-level walkthrough is in CKE’s Audio Kernels Deep Dive; the conceptual framing is in How Audio Transformers Work; and the CPU-and-smaller-models argument behind all of it is in why my strategic bet is CPUs, smaller models, and less compute.