I keep allocating very large buffers in CKE, but malloc returning a pointer does not mean those bytes are already sitting in DRAM. It means the allocator has promised me a range of virtual addresses. The physical pages — the actual DRAM — show up later, and when they show up is a choice I get to make as a runtime author, whether I make it deliberately or by accident. For a long time I made it by accident: allocate, immediately run a timed kernel pass, and then wonder why the first pass was slow and the second was fast. The first pass was not running my kernel. It was paying the page system.

This became impossible to ignore while building C-Kernel-Engine (CKE), my CPU-first AI runtime that compiles models into explicit, generated C kernels instead of leaning on a framework. A model is a memory workload as much as a math workload: gigabytes of weights, an activation arena, a KV cache that grows with every token, scratch buffers, and — on the training side — gradients and optimizer state. The CPU does not multiply vibes. It loads bytes, translates addresses, fills cache lines, issues vector instructions, and repeats that cycle billions of times. If a page faults inside a timed loop, if the TLB thrashes on a huge arena, or if memory landed on the wrong NUMA node, the arithmetic units wait. I wrote the top-down version of this argument in Linux system programming for AI kernels; this post goes one layer lower, into the machinery underneath those controls: pages, faults, the TLB, and why madvise is advice and not a promise.

This is the mental model I actually use when I plan memory for a CKE run — pointer to VMA to first touch to TLB to DRAM to NUMA node — written down so each step can be made explicit instead of hoping the OS guesses right. It is for kernel and runtime engineers, for anyone who has ever watched a first benchmark pass run slow and not known why, and for people following CKE who want to see what the memory planner does and what it deliberately does not do.

One Allocation: What malloc Actually Promises

malloc is a user-space allocator API, not a syscall that fetches DRAM. What it promises is narrower than most of us casually assume: a usable range of virtual addresses in your process. How the allocator fulfills that promise varies. It may carve the block out of a freed region in one of its arenas, extend the heap with brk/sbrk, or — for larger requests — ask the kernel for a separate mmap mapping. glibc's allocator switches strategies based on request size, but the threshold is a tunable (M_MMAP_THRESHOLD), it can change dynamically at runtime, and other allocators make different choices entirely. Treat any specific threshold as a local detail of your libc and settings, not as a universal constant of C.

In every one of those paths, the thing you receive is virtual. Linux gives each process its own virtual address space; the CPU's memory-management unit translates virtual addresses into physical ones on every single load and store. That abstraction buys isolation, lazy allocation, file mapping, and copy-on-write — and it costs a translation on every access. The compiler can emit a perfect kernel, but if the memory underneath it is unplanned, the number you measure is not the kernel. It is the page system.

Following The First Access

Here is the full chain for one large buffer, slowed down. When mmap succeeds, Linux creates a virtual memory area — a VMA — an entry in the kernel's map of your address space describing the range, its permissions, and its backing. The VMA existing means the addresses are reserved. It does not mean any physical page frame backs them. Then your code writes p[i] = 0 and the CPU takes over: it looks up the virtual page in the TLB, the tiny hardware cache of recent translations. On a TLB hit, the physical address comes back immediately and the store proceeds. On a TLB miss, the hardware walks the page tables — a multi-level lookup that costs real cycles. And if the walk discovers the page is simply not present, the CPU raises a page fault and hands control to Linux.

The fault handler checks that the access is legal against the VMA, finds or allocates a physical page frame — for a fresh anonymous page, that means grabbing a free frame and zeroing it — installs the mapping in the page tables, and returns from the fault so the instruction replays, this time successfully. One more decision hides inside that allocation: on a multi-socket machine, Linux's default first-touch NUMA policy places the physical page on the node local to whichever CPU ran the faulting instruction. Placement is decided once, at first touch, and it is sticky. If your worker threads live on node 1 but your initialization thread ran on node 0, every access for the rest of the run crosses the interconnect.

float *x = malloc(n * sizeof(float));

Distinctions That Save You Hours

Most confusion in this layer comes from collapsing pairs of things that share a name or a neighborhood. Five separations do most of the work.

The TLB is not the CPU data cache. They sit next to each other in the memory path and both are "small fast hardware caches," but they cache different things. The TLB caches translations — virtual page to physical frame. The L1/L2/L3 caches cache data — the actual bytes of your tensors, in 64-byte lines. You can have a data-cache hit and a TLB miss on the same load. A workload can be bandwidth-starved (data caches can't hold the working set), translation-starved (the TLB can't hold the page coverage), both, or neither — and the fixes are different, so you have to know which one you are.

A TLB miss is not a page fault. A TLB miss means the translation exists in the page tables but not in the TLB, so the hardware walks the tables — expensive, but handled entirely in hardware, no OS involvement. A page fault means the page tables could not satisfy the access at all, and control transfers to the Linux fault handler. A TLB miss costs tens of cycles; a fault costs a kernel entry, an allocation or lookup, and a return. Confusing them leads to "optimizing" the wrong layer.

A minor fault is not a major fault. The word "page fault" hides two very different bills. The cheap kind is the one you hit on first touch: the mapping exists, no physical page is behind it yet, the kernel hands you a free page and moves on. That is a minor fault — one-time, and you can pay it on purpose during warm-up instead of inside a measured loop. The expensive kind arrives later, under memory pressure: DRAM fills, the OS pushes a page out to swap, and touching it again means reading it back from disk. That is a major fault: real I/O, orders of magnitude slower, repeatable as long as you oversubscribe memory. For CKE the implication is direct — a GB-scale KV cache or weight buffer that gets paged out mid-run turns a fast kernel into a disk benchmark. The fix is not to fear faults; it is to move the minor ones into warm-up and make the major ones impossible.

Reserved is not resident. After a successful mmap of 8 GiB, the process VmSize grows by 8 GiB and VmRSS barely moves. The address space is reserved; the resident set — the pages actually occupying DRAM — only grows as pages are faulted in. Any monitoring that reads virtual size and concludes "the process is using 8 GB" is reading the wrong column.

Anonymous is not file-backed. An anonymous mapping (the usual MAP_ANONYMOUS buffer) starts life as zero pages allocated on first touch. A file-backed mapping points at pages of an actual file, served through the page cache — which is how a runtime can map a multi-GB GGUF weight blob and let the OS pull pages in lazily as layers are touched, instead of copying the whole file into a heap buffer. The two kinds fault differently, advise differently, and behave differently under memory pressure, and a model runtime uses both on purpose.

Page Geometry: One GiB, Three Ways

A page is the unit of virtual-memory management, and on most Linux systems the ordinary size is 4 KiB. That makes the arithmetic of large buffers uncomfortable. One GiB of tensor data is:

\[ \text{page count} = \left\lceil \frac{\text{buffer bytes}}{\text{page size}} \right\rceil \]

1 GiB = 262,144 pages at 4 KiB  ·  512 pages at 2 MiB  ·  1 page at 1 GiB

Every page in that buffer wants a translation, and the TLB is small. A streaming kernel over a 1 GiB arena at 4 KiB pages touches 262,144 distinct pages; the TLB cannot hold that set, so the run keeps re-walking page tables. The same arena at 2 MiB pages touches 512 — a set a modern TLB hierarchy can plausibly cover. That is the entire argument for larger pages: fewer pages means more translation coverage per TLB entry. One honest caveat: page sizes do not all draw from the same TLB pool. CPUs typically have separate entry counts for 4 KiB and 2 MiB/1 GiB translations (and the counts differ again between them), so "the TLB holds N entries" is never a size-independent statement. Check your specific CPU's numbers before doing this arithmetic for real.

The Three Huge-Page Paths

There are three ways a buffer can end up with bigger page geometry, and they have genuinely different costs.

Ordinary 4 KiB pages. The default. No setup, no waste, no surprises — and a quarter-million translations per GiB. For small or short-lived buffers this is the right answer; huge-page machinery has its own costs and there is no reason to pay them for scratch that lives for microseconds.

Transparent Huge Pages (THP). The kernel tries to promote suitable 2 MiB-aligned regions into huge pages automatically, in the background, via khugepaged. When it works, your code never changes. When it doesn't, you pay in less visible ways: promotion itself is work, and if the region is fragmented the kernel may compact memory — moving pages around, sometimes synchronously inside your allocation path, which shows up as a latency spike nobody scheduled. THP also operates at 2 MiB granularity, so a sparse or partially used region can waste real memory. Whether it helps depends on the enabled mode (always/madvise/never), the fragmentation state of the system, and your access pattern. "Huge pages are faster" is not a universal claim; "fewer translations help large, stable, streaming arenas" is.

Explicit hugetlb pages. The administrator reserves a pool of huge pages at boot or runtime (/proc/sys/vm/nr_hugepages), and the program allocates from it with mmap(..., MAP_HUGETLB) or hugetlbfs. You get guaranteed 2 MiB (or 1 GiB) geometry — no promotion race, no compaction surprise — at the price of a reserved pool that cannot be used for anything else, plus startup cost to configure it, plus the need for a fallback path when the pool is exhausted or was never configured.

This is exactly the decision tree CKE's allocator encodes today. ck_huge_alloc in src/ckernel_alloc.c first tries the explicit path — mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0) — and if that fails (no pool configured, pool exhausted), it falls back to a plain 2 MiB-aligned aligned_alloc plus a best-effort madvise(q, len, MADV_HUGEPAGE), with the error deliberately ignored. The buffer works either way; the page geometry is the best the system could offer. Fallback behavior is not an edge case — it is the common case on machines where nobody reserved a pool.

madvise Is Advice, Not A Wand

madvise lets a program tell Linux how it expects to use a region. The kernel cannot perfectly infer intent, and a model runtime holds regions with wildly different intents, so this channel matters. But the semantics are in the name: it is advice. The kernel is free to ignore it, and sometimes does.

The correction I have to keep making in my own head: MADV_WILLNEED does not guarantee residency, and MADV_HUGEPAGE does not guarantee huge pages. They express intent. If the runtime needs stronger guarantees, the tools are different, and they are not interchangeable:

MAP_POPULATE is an mmap flag that faults the pages in at map time — you pay the minor faults up front, in the mapping call, instead of lazily. Manual pre-touching is the user-space version of the same idea: write (or read) one byte per page before the hot path. Crude, portable, effective, and — unlike MAP_POPULATE — it also lets you control which thread does the touching, which is how you control NUMA first-touch placement. mlock/mlock2 (man page) asks the kernel to keep pages resident — never swapped, never reclaimed. That is the only mechanism here that speaks to major faults, and it comes with hard deployment constraints: it is capped by RLIMIT_MEMLOCK, which defaults to a fraction of RAM (historically 64 KiB, commonly 8 MiB on modern distros), so locking a multi-GB KV cache needs a raised limit, a capability, or an ops conversation — in a container or on shared CI you may simply not get it. And NUMA binding (set_mempolicy/mbind, or numactl from outside) pins which nodes pages may be allocated from at all — a stronger, more rigid version of relying on first touch. Each of these answers a different question; madvise answers none of them with certainty.

Prove It: Measuring What Linux Actually Did

Everything above is a model of behavior; the tools below are how you check the model against your machine. I never ship a memory-policy change without at least one of these saying the thing I hoped it would say — and I never quote numbers I did not just measure, because these counters move with CPU generation and kernel version.

/proc/meminfo gives the system view: AnonHugePages (how much anonymous memory currently sits in THP), HugePages_Total/HugePages_Free and Hugepagesize for the explicit pool. /proc/<pid>/smaps gives the per-mapping truth, and it is the tool I reach for first: for each VMA you get its Size versus Rss (reserved versus resident, per mapping), AnonHugePages for that range, KernelPageSize versus MMUPageSize (the geometry the kernel actually used versus the hardware base), and THPeligible (whether the range qualifies for promotion under current policy). perf stat answers what the hot path paid: page-faults, minor-faults, major-faults, and the dTLB events (dTLB-load-misses, dTLB-store-misses where the CPU exposes them). numastat answers placement: per-node hit/miss/foreign counts that tell you whether first touch put pages where the workers are.

# What did the run pay, in faults and translation stalls?
perf stat -e page-faults,minor-faults,major-faults,\
dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses ./ck-infer-smoke

# Reserved vs resident, page geometry, THP eligibility — per mapping.
grep -E 'Size|Rss|AnonHugePages|KernelPageSize|MMUPageSize|THPeligible' \
  /proc/$(pidof ck-infer-smoke)/smaps

# Where did the pages land?
numastat -p $(pidof ck-infer-smoke)

Two warnings that keep me honest. First, hardware-counter names and availability vary by CPU vendor, generation, and kernel — dTLB-store-misses may not exist on your part, and what a miss counter includes varies too; check perf list on the machine in front of you rather than trusting any blog post's event list, including this one. Second, never fabricate the output. If you have not run the experiment, you do not have the numbers — which brings me to the honest status of this post's performance claims at the end.

grep -E 'Rss|AnonHugePages|THPeligible' /proc/<pid>/smaps

What This Looks Like Inside CKE

CKE's memory planner works at a different layer than everything above, and the distinction is the point of this section. The planner (version/v8/scripts/memory_planner_v8.py) walks the IR dataflow graph and assigns tensor slots to physical buffers — A_LAYER_INPUT, A_RESIDUAL, A_ATTN_SCRATCH, A_KV_CACHE and friends — deciding lifetimes, reuse, and offsets. Its output is which logical bytes live where, for how long, in virtual space. Linux page policy determines how those virtual ranges become physical memory: when they fault in, with what geometry, on which node, with what translation cost. Connected, but not the same mechanism. The planner can produce a perfect lifetime schedule and still hand you a slow run if every buffer faults inside the timed loop.

mmap(weights) + MADV_SEQUENTIAL

Honesty about what exists in the code today versus what is still policy work. Today: the v8 memory planner assigns buffers from dataflow; ck_huge_alloc implements the explicit-hugetlb-with-THP-fallback path described above; the v7 loader family can run against pre-allocated (mmap'd or malloc'd) weight regions; and the config tooling already reads NUMA topology for diagnostics. Not yet: there is no mlock call, no MAP_POPULATE, no set_mempolicy binding, and no per-buffer-class madvise policy in the runtime — the matrix above is the design those policies will follow, and the runtime-ownership argument is why they belong inside the runtime rather than in a wrapper script. The execution context matters too: the cost of getting this wrong differs by phase, which is why prefill and decode are planned separately, and why the Qwen3.6 prefill work treated memory behavior as part of the optimization, not an afterthought.

The Experiment I Owe You

I do not have a measured CKE before/after for huge-page policy yet, so this post makes no speed claim. What I have is a design, and I am labelling the evidence as pending until it runs. The experiment: one model, one fixed fixture, one thread count, three memory configurations — ordinary pages, THP with MADV_HUGEPAGE, and explicit hugetlb via ck_huge_alloc — each run with and without warm-up pre-touching. Measurements: perf stat minor/major faults and dTLB misses over the identical workload, smaps before and after to confirm the geometry actually changed (a config that didn't change KernelPageSize is a config that didn't happen), and wall time of a fixed decode length with the first-pass numbers reported separately from steady state. If the geometry changed and the fault count didn't drop, the claim dies; that is what makes it an experiment rather than a rumor.

The Five Questions

Everything in this post exists to make five questions answerable about any buffer in any run. What did Linux reserve? The VMA: virtual range, permissions, backing — visible in smaps as Size. What is physically resident? The pages faulted in so far — Rss in the same file, VmRSS process-wide. How is it translated? Through the TLB, falling back to a hardware page-table walk, with geometry shown by KernelPageSize and misses counted by perf. Where is it placed? On the NUMA node of whoever touched it first, unless you bound it — numastat tells the truth. How do I prove each answer? With the four tools above, on the machine in front of you, quoted from your own terminal and nobody else's.

malloc gives a pointer. mmap gives a mapping. Pages give virtual memory its unit of management. Faults make lazy memory real. The TLB makes translation fast until it misses. Huge pages trade translation pressure for setup and waste. madvise states intent. Pre-touching, population, locking, binding, and measurement are how a runtime turns intent into fact. Memory is not the enemy; unknown memory behavior is.