Today I want to write about the layer most people skip: what actually happens between the moment a C program says "give me memory" and the moment the CPU can stream gigabytes of it through its execution units. This is a direct follow-up to my earlier post on Linux system programming for AI kernels, where I made the argument that Linux tuning is not a side quest — it is part of kernel engineering. That post covered core pinning, NUMA, huge pages, TLBs, and memory discipline from the top down. This one goes one layer lower and looks at the machinery underneath those controls: pages, page faults, the TLB, and why madvise is advice and not a promise.

Here is why I keep coming back to this. In C-Kernel-Engine (CKE) — my CPU-first AI compiler that turns a model into explicit, generated C kernels instead of leaning on a big framework runtime — my whole job is to make CPUs run AI predictably. If you have not seen the series before, that link is the place to start; the short version is that CKE compiles models down to C so the machine's behavior is inspectable rather than magic. A model is a memory workload as much as a math workload — the CPU does not multiply vibes. It loads bytes, translates addresses, fills cache lines, issues vector instructions, writes results, and repeats that cycle billions of times. If a page faults inside a timed loop, if the TLB is thrashing on a huge buffer, or if the runtime placed memory on the wrong NUMA node, the arithmetic units just wait. And when the units wait, no clever kernel saves you. That is the reason Linux tuning is such a large part of CKE's work: the compiler can emit a perfect kernel, but if the memory underneath it is unplanned, the number you measure is a lie.

So this post is me writing down the mental model I actually use when I plan memory for a CKE run — pointer to page to TLB to DRAM to execution unit — so the runtime can make each of those steps explicit instead of hoping the OS gets it right.

pointer -> TLB -> DRAM -> execution units

The mental model is this: memory allocation is not the same thing as physical memory readiness. A pointer can exist before physical DRAM is fully committed. A mapping can exist before pages are resident. A buffer can be legal C memory and still be a terrible hot-loop buffer until the runtime touches it, places it, advises it, locks it, or backs it with better page geometry. core idea The kernel engineer thinks in bytes, pages, translations, cache lines, and stalls.

1. Virtual Memory: The Address Is Not The DRAM

In user-space C, a pointer is a virtual address. It is not directly a DRAM coordinate. Linux gives each process its own virtual address space, and the CPU memory-management unit translates those virtual addresses into physical addresses. This abstraction is one of the great inventions of operating systems. It gives isolation, security, lazy allocation, file mapping, copy-on-write, and process independence.

But abstractions have costs. Every load from a pointer needs address translation. The CPU has hardware to make this fast, but fast does not mean free. The translation path uses page tables and the TLB. When the TLB has the mapping, memory access can proceed quickly. When it misses, the hardware has to walk page tables. If the page is not present, Linux handles a page fault.

2. What Is A Page?

A page is the unit of virtual memory mapping. On most Linux systems the normal page size is 4 KiB. The operating system maps virtual pages to physical page frames. Your program sees a large address range. Underneath, Linux and the CPU translate pieces of that range page by page.

If a program allocates a 1 GiB buffer using normal 4 KiB pages, that buffer spans 262,144 pages. Each page needs a translation. The CPU cannot keep hundreds of thousands of translations in the small fast TLB. So a big streaming workload can become not only a bandwidth problem, but also an address-translation problem.

\[ \text{page count} = \left\lceil \frac{\text{buffer bytes}}{\text{page size}} \right\rceil \]

A 1 GiB buffer is 262,144 normal 4 KiB pages, 512 huge 2 MiB pages, or 1 gigantic 1 GiB page.

3. What malloc Really Means

malloc is a user-space allocator API. It does not mean "ask DRAM for exactly this many bytes right now." It means "give me a usable region from the process heap or from a larger memory mapping that the allocator manages." The allocator may reuse freed blocks, extend the heap with brk/sbrk, or request a separate mapping with mmap.

Small allocations usually come from arenas managed by the allocator. Large allocations are often backed by mmap. The exact threshold depends on the allocator and runtime settings. This is why two C programs both using malloc can have very different memory behavior. The allocator is a policy layer.

#include <stdlib.h>

float *x = malloc(n * sizeof(float));
if (!x) {
    /* Allocation failed. */
}

/* This pointer is usable, but pages may still fault when first touched. */
for (size_t i = 0; i < n; i++) {
    x[i] = 0.0f;
}

The loop at the end is not just initialization. It also touches pages. On many Linux systems, physical pages become committed on first write. That means the first pass can include page fault handling and zero-page allocation. If you benchmark that pass as if it were pure kernel execution, the number is lying.

4. What mmap Does

mmap creates a mapping in the process virtual address space. The mapping can be anonymous memory or file-backed memory. Anonymous mappings are common for large buffers. File-backed mappings are useful for model weights, memory-mapped datasets, and runtime artifacts.

#include <sys/mman.h>
#include <unistd.h>

void *ptr = mmap(
    NULL,
    bytes,
    PROT_READ | PROT_WRITE,
    MAP_PRIVATE | MAP_ANONYMOUS,
    -1,
    0
);

if (ptr == MAP_FAILED) {
    /* Handle failure. */
}

The mapping reserves a virtual address range and defines permissions and backing behavior. It does not necessarily mean every page has already been physically allocated. That is why page faults still matter.

For model weights, file-backed mmap can be powerful. A runtime can map a contiguous weight blob and let the OS page cache serve it. The program avoids copying the whole model into a separately allocated heap buffer. But the first access pattern still matters: pages may be pulled in lazily as the runtime touches them.

5. When Does DRAM Actually Get Involved?

Physical memory usually gets involved when a page is touched and the kernel needs to back the virtual page with a physical page frame. The exact behavior depends on mapping type, permissions, overcommit policy, copy-on-write behavior, file cache state, and whether the memory is already resident. But the practical kernel-engineering rule is: do not assume allocation time equals physical readiness time.

If a benchmark allocates 8 GiB and immediately times the first kernel pass, that pass may include: page faults, zero-fill-on-demand, file-backed page-in, page table construction, transparent huge page promotion attempts, and NUMA first-touch placement. Some of those costs are one-time costs. Some can reappear if memory pressure causes eviction.

static void prefault_write(char *ptr, size_t bytes) {
    const size_t page = 4096;
    for (size_t i = 0; i < bytes; i += page) {
        ptr[i] = 0;
    }
}

This simple loop forces write access to each page. It is crude, but it makes the distinction visible: mapping memory is not the same thing as touching memory. A serious runtime can use more refined versions of this idea during warmup and memory planning.

6. What Is A Page Fault?

A page fault happens when the CPU tries to access a virtual page and the current page table entry cannot satisfy the access. The page may not be present. The permissions may not allow the access. The page may be copy-on-write. Or the mapping may need Linux to bring data from storage or allocate a fresh physical page.

Page faults are not always errors. Minor faults can happen when the page is already in memory but needs a mapping update. Major faults can require I/O, which is far more expensive. For an AI runtime, the key concern is not "page faults bad" in a moral sense. The concern is whether faults happen inside the hot path.

The word "page fault" hides two very different events, and the difference is the whole story for a runtime. The first kind is the one you hit the moment you first touch freshly allocated memory. The mapping exists, but no physical page is behind it yet, so the kernel simply hands you a free page and moves on. That is a minor fault: cheap, one-time, and something you can pay on purpose during warm-up instead of during a measured loop. The second kind shows up later, under memory pressure: DRAM fills up, the OS pushes a page you were not using out to disk (swap), and then — when you touch it again — it has to read that page back from the drive. That is a major fault: real I/O, orders of magnitude slower, and it can happen again and again if you keep oversubscribing memory.

In plain terms: a minor fault is Linux saying "you asked for this page, here it is." A major fault is Linux saying "I gave your page away because I ran out of room, hold on while I fetch it back from the disk." For CKE this is why memory sizing and mlock matter — a GB-scale KV cache or weight buffer that quietly gets paged out mid-run turns a fast kernel into a disk benchmark. The fix is not to fear faults; it is to move the minor ones into warm-up and keep the major ones from ever happening.

# Count minor/major faults and other events.
perf stat -e minor-faults,major-faults,page-faults ./ck-infer-smoke

# Process-level memory and fault info.
cat /proc/$(pidof ck-infer-smoke)/status | grep -E 'Vm|voluntary|nonvoluntary'

7. What Is The TLB?

The TLB is the Translation Lookaside Buffer. It is a hardware cache for virtual-to-physical address translations. Without the TLB, every load would require expensive page-table walks. With the TLB, recently used translations are available quickly.

AI kernels often stream over large contiguous regions: weights, activations, KV cache, gradients, and optimizer states. If the working set spans many pages and the access pattern jumps around, the TLB may not hold enough translations. Then the CPU spends more time walking page tables instead of feeding vector units.

perf stat \
  -e dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses \
  ./ck-infer-smoke

This is one reason huge pages exist. A larger page covers more memory with one translation. Fewer translations can mean fewer TLB misses. That does not automatically make every program faster, but it can matter for GB-scale streaming buffers.

8. Huge Pages And Transparent Huge Pages

A huge page is a page larger than the normal 4 KiB page. Common huge page sizes are 2 MiB and, on some systems, 1 GiB. Linux supports explicit huge pages through hugetlbfs and also Transparent Huge Pages, usually called THP.

Transparent Huge Pages try to promote suitable memory regions into huge pages automatically. This can help programs without changing their code. But "transparent" does not mean "free" or "always optimal." THP can introduce promotion overhead, fragmentation behavior, and latency spikes if the kernel has to compact memory. For low-latency runtimes, explicit policy is often better than hope.

# See THP mode.
cat /sys/kernel/mm/transparent_hugepage/enabled

# See huge page counters.
grep -i huge /proc/meminfo

# See page size used by a process mapping.
grep -A20 -B2 -i huge /proc/$(pidof ck-infer-smoke)/smaps

In user space, madvise(ptr, bytes, MADV_HUGEPAGE) asks Linux to consider huge pages for a mapping. MADV_NOHUGEPAGE asks Linux not to. These are policy hints, not absolute commands.

9. madvise: Advice, Not A Magic Wand

madvise lets a program tell Linux how it expects to use a memory region. This is useful because the kernel cannot perfectly infer intent. A model-weight mapping, a KV cache, an activation scratch arena, and a dataset streaming buffer may all want different policies.

The important correction is this: madvise does not by itself guarantee physical DRAM residency. MADV_WILLNEED can trigger readahead or paging work depending on the mapping and kernel behavior, but it is still advice. If the runtime needs stronger certainty, it should touch pages, check faults, use mlock where appropriate, and measure.

10. What Actually Guarantees A Page Is Ready?

In practice, a runtime builds confidence in layers. mmap creates the mapping. madvise communicates intent. A pre-touch loop forces page faults before the hot path. mlock asks Linux to keep pages resident. Huge page policy reduces translation pressure. NUMA policy places memory near the workers. perf verifies whether faults and TLB misses actually went down.

void prepare_hot_buffer(void *ptr, size_t bytes) {
    madvise(ptr, bytes, MADV_WILLNEED);
    madvise(ptr, bytes, MADV_HUGEPAGE);

    /* Force physical backing before timing the hot loop. */
    volatile char *p = (volatile char *)ptr;
    for (size_t i = 0; i < bytes; i += 4096) {
        p[i] = p[i];
    }

    /* Optional and system-limit dependent. */
    mlock(ptr, bytes);
}

Even this is not a universal recipe. Some file-backed mappings, read-only mappings, NUMA policies, and huge-page modes require different handling. The point is not to memorize one magic incantation. The point is to understand the layers and make the runtime explicit.

11. Why This Speeds Up GB-Scale Data Cycling

AI kernels want to cycle huge amounts of data through execution units: load weights, load activations, compute dot products, update accumulators, write outputs, repeat. The limiting factor is often not whether the CPU can multiply. The limiting factor is whether the CPU can continuously receive the right data with low enough latency and high enough bandwidth.

Good memory policy helps in several ways: fewer page faults inside timing windows, fewer TLB misses on large arenas, better NUMA locality, more predictable cache behavior, lower scheduler noise, and cleaner profiling signals. None of this replaces better kernels. It makes better kernels measurable and feedable.

12. The Takeaway

malloc gives a pointer. mmap gives a mapping. Pages give virtual memory its unit of management. Page faults make lazy memory real. The TLB makes address translation fast until it misses. Huge pages reduce translation pressure by covering more memory per page. madvise tells Linux what you intend, but it is not a guarantee. Pre-touching, mlock, NUMA placement, and measurement are how a runtime becomes more deterministic.

This is the level where "memory is your friend, not your enemy" becomes concrete. Memory is not bad. Unknown memory behavior is bad. A kernel engineer learns to make memory visible, placed, warmed, aligned, translated, and measured.