In 2002, researchers showed that compressed file lengths alone could recover meaningful structure between languages and authors. No transformer, embeddings, or linguistic parser was required. That surprising result is our practical entrance into why compression, probability, and language-model training share the same mathematics.

This article began with a question raised by Grant Sanderson's 2026 3Blue1Brown work on cross-entropy: for a source of data, is there a mathematically defined minimum loss, and can training reach it? The animation source in the 3Blue1Brown videos repository makes the predictor-compressor connection especially vivid.

The short answer is precise but easy to misuse:

If the true conditional distribution is p, the population cross-entropy is minimized when the model distribution q equals p. The minimum is the entropy of the source. A finite dataset does not reveal p exactly, so its lowest training loss is not automatically the lowest loss on new data.

We will build that statement from bits rather than treating cross-entropy as a formula imported from a machine-learning library.

1. Information Begins With Surprise

Before The Equation: What Is An Outcome?

Start with an experiment: choose one integer uniformly from 0 through 1023. Before the choice, there are 1024 possible outcomes. After the choice, suppose the observed outcome is:

\[X=731\]

The outcome is 731. The number \(1/1024\) is not the outcome; it is the probability assigned to that outcome before the choice:

\[P(X=731)=\frac{1}{1024}\]

The same separation applies elsewhere. In “it rained today,” the outcome is rain and the probability is our prior chance of rain. In language modeling, the outcome is the actual next token and the probability is the model's belief about that token before seeing it.

Why Does A Rare Outcome Have More “Surprise”?

Here “surprise” is a technical shorthand for unexpectedness before observation, not excitement and not sparsity. A rare outcome has a low probability of occurring. Most of the time it does not happen. On the unusual occasion when it does happen, our prior expectation was wrong by more, so observing it removes more uncertainty.

Compare two statements received as new information:

• “The sun rose this morning.” Its probability was already near one, so learning it removes almost no uncertainty.
• “Ticket 731 won a lottery with 1024 equally likely tickets.” Before the draw, 1024 results were possible. Learning the exact winner removes ten binary levels of uncertainty.

Information theory gives this unexpectedness the less conversational name self-information. Suppose an outcome \(x\) occurs with probability \(p(x)\). A useful measure should satisfy three requirements:

• A certain event, \(p=1\), should provide zero surprise.
• A lower-probability outcome should remove more prior uncertainty when it occurs.
• Independent surprises should add: observing events with probabilities \(p_1\) and \(p_2\) should provide the sum of their information.

The logarithm supplies exactly that additive rule because \(\log(p_1p_2)=\log p_1+\log p_2\). With base two, self-information is measured in bits:

\[I(x)=-\log_2 p(x)\]

An outcome with probability \(1/2\) has one bit of self-information. Probability \(1/4\) gives two bits. Probability \(1/1024\) gives ten bits because \(2^{10}=1024\): ten binary yes/no distinctions can isolate one result among 1024 equally likely possibilities.

The distinction is even clearer for a binary source. Suppose the literal outcome 0 has probability \(1/1024\), while 1 has probability \(1023/1024\). A raw machine bit can store either value in one bit. But an optimal compressor operating across a long sequence exploits the many predictable ones and spends additional coding capacity when the rare zero occurs. The approximately ten-bit self-information of that zero describes its contribution inside that long-message coding model; it does not change the physical width of the value 0.

This does not mean that the numerical probability itself takes more bits to store. The probability is used to design a code; the resulting code assigns more bits to each occurrence of a rare outcome. These are the quantities we must keep separate:

• Outcome \(x_i\): the actual identity or value, such as phone, donkey, or Tom.
• Count \(n_i\): how many times outcome \(i\) appeared in \(N\) observations.
• Probability \(p_i\): the underlying chance of outcome \(i\), estimated from data as \(\hat p_i=n_i/N\).
• Code length \(\ell_i\): the ideal bits charged when outcome \(i\) occurs, derived from probability as \(\ell_i=-\log_2p_i\).

Only the first three are distinct inputs to this empirical description; the ideal code length is calculated from the probability. “More information” here means more statistical surprise and a longer ideal representation, not greater semantic importance.

2. Entropy Is The Average Irreducible Coding Cost

Now let a source produce symbols according to a true distribution \(p\). Before observing the next symbol, we do not know which self-information value we will pay. The expected value is Shannon entropy:

\[H(p)=\mathbb{E}_{x\sim p}[-\log_2 p(x)]=-\sum_x p(x)\log_2 p(x)\]

Consider 16 observations of four possible values. The labels can represent words, tokens, sensor states, or any other mutually exclusive outcomes:

The code does not store the text phone in one literal bit. Both encoder and decoder first agree that codeword 0 means the outcome phone in this codebook. Because that outcome occurs eight times, those eight occurrences cost eight coded bits. The two occurrences of Tom cost three bits each, or six coded bits. The identity, frequency, and per-occurrence price remain separate.

The expected length is:

\[H(p)=\tfrac12(1)+\tfrac14(2)+\tfrac18(3)+\tfrac18(3)=1.75\text{ bits/symbol}\]

Claude Shannon's 1948 A Mathematical Theory of Communication established entropy as the fundamental rate for lossless source coding. There is an important qualification: individual binary codewords must have integer lengths, while \(-\log_2 p\) can be fractional. Huffman coding approaches the bound symbol by symbol; arithmetic coding and long block codes can approach the entropy rate arbitrarily closely under the theorem's assumptions.

3. Cross-Entropy Is The Bill For Believing The Wrong Distribution

Up to this point, one distribution \(p\) has played two roles: it described how frequently reality emits each outcome, and it supplied the ideal bit price \(-\log_2p(x)\). Entropy is the average cost when those two roles agree.

A real learner does not know \(p\). It estimates another distribution \(q\). Reality still emits outcomes according to \(p\), but the model allocates code lengths according to \(q\). Separating those two roles is exactly what creates cross-entropy.

Here, a lossless binary coding scheme does not mean program source code. It means an agreed rule for representing outcomes as bits so that a decoder can recover the original sequence exactly.

Huffman coding constructs explicit integer-length codewords. Arithmetic coding encodes a sequence using intervals whose sizes follow predicted probabilities and can approach fractional average lengths. In either case, \(q\) determines how bit space is allocated. That is what “a code designed for distribution \(q\)” means: the model's beliefs determine the price, while reality determines how often each price is paid.

In practice, we build a model \(q\). If \(q(x)\) is the probability our model assigns to \(x\), its ideal code length is:

\[\ell_q(x)=-\log_2 q(x)\]

But the source still emits symbols according to \(p\). Therefore the average bill is:

\[H(p,q)=\mathbb{E}_{x\sim p}[-\log_2 q(x)]=-\sum_x p(x)\log_2 q(x)\]

This is cross-entropy. The word cross matters: outcomes are weighted by one distribution, \(p\), but priced using another, \(q\).

What Do \(i\), \(p_i\), And \(q_i\) Actually Mean?

Neither \(p_i\) nor \(q_i\) is the word itself. The index \(i\) identifies one possible outcome, such as the token cat. The value \(p_i\) is how often that outcome occurs in reality, while \(q_i\) is the probability assigned to that same outcome by the model or coding scheme.

If a dataset contains 100 relevant next-token examples and cat is the true next token 50 times, its empirical probability is:

\[\hat p_{\mathrm{cat}}=\frac{n_{\mathrm{cat}}}{N}=\frac{50}{100}=0.5\]

Suppose the model assigns cat probability \(q_{\mathrm{cat}}=0.25\). Its ideal code length for that outcome is \(-\log_2(0.25)=2\) bits. Because cat occurs half the time, its contribution to the average cross-entropy is:

\[p_{\mathrm{cat}}[-\log_2 q_{\mathrm{cat}}]=0.5\times 2=1\ \text{bit per example}\]

The average is \(0.5(2)+0.3(1)+0.2(2)=1.7\) bits per token. The counts determine the empirical \(p\); the model supplies \(q\). For one individual training example, the observed target is a one-hot vector. Across many examples with the same context, those observations estimate the underlying distribution \(p\).

Suppose the source still uses \(p=(1/2,1/4,1/8,1/8)\), but our model assumes the uniform distribution \(q=(1/4,1/4,1/4,1/4)\). It assigns every symbol a two-bit code. The average cost is now 2 bits/symbol instead of 1.75. We pay 0.25 extra bits per symbol because our model failed to exploit the source's structure.

4. KL Divergence Is The Avoidable Part

Add and subtract \(\log_2 p(x)\) inside the cross-entropy:

\[\begin{aligned}H(p,q)&=-\sum_x p(x)\log_2 q(x)\\&=-\sum_x p(x)\log_2 p(x) +\sum_x p(x)\log_2\frac{p(x)}{q(x)}\\&=H(p)+D_{\mathrm{KL}}(p\|q)\end{aligned}\]

KL divergence is non-negative, and it is zero when \(q=p\) on the support of \(p\). Therefore:

\[H(p,q)\ge H(p)\]

This answers the ideal mathematical question. If we know the true data-generating distribution and may choose any valid distribution \(q\), cross-entropy reaches its global minimum at \(q=p\). The minimum value is \(H(p)\).

There is also a geometric way to see it. Hold the true binary distribution \(p\) fixed and vary the model's reported probability \(q\). Cross-entropy forms a curve with one minimum at \(q=p\). Move \(p\) and trace each curve's minimum; those minimum points draw the binary entropy curve itself.

KL divergence is not a distance: it is asymmetric and does not obey the triangle inequality. Its operational meaning here is more useful than calling it a distance. It is the expected number of extra bits paid per outcome because our probabilistic model is mismatched to the source.

Why The Direction \(D_{\mathrm{KL}}(p\|q)\) Matters

The expectation is taken under \(p\), so reality decides which mistakes are encountered and how often. Swapping the arguments asks a different question:

\[D_{\mathrm{KL}}(p\|q)=\sum_i p_i\log\frac{p_i}{q_i}\qquad\ne\qquadD_{\mathrm{KL}}(q\|p)\]

A single term \(p_i\log(p_i/q_i)\) may be negative when the model overestimates outcome \(i\), but the sum over the complete distribution cannot be negative. Underestimates and overestimates do not cancel arbitrarily because both \(p\) and \(q\) must remain normalized probability distributions.

The most severe failure is assigning \(q_i=0\) to an outcome for which \(p_i>0\). Then \(-\log q_i\) and \(D_{\mathrm{KL}}(p\|q)\) become infinite: the code has no representation for an event reality can produce. Neural language models use softmax probabilities, label smoothing, clipping, or finite-precision safeguards partly because exact zero probability makes the log loss undefined.

Where KL Appears Beyond Pre-Training

In distillation, \(p\) can be the teacher model's softened token distribution and \(q\) the student distribution. The student then learns from the teacher's complete ranking of alternatives, not only from one observed token. In reinforcement learning and preference optimization, KL is often used as a separate regularizer between a changing policy and a reference policy. There it limits how far optimization moves the model away from a known behavior distribution. Cross-entropy therefore does not belong only to pre-training, and KL does not belong only to compression; their precise roles depend on which distribution supplies the expectation and which distribution is being optimized.

5. Language Trees: When A General Compressor Detects Structure

The compression interpretation was useful for natural-language analysis before modern neural language models. In 2002, Dario Benedetto, Emanuele Caglioti, and Vittorio Loreto published Language Trees and Zipping. They used general-purpose compressors to estimate how readily patterns learned from one text transfer to another. Their experiments addressed language recognition, authorship attribution, and language-family clustering without embedding a linguistic grammar in the comparison function.

Let \(L(A)\) be the compressed length of document \(A\). Append a short sample \(b\) from document or language \(B\), then measure:

\[\Delta_{Ab}=L(A+b)-L(A)\]

If the compressor's dictionary and context learned from \(A\) also describe \(b\) well, the incremental cost is small. If \(b\) contains unfamiliar sequences, the cost is larger. The published distance construction includes normalization and comparison controls; it is not simply one raw file-size subtraction.

This is related to cross-entropy, but it is not an exact cross-entropy measurement. A gzip-style compressor is finite, implementation-specific, and far from an oracle at the Shannon limit. Documents are samples rather than known distributions. Header costs, window sizes, sample order, text length, and compressor choice matter. The paper's result is valuable precisely because even this imperfect engineering proxy exposed useful statistical structure. It also attracted contemporary technical criticism and replies, which is a reminder to treat compression distance as an empirical method with controls, not a universal semantic metric.

6. Supervised Learning Uses Conditional Cross-Entropy

In machine learning, an input \(X\) is observed and a target \(Y\) must be predicted. The unknown data-generating process has a conditional distribution \(p(y\mid x)\). A model with parameters \(\theta\) produces \(q_\theta(y\mid x)\). Its population risk is:

\[R(\theta)=\mathbb{E}_{(X,Y)\sim P}\left[-\log q_\theta(Y\mid X)\right]\]

The same decomposition now applies at each input:

\[R(\theta)=H_P(Y\mid X)+\mathbb{E}_{X}\left[D_{\mathrm{KL}}\big(p(\cdot\mid X)\|q_\theta(\cdot\mid X)\big)\right]\]

This separates two sources of loss:

• Irreducible uncertainty: \(H_P(Y\mid X)\). Even the true predictor cannot know a genuinely random target before it occurs.
• Model mismatch: the expected KL term. This is the part a better model, representation, dataset, or optimization process may reduce.

If labels are perfectly deterministic given every relevant input, the theoretical conditional entropy can be zero. Real datasets often contain missing variables, ambiguous labels, measurement noise, inconsistent annotation, and future events that are genuinely uncertain. A nonzero validation loss is not automatically proof that optimization failed.

7. Why One-Hot Cross-Entropy Becomes Negative Log-Likelihood

For one training example with observed class \(k\), the target distribution is one-hot: \(y_k=1\) and all other \(y_i=0\). The categorical cross-entropy reduces to:

\[L=-\sum_i y_i\log q_i=-\log q_k\]

The model is charged only for the probability it assigned to the event that actually occurred. Predict 0.9 for the correct class and the charge is small. Predict 0.001 and the charge is severe. Confidently wrong predictions are expensive because they correspond to codes that were absurdly long for events the data source actually produced.

With softmax logits \(z\), the backward result is unusually clean:

\[\frac{\partial L}{\partial z_j}=q_j-y_j\]

This is why fused log-softmax plus negative-log-likelihood kernels are so common. The mathematical loss, numerical stability of log-sum-exp, probability normalization, reduction order, and gradient kernel all meet at one execution boundary.

8. Why A Local Proper Loss Is Forced To Be A Logarithm

Now we can address the derivation in Grant Sanderson's animation. Imagine that event \(i\) occurs with true probability \(p_i\), while a forecaster reports probability \(q_i\). Suppose the loss paid after observing event \(i\) depends only on the probability assigned to that observed event:

\[L(i,q)=f(q_i)\]

This is called an outcome-local scoring rule. Its expected loss under reality \(p\) is:

\[\mathcal{L}(q;p)=\sum_i p_i f(q_i)\]

We want honest probabilities to be optimal: for every true distribution \(p\), the unique minimum over valid reports \(q\) should occur at \(q=p\). This property is called strict propriety. The report must also remain on the probability simplex:

\[\sum_i q_i=1,\qquad q_i>0\]

Step 1: Add The Constraint With A Lagrange Multiplier

We cannot independently change every \(q_i\), because increasing one probability requires decreasing another. Introduce a multiplier \(\lambda\):

\[\mathcal{J}(q,\lambda)=\sum_i p_i f(q_i)+\lambda\left(\sum_i q_i-1\right)\]

At an interior constrained optimum, each partial derivative with respect to \(q_i\) must vanish:

\[\frac{\partial\mathcal{J}}{\partial q_i}=p_i f'(q_i)+\lambda=0\]

Equivalently:

\[p_i f'(q_i)=-\lambda\qquad\text{for every }i\]

The right side is the same scalar for every coordinate. Geometrically, the gradient of expected loss is parallel to the normal vector of the simplex constraint. A feasible first-order movement along the simplex therefore cannot reduce the loss.

Step 2: Require The Optimum To Be The Truth

Strict propriety requires \(q_i=p_i\) at the optimum. Substitute that condition:

\[p_i f'(p_i)=-\lambda\qquad\text{for every }i\]

Now consider any interior distribution containing several different probability values. The product \(x f'(x)\) must have the same value for all of them. Requiring this for every possible distribution forces:

\[x f'(x)=C\]

Therefore:

\[f'(x)=\frac{C}{x}\;\Longrightarrow\;f(x)=C\log x+D\]

We are minimizing loss, so assigning a larger probability to the observed event must reduce the penalty. Thus \(C<0\). Write \(C=-a\) with \(a>0\):

\[f(q)=-a\log q+D\]

The additive constant \(D\) changes every score equally, and the positive scale \(a\) changes units or learning-rate scale. Neither changes which \(q\) minimizes expected loss. Up to those irrelevant affine choices, the only smooth local strictly proper loss is negative log probability.

What The Derivation Does Not Claim

• It assumes locality. If loss may inspect the entire predicted vector, other strictly proper rules exist. The Brier score, \(\sum_i(q_i-y_i)^2\), is the standard counterexample.
• It assumes smooth interior probabilities. Boundary values need limits; log loss diverges when the model assigns zero probability to an event that occurs.
• The clean uniqueness result needs more than a binary shortcut. With only two outcomes, \(q_2=1-q_1\) leaves less independent variation, and additional local proper forms can satisfy the binary condition. The standard finite-outcome uniqueness statement applies for more than two mutually exclusive outcomes under its regularity assumptions.
• It proves truthful probability elicitation, not universal task utility. A cost-sensitive decision problem may rationally use a different objective because false positives and false negatives have different consequences.

This is the deeper answer to "why logarithms?" Compression supplies one route: independent probabilities multiply while code lengths add. Proper scoring supplies another: if a local loss must reward honest probability distributions for every possible source, constrained optimization forces the logarithm. The same function appears because both problems are asking for a coherent numerical price for probability.

9. A Language Model Is A Conditional Compressor

An autoregressive language model factorizes a sequence probability:

\[q_\theta(x_{1:T})=\prod_{t=1}^{T}q_\theta(x_t\mid x_{<t})\]

Taking negative logarithms converts that product into a sum:

\[-\log q_\theta(x_{1:T})=\sum_{t=1}^{T}-\log q_\theta(x_t\mid x_{<t})\]

That is both sequence negative log-likelihood and ideal compressed length under the model. If an arithmetic coder uses the model's next-token probabilities, better predictions produce shorter encodings. Prediction and compression are not merely analogous; under this coding construction they are two views of the same probability assignment.

The average token loss is:

\[\bar L=-\frac{1}{T}\sum_t\log q_\theta(x_t\mid x_{<t})\]

If natural logarithms are used, loss is measured in nats and perplexity is \(\exp(\bar L)\). If base-two logarithms are used, loss is bits per token and perplexity is \(2^{\bar L}\). Perplexity can be interpreted as the model's geometric-average effective branching factor, but comparisons are only meaningful under compatible tokenizers and evaluation protocols.

Pre-Training Samples One Token; Distillation Supplies A Distribution

For an ordinary pre-training position, the dataset shows one realized next token. The target vector is one-hot, so the loss is \(-\log q_\theta(x_t\mid x_{<t})\). Across many related contexts and examples, those individual outcomes form an empirical sample of the unknown language distribution. Exact prefixes are usually observed only once, so the model cannot literally tabulate a reliable next-token distribution for every context. It must learn structure that transfers across related contexts.

Knowledge distillation provides a denser target. A teacher model emits a complete probability distribution \(r_i\) over possible next tokens, and a student emits \(q_i\). The student can be trained with:

\[H(r,q)=-\sum_i r_i\log q_i\]

One example now conveys relative preference among many alternatives, not only the identity of the sampled token. If the teacher distribution is fixed, replacing cross-entropy with forward KL divergence does not change the gradient with respect to the student:

\[D_{\mathrm{KL}}(r\|q)=H(r,q)-H(r)\]

The teacher entropy \(H(r)\) is constant with respect to student parameters. Cross-entropy and forward KL therefore have the same student minimizer and parameter gradients, although their reported scalar values differ. Temperature scaling, masking, reverse-KL direction, auxiliary hard-label loss, and teacher calibration can change the practical contract.

Where Cross-Entropy Fits In SFT, Preference Learning And RLVR

Cross-entropy does not disappear after pre-training, but it stops being the whole objective in several post-training methods:

For supervised fine-tuning, nothing fundamental changes: the desired answer is treated as a sequence of target tokens, and the model minimizes their negative log probabilities.

Why Not Use Ordinary Cross-Entropy For Everything?

Ordinary token cross-entropy needs a target token, or a complete target distribution over tokens. Pre-training has the next token from the dataset. SFT has the next token from a demonstration. Distillation has the teacher's distribution. In those settings, cross-entropy asks exactly the right question: how much probability did the model assign to what the target says should happen?

A preference example supplies different information. It may say that response \(y^+\) is better than response \(y^-\), but it does not specify the one correct token at every position. A common pairwise objective is therefore logistic:

\[\mathcal{L}_{\mathrm{pref}}=-\log\sigma\!\left(s_\theta(y^+)-s_\theta(y^-)\right)\]

The score may come from a separate reward model or from differences between policy and reference-model log probabilities, as in DPO. The objective teaches an ordering between complete responses rather than pretending that the preferred response is the only linguistically valid sequence.

A verifier reward supplies less token-level information still. It may report only that the final answer passed a test, compiled, or matched a mathematical result. There is no target next-token distribution to insert into ordinary cross-entropy. Policy-gradient methods instead ask whether sampled decisions made the final reward more or less likely. A practical objective often combines reward optimization with a constraint:

\[\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{policy}}+\beta D_{\mathrm{KL}}\!\left(\pi_\theta\|\pi_{\mathrm{ref}}\right)\]

So yes: post-training continues adjusting many of the same model weights, but it may use another loss because the available supervision changed. Cross-entropy has not become weak. It is simply unable to invent token-level targets that the preference label or verifier never provided. KL regularization, small updates, reference policies, and mixtures with SFT data help the model gain the new behavior without needlessly destroying the language distribution learned during pre-training.

In reinforcement learning with verifiable rewards (RLVR), the model samples a complete response \(y\sim\pi_\theta(\cdot\mid x)\). A verifier scores the result, for example by checking a mathematical answer or executing code. The basic score-function identity is:

\[\nabla_\theta\,\mathbb{E}_{y\sim\pi_\theta}[R(y)]=\mathbb{E}_{y\sim\pi_\theta}\left[R(y)\nabla_\theta\log\pi_\theta(y\mid x)\right]\]

An implementation therefore contains terms resembling:

\[\mathcal{L}_{\mathrm{PG}}=-A(y)\sum_t\log\pi_\theta(y_t\mid x,y_{<t})\]

This looks like weighted negative log-likelihood, but it is not ordinary cross-entropy against a fixed target distribution. The sequence was sampled by the policy itself, and the advantage \(A\) can be positive or negative. A positive advantage increases the sampled tokens' probability; a negative advantage decreases it. PPO and GRPO add importance ratios, clipping, group-relative baselines, and other stabilization terms.

Many RL post-training objectives also penalize drift from a reference policy:

\[D_{\mathrm{KL}}(\pi_\theta\|\pi_{\mathrm{ref}})=\mathbb{E}_{y\sim\pi_\theta}\left[\log\pi_\theta(y\mid x)-\log\pi_{\mathrm{ref}}(y\mid x)\right]\]

Again the same probability primitives appear, but this KL term is not identical to teacher-student cross-entropy because the expectation is under the changing policy \(\pi_\theta\). RLVR changes where the supervision comes from: a verifier supplies sequence-level reward rather than a dataset supplying the one correct next token. The difficult new problem is credit assignment: which token decisions deserve responsibility for the final verified outcome?

DPO takes another route. It turns a preference pair into a binary classification-style objective over differences between policy and reference-model log probabilities. Its outer loss is not next-token cross-entropy, even though every response score is assembled from the same autoregressive log-probability kernels.

Cross-entropy remains the probability engine underneath modern language-model training. Pre-training and SFT use it directly. Distillation compares distributions. Preference optimization and RLVR reuse log probabilities inside objectives that also encode rewards, comparisons, trust regions, or KL constraints.

10. A Dataset Does Not Reveal The True Minimum

We do not train on the population distribution \(P\). We train on \(n\) samples \(D=\{(x_i,y_i)\}_{i=1}^n\). The optimizer sees empirical risk:

\[\hat R_D(\theta)=\frac{1}{n}\sum_{i=1}^{n}-\log q_\theta(y_i\mid x_i)\]

Population risk \(R(\theta)\) and empirical risk \(\hat R_D(\theta)\) are different functions. A sufficiently expressive model may memorize the sample and drive training loss near zero while assigning poor probabilities to unseen examples. That does not beat Shannon's bound. It means the model encoded peculiarities of the finite sample rather than recovering the source distribution.

The generalization gap is:

\[\mathrm{gap}(\theta)=R(\theta)-\hat R_D(\theta)\]

Because \(R\) depends on unseen data, it cannot generally be calculated exactly from one finite dataset. We estimate it with validation and test samples, confidence intervals, repeated runs, and assumptions about how data were sampled. Generalization theory can bound the gap using model capacity, sample size, margins, stability, priors, or compression arguments, but no distribution-free calculation reads the exact future minimum directly from an arbitrary dataset.

11. Can Gradient Descent Reach The Minimum?

There are four different obstacles, and calling all of them "optimization" hides the diagnosis:

1. Irreducible uncertainty. The source itself may have positive conditional entropy.
2. Approximation error. The chosen architecture may be unable to represent \(p(y\mid x)\).
3. Estimation error. Finite or biased data may not identify the population distribution.
4. Optimization error. The training algorithm may fail to find the best parameters available inside the model class.

For an unconstrained distribution over a known finite alphabet, minimizing population cross-entropy is simple: choose \(q=p\). For a deep network, \(q_\theta\) is a complicated constrained family and its loss surface is non-convex in \(\theta\). Training can still find useful solutions, but there is no general promise that one run reaches the global population optimum.

The practical objective is therefore not "make training loss as small as numerically possible." It is:

Find a model whose probabilities compress held-out data well, remain calibrated under the intended deployment distribution, and achieve that result within compute, memory, latency, and numerical constraints.

12. Why This Matters To A Kernel Engine

In the C-Kernel-Engine (CKE), training support cannot stop at exposing a function named cross_entropy. The numerical contract includes:

For one target class \(y\) and logits \(z\), the stable forward computation is:

\[\mathcal{L}=m+\log\sum_j e^{z_j-m}-z_y,\;\; m=\max_j z_j\]

float m = logits[0];
for (int j = 1; j < vocab; ++j) m = fmaxf(m, logits[j]);

float sum_exp = 0.0f;
for (int j = 0; j < vocab; ++j) sum_exp += expf(logits[j] - m);

float loss = m + logf(sum_exp) - logits[target];

The subtraction by \(m\) is not algebraic decoration. It prevents large positive logits from overflowing the exponential while preserving the exact mathematical result. A production kernel must then define accumulation precision, vector reduction order, masking, ignored targets, reduction semantics, and the fused backward result.

• stable log-sum-exp rather than exponentiating large logits directly;
• the exact reduction axis and masking policy;
• mean versus sum reduction and ignored-token handling;
• FP32 accumulation boundaries for BF16 or quantized paths;
• the fused softmax-cross-entropy backward result;
• parity against a trusted framework on logits, loss, and gradients;
• separate train, validation, and held-out compression measurements.

A kernel can be fast while returning a subtly different log-sum-exp, and that difference can change both the reported loss and every gradient downstream. The information-theory interpretation tells us what the number means. The kernel contract determines whether the machine actually computed that number.

13. The Original Question, Answered Carefully

For a known true distribution: yes. The minimum expected cross-entropy is mathematically defined. It occurs at \(q=p\) and equals the source entropy.

For a finite dataset: we can calculate the minimum empirical loss within a chosen model class, but that is not automatically the minimum generalizing loss. The dataset is evidence about the source, not the source itself.

Can training reach it? training may approach the best solution representable by the model, but irreducible uncertainty, finite data, architecture limits, distribution shift, numerical behavior, and optimization all create separate gaps.

What is cross-entropy? it is the expected surprise of real outcomes when probabilities are supplied by a model. In coding language, it is the average number of bits paid when data comes from \(p\) but the compressor believes \(q\). In machine learning, minimizing it teaches \(q\) to approximate \(p\).

14. References And Further Study

• Grant Sanderson, But What Is Cross-Entropy? | Compression Is Intelligence Part 2, 3Blue1Brown, July 16, 2026.
• 3Blue1Brown cross-entropy animation source, by Grant Sanderson and contributors. This article uses the predictor-compressor question as a teaching prompt; its diagrams and derivation are original.
• Claude Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, 1948.
• Dario Benedetto, Emanuele Caglioti, and Vittorio Loreto, Language Trees and Zipping, Physical Review Letters, 2002.
• Dawid, Lauritzen, and Parry, Proper Local Scoring Rules On Discrete Sample Spaces, for the formal scoring-rule context and more general notions of locality.
• Shao et al., DeepSeekMath, which introduced Group Relative Policy Optimization (GRPO).
• Rafailov et al., Direct Preference Optimization, for a preference objective built from policy log-probability ratios.
• Google Machine Learning Crash Course: log loss and regularization.
• Google Machine Learning Crash Course: overfitting and generalization curves.
• How The Jacobian Matrix Works, for the derivative structure behind vector-valued model outputs.
• Muon, SGD And AdamW, for how an optimizer uses gradients after the loss kernel produces them.