The questions that sent me down this path

1. How is TabPFN different from the Clef-style model we have been working on?
2. What does its architecture actually look like?
3. If it uses a transformer, what makes it different from the language models I already know?
4. Where does its training data come from, and what is the training recipe?
5. Could I build a small version from scratch to understand the moving parts?

I am going to answer those by following one little table. It is much easier for me to understand a model when I can see what goes in, what changes inside it, and what comes out.

First, what kind of decision are we talking about?

With Clef-style decision heads, I give the model some state and a set of allowed choices. It scores the choices. TabPFN starts somewhere else: I give it a table with known examples, then ask about another row. That difference sounds small until you look at the input.

Say I maintain servers and want to estimate whether one needs attention this week. I have temperature, recent error count and machine family. Four machines have known outcomes; the fifth is the one I am asking about.

Looking at E, I would probably guess “yes.” But TabPFN does not get my guess. It gets the four labelled rows as context and E's features without its answer. Its job is to predict that missing answer. I made up these rows so we can follow the idea; I have not run this particular example through the model.

Where did it learn to do this?

This was the part that took me a moment. In a traditional tabular workflow I might train a new XGBoost model on each dataset. TabPFN has already spent a lot of compute learning across many generated tables. During that offline training, it repeatedly sees a synthetic task: some rows with labels, other rows to predict. The weights change across those tasks. Later, my server table becomes the context for a forward pass. I am not starting a fresh neural-network training run every time I bring it a new table. The original TabPFN paper calls this amortized Bayesian-style inference: pay for the broad learning up front, then use it on a new problem.

The useful thing to notice in that shorthand is what changes. The new table changes the context; the pretrained weights \(\theta^*\) can stay put. This is my simplified way of writing the idea, not the complete training objective from Prior Labs.

The 3.5 report's synthetic-prior section says the generator puts more emphasis on high-cardinality categories, tables with many features, and grouped data where the test group differs from the training group. That matters to me: a model trained only on tidy little random tables may learn the wrong habits for real datasets. The paper does not hand us a one-paragraph recipe that reproduces its weights, though. Building a small teaching version is a different project from reproducing TabPFN-3.5.

So what is inside TabPFN-3.5?

The 3.5 technical report and the released architecture code show a pipeline with more structure than “feed a CSV into a transformer.” The model must represent columns whose scales, missing values and categories differ, combine information across features, and then compare rows with and without known targets.

1. Turn each cell into more than one number

For each column, preprocessing records missing or infinite values before imputing them from the training column. It keeps a standardized value, but 3.5 adds two useful views of that value. A learned Fourier encoding applies sine and cosine at multiple frequencies; an empirical-CDF (ECDF) signal places the value within the training column's distribution. If 75 degrees is high for one machine family but ordinary for another dataset, the rank-like signal supplies context that a raw number alone cannot. The report says the ECDF is computed against training rows of the same column, avoiding test-label leakage. In the released configuration, the Fourier encoder uses 32 learned frequencies and the ECDF expansion uses four harmonics. Missingness remains explicit rather than being silently interpreted as a normal temperature. [report, section 3.1]

The exact released implementation has several preprocessing details; those equations are a reading aid, not drop-in source code.

2. Group features, then learn a column distribution

TabPFN-3.5 groups three feature positions at a time using circular offsets. The resulting group embedding combines the value signals above. A per-column distribution embedder then lets the model learn what values in a feature group look like across rows. It uses induced self-attention blocks: a bounded set of learned inducing points can summarize a larger column distribution without treating every cell as an independent vocabulary token. In the default 3.5 configuration, this stage has three blocks and 128 inducing points. [architecture config]

3. Assemble a row, then compare rows

Feature aggregation brings information across columns into a row representation using eight aggregation tokens per row. The in-context transformer then relates rows to the labelled examples. TabPFN-3.5's default configuration has 24 in-context layers, a 1,024-wide row representation and 16 attention heads. I initially mixed up the eight aggregation tokens with the 16 attention heads. They do different jobs: the tokens help build a row representation; the heads split the attention computation. The report describes the width increase from TabPFN-3's 512 to 1,024, enabled by four becoming eight aggregation tokens. [report, section 3.1]

The task-specific output differs by problem. Classification uses the known class labels to score classes; regression predicts a distribution over bins. The backbone stages are shared, while label encoding and output head depend on task type. That is how the 3.5 report describes a single checkpoint trained jointly for classification and regression. [report, section 3.1]

Is this just a transformer for spreadsheets?

It does use attention, but the unit of work is not a stream of next-word tokens. Before the row transformer gets involved, TabPFN has to turn messy columns into useful cell representations, learn something about each column's distribution, and gather features into a row. The row transformer can then relate an unlabelled row to the labelled ones. Its attention boundary matters: the answer I am trying to predict cannot become evidence for itself. A language model's usual next-token story does not explain this table-specific front end, the labelled-row context, or the task-specific prediction head.

Underneath, I still recognize familiar operations from CKE work: projections, attention, normalization and matrix multiplies. What changes is how the data is represented and how the attention is allowed to flow. That is why I find this model interesting rather than treating “transformer” as the end of the explanation.

Let me follow one prediction through the math

The architecture diagram is useful, but I wanted to know what the numbers are doing. I will keep the five-machine table above and follow machine E. The equations below are a small teaching model of the operations, not a transcription of every layer in TabPFN-3.5. The architecture section of the report and the released configuration are the references for the actual design.

1. A number becomes a cell representation

E's temperature is 75. A raw 75 is not enough: it has a scale, a position among the known temperatures, and possibly a missing-value flag. The four known temperatures are 48, 77, 52 and 73. Three of those are below 75, so a simple empirical rank is 3/4. In TabPFN-3.5 the in-context ECDF is computed against the training values of that column. The model also receives a standardized magnitude and learned-frequency sine/cosine features. The actual midrank and edge-case handling live in the implementation; here I am showing the intuition for a value not tied with any training value.

The function \(\phi\) stands for the value, missingness, Fourier and ECDF encodings plus learned projections. It is not one GEMM: ranking and handling missing values are preprocessing; sine and cosine are elementwise math; the projections and learned combinations use matrix operations.

2. Cells become a row, and rows become context

The model groups feature positions, learns distribution information across rows, and aggregates features into a row representation. In the default 3.5 configuration, eight aggregation tokens help build each row; the in-context transformer has a 1,024-wide representation. For our five rows, I can sketch the result as \(H\in\mathbb{R}^{5\times1024}\): four known rows and one query row. That is a shape for reasoning about this stage, not a promise that the implementation materializes exactly one flat matrix at every intermediate step.

The feature-distribution stage matters here. If I throw away the column context and treat 75 as just another token ID, I have lost what makes tabular data different. A temperature, an error count and a category have different distributions. TabPFN's cell encoder and induced-attention distribution embedder do work before the row transformer gets its input.

3. Attention lets E read the labelled rows

Take one attention head as a simplified view. Let \(h_E\) be E's row vector and \(H_{A:D}\) the four known-row vectors. Linear projections create a query, keys and values. A dot product scores how strongly E should read each known row; softmax turns the four scores into weights; another matrix multiply mixes the value vectors.

For this toy shape, \(q\) is \(1\times d_h\), \(K\) and \(V\) are \(4\times d_h\), \(qK^{\mathsf T}\) is \(1\times4\), and \(z\) is \(1\times d_h\). TabPFN-3.5's actual in-context stack has 16 heads, with a 64-dimensional per-head width in its default configuration, plus its own masking, normalization and grouped-query choices. The equations expose the operation; they are not a claim that the released model uses this exact four-row attention matrix at every layer.

4. What does it generate?

Not a sentence. For classification, the model gives scores or a probability distribution over the classes present in the task. For our table the classes are “needs attention” and “does not.” The paper describes a classification readout that attends over the known one-hot labels. To make that concrete, imagine that a readout assigned weights 0.10, 0.20, 0.15 and 0.55 to A, B, C and D. A and C are “no”; B and D are “yes.” Then the teaching calculation is \(p(\text{yes})=0.20+0.55=0.75\) and \(p(\text{no})=0.10+0.15=0.25\). Those weights and probabilities are invented here, not generated by TabPFN.

For regression, the output is different: the report describes a predicted distribution over fixed target bins, from which a numeric prediction can be derived. In neither case is the model deciding to shut down a server or open a ticket. It predicts a target; an application would still have to decide what to do with that prediction.

5. What changes during training?

Now replace my one table with many synthetic tasks. In each task, the generator supplies known rows and hides some query labels. The network predicts those hidden labels. For a classification query with true class \(y\), a simple loss is \(L=-\log p_\theta(y\mid X_{\mathrm{known}},y_{\mathrm{known}},x_{\mathrm{query}})\). Backpropagation moves through the output readout, row attention, aggregation and cell encoders, and the optimizer updates shared weights \(\theta\). The real 3.5 training run mixes classification and regression and uses task-specific label encoders and heads, so that single cross-entropy expression is one slice of the training picture, not its complete loss recipe.

When I later call fit on a new table through the package API, that is not the same as repeating this offline gradient-training program. The new table supplies context for the pretrained weights. This was the distinction I needed to get straight when I first read “prior-data fitted.”

Where do the kernels show up?

That table is why I would not say “it is all just GEMM.” GEMM carries much of the dense arithmetic, but ranking a column, building an attention mask and normalizing scores are different operations with different memory-access patterns. If I tried to bring a teaching version into CKE, I would first make these shape and numerical contracts explicit, then test each stage before worrying about speed. CKE does not currently have a TabPFN-3.5 implementation; this is the kernel map I would use to investigate one.

A practical trick: reuse the table you already know

If I ask about a hundred new machines against the same labelled history, repeatedly recomputing the training rows is wasteful. The 3.5 design can cache intermediate states for the known rows: distribution-embedder inducing states, keys and values in the in-context transformer, and decoder inputs. It also uses a bucketed ECDF cache. The report says this largely preserves cache size compared with TabPFN-3, despite a much larger model. A new test row can then read the prepared context. That explains why cached single-row latency and full-table first-pass latency should not be mixed in a headline. For large training sets, the report also notes quadratic scaling beyond 128k rows; “one forward pass” does not mean constant time. [report, inference benchmarks]

The report lists 220 million parameters for TabPFN-3.5 and 84 million for 3.5-Fast, with recommended limits of up to one million rows and 6,000 features. Those are reported model configurations and recommendations, not measurements from our machines or a guarantee for every dataset. It also distinguishes the open 3.5 and Fast checkpoints from the API-only Plus and Thinking variants. I would not quote a Thinking benchmark as if it were the base checkpoint. [report, model table]

Now put it next to Clef

In my Clef article, I used a support-ticket example: the model reads a state and a schema of allowed options, then returns logits for choices such as billing or technical. TabPFN is not doing that. It receives a table, labelled examples from that same task, and unlabelled rows. The target is a table prediction, not a dynamically worded question schema.

Here is the architectural split I was looking for. Cloudflare describes Clef as a Qwen-based, prefill-only backbone followed by option-specific evidence routing, cross-field attention and schema-bound scoring. TabPFN-3.5 instead has a table-specific cell encoder, a distribution embedder, feature aggregation and an in-context row transformer before its task-specific readout. Both contain attention and output scores, but they are not the same network with different labels on the last layer.

Could both help a decision system? Yes. One could predict failure risk from machine history; the other could score a typed routing question using an incident description. But that is a possible application design, not a reported TabPFN–Clef benchmark or a component we have integrated. They are not interchangeable heads on the same network. [Cloudflare's Clef architecture account]

Could I build one to understand it?

I think so, if I keep the goal honest. I would start with a tiny synthetic-table generator: sample a few features, choose a hidden rule that creates labels, split the rows into visible examples and queries, and train a small network to predict the query labels. Then I would add a cell encoder, a simple way to combine columns into rows, and attention that lets query rows read the labelled context without reading their own missing labels. Finally I would compare it against a small tree baseline on new synthetic rules. That would teach me the mechanism. It would not be a reproduction of Prior Labs' data generator, scale, training schedule or reported performance.

Only after that would I try the released checkpoint on a real table. At that point I would be asking a different question: how well does their trained model work for my data, on my hardware?

What would that first local test look like?

The official quick start uses a scikit-learn-like fit and predict interface. Here is the shape of the five-row example above, using only the numerical columns. This snippet is illustrative and has not been executed for this article. In the real experiment, I would encode the family column using the package's supported preprocessing path and use far more than four labelled rows.

import numpy as np
from tabpfn import TabPFNClassifier

X_train = np.array([[48, 0], [77, 9], [52, 1], [73, 7]])
y_train = np.array([0, 1, 0, 1])
X_test = np.array([[75, 8]])

model = TabPFNClassifier()  # current package default: TabPFN-3.5
model.fit(X_train, y_train)
probabilities = model.predict_proba(X_test)
print(model.classes_, probabilities)

fit here prepares the table context for the pretrained model; it should not be read as proof that a new neural network was optimized on four examples. The first call may download weights. Prior Labs recommends a GPU for larger workloads and describes the ordinary CPU path for TabPFN-3.5 as suitable for up to about 5,000 samples; its one-million-row model figure is not a promise that my laptop can run a million rows comfortably. Check the current package instructions and the weights license before using it in a product. [Prior Labs README]

What I want to try next

For this server example, I would compare TabPFN-3.5 with a tuned tree model, a simple baseline and, if the task is textual decision-making, a separate decision model. I would split by future time or machine family if that matches deployment. A random row split can put near-duplicate observations from the same machine on both sides and make any model look stronger than it is. I would record accuracy or error, calibration, false-negative costs, runtime with and without cache, and how missing sensors or new hardware families affect the answer. The Prior Labs report includes grouped and temporal benchmarks; that is useful evidence, but it does not replace testing on our own split. [TabPFN-3.5 technical report]

The released Python implementation is open-source, but its 3.5 model weights carry a separate license. The report permits research and evaluation while restricting commercial and production use of the released weights without the appropriate commercial arrangement. That matters if this ever becomes more than an experiment. [license section]

The part I keep coming back to is where the learning happens. TabPFN learns across many generated tables and then uses my new labelled table as context. Clef reads a state and a question schema to score the options I gave it. I want to build a small teaching version of the first idea, then run the released model on a table I actually care about. That should tell me more than staring at another leaderboard.