When to adapt a model—and what changes inside it

Before asking where to insert LoRA, I would ask what the model is getting wrong. Does it lack information? Does it misunderstand the instructions? Or does it repeatedly struggle with a task even when the information and instructions are there? Those lead to different next steps.

When would I actually want LoRA?

LoRA is useful when I want to learn that task behaviour while training and storing a relatively small set of parameters. It is not a requirement for using a local model, and it is not something I would add simply because the option exists.

These approaches can work together. An adapted model might still retrieve documents and call tools. Fine-tuning can teach useful patterns from data, but I would not treat its weights as a dependable database for frequently changing facts.

Fine-tuning, full fine-tuning and PEFT

Fine-tuning means starting with a pretrained model and continuing training for a particular purpose. In a supervised example, the model processes an input, a loss measures its prediction against the desired target, and backpropagation computes gradients used by an optimizer to update the trainable parameters. We are continuing from learned weights, not training the whole model from a random start.

Full fine-tuning makes all the original model parameters eligible for updates. That gives the optimizer a much larger space to change, with a larger gradient and optimizer-state memory budget. It can be worth evaluating when the task, available data and compute justify it. It does not automatically produce a better model: overfitting and degradation of previously useful behaviour still need evaluation.

PEFT means Parameter-Efficient Fine-Tuning. It is a family of methods that train a relatively small number of parameters while keeping most of the pretrained model fixed. Some learn added parameters; others selectively train existing ones. LoRA is one member of that family. PEFT is also the name of Hugging Face's implementation library; the general idea is broader than that library.

How would I start?

I would first save a baseline on a small, representative evaluation set. Then build clean training examples, keep evaluation examples out of training, and choose a supported base model. Try LoRA when the unquantized base fits the training budget; consider QLoRA when base-weight memory is a constraint and the runtime supports the required quantized training path. Compare task quality, peak memory and speed rather than assuming either choice wins.

The detailed experiment plan comes later in this article. First, let us see what the adaptation actually adds: where it lives inside the LLM, why it is called low rank, and how its small matrices connect to attention, the MLP and the kernels underneath.

What are LoRA and QLoRA?

LoRA means Low-Rank Adaptation. It is a way to fine-tune a pretrained model by learning small corrections to selected weight matrices. In the basic setup, we keep the original model weights fixed and train the added matrices using examples of the task we want the model to learn.

QLoRA combines this adaptation with a quantized frozen base. The original QLoRA recipe stores eligible base weights in four bits, while the adapter and much of the computation use higher precision. The important distinction is simple: LoRA changes which parameters we train; QLoRA also changes how we store the base model. See the original LoRA and QLoRA papers.

Explore the model: open a block, then a projection

Start with block 1. Open Attention or Gated MLP, then open an individual projection. The red branch belongs to that particular weight matrix. Change the rank, enable other projections, or apply Q and V adapters across the whole stack.

About this teaching model · terms and assumptions

What is a block? Here, one block means one transformer layer: attention plus an MLP, with normalization and residual connections. Its output becomes the next layer’s input. Ten blocks means ten repeated transformer layers—not ten attention heads or ten adapters.

Model width is the number of features per token in the residual stream. Adapter rank r is the intermediate width between its two GEMMs: d → r → m. It is also the shared dimension when multiplying A [d × r] by B [r × m]. It is not the size of the whole GEMM, nor the number of kernels.

Teaching model: a pre-normalized causal decoder with equal-width Q/K/V, RoPE and a gated MLP expanded to four times the model width. Five or ten blocks are illustrative, not a specific BERT or Qwen configuration. Bias, dropout and training execution are omitted.

Transformer block → attention and MLP

The input passes through normalization and attention, a residual addition, then normalization and a gated MLP followed by another residual addition.

Attention → Q, K, V and output projections

LoRA can adapt each selected projection separately. Q and K determine attention scores; V supplies the combined features. The output projection receives concatenated head outputs.

MLP → gate, up and down projections

Gate and up projections consume the same normalized input. SiLU(gate) multiplied by up produces H. The down projection and its adapter consume H.

One projection → frozen weight plus trainable adapter

Y = XW + s(XA)B. With a 4096 × 4096 weight and rank 1, A is 4096 × 1 and B is 1 × 4096: 8192 trainable parameters. QLoRA changes frozen-weight storage; the attachment points remain the same.

Enable JavaScript for the configurable graph. The expandable explanation and the article diagrams remain available without it.

A second view: follow the computational graph

The explorer above answers where can I attach an adapter? These graphs answer which operation consumes each tensor? Small nodes represent operations; labelled arrows show tensor shapes. Branches and joins are actual data dependencies. The layout takes inspiration from the compact network views in the Wolfram Neural Net Repository; the model here remains our illustrative causal decoder, not BERT.

Read left to right. Scroll wide graphs horizontally, or open the full-size SVG. Blue-grey nodes handle the base computation; red outlines identify projection targets or adapter operations; violet marks nonlinear operations. Dashed arrows supply parameters.

Expand attention: Q, K, V → scores → probabilities → weighted values

Here the batch size is one, with eight token positions, model width 512 and eight heads of width 64. After splitting heads, the axes are [heads, tokens, head width]. The score multiplication and value aggregation are batched over heads. Softmax runs over the key-token axis. The causal mask blocks access to future positions.

A Q adapter changes Q before reshaping and RoPE; a K adapter changes K at the corresponding point. A V adapter changes the values before aggregation. The O adapter acts on the concatenated result. The softmax and the two attention matrix products have no separate projection weights to adapt in this example.

Expand the MLP: parallel projections → SiLU → elementwise product → down

The symbol ⊙ is an elementwise multiplication, not GEMM. Both inputs have shape 8 × 2048. The down projection compresses this result back to 8 × 512 before the residual addition shown in the parent graph.

Expand one projection: frozen GEMM plus the two adapter GEMMs

This fixed numerical example uses rank 4 and alpha 16, so the usual scale alpha/r is 4. The base path and adapter path receive exactly the same X. For QLoRA, the W₀ parameter is stored as quantized codes and metadata; weight reconstruction may be fused into the base multiplication. The adapter topology remains the same.

Where does it sit in the whole LLM?

A typical decoder-only language model maps tokens to vectors, passes those vectors through a stack of transformer blocks, and uses a final normalization and output projection to produce token logits. Each block contains attention and a feed-forward network, often called the MLP. Inside both are learned matrix multiplications.

LoRA usually targets some of those matrix multiplications. It does not have to add another transformer block or a separate reasoning stage after the model. A configuration can select particular projections in every block, or only selected blocks. Each selected weight generally gets its own adapter parameters; all the red locations in the diagrams below are possible targets, not mandatory ones.

Why “low rank”? Is rank a layer count?

Suppose a layer maps 4,096 input features to 4,096 output features. A full correction could contain 4,096 × 4,096 independently adjustable entries. At rank 16, LoRA instead learns a mapping from 4,096 features into 16 intermediate features, followed by a mapping from those 16 features back to 4,096 outputs. The correction can combine at most 16 independent directions. The rest of the original layer is still there.

At rank one, the adapter extracts one learned scalar per token and uses it to scale one learned output direction. At rank sixteen, it can mix sixteen such directions. Both the input combinations and output directions are learned. That is the useful intuition behind the two matrices A and B. Their product has rank at most the chosen rank; training can produce a lower actual rank.

A few questions I had about rank

Once I look at the two matrix multiplications, these are the questions I want answered before moving on.

“So rank is basically the GEMM dimension?”

Yes—the configured rank r is the intermediate dimension between the adapter’s two matrix multiplications. With row-vector notation, X [n × d] multiplies A [d × r] to produce XA [n × r]. That result multiplies B [r × m], giving a correction of shape n × m.

There is a useful distinction: r is the chosen bottleneck width; the mathematical rank of AB is at most r. The learned factors could have dependent directions, giving a smaller actual rank. We are restricting the correction’s capacity, not counting transformer blocks, attention heads or model quality.

“Can each LoRA in each layer have a different rank—layer 1 rank 4, layer 5 rank 10?”

Absolutely. Nothing mathematically requires uniform ranks. We can go further: the Q projection and V projection inside the same transformer block can have different ranks. Each adapted projection owns its own pair of trainable matrices.

For a 512-to-512 projection, rank 4 means A is 512 × 4 and B is 4 × 512. Rank 10 means A is 512 × 10 and B is 10 × 512. Both branches return a n × 512 correction. That is why the surrounding transformer does not need to change. In an MLP with different input and output widths, use that projection’s own d and m instead.

Rank 10 is valid. Some hardware and kernels work more efficiently with particular dimensions, but that is a performance consideration, not a mathematical restriction.

“Then why keep the rank uniform, apart from more complicated programming and training?”

Uniform rank is a practical starting point. It is not a claim that every projection needs equal capacity.

• A predictable parameter budget: an adapter has r(d + m) trainable parameters. Sum that over the selected projections. Equal ranks do not imply equal parameter counts when projection dimensions differ.
• Fewer tuning decisions: one rank setting is easier to investigate than a different setting for every projection in every block.
• A clear experimental baseline: keep the allocation fixed while comparing data, learning rates or training duration. Otherwise, several choices change at once.
• More regular computation: compatible shapes can make batching and kernel execution easier. A speed benefit is implementation-dependent, not guaranteed.

Different projections may benefit from different amounts of adaptation. A larger rank where it helps can be a better use of memory than increasing every rank. But finding that allocation takes evaluation or an allocation method. A deeper block does not automatically need a higher rank, and a higher rank does not guarantee a better model.

This is already supported in tooling: Hugging Face PEFT exposes rank_pattern and alpha_pattern for assigning ranks and scaling parameters to particular modules. Use the actual module names of the model you are adapting. See the PEFT guide to per-module ranks and scaling.

“There is no nonlinear activation between the down and up GEMMs?”

Correct. Standard LoRA computes (XA)B without a nonlinear activation between A and B. That lets us write the same correction as X(AB). Adding an activation there would change the function and generally prevent representing it as one fixed weight correction.

The transformer’s existing nonlinear operations remain in place. For example, an adapted MLP gate projection still feeds its result into SiLU. Optional adapter dropout during training is a separate issue; it is not an activation inserted between A and B. The simple diagrams here omit dropout.

“What is ×4? Are there always two adapters per layer?”

Two matrices form one adapter. A reduces the feature dimension and B expands it to the projection’s output width. These operations change dimensions; they do not necessarily shrink or amplify the numerical values.

The ×4 in the computational graph is a separate scalar multiplier. For ordinary LoRA, s = α/r. In that example, α = 16 and r = 4, so s = 4. It does not mean four adapters or four layers.

If we adapt Q and V, that is two adapters and four trainable factor matrices per block. If we adapt Q, K, V and O, that is four adapters and eight matrices. In the illustrated gated-MLP architecture, adapting those four plus gate, up and down gives seven adapters and fourteen matrices. These are choices about targets, not a requirement to adapt every projection.

“If I change ranks, should I also think about alpha?”

Yes. Under s = α/r, keeping alpha fixed while increasing rank changes the multiplier too. With alpha 16, rank 4 gives scale 4, while rank 16 gives scale 1. Account for that when comparing experiments. Keeping alpha proportional to rank holds this explicit multiplier constant, but does not guarantee identical update magnitudes or learning behaviour: the factors, initialization and optimization also matter. Some LoRA variants use a different scaling rule.

Without training-time dropout, the linear correction is ΔW = sAB. It can be merged into compatible base weights for inference as Wmerged = W + ΔW, subject to the precision and quantization considerations discussed later.

Where does the adapter actually attach?

“Where do ranks or LoRA normally attach—Q, K, V, MLP up or down, or LayerNorm?”

The adapter attaches to a selected computation; rank is a setting of that adapter. In a transformer, the usual targets are learned linear projections: Q, K, V and the attention output projection O, plus the MLP’s up, down and, in a gated MLP, gate projection. We do not attach a rank to the whole block as a new operation.

Each selected projection gets a correction with the same output shape as its base calculation. The attention score product QKᵀ is also a matrix multiplication, but its inputs are activations, not a stored projection weight W. It is therefore not the usual place to attach a standard LoRA weight update. Softmax, masking, residual addition and the elementwise MLP product are not ordinary LoRA targets either.

This is a pre-normalized decoder block with a gated MLP. Red means eligible in this example, not that every training run must adapt every red box. Normalization order, positional operations and MLP structure differ between model families.

“Does it have to attach to actual weights, or can it be its own side transformer block?”

Standard LoRA describes a low-rank correction to a selected weight matrix: W + sAB. During training, the runtime can implement that as a separate branch s(XA)B, while leaving W frozen. It is a side branch in the computation graph, but it still belongs to that particular projection. It consumes the same local input as the base projection and adds its result to that projection’s output.

You can design a separate trainable transformer block or side network. That is a different architectural choice; it is not standard LoRA merely because it sits alongside a frozen model. A transformer block brings attention, normalization and nonlinear transformations. The ordinary LoRA branch has two linear factors and scaling, without its own attention mechanism.

You could also apply LoRA to compatible projections inside an added block. In that case, adding the block and using LoRA within it are two separate design decisions.

“So is LoRA mostly represented as GEMM?”

For dense transformer projections, yes. In an unmerged implementation, the base multiplication XW is accompanied by two smaller multiplications, XA and (XA)B, followed by scaling and addition. GEMM names a matrix-multiplication operation; it is not a requirement to launch exactly three separate GPU kernels. A runtime can fuse operations, and a single-token calculation may use a matrix-vector path.

The idea is not restricted to one kernel name or only dense transformer layers. Implementations can support low-rank updates for embeddings or convolutional weights too, with the appropriate layouts and operations. For this article, the dense projection is the clearest place to see the math.

“What about LayerNorm? It has learned parameters too.”

Having parameters does not make every operation a natural LoRA target. LayerNorm normally has a learned scale vector and a bias vector; RMSNorm commonly has a scale vector. These act element by element around normalization, rather than as the large dense d-by-m projection we factorize here.

Training those normalization parameters is a valid fine-tuning choice, but it is not the same operation as attaching an ordinary A/B LoRA branch. A recipe may combine normalization tuning with LoRA. Make that explicit in the trainable-parameter list instead of calling every trainable component a LoRA adapter.

“Where do traditional LoRA adapters normally go?”

There is no universal target list. The original LoRA work focused on attention weights in its transformer experiments and compared different choices. Q and V are a familiar economical starting configuration. Adapting all four attention projections gives broader attention coverage; including the MLP expands coverage further.

The PEFT guide describes targeting all linear layers for QLoRA-style training. The QLoRA paper is the primary reference for the quantized-base training method. Broad targeting is a recipe choice, not a mathematical requirement of four-bit storage. The actual eligible modules and output-head treatment depend on the model and library. Inspect the matched modules before training.

• Q/V only: a smaller adaptation budget and fewer targets to inspect.
• Q/K/V/O: corrections across the attention projections.
• Attention plus MLP: broader coverage, with additional trainable parameters and computation.

These options are experiments, not a quality ranking. Compare them on held-out examples under a clear budget. Do not assume that more targets always improve results. Names such as q_proj are architecture-specific, and fused QKV weights need support for selecting the intended pieces.

“Are those the same as the older bottleneck adapters?”

The word adapter covers more than LoRA. A classic bottleneck adapter is a small module that takes hidden states, projects them down, applies a nonlinear activation, projects them back up, and adds a residual connection. In row-vector shorthand, omitting biases: H + f(H D) U.

Houlsby-style adapters were inserted around the attention and feed-forward sublayer outputs. Their two projections resemble LoRA’s two factors visually, but the nonlinearity and insertion point change the computation. Other adapter families use other placements.

For standard LoRA, the correction sAB can be folded into compatible base weights for inference. A nonlinear bottleneck adapter generally cannot be folded into one fixed update to the preceding weight matrix. Neither of these small adapter modules is, by itself, a complete transformer block.

My practical check is: which exact tensor enters the adapter, which weight or hidden-state transformation does it modify, and where is the result added? Those three questions make the attachment point concrete. The attention and MLP diagrams below follow those tensors in more detail.

Inside attention: which multiplication changes?

Let U be the normalized input to attention. The query, key and value projections produce Q, K and V. Adapting q_proj means computing Q = U Wq + sq(U Aq)Bq. Adapting v_proj does the same for V, with its own weights and adapter. The corrected Q, K and V then enter the usual attention calculation.

• Query projection: changes the query representation used to score keys.
• Key projection: changes the key representation against which queries are compared.
• Value projection: changes the features combined using the attention weights.
• Output projection: changes the mapping of the concatenated head outputs back into the residual-stream width. Its adapter receives that concatenated attention output, not the original U.

So the adapter is attached to a learned projection, before its result is consumed by the next operation. It is not normally attached to softmax or to the token-by-token attention score matrix itself. In grouped-query attention, Q and K/V can have different widths; each adapter must match its own projection. Some implementations fuse Q, K and V into one weight object, so selecting the right subprojections requires checking the implementation rather than guessing from a module name.

Inside the MLP: before or after the activation?

If we adapt the gate projection, its correction is added before SiLU. If we adapt the up projection, its correction is added before the elementwise product. If we adapt the down projection, its adapter takes the already gated intermediate features as input, and its correction is added to the down-projection output before the residual addition.

That last detail matters when implementing the kernels. Every adapter consumes the input of the particular linear layer it modifies. We cannot feed the original block input into every adapter and expect the same calculation. A conventional two-layer MLP without gating has different names and fewer projections, but the same rule applies at its expansion and contraction weights.

Is LoRA a new GEMM kernel?

LoRA defines a parameterization and a computation, not a mandatory new hardware instruction. GEMM means general matrix multiplication. With an unmerged adapter, a selected layer has its original multiplication, two smaller adapter multiplications, a scale and an addition. A runtime can execute those with existing matrix-multiplication kernels or specialize and fuse parts of the path.

For one token, these shapes can behave more like matrix-vector operations. For a batch of tokens, they are matrix-matrix operations. This is why a low parameter count does not automatically imply zero latency overhead. Dispatch, data layout and small-matrix efficiency still matter. For compatible inference paths, the correction can be merged into the base weight ahead of time; we will return to the precision trade-offs later.

Where does QLoRA enter this picture?

Keep the same transformer diagrams and the same adapter attachment points. Now store eligible frozen base matrices as quantized codes plus reconstruction metadata. The base path reconstructs weight values for computation, potentially inside a fused kernel. The adapter path still learns its higher-precision A and B matrices. The two paths still add their outputs.

There is no extra “QLoRA layer” after attention. Quantization is a storage-and-computation choice for the frozen base, while LoRA supplies the trainable correction. Activations, normalization parameters and other tensors do not all become four-bit. Nor does a four-bit checkpoint by itself guarantee that a runtime can train adapters on it.

Can adapters be inserted elsewhere?

The most common targets are attention and MLP projections, but supported implementations can also adapt embeddings, output heads, convolutions or expert projections. The shapes and support rules differ. Normalization scales and biases can sometimes be trained separately; that is a distinct configuration choice from attaching a standard two-matrix LoRA branch.

In PEFT, target selection is exposed through target_modules and related configuration. Module names differ across models; an all-linear selection is a convenient policy, not proof that every model tensor or output head is included. Inspect the actual selected modules and trainable parameters. The LoRA configuration reference and custom-model guide explain those selection rules.

Now we have the map. Let us work through one adapted layer numerically, then follow its gradients and count the memory.

1. Put LoRA inside one matrix multiplication

I will use row vectors and batches of rows throughout. This keeps the notation close to the matrix Wx+b and GEMM walkthrough, with the orientation made explicit. Let X contain n token representations, each of width d. The layer produces m features:

X:  n × d
W₀: d × m
Y = XW₀                 # bias omitted for clarity
Y:  n × m

Here n can stand for the batch and sequence positions flattened together. A dense correction would contain another d × m numbers. LoRA writes that correction as the product of two smaller matrices:

A: d × r
B: r × m
s = α / r
ΔW = sAB
Y = XW₀ + s(XA)B

First XA maps the input into an r-wide space. Then multiplication by B maps it back into the layer's output width. We add this contribution to the frozen base output. The scalar s controls the contribution's scale; the usual original convention is α/r. Other scaling conventions exist, so an adapter's configuration is part of its meaning.

The shape check is useful: (n × d)(d × r)(r × m) ends at n × m, exactly the shape of the base branch. If those shapes do not agree, the addition cannot represent this layer.

In PyTorch, a standard linear layer stores its weight as out_features × in_features and applies its transpose. That is why implementations often show BA instead of the AB used here. The letters are less important than the dimensions. Check the storage convention before translating an equation into a kernel. See the Linear documentation.

2. What does “low rank” actually restrict?

For a single input row, xA produces r coefficients. Those coefficients mix the r rows of B to construct the output correction. The adapter therefore has at most r independent output directions at that layer.

Rank does not mean “the number of facts the model can learn.” It is a linear-algebra constraint on one update matrix. Many adapters distributed through a nonlinear network can have a substantial effect on the overall function.

The parameter count is easy to calculate:

Dense matrix:          d × m
LoRA matrices:         d × r + r × m = r(d + m)
For d = m = 4096:
  dense parameters:    16,777,216
  LoRA at r = 16:         131,072
  adapter / dense:       0.78125%

These are counts for one targeted matrix, not the whole model. Increasing rank, targeting more layers, or training an additional output head all increase the trainable total. Small rank is also not automatically economical for a tiny layer: the saving depends on whether r(d+m) is smaller than dm.

To see the constraint itself, take a synthetic target correction with singular values [4, 2, 1, 0.5]. A rank-one matrix cannot reproduce all four nonzero directions. The best rank-one approximation retains the largest direction and leaves residual Frobenius norm √(2² + 1² + 0.5²) ≈ 2.291. Retaining more directions reduces that approximation error.

This chart is an exact linear-algebra illustration. It is not a measured LoRA training curve. Real training optimizes a task loss through the network, not the distance to a known dense update, and does not necessarily find the best rank-constrained solution.

3. A complete forward pass with numbers

Let us make the layer small enough to inspect. Use three input features, two outputs, rank one and scale s = 0.5:

x  = [2, -1, 3]

W₀ = [[1,  0],
      [0,  1],
      [1, -1]]

A  = [[1],
      [0],
      [1]]

B  = [[0.2, -0.1]]

The base output is xW₀ = [5, -4]. The rank-one activation is xA = 5. Multiplying that scalar by B gives [1, -0.5]. Scaling by one half gives the adapter contribution [0.5, -0.25]. Add it:

y = [5, -4] + [0.5, -0.25]
  = [5.5, -4.25]

The same calculation can be expressed as a merged matrix:

sAB = [[0.1, -0.05],
       [0.0,  0.00],
       [0.1, -0.05]]

x(W₀ + sAB) = [5.5, -4.25]

That equivalence follows from associativity and distributivity. In finite precision, separate and merged computations can round differently. The example also explains why rank one is not “change one weight”: its outer product can change many entries together.

4. Frozen weights do not remove the backward path

This is the part I pay attention to when thinking about a runtime. Freezing a parameter means that we do not optimize it. It does not mean that we can discard every derivative involving the operation that uses it.

Let D = ∂L/∂Y, the gradient arriving from the rest of the network. For our layer, without dropout or bias, the derivatives are:

dB = s(XA)ᵀD             # r × m
dA = sXᵀDBᵀ              # d × r
dX = DW₀ᵀ + sDBᵀAᵀ      # n × d

The optimizer needs gradients for A and B. If earlier adapters need an input gradient, the base term DW₀ᵀ must still be computed. The base weight-gradient product XᵀD need not be materialized solely to update a frozen W₀.

For our worked example, choose an incoming gradient D = [0.4, -0.2]. Then dB = [1, -0.5], dA = [0.1, -0.05, 0.15]ᵀ, and dX = [0.45, -0.2, 0.65]. I checked these against finite differences of the scalar objective sum(Y * D); the maximum absolute discrepancy was below 3 × 10⁻¹⁰ in float64.

A common initialization uses a random A and zero B. Then the adapter starts with zero output. Notice what the derivatives predict: initially dA is zero because it contains Bᵀ, while dB can already be nonzero. Once B changes, gradients can update both factors. Initializing both factors to zero would instead block both gradients in this simple parameterization. The reference LoRA implementation is useful for checking initialization and merge behaviour.

5. Where do adapters go in a transformer?

A transformer contains many learned projections: query, key, value and output projections in attention, plus the up, gate and down projections of its feed-forward blocks. Adapters can target selected projections. An implementation must identify the actual modules and shapes; naming conventions differ across architectures.

Changing an attention projection changes how information is represented or routed. Changing a feed-forward projection changes another part of the transformation applied to each token. These are different intervention points. Equal ranks on different sets of modules do not describe equivalent experiments.

Before training, print the trainable parameter names, shapes and count. Check that the intended layers received adapters, that the original weights are frozen, and that any deliberately trainable head is included. A script that runs without errors can still be training the wrong parameters.

This also connects to the Clef-style decision-head walkthrough: training a task head and adapting the backbone are separate choices. A new head does not by itself establish that LoRA is enabled, and LoRA does not define the head's architecture.

6. QLoRA: change how the frozen base is stored

LoRA reduces the number of trainable parameters, their gradients and their optimizer state. The frozen base can still be large. QLoRA attacks that remaining weight-storage cost.

The QLoRA paper by Dettmers and colleagues combines low-rank adaptation with a frozen four-bit base. Its recipe includes NF4, double quantization and paged optimizers. These address different parts of the memory problem; they should not be collapsed into “everything is four-bit.”

Ŵ₀ = dequantize(codes, quantization_metadata)
Y  = XŴ₀ + s(XA)B

Frozen:   base codes and their quantization representation
Trainable: adapter factors A and B

NF4 is a nonuniform, codebook-based representation designed around a normal-distribution assumption for weights. A four-bit code selects a reconstruction level. Block scales relate those normalized levels to the values in a particular block. Double quantization compresses quantization constants to reduce metadata overhead. Paged optimizers use memory-management mechanisms to handle memory pressure; they do not make optimizer state disappear.

The quantized linear layer reconstructs values for computation, potentially within a fused kernel. It does not necessarily allocate an entire dense dequantized model at once. The bitsandbytes Linear4bit interface exposes storage and compute settings separately.

The adapter branch, activations, reductions and gradients have their own dtypes. BF16 may be used for parts of the computation; some operations or optimizer states can remain FP32. Exact choices depend on the implementation and hardware.

NF4 is not another spelling of MXFP4 or NVFP4

Our earlier four-bit quantization article follows the contracts behind Q4_K, MXFP4 and NVFP4. NF4 has a different encoding and reconstruction contract. “Four bits” tells us the code width, not the codebook, scale layout, arithmetic or supported kernels.

For a runtime, that distinction becomes concrete: which bytes encode the weights, how are scales located, which values must be reconstructed, and which multiply consumes them? A kernel that handles one four-bit format does not automatically support every other format.

“How does the quantization train? Does the adapter dequantize and quantize again?”

In standard QLoRA, the quantized base is frozen; the adapter trains. The adapter does not decide how to decode weights. The quantized layer and its runtime kernels already know the format, scales and reconstruction rules.

1. Load the base: quantize eligible pretrained weights into the chosen format, or load an already compatible quantized checkpoint.
2. Compute: the runtime reconstructs approximate base values as needed for multiplication. This can happen inside a fused kernel rather than creating a full dense copy. The adapter computes its separate correction.
3. Backpropagate: the loss supplies gradients to A and B and, where needed, earlier activations. Frozen weights still participate in those calculations; they do not receive optimizer updates.
4. Update: the optimizer changes the adapter factors. Standard QLoRA does not repeatedly update and requantize the base after each step.

The adapter learns from the predictions of the quantized base plus its own correction. It can learn useful task changes in that setting, but it does not recover the exact original weights or guarantee removal of quantization error. Double quantization means compressing quantization constants; it does not mean an adapter repeatedly quantizes itself during training.

“Which quantization formats can we train with?”

Separate two questions: how are the frozen weights stored, and does this backend support the backward path needed to train adapters? Being able to load a format for inference is not enough. The PEFT quantization guide documents supported combinations; support is specific to the library, model, hardware and version.

Base representation	What it means here	Training interpretation
BF16 / FP16	16-bit floating-point base weights	Ordinary LoRA baseline; the base need not be quantized.
NF4	4-bit codes selecting nonuniform reconstruction levels, with block scales	The original QLoRA recipe’s base format; adapters remain higher precision.
bitsandbytes FP4	A different 4-bit floating-point encoding	An alternative supported by its 4-bit linear implementation; not interchangeable with NF4.
8-bit base	Lower-precision base storage, with method-specific metadata	Adapter training is supported in documented configurations; not the original four-bit QLoRA recipe.
Other quantized checkpoints	For example, GPTQ-based representations	Use only a documented training-compatible integration. The quantization method alone does not establish backward support.

The bitsandbytes four-bit layer reference distinguishes NF4 and FP4 and exposes compute precision separately. NF4, bitsandbytes FP4, Q4_K, MXFP4 and NVFP4 are not synonyms. Do not assume a serving kernel for one of these formats provides a usable adapter-training path.

Quantization changes the bytes used to represent W, not its logical input and output dimensions. A 4,096-by-4,096 projection remains that shape after quantization. Its rank-16 adapter still has A shaped 4,096-by-16 and B shaped 16-by-4,096. Physical storage may pack several codes into a byte and store scales separately.

“What gets saved after training?”

A typical adapter export saves A and B, their target-module configuration and scaling, with enough information to identify the compatible base model and quantization setup. Those adapter weights are not automatically converted to four-bit just because the base uses NF4. A resumable training checkpoint additionally needs optimizer, scheduler and other training state.

For inference, keep that adapter with the compatible base, or use a supported merge/export process. Merging into a quantized model may require reconstructing weights, adding the correction and quantizing again; that is a separate export step with numerical trade-offs, not the usual training loop.

This is the short version. In the next article we will follow the gradients through the two branches and check the backward calculations with small matrices. The original QLoRA paper explains the frozen four-bit base, NF4, double quantization and memory-management choices behind the method.

7. Count memory in separate buckets

I find a ledger more helpful than a single headline number. Consider an illustrative seven-billion-parameter base and a twenty-million-parameter adapter. Assume two-byte base weights and gradients where present, two four-byte Adam moment buffers per trainable parameter, and two-byte adapter weights and gradients. Ignore master-weight copies for this first calculation.

These are decimal gigabytes and deliberately incomplete totals. They exclude activations, quantization metadata, master copies, attention buffers, temporary dequantization storage, kernel workspace and allocator overhead. Actual precision policies may also change the counts. The QLoRA column is not a claim that this training job fits on a four-gigabyte GPU.

Sequence length and microbatch size can make activations dominate. Gradient checkpointing trades recomputation for saved activation memory. Gradient accumulation combines several microbatches before an optimizer step, but each microbatch still needs its own forward and backward execution. Measure peak allocated and reserved memory on the actual workload.

8. What the kernels have to do

At one adapted linear layer, the forward schedule contains a base multiplication, an input-to-rank multiplication, a rank-to-output multiplication, scaling and addition. In QLoRA the base path also needs the quantization reconstruction contract.

Base path:       X × W₀                 # or quantized-base kernel
Adapter path:    Z = X × A
                 U = Z × B
Combine:         Y = base + s × U

The intermediate Z is n × r. Whether it is stored, recomputed or consumed through a fused operation affects bandwidth and memory. Small-rank matrix multiplies have different shapes from the large base multiplication. Their efficiency depends on batch size, layout, kernel dispatch and scheduling—not just the reduction in trainable parameter count.

Backward needs the products shown earlier, plus any activation functions, attention derivatives and normalization derivatives surrounding the layer. Freezing the base can avoid its optimizer state and weight-gradient accumulation; it does not remove the whole network from the training graph.

This is the connection to my C-Kernel-Engine work: follow the tensors and their contracts before deciding where to fuse or optimize. For an adapter implementation, I would compare the forward output, input gradient, factor gradients and parameter updates against a reference, then confirm that frozen weights stayed unchanged. The equations here explain the required computation; they are not a claim that every CKE serving path already supports QLoRA training.

9. A practical way to start an experiment

Define the task before the rank

Write down an input, the desired output and how you will judge it. For structured output, measure schema validity as well as task correctness. For classification, inspect confusion between classes. For generated answers, review factual accuracy and failure cases alongside any automated metric.

Keep related examples and duplicated templates from leaking across training and evaluation. Hold out a set that represents how the model will actually be used. A loss curve alone cannot tell us whether an adapter improved the behaviour we care about.

Make one configuration inspectable

Record the base model revision, tokenizer, prompt/chat format, target modules, rank, scaling, dropout, sequence length, loss masking, optimizer, learning rate and precision. If the base is quantized, record its format and configuration too. The PEFT quantization guide shows the supported preparation flow and adapter configuration for quantized models.

For supervised chat fine-tuning, decide which tokens contribute to the loss. Training on the full conversation and training only on the assistant response are different objectives. Truncation can silently remove the answer you wanted to supervise. Inspect a tokenized and labelled sample before launching a long run.

Use small runs to find mechanical mistakes

1. Verify that the base model produces the expected baseline output.
2. Check that adding a zero-output adapter preserves that output within the chosen numerical tolerance.
3. Inspect the trainable parameter list and gradient presence.
4. Overfit a tiny, deliberately limited batch to test the training machinery.
5. Run a separate held-out evaluation to assess generalization.
6. Save, reload and compare the restored model on fixed inputs.

Those checks answer different questions. Overfitting a tiny batch demonstrates that parameters can learn that batch. It does not demonstrate a useful model. A checkpoint that cannot be reloaded reproducibly is also not a finished training result.

What should change first when results are poor?

Inspect the data, formatting, labels and evaluation before increasing rank. Then change one variable at a time: target modules, rank, learning rate or training duration. Preserve the baseline so improvements are attributable. If a model repeatedly produces the wrong output structure, first check whether the examples and loss actually teach the desired structure.

LoRA is also separate from the optimizer. AdamW can update the factor matrices; that does not make LoRA an AdamW variant. Our Muon matrix-update article explains a different choice: how gradients become parameter updates. Applying a different optimizer to low-rank factors needs its own evaluation; the effect on AB is not the same as optimizing a dense correction directly.

10. Keep the adapter separate, or merge it?

With a compatible dense base, the algebra suggests Wmerged = W₀ + sAB. In evaluation mode, with adapter dropout inactive and the correct scaling, this can remove the extra adapter branch from inference. The PEFT LoRA documentation describes adapter configuration and merging options.

Keeping adapters separate is useful when several tasks share a base. It avoids making a full dense model copy for each adapter, but serving must select the correct adapter and execute its contribution. That introduces scheduling, batching and memory-management choices.

For a quantized base, merging is more involved. Adding a dense correction and encoding the result again introduces another quantization step. Support varies by backend and format. The merged quantized output need not match the separate-branch output exactly. Validate both task quality and numerical differences after any conversion.

Package the adapter with enough information to resolve the correct base, tokenizer, target module names, shapes and scaling. “It is a rank-16 adapter” is not enough to determine what it belongs to.

11. Which approach would I choose?

I come back to three questions: what changes, what stays frozen, and what still has to compute? LoRA restricts the learned correction. QLoRA reduces the storage of the frozen foundation. The forward and backward paths explain the remaining work.

Once those pieces are visible, the acronyms become much less mysterious. We can inspect the shapes, reproduce a small result, trace the gradients and then decide whether the larger system behaves as intended.

Next: backpropagation through LoRA and QLoRA

In the next article, we will follow the backward pass step by step. Starting with a small numerical example, we will trace how the loss produces gradients for A and B, why the frozen base weights still participate in computing gradients for earlier layers, and how the optimizer updates the adapter.

Then we will repeat the walkthrough for QLoRA: what is stored in four bits, what is reconstructed for computation, and how gradients flow through the calculation without training the frozen quantized weights. We will use computational graphs and worked matrices so we can check each step.

Keep following the matrices

• Matrix Wx+b and the GEMM underneath
• Muon: from wx+b to matrix updates
• The contracts behind four-bit quantization
• From language-model heads to decision heads

If these walkthroughs are useful, subscribe to ShivasNotes. I write about the math, kernels and runtime decisions behind models, working from small examples toward systems we can inspect and test.

Primary references

• Houlsby et al., Parameter-Efficient Transfer Learning for NLP—the nonlinear bottleneck-adapter design, distinct from LoRA.
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models.
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs.
• Microsoft, reference LoRA implementation.
• Hugging Face, PEFT quantized-model training guide and LoRA configuration.
• bitsandbytes, four-bit linear layers.