Softmax and softplus sound like close relatives. They both contain exponentials, both are smooth, and both appear in neural networks. That resemblance can make the wrong intuition feel reasonable: perhaps softplus is simply a softer softmax, or perhaps either function could be used wherever a model needs a gate.

They answer different questions. Softmax asks how a fixed total should be divided among alternatives. Softplus asks how strongly one independent signal should be enabled while keeping that strength positive. The difference becomes concrete in Poolside's Laguna S 2.1: token positions still compete through attention softmax, while each attention head is subsequently multiplied by its own softplus gate.

1. Two Functions, Two Questions

Suppose a model produces three raw values:

\[ x=[1,2,3] \]

These values are often called logits: unrestricted scores that have not yet been converted into the form required by the next operation.

The words competition and independence are the central distinction. Softmax has one denominator shared by every output. Softplus has no cross-element denominator, so one output can grow without mathematically forcing another output to shrink.

2. Softplus Creates A Positive Strength

For one scalar \(x\), softplus is:

\[ s(x)=\log(1+e^x) \]

It is a smooth approximation to ReLU, but unlike ReLU it has no sharp corner at zero:

\[ s(x)\approx \begin{cases} e^x & x\ll 0\\ x & x\gg 0 \end{cases} \]

At zero:

\[ s(0)=\log 2\approx0.693 \]

A negative gate logit therefore approaches zero smoothly. A large positive gate logit behaves approximately linearly. Crucially, softplus is not bounded above. A gate can attenuate an attention head with a multiplier below one or amplify it with a multiplier above one.

3. Softmax Creates Competing Shares

For logits \(z_1,\ldots,z_n\), softmax is:

\[ p_i=\frac{e^{z_i}}{\sum_{k=1}^{n}e^{z_k}} \]

The denominator couples the vector. With \(z=[1,2,3]\):

\[ \operatorname{softmax}(z)\approx[0.090,0.245,0.665] \]

The values are positive and sum to one. Increasing \(z_3\) increases its share, but because the total remains one, it also reduces the relative shares of the first two entries.

In causal attention, one query produces a row of compatibility scores against allowed keys:

\[ S=\frac{QK^T}{\sqrt{d}} \]

After masking future or out-of-window positions, softmax turns each row into attention weights:

\[ P=\operatorname{softmax}(S),\qquad A=PV \]

Here competition is intentional. The query has one normalized distribution over the keys it can read.

4. Laguna Uses Both; Gemma 4 Is The Control Case

Laguna S 2.1 and Gemma 4 both alternate sliding-window and global attention and both use different RoPE configurations for those layer types. That architectural resemblance does not imply identical attention blocks.

Laguna declares an additional learned gate projection. To avoid overloading the symbol \(x\), let \(H\in\mathbb{R}^{B\times T\times D}\) denote the normalized hidden states entering attention, where \(B\) is batch size, \(T\) is token count and \(D\) is model width. The ordinary projections and the gate projection all read this same activation tensor:

\[ Q=HW_Q,\qquad K=HW_K,\qquad V=HW_V,\qquad Z_g=HW_g \]

The \(x\) in the generic formula \(\operatorname{softplus}(x)=\log(1+e^x)\) is therefore a gate logit from \(Z_g\), not an attention score from \(QK^T\). For Laguna S 2.1's per-head mode, \(W_g\in\mathbb{R}^{D\times H_q}\), so:

\[ Z_g\in\mathbb{R}^{B\times T\times H_q},\qquad G=\operatorname{softplus}(Z_g) \]

The values in \(W_g\) are trained model parameters. A different token hidden state produces different gate logits, so the model learns when each head should be weak or strong. Poolside's published implementation calculates softplus(g_proj(hidden_states)) in FP32, converts it to the attention-output dtype, broadcasts one gate across each head's channels, and applies it before o_proj. See the public modeling_laguna.py and configuration_laguna.py.

Full Attention First, Softplus Gate Second

For prefill, after Q/K/V projection and head splitting, each query head calculates:

\[ S_h=\frac{Q_hK_h^T}{\sqrt{d_h}}+M,\qquad P_h=\operatorname{softmax}(S_h),\qquad A_h=P_hV_h \]

With \(T\) prompt tokens, each score and probability matrix has shape \(T\times T\) per head. That matrix answers which previous token positions each query position should read. During decode, query length is one and key length is the cached history, so the score shape becomes \(1\times T_{past}\).

Only after the weighted value sum has produced \(A\in\mathbb{R}^{B\times T\times H_q\times d_h}\) does Laguna apply:

\[ A'_{b,t,h,d}=A_{b,t,h,d}\,G_{b,t,h} \]

The subscript \(h\) identifies a query head and \(d\) a channel inside that head. One scalar \(G_{b,t,h}\) is broadcast across all \(d_h\) channels. The gated heads are then concatenated and passed through the ordinary output projection:

\[ Y=\operatorname{Concat}(A'_1,\ldots,A'_{H_q})W_O \]

What Computational Delta Does Softplus Add?

The gate does not create another \(T\times T\) matrix, does not compare tokens and does not make full attention linear. Full prefill attention still has the score and value work that grows quadratically with context length. The Laguna gate adds:

• one projection \(HW_g\), producing \(B\times T\times H_q\) values for per-head gating;
• one elementwise softplus over those gate logits;
• one broadcast multiplication across the \(d_h\) channels of each attention-head output.

For Laguna S 2.1, the published configuration has \(D=3072\), \(H_q=48\), \(H_{kv}=8\) and \(d_h=128\). The per-head gate projection therefore emits 48 values per token, compared with 6,144 concatenated attention channels. The gate is meaningful model behavior, but its tensor is not the quadratic attention map.

The sequence is therefore:

attention scores
→ token-position softmax
→ weighted value sum
→ hidden-state gate projection
→ per-head softplus gate
→ gated attention output
→ output projection

Gemma 4 provides the useful contrast. Its text decoder also has hybrid local/global attention and dual RoPE, but its published configuration and Transformers implementation do not define Laguna's per-head softplus attention gate. Gemma 4 has other distinctive contracts, including layer-dependent attention geometry, Q/K normalization, global KV behavior, multimodal paths and optional MoE.

Softplus is not required by hybrid attention. It is a separate Laguna design choice layered on top of attention.

Which Model Families Use Softplus?

Softplus appears in several modern families, but often for a completely different variable. Saying that two models “use softplus” does not establish architectural equivalence.

This table is representative rather than an exhaustive census of every neural architecture. The crucial distinction is the operation's contract. Laguna uses softplus to scale an already-computed attention output. Qwen3.5, Mamba-family and Nemotron-H recurrent layers use softplus to convert unconstrained values into valid timing or decay parameters.

This is why a runtime should not identify a model only by broad labels such as “sliding attention,” “softplus” or “Gemma-like.” Shape, source tensor, axis, placement and downstream consumer are all part of the operation.

5. Softplus Backward Pass

Start with:

\[ s(x)=\log(1+e^x) \]

Differentiate:

\[ \frac{ds}{dx} =\frac{1}{1+e^x}e^x =\frac{e^x}{1+e^x} =\frac{1}{1+e^{-x}} =\sigma(x) \]

The derivative of softplus is sigmoid. If the gradient arriving from later computation is:

\[ \bar{s}=\frac{\partial L}{\partial s} \]

then reverse-mode differentiation gives:

\[ \bar{x}=\frac{\partial L}{\partial x}=\bar{s}\,\sigma(x) \]

This backward pass is elementwise. No neighboring element is required. For very negative \(x\), the derivative approaches zero; for very positive \(x\), it approaches one.

The second derivative is:

\[ \frac{d^2s}{dx^2}=\sigma(x)(1-\sigma(x)) \]

It is positive, so softplus is convex. The curvature is largest around zero and fades in both tails.

6. Softmax Backward Pass Requires A Jacobian

Softmax output \(p_i\) depends on every input logit \(z_j\). We therefore calculate:

\[ \frac{\partial p_i}{\partial z_j} =p_i(\delta_{ij}-p_j) \]

There are two cases:

\[ \frac{\partial p_i}{\partial z_i}=p_i(1-p_i) \]

and, when \(i\ne j\):

\[ \frac{\partial p_i}{\partial z_j}=-p_ip_j \]

The negative off-diagonal derivative is the calculus form of competition: increasing logit \(z_j\) decreases every other probability \(p_i\).

The complete Jacobian is:

\[ J=\operatorname{diag}(p)-pp^T \]

A runtime should not construct this \(n\times n\) matrix. Given upstream gradient \(u_i=\partial L/\partial p_i\), multiply by the Jacobian algebraically:

\[ \frac{\partial L}{\partial z_j} =p_j\left(u_j-\sum_i u_ip_i\right) \]

In vector notation:

\[ \bar{z}=p\odot\left(u-\langle u,p\rangle\right) \]

The backward kernel needs one dot-product reduction \(\langle u,p\rangle\), followed by an elementwise update. Softplus backward needs only the elementwise update. Their computational structures are therefore as different as their semantics.

7. Why Softmax Plus Cross-Entropy Simplifies

For a target distribution \(y\), cross-entropy is:

\[ L=-\sum_i y_i\log p_i \]

When \(p=\operatorname{softmax}(z)\), the combined derivative simplifies to:

\[ \frac{\partial L}{\partial z_i}=p_i-y_i \]

This compact result does not mean softmax has an elementwise derivative. The cancellation occurs because cross-entropy and softmax are differentiated together. If a different loss or upstream computation consumes the probabilities, the general Jacobian-vector product is still required.

The relationship to information theory is developed in Where Does LLM Cross-Entropy Loss Come From?.

8. Backpropagating Through Laguna's Gated Attention

Now combine both ideas. Let \(H\) be the normalized hidden state entering both branches:

\[ A=\operatorname{Attention}(H),\qquad Z_g=HW_g,\qquad G=\operatorname{softplus}(Z_g) \]

For a per-head gate:

\[ Y_{h,d}=A_{h,d}G_h \]

Let the upstream gradient be:

\[ U_{h,d}=\frac{\partial L}{\partial Y_{h,d}} \]

Gradient Into Attention Output

Because \(Y_{h,d}=A_{h,d}G_h\):

\[ \frac{\partial L}{\partial A_{h,d}}=U_{h,d}G_h \]

The learned gate therefore scales not only the forward attention signal but also the gradient returning into the attention computation.

Gradient Into The Broadcast Gate

One gate affects every channel in its head. Those contributions must be accumulated:

\[ \frac{\partial L}{\partial G_h} =\sum_d U_{h,d}A_{h,d} \]

This reduction is easy to miss. Treating \(g_h\) as though it had an independent value for every channel would implement a different architecture and produce a differently shaped gate projection.

Through Softplus

Since \(G_h=\operatorname{softplus}(Z_{g,h})\):

\[ \frac{\partial L}{\partial Z_{g,h}} =\frac{\partial L}{\partial G_h}\sigma(Z_{g,h}) \]

Through The Gate Projection

For \(Z_g=HW_g\), using row-major activation notation:

\[ \frac{\partial L}{\partial W_g}=H^T\bar{Z}_g \]

and:

\[ \left.\frac{\partial L}{\partial H}\right|_{gate}=\bar{Z}_gW_g^T \]

The hidden state also receives a gradient through \(A=\operatorname{Attention}(H)\). The final hidden-state gradient is the sum of both branches:

\[ \frac{\partial L}{\partial H} =\left.\frac{\partial L}{\partial H}\right|_{attention} +\left.\frac{\partial L}{\partial H}\right|_{gate} \]

Inside the attention branch, gradients continue through the output projection, the value-weighted sum, token-position softmax, score scaling, and Q/K/V projections. Softmax and softplus therefore participate in one backward graph but solve different local derivative problems.

9. Stable CPU Implementations

Stable Softplus

The direct expression log1p(exp(x)) can overflow when \(x\) is large. A stable identity is:

\[ \operatorname{softplus}(x)=\max(x,0)+\log(1+e^{-|x|}) \]

static inline float softplusf_stable(float x) {
    return fmaxf(x, 0.0f) + log1pf(expf(-fabsf(x)));
}

static inline float sigmoidf_stable(float x) {
    if (x >= 0.0f) {
        float e = expf(-x);
        return 1.0f / (1.0f + e);
    }
    float e = expf(x);
    return e / (1.0f + e);
}

void softplus_backward(const float *x,
                       const float *grad_out,
                       float *grad_x,
                       size_t n) {
    for (size_t i = 0; i < n; ++i) {
        grad_x[i] = grad_out[i] * sigmoidf_stable(x[i]);
    }
}

Production implementations may use range-based approximations and vectorized exponential/logarithm providers. The numerical contract must specify acceptable error rather than silently replacing the scalar function.

Stable Softmax

Softmax uses the maximum-shift identity:

\[ p_i=\frac{e^{z_i-m}}{\sum_k e^{z_k-m}},\qquad m=\max_kz_k \]

void softmax_forward(const float *z, float *p, size_t n) {
    float max_z = -INFINITY;
    for (size_t i = 0; i < n; ++i)
        max_z = fmaxf(max_z, z[i]);

    float sum = 0.0f;
    for (size_t i = 0; i < n; ++i) {
        p[i] = expf(z[i] - max_z);
        sum += p[i];
    }

    float inv_sum = 1.0f / sum;
    for (size_t i = 0; i < n; ++i)
        p[i] *= inv_sum;
}

void softmax_backward(const float *p,
                      const float *grad_out,
                      float *grad_z,
                      size_t n) {
    float dot = 0.0f;
    for (size_t i = 0; i < n; ++i)
        dot += grad_out[i] * p[i];

    for (size_t i = 0; i < n; ++i)
        grad_z[i] = p[i] * (grad_out[i] - dot);
}

For BF16 or FP16 model storage, reductions are commonly accumulated in FP32. Parallel implementations must define reduction order, masking behavior, all-masked-row behavior and whether fused multiply-add contraction is part of the numerical reference.

Potential Laguna Fusion

Laguna's gate path can initially use ordinary primitives:

GEMM/GEMV gate projection
→ softplus
→ broadcast multiply

After correctness is established, a fused kernel could combine softplus and the broadcast multiplication. The projection itself may also be fused into a wider projection schedule, but that changes packing, threading and memory-planning requirements. Profiling should decide whether fusion is worthwhile.

10. What C-Kernel-Engine Must Represent

C-Kernel-Engine (CKE) is a Linux CPU inference and training runtime that represents model families as explicit mathematical circuits connected to concrete C kernels. Laguna demonstrates why that separation matters.

The low-level kernel registry may already contain enough primitives:

• GEMM and GEMV for gate projection;
• softmax forward and backward;
• softplus forward and sigmoid-based backward;
• broadcast multiplication and reduction;
• grouped MoE GEMM and routing primitives.

But the Laguna circuit must still state:

• whether gating is per-head or per-element;
• the gate-projection output shape for each layer;
• where the gate is applied relative to attention concatenation and output projection;
• which axis receives the backward reduction;
• the dtype used for the gate and reduction;
• whether the forward intermediates or recomputed values feed backward;
• the exact tolerances against the reference implementation.

This is not necessarily a brand-new mathematical kernel. It is a new circuit contract assembled from familiar operations. CKE should first prove the compositional path, then fuse only the measured bottleneck.

11. Practical Takeaway

Softmax and softplus belong to the same broad exponential family, but they do different engineering work:

• Softmax: make alternatives compete for one normalized total.
• Softplus: turn one unrestricted scalar into an independent positive strength; in Laguna that scalar is produced by the learned projection \(Z_g=HW_g\).
• Softmax backward: account for coupling through a reduction or Jacobian-vector product.
• Softplus backward: multiply elementwise by sigmoid.
• Laguna: use softmax across token positions, calculate the weighted value sum, then use softplus to gate the resulting attention heads before \(W_O\).
• Gemma 4: share hybrid attention and dual-RoPE ideas without Laguna's published per-head softplus gate.

The deeper lesson is not to identify a model by one fashionable architectural label. Two models may both advertise sliding attention, global attention and MoE while differing at exactly the operation that controls signal flow and gradient flow. Kernel engineering begins by making those contracts explicit.

References And Further Study

• Poolside Laguna S 2.1 model card and configuration
• Poolside Laguna implementation: attention softmax and softplus output gate
• Poolside: Introducing Laguna S 2.1
• Google Gemma 4 26B-A4B model card
• Hugging Face Transformers Gemma 4 implementation
• Hugging Face Transformers Qwen3.5 DeltaNet implementation
• Hugging Face Transformers Mamba2 implementation
• Hugging Face Transformers Nemotron-H implementation
• Attention: The Core Of The Transformer
• Gemma4 In CKE: Four Attention Paths, Shared KV, And Sliding Windows
• Activation Functions: The Nonlinearity Inside Neural Networks