| softmax | \(p_i=e^{z_i}/\sum_j e^{z_j}\) | tensor.go Softmax |
| log-sum-exp | \(\operatorname{LSE}(z)=M+\ln\sum_j e^{z_j-M}\) | sample.go shiftAndLogSum |
| logprob | \(\log p_i=z_i-\operatorname{LSE}(z)\) | sample.go LogProbAt |
| 熵 | \(H(p)=-\sum_i p_i\ln p_i\) | 概念 |
| 交叉熵 | \(H(p,q)=-\sum_i p_i\ln q_i\) | 训练损失 |
| KL | \(D(p\|q)=\sum_i p_i\ln(p_i/q_i)\) | 概念 |
| 困惑度 | \(\operatorname{PPL}=\exp(\frac1N\sum_t -\ln q_t)\) | 概念 |
| RMSNorm | \(y_i=x_i\gamma_i/\sqrt{\frac1n\sum x_j^2+\epsilon}\) | norm.go RMSNorm |
| SiLU | \(x\sigma(x)=x/(1+e^{-x})\) | tensor.go SiLU |
| SwiGLU | \(W_d(\operatorname{SiLU}(W_gx)\odot W_ux)\) | mlp.go MLP |
| 缩放点积 | \(\operatorname{softmax}(QK^\top/\sqrt d+M)V\) | attn.go Attention |
| RoPE | \(q'_m=R(m\omega)q,\ \langle q'_m,k'_n\rangle=f(n-m)\) | rope.go ApplyRoPE |
| GQA 映射 | \(g(h)=\lfloor h/(H/H_{kv})\rfloor\) | config.go KVGroupSize |
| KV 内存 | \(2LH_{kv}d\cdot\text{bytes}\) | kv.go BytesPerToken |
| 分页地址 | \(\text{phys}=\text{table}[\ell][\lfloor p/B\rfloor],\ \text{slot}=p\bmod B\) | pagedkv.go kSlot |
| online softmax | \(m{=}\max(m,m_b),\ o{=}oe^{m-m'}{+}o_be^{m_b-m'}\) | pagedattn.go attentionPaged |
| 温度 | \(p_i\propto e^{z_i/T}\) | sample.go Sample |
| top-p | 最小 \(m\) 使 \(\sum_{j\le m}p_{(j)}\ge p\) | sample.go topPFloat64 |
| 逆变换 | \(X=\min\{i:u<C_i\}\) | sample.go sampleOneFloat64 |