Neural API Examples

Start here — smallest demos

• Only two layers (XOR) — trains a neural network containing only two layers to learn the XOR boolean operation, about the smallest non-linearly-separable demo in the tree. Pure CPU, instant.
• Sine regression — smallest possible "does the library still train?" demo: a two-layer MLP fits y = sin(pi*x) on x in [-1, 1] with hand-rolled mini-batch SGD; prints periodic MSE and an 11-point predicted-vs-truth table.
• Binary adder — tiny "learns to add two binary numbers" demo: a small MLP learns 4-bit + 4-bit addition over all 256 input combinations (5-bit sigmoid output for carry), trains in a few seconds, reaches 100% exact-bit accuracy, and prints sample additions in binary.
• Self-test — runs the library's built-in correctness self-tests: TestTNNetVolume() (the volume/tensor API), TestKMeans(), and TestConvolutionAPI(). A quick "is my build of the library sane?" smoke check.
• Delphi template — minimal Delphi FMX GUI project (NeuralInDelphi) showing the library compiling and running under Delphi/FireMonkey rather than only FPC/Lazarus — a starting skeleton for embedding neural-api in a Delphi desktop app.
• Model-summary demo (PrintSummary smoke test) — constructs three structurally-distinct nets (a small MLP, a CIFAR-style conv stack, and a pre-norm residual block via TNNet.AddPreNormResidual) and prints each via TNNet.PrintSummary / SummaryString (the Keras-style Idx / Layer / Output Shape / Params / Neurons table with a Totals: footer). It doubles as a format smoke test: it parses each summary string and asserts the header is present, the body has exactly CountLayers() rows, the per-row Params/Neurons sum to CountWeights()/CountNeurons(), and the footer totals match — Halt(1) on any mismatch. No training. Pure CPU, deterministic, < 1 s.
• Euclidean-norm-reciprocal head (layer composition) — a teaching demo that the in-tree elementwise transcendental layers compose into a Euclidean-norm head: Input(1,1,8) -> TNNetSquare -> TNNetFullConnectLinear(1)[all-ones, frozen sum] -> TNNetSqrt -> TNNetReciprocal computes 1/||x||_2 exactly (an all-ones frozen FC is used for the sum_i x_i^2 reduction so there is no N^2 pool-scaling to undo). Self-checks (Halt(1) on failure) over 200 random vectors: forward matches the analytic 1/sqrt(sum x_i^2) to 0 float32 error, a unit vector returns 1.0, the x*(1/||x||) L2-normalize extension has norm 1 within 1.2e-7, and gradients flow through the stack without NaN. The README contrasts the composition with the dedicated TNNetL2Normalize (which ships the exact Jacobian and an eps guard). Pure CPU, deterministic, < 1 s.
• Involution demo — shows that TNNetReverseChannels, TNNetReverseXY, TNNetFlipX, and TNNetFlipY are involutions: applying each twice round-trips to the input exactly (per-element).
• Seeded reproducibility — trains a tiny MLP twice with the same RandSeed and MaxThreadNum := 1, snapshots every weight and bias, and prints PASS/FAIL on bit-for-bit equality (non-zero exit code on FAIL — CI-friendly).
• Quick-Start Sequence — "hello world" for sequence learning — the shortest possible "it learns a sequence" demo: a tiny 4-layer char-level next-token model (Input(2*10) -> FullConnectReLU(16) -> FullConnectLinear(10) -> SoftMax, a few hundred weights) learns a fixed counting sequence 0,1,2,...,9,0,1,... from its last two symbols, trained with a plain hand-written SGD loop (NN.Compute + NN.Backpropagate, no TNeuralFit, no threads, no data files). Seeded with 01 it regenerates three full cycles by feeding its own predictions back in (100% next-symbol accuracy). Pure CPU, ~1 s — a beginner-friendly counterpart to the full transformer in SimpleNLP.

Activations & gated feed-forwards

• PReLU vs LeakyReLU vs ReLU — three-config activation comparison on the toy hypotenuse regression task: TNNetReLU (no negative leak), TNNetLeakyReLU (fixed slope 0.01), and TNNetPReLU (single learnable scalar slope shared across all elements, He et al. 2015, init alpha = 0.25). Pure CPU.
• Mainstream activation bake-off on the hypotenuse toy — companion to the hyperbolic bake-off comparing 15 ReLU-family activations (TNNetReLU, TNNetLeakyReLU, TNNetVeryLeakyReLU, TNNetReLU6, TNNetPReLU, TNNetELU, TNNetSELU, TNNetCELU, TNNetSwish, TNNetSiLU, TNNetGELU, TNNetHardSwish, TNNetMish, TNNetSoftPlus, TNNetAconC) on the same tiny MLP / RNG seed and prints a CSV of final MSE and epochs-to-converge.
• Hyperbolic activation bake-off on the hypotenuse toy — compares 8 hyperbolic-family activations (TNNetHyperbolicTangent, TNNetLeCunTanh, TNNetSinhAct, TNNetArcSinh, TNNetLisht, TNNetBentIdentity, TNNetTanhExp, TNNetLogCoshActivation) on the same tiny MLP / RNG seed and prints a CSV of final MSE and epochs-to-converge.
• APL activation bake-off — does the extra piecewise capacity of the Adaptive Piecewise Linear activation TNNetAPL (Agostinelli et al. 2015) buy a lower final loss than plain TNNetReLU and TNNetPReLU? Trains all three on the same hypotenuse toy at matched seed/data and sweeps the number of per-channel learnable hinges S to chart capacity vs final MSE. Pure CPU.
• SwiGLU feed-forward — tiny Dense(2*D) → TNNetSwiGLU → Dense(1) block trained on a synthetic regression task; demonstrates the gated-FFN half of a transformer encoder using existing layers only.
• GLU feed-forward — the same gated-FFN block built with NN.AddGLUFeedForward(D_in, D_hidden, D_out), using the plain sigmoid-gated TNNetGLU (GLU(a, b) = a * sigmoid(b)); sibling of the SwiGLU and GEGLU feed-forward builders.
• GEGLU feed-forward — smallest possible end-to-end demo of the transformer feed-forward block: a Dense -> TNNetGEGLU -> Dense sandwich trained on a synthetic regression target (no attention, no embedding, no positional encoding), the canonical "GEGLU FFN" (Shazeer 2020) shrunk to CPU-friendly size so the gated activation is visible in isolation. Sibling of the GLU/SwiGLU feed-forward demos.
• Gated-FFN bake-off — head-to-head comparison of all five in-tree gated feed-forward layers (TNNetGLU / TNNetReGLU / TNNetGEGLU / TNNetSwiGLU / TNNetTanhGLU) on ONE synthetic per-position sequence-regression task. The same FFN block (PointwiseConvLinear(2*d_ff) -> GATE -> PointwiseConvLinear(1)) is built five times, identical except the gate — the gates are parameter-free so all arms have identical parameter counts — and trained at matched seed/LR/epochs/data. Prints a comparison table (initial vs final MSE, wall-clock seconds, epochs-to-converge) plus two built-in sanity checks (no NaN/Inf, every arm beats its pre-training baseline). The README notes the ranking is seed-dependent — the value is the apples-to-apples harness, not a universal winner. Pure CPU, ~4 s.
• KAN-vs-MLP toy fit — puts the headline KAN claim (Liu et al. 2024) on a wiggly 1D regression y = sin(3x) + 0.3*sin(11x): the same tiny MLP with its ReLUs swapped for the per-channel learnable TNNetSplineActivation, at a parameter count matched against a wider ReLU arm (TNNet.CountWeights: 21 vs 20). The spline arm reaches ~67% lower MSE and the example dumps the learned per-channel control points + sampled activation, showing one channel bent away from the identity that an untrained spline starts at (a sparse KAN fit). Pure CPU, a few seconds.
• KAN dense layer vs ReLU-MLP toy fit — the WEIGHT-space sibling of SplineActivationKAN: uses the true Kolmogorov-Arnold dense layer TNNetKANLayer (Liu et al. 2024), where every input→output edge carries its own learned Chebyshev univariate function phi_{ij}(x)=sum_k c_{ijk} T_k(tanh(x)) and y_j=sum_i phi_{ij}(x_i) (no weight matrix). On the same wiggly 1D fit y = sin(3x) + 0.3*sin(11x), a KANLayer(8,K=4) arm is param-matched against a wider ReLU arm (TNNet.CountWeights: an exact 48 vs 48) and reaches ~27% lower MSE. Pure CPU, under a second.
• KAN convolution per-edge nonlinear bake-off — the CONVOLUTIONAL sibling of KANLayer: the new TNNetKANConv replaces a convolution's linear dot product over each receptive-field patch with a sum of learned univariate Chebyshev edge functions (one per kernel-position × input-channel), y_filter(x,y)=sum_p phi_p(input_p), phi_p(t)=sum_k c_{p,k} T_k(tanh(t)), with NO output bias (the c_0 term plays that role) — reusing the exact Chebyshev forward/backward of TNNetKANLayer over a sliding window with the usual padding/stride plumbing inherited from TNNetConvolutionLinear. On a synthetic $8 \times 8$ 1-channel task whose target pixel is a per-3×3-window NONLINEAR map (y=sum_p a_p·f_p(neighbour_p), each f_p a distinct sin/tanh/square/cube/abs), a TNNetKANConv(1,3,1,1,K=4) arm (45 weights) reaches val-MSE ≈ 0.0006 while a comparable linear-conv baseline 4→1 (40 weights) plateaus around 0.05 — the per-edge learnable nonlinearity directly fits what a linear conv can only linearise. Gradient-checked (input + every coefficient) and serialization-round-tripped. Pure CPU, well under a minute.
• Spline knot-count sweep — a capacity study follow-up to SplineActivationKAN: the per-channel learnable TNNetSplineActivation (KAN-style piecewise-linear activation, Liu et al. 2024) can fit a wiggly 1D target, so this asks the obvious next question — how many knots do you need, and what does extra capacity (more knots / wider range) actually buy? Pure CPU.
• SIREN (periodic-activation 1D fit) — the headline Sitzmann et al. 2020 SIREN result from existing layers only: the SAME small coordinate-MLP (Input(1) -> [FullConnectLinear(24) -> act]x3 -> FullConnectLinear(1)) fits a high-frequency target y = sin(3x) + 0.3*sin(11x) once with TNNetSin activations and once with a TNNetHyperbolicTangent baseline at matched width/depth/seed/epochs. The SIREN-specific init is reproduced by hand via TNNetLayer.InitUniform (first layer folds the omega_0=12 frequency into the weights, later sine-feeding layers use sqrt(6/fan_in)/omega_0). The sine arm reaches ~100% lower MSE (0.000016 vs 0.053), demonstrating that periodic activations escape the spectral bias that limits the tanh/ReLU MLP. Self-gating (Halt(1) on failure), pure CPU, deterministic, ~12 s.
• Gumbel-softmax demo — demonstrates TNNetGumbelSoftmax, a differentiable categorical sampling head computing y = softmax((logits + g)/tau) with g ~ Gumbel(0,1). Part (a) sweeps the temperature tau ∈ {2.0, 1.0, 0.5, 0.1} on the deterministic (no-noise) inference path and prints the resulting softmax distribution plus its Shannon entropy (as tau shrinks the distribution sharpens toward one-hot, entropy falls); part (b) shows hard straight-through mode emitting a one-hot output. Pure CPU.
• Gumbel-Softmax temperature annealing autoencoder — a temperature-annealing micro-experiment for the TNNetGumbelSoftmax bottleneck (distinct from GumbelSoftmaxDemo, which only sweeps tau on a fixed logit vector at inference). Here a tiny discrete-latent autoencoder (encoder MLP → K=6 logits → TNNetGumbelSoftmax → decoder MLP) is TRAINED end-to-end on a genuinely K-category-structured synthetic set (6 well-separated cluster prototypes + noise) while tau is ANNEALED 2.0→1.0→0.5→0.25→0.1 across phases. The net is built ONCE and tau is lowered IN-PLACE each phase via the public TNNetGumbelSoftmax.SetTemperature(tau) setter (no more rebuild + TNNet.CopyWeights per phase). Headline: a (tau, recon-MSE, mean bottleneck ENTROPY) table with a graded PASS/FAIL verdict — the categorical sharpens as tau drops (entropy collapses 0.064→0 nats) while reconstruction holds at the ~0.01 noise floor. Honest caveat (in output + README): on this well-separated dataset confident routing means even the highest-tau entropy starts modest, far below ln(K)=1.79; the verdict checks the trend, not an unrealistically uniform start. Pure CPU, ~2 s.

Losses & output heads

• LogCosh dual experiment — pairs TNNetLogCoshActivation (hidden) with TNNetLogCoshLoss (output head) vs. a plain MSE head on the hypotenuse toy task; prints a CSV comparing final validation MSE and epochs-to-converge.
• Loss-family bake-off (output heads) — head-to-head of robust regression loss heads (TNNetHuberLoss, TNNetSmoothL1Loss, TNNetCharbonnierLoss, TNNetLogCoshLoss) vs. a plain MSE baseline on a noisy hypotenuse task with injected outliers, at matched trunk/seed/LR/epochs; prints clean-test MSE/MAE per arm and shows the robust heads beating MSE (which over-weights the squared outlier residuals).
• Energy heads (L1 vs L2) — side-by-side comparison of two final-feature "energy" heads on a tiny regression task: TNNetAbs (elementwise |x|, an L1-energy head) vs TNNetSquare (elementwise x^2, an L2-energy head). Both nets share an identical body and training data, so the final test-MSE gap isolates the effect of the energy layer. Pure CPU.
• Quantile (pinball) regression + joint multi-quantile head — fits prediction intervals on a heteroscedastic 1-D dataset (y = sin(x) + 0.3x + N(0, sigma(x)), noise growing with x) with the pinball-loss head TNNetQuantileLoss. Arm 1 trains three independent MLPs (one per q in {0.1, 0.5, 0.9}); arm 2 trains ONE model whose 3-wide output predicts all three quantiles jointly in a single forward pass via the new TNNetMultiQuantileLoss (the same scalar target replicated across channels). Both reach ~80% held-out [q=0.1, q=0.9] band coverage, and the example exercises the non-differentiable inference-time monotonicity guard TNNetMultiQuantileLoss.SortAscending (sorts each N-channel group so q=0.1 never crosses q=0.9), reporting crossing counts with vs without the guard. Pure CPU, ~40 s.
• Tversky α/β Sweep - Asymmetry knob study on the TNNetTverskyLoss segmentation head (Salehi et al. 2017): on a deliberately class-imbalanced mask, sweeps (α,β) ∈ {(0.5,0.5),(0.3,0.7),(0.7,0.3)} and prints a precision/recall/FN table showing how β>α (penalising false negatives harder) trades precision for recall — recall rises and FN falls monotonically. Pure CPU, ~1s.
• Mixture density network — a self-contained pure-CPU reproduction of Bishop's Mixture Density Network (Bishop 1994) on the classic inverse-problem toy, contrasting a plain MSE regressor (which provably collapses to the conditional mean) against an MDN head that recovers the full MULTIMODAL conditional p(y|x). No new layer — uses existing layers plus hand-rolled gradient surgery for the mixture loss.
• Mixture density network (multi-valued inverse map) — the headline "predict a distribution, not a point" win for the new TNNetMixtureDensity probabilistic regression head (Bishop 1994, Mixture Density Networks). Data comes from the non-monotonic forward map x = y + 0.3·sin(2π·y) + noise, so the inverse y|x is multi-valued — for many x there are several valid y branches. Two models share a x → 32 → 32 trunk: an MDN head TNNetFullConnectLinear(1,1,K·(1+2·D)) → TNNetMixtureDensity(K=5, D=1) that maps the trunk output to a K-component diagonal-Gaussian mixture (π=softmax mixing weights, μ=raw means, σ=softplus scales), owns the negative-log-likelihood loss (its Backpropagate emits the exact responsibility-weighted dNLL/dparam via a numerically-stable log-sum-exp), and exposes a SampleMixture inference helper; versus a plain MSE TNNetFullConnectLinear(1) baseline. The MDN target is read from the first D channels of the seeded target volume, and the batch-update deltas are scaled to the mean gradient so no component collapses; the head's mean biases are initialised spread across the target range to break symmetry (the classic MDN mode-collapse pitfall). On the folded region the MDN spreads its active components across the true branches — branch-coverage error ≈0.11 vs ≈0.20 for the single MSE point, which can only sit near one branch and otherwise lands in the gap-filling conditional mean — and drawn samples scatter onto the separate branches. Analytic backward finite-difference verified (TestMixtureDensityGradient). Pure CPU, ~25 s.
• Deep evidential regression (closed-form epistemic uncertainty) — the headline "uncertainty from one deterministic pass" win for the new TNNetEvidentialRegression head (Amini et al., NeurIPS 2020, Deep Evidential Regression). Instead of a point estimate, the head emits the 4 parameters of a Normal-Inverse-Gamma prior per scalar target — gamma (mean, linear), nu (>0), alpha (>1), beta (>0) via softplus links over 4·D raw channels — and owns the NIG negative-log-likelihood + evidence-regularizer loss (closed-form Student-t marginal using a local Lanczos lnGamma/digamma; its Backpropagate emits the exact dL/d{gamma,nu,alpha,beta} chained through the softplus). From one forward pass it reads off, in closed form, prediction=gamma, aleatoric var=beta/(alpha-1) and epistemic var=beta/(nu·(alpha-1)) — distinct from TNNetMixtureDensity (aleatoric-only mixture) and TNNetKalmanFilterCell (recursive sequential covariance), this is the only head giving a closed-form epistemic estimate with no sampling and no ensemble. The demo learns f(x)=sin(1.5x)+0.2x from a central band [-3,3] with the outer tails held out; a deliberately saturating tanh trunk makes far-OOD inputs fall back to the head's low-evidence (high-epistemic) bias prior rather than linearly extrapolating the NIG params, and the nu bias starts low so evidence only grows where data is seen. Headline: mean epistemic variance is ~2× larger in the held-out tails than in-distribution (deepest in the far tail, ~5× the band centre). Analytic backward finite-difference verified (TestEvidentialRegressionGradient). Pure CPU, ~1 min, ~4 MB.
• Evidential (Dirichlet) classification (uncertainty out of distribution) — the classification sibling of the above for the new TNNetEvidentialClassification head (Sensoy et al., NeurIPS 2018, Evidential Deep Learning to Quantify Classification Uncertainty). Treats the previous layer's K raw outputs as evidence for a Dirichlet over the class simplex (alpha_k = 1 + softplus(raw_k)) and reads off, from one deterministic pass, p_k = alpha_k/S and an uncertainty mass u = K/S in [0,1] (u→1 = the network abstains). It owns the EDL loss — the Bayes-risk expected-MSE (Eq. 5) plus a lambda·KL-to-uniform regularizer on the misleading-evidence Dirichlet (local Lanczos lnGamma/digamma/trigamma; its Backpropagate emits the exact dL/d(raw) chained through softplus) — and exposes Alpha/Prediction/Uncertainty. The demo trains on two side-by-side upper-half-plane blobs (a geometry that makes the whole lower half-plane provably out-of-distribution, unlike antipodal blobs) with a saturating tanh trunk; headline: mean uncertainty ~5–6× larger out of distribution (u reaches exactly 1.0 below the blobs) than in-distribution (u≈0.15). Analytic backward finite-difference verified (TestEvidentialClassificationGradient). Pure CPU, <1 min.

Dense prediction & detection

• Dice-loss segmentation — a tiny synthetic binary-mask segmentation task (predict the filled disc/box mask from a noisy 16x16 grid) trained two ways at identical fully-convolutional architecture: a Sigmoid -> TNNetDiceLoss head vs an MSE-head baseline. After training it reports mean Dice and IoU (threshold 0.5) on a held-out set and renders one ASCII (input, ground-truth, Dice-prediction, MSE-prediction) sample — the Dice head reaches markedly higher IoU on the class-imbalanced foreground. Pure CPU, finishes in a few seconds.
• U-Net segmentation — trains a real symmetric U-Net built with the reusable TNNet.AddUNet builder (DiceSegmentation is a FLAT fully-convolutional net with no down/upsampling) on a self-contained synthetic-shapes task and reports Dice/IoU on a held-out split. AddUNet(Depth, BaseFeatures, OutputChannels, out EncoderTaps, UseNorm) constructs Depth encoder stages (each 2x [Conv3x3 -> (Norm) -> ReLU] then a 2x2 stride-2 TNNetMaxPool, doubling features and recording a skip tap per stage), a bottleneck, Depth decoder stages (nearest x2 upsample via TNNetDeMaxPool -> TNNetDeepConcat with the matching encoder tap -> 2x conv, halving features), and a 1x1 conv head — and returns the encoder-tap layer indices in EncoderTaps so callers can inspect/re-wire the skips. Output spatial size equals input size (the defining U-Net property), so the 1-channel logit map feeds a Sigmoid + TNNetDiceLoss head (the same Dice loss as DiceSegmentation). Input SizeX/SizeY must each be divisible by 2^Depth. Reports per-eval Dice/IoU (threshold 0.5), renders one ASCII (input/ground-truth/prediction) sample and writes a dependency-free PPM strip. Pure CPU; the default SMOKE run (depth 3, ~1 min) reaches Dice ≈ 0.97 / IoU ≈ 0.95, with a --full mode for longer training.
• SegFormer semantic segmentation — the demo for BuildSegformerFromSafeTensors (neuralpretrained.pas), a semantic-segmentation importer (model_type segformer: nvidia/segformer-b0-finetuned-ade-512-512 and the MiT-b0…b5 family). SegFormer (Xie et al. 2021) is a hierarchical Mix-Transformer (MiT) encoder — overlap-patch embeddings (strided convs, NOT a CLS token), spatial-reduction efficient attention (the rectangular SeqLen × (SeqLen/sr²) score matrix that makes attention affordable at image resolution), a Mix-FFN with a depthwise positional conv (so the model needs NO positional embedding) — feeding a lightweight all-MLP decode head: a per-stage $1 \times 1$ Linear, bilinear upsample of every stage back to input/4 via the new TNNetBilinearUpsample, a reversed channel concat, a fuse conv with folded BatchNorm, and a $1 \times 1$ classifier that emits a per-pixel logit map at input/4 resolution. The demo loads the committed pico fixture (tests/fixtures/tiny_segformer.safetensors, a tiny MiT-b0-shaped net), runs it on a synthetic image, takes the per-pixel argmax and renders the label map as a colored ASCII palette; a real run loads the nvidia checkpoint the same way and colorizes a photo. Parity is asserted to max |diff| < 1e-4 vs the real transformers float64 forward over the whole logit map (TestSegformerSemanticSegmentationParity, fixture generator tools/make_pico_segformer_fixture.py). Pure CPU, <1 s on the fixture.
• DPT / Depth-Anything monocular depth estimation — the demo for BuildDPTFromSafeTensors (neuralpretrained.pas), a monocular-depth importer producing a dense per-pixel regression (a continuous depth value per pixel, not a class label; model_type depth_anything / dpt: depth-anything/Depth-Anything-V2-Small-hf and the DPT-hybrid family). Depth Anything (Yang et al. 2024) and DPT (Ranftl et al. 2021) pair a plain ViT / DINOv2 transformer backbone (REUSED — the encoder is built from the existing BuildDINOv2FromSafeTensors path, tapped at four intermediate stages with a SHARED final LayerNorm) with a convolutional reassemble + fusion neck and a small depth head. The new code is the decoder neck: a reassemble stage that per-stage $1 \times 1$-projects each hooked feature and resizes it by the reassemble_factors (4,2,1,½) — the up-sampling ConvTranspose2d (kernel = stride = factor) is realized as a pointwise expansion feeding the existing TNNetPixelShuffle, the ½ step is a strided $3 \times 3$ down-conv — building a 4-level pyramid; bias-free $3 \times 3$ neck convs unify the channel count; then RefineNet-style additive fusion blocks (pre-activation residual conv units + the new TNNetBilinearResize, the absolute-target bilinear resize supporting PyTorch's align_corners=True/False) merge the pyramid coarse-to-fine; and a 3-conv head (with a bilinear upsample to full resolution) emits a single-channel depth map. The demo loads the committed pico fixture (tests/fixtures/tiny_dpt.safetensors, a tiny DINOv2-backbone net), runs it on a synthetic image and renders the depth map as an ASCII grayscale ramp (near = bright, far = dark); a real run loads the Depth-Anything-V2 checkpoint the same way and processes a photo. Parity is asserted to max |diff| < 1e-4 vs the real transformers float64 forward over the whole depth map (TestDPTDepthEstimationParity, fixture generator tools/make_pico_dpt_fixture.py). Pure CPU, <1 s on the fixture.
• Depth Anything V2 → normalized depth-map image — the demo for the NAMED BuildDepthAnythingV2FromSafeTensors entry point (neuralpretrained.pas; model_type depth_anything: depth-anything/Depth-Anything-V2-Small/Base/Large-hf). Depth Anything V2 (Yang et al. 2024) IS the DPT dense-prediction stack on a DINOv2 ViT backbone (S/B/L); this importer is the wiring of the landed BuildDINOv2* backbone into the landed DPT reassemble + fusion neck + 3-conv depth head, asserting the depth_anything family. The wiring honors the backbone's out_indices (the four SELECTED encoder stages that feed the neck; 1-based stage K = encoder block K−1, default last-4 [N−3..N]) so non-default hooks load correctly. Distinct from the DepthEstimation ASCII-ramp DPT demo: it loads via the named V2 entry point, runs the pico fixture whose backbone hooks non-last-4 stages (out_indices=[2,3,5,6]), and WRITES image files — a min/max-normalized 8-bit grayscale PGM (P5) plus a color PPM (P6) depth visualization (near = warm, far = cool) — alongside an ASCII preview. Default is a SMOKE run on the committed pico fixture (tests/fixtures/tiny_depth_anything_v2.safetensors); pass a .safetensors path to run a real checkpoint (config beside it). Parity is asserted to max |diff| < 1e-4 vs the real transformers float64 DepthAnythingForDepthEstimation forward over the whole depth map (TestDepthAnythingV2Parity, fixture generator tools/make_pico_depthanythingv2_fixture.py). Pure CPU, <1 s on the fixture.
• ViTPose human-pose keypoint estimation — the demo for BuildViTPoseFromSafeTensors (neuralpretrained.pas), a keypoint / human-pose importer with a heatmap output modality (a stack of per-joint 2-D heatmaps, not a class label, box or dense class map; model_type vitpose, backbone vitpose_backbone: usyd-community/vitpose-base-simple and the ViTPose family). ViTPose (Xu et al. 2022) is a top-down single-person estimator: a plain ViT transformer backbone (REUSED — the encoder is the same separate-q/k/v pre-LN block path as the BuildViTFromSafeTensors image classifier, over the cropped person image) feeds a small "simple" deconvolution head. The backbone has two ViTPose quirks faithfully reproduced: the patch conv carries a hardcoded padding=2 (for patch_size ≥ 5 the floored output grid still equals image // patch), and there is NO class token — each patch token instead gets position_embeddings[1:] PLUS the broadcast cls-position row [:1], folded into a single learned position table at load. The new code is the patch-grid reshape + the deconvolution head (ReLU → bilinear upsample by scale_factor (the existing TNNetBilinearUpsample, align_corners=False) → $3 \times 3$ conv to num_labels channels) emitting one (H·scale, W·scale) heatmap per joint, and the CPU spatial-argmax read-out DecodeViTPoseKeypoints (per-channel max → (x,y) peak + score; a plain helper, NOT a leaf layer). The demo loads the committed pico fixture (tests/fixtures/tiny_vitpose.safetensors, a tiny ViT-backbone net), runs it on a synthetic image, decodes the per-joint peaks and renders them as an ASCII plot (each joint a digit at its argmax) over the heatmap grid; a real run loads the vitpose-base-simple checkpoint the same way and processes a detector person crop. Parity is asserted to max |diff| < 1e-4 vs the real transformers float64 forward over the whole heatmap stack (TestViTPosePoseEstimationParity), the argmax decode is verified against the oracle peaks (TestViTPoseKeypointDecode), fixture generator tools/make_pico_vitpose_fixture.py. Pure CPU, <1 s on the fixture.
• DETR object detection (draws boxes) — the demo for BuildDetrFromSafeTensors (neuralpretrained.pas), an object-detection importer (model_type detr: facebook/detr-resnet-50). DETR (Carion et al. 2020) pairs a ResNet-50 backbone (conv + folded FrozenBatchNorm) with the DETR transformer encoder-decoder: an input-projection $1 \times 1$ + flatten, a 2-D sinusoidal spatial position embedding (normalize=True, added to the queries+keys but NOT the values of every attention), a fixed set of learned object queries, post-LN encoder/decoder blocks with cross-attention, and a sigmoid-cxcywh box head + class head emitting, for each of num_queries object slots, a (num_labels + 1 + 4) vector (the class logits with a trailing "no-object" slot, then the normalized cxcywh box). Inference only — there is no Hungarian matcher (training-only). The demo runs a full DETR forward on ONE image, decodes the predictions with DecodeDetrDetections (softmax the class logits, drop the no-object slot, threshold on confidence, convert each normalized cxcywh box to pixel xyxy) and DRAWS the surviving boxes as colored rectangle outlines into the image, writing the annotated result to object_detection.ppm and printing the (class, score, box) list. It loads the committed pico fixture (tests/fixtures/tiny_detr.safetensors, a tiny random DetrForObjectDetection) by default so it runs fully OFFLINE on its pinned synthetic image; pass a real facebook/detr-resnet-50 checkpoint + config.json (+ optional confidence threshold) for real detections through the same path. Because the fixture has random weights the "detections" are not meaningful objects — the demo self-reports instead (asserts no NaN/Inf in the output and that every decoded box lands in the sigmoid [0,1] range / in-range pixel coords), exercising the full decode+draw pipeline end to end. Parity of the importer is asserted to max |diff| < 1e-4 vs the real transformers float64 forward over both the class-logits and box tensors (TestDetrObjectDetectionParity), the decode read-out is verified against a manual softmax+argmax (TestDetrDetectionDecode), fixture generator tools/make_pico_detr_fixture.py. Pure CPU, <1 s on the fixture.
• YOLOv8 single-shot object detection (draws boxes) — the demo for BuildYoloFromSafeTensors (neuralpretrained.pas), an anchor-free fully-convolutional one-stage detector (model_type yolov8: ultralytics yolov8n), structurally distinct from DETR (no transformer, no learned object queries). The importer reuses the conv + folded-BatchNorm loader path (every ultralytics Conv = conv2d(no bias) → BatchNorm2d → SiLU), and the new wiring is: the C2f cross-stage block (a $1 \times 1$ conv → channel split into two halves → n chained Bottlenecks keeping every intermediate → DeepConcat → $1 \times 1$), the SPPF block ($1 \times 1$ → 3 chained k5 s1 maxpools all kept → concat → $1 \times 1$), the PANet feature-pyramid neck (top-down nearest-$2 \times$ upsample + concat + C2f, then bottom-up stride-2 conv + concat + C2f), and the decoupled DFL detect head over 3 strides (a box branch emitting 4*reg_max distribution logits + a class branch emitting num_classes logits per grid cell). The net outputs the raw head flattened to a (Σ Hᵢ·Wᵢ, 1, 4*reg_max + num_classes) per-cell tensor (the same per-cell layout as DETR's per-query tensor). DecodeYoloDetections is the CPU post-process: sigmoid the class logits, DFL-decode each of the 4 box sides (softmax the reg_max bins → expected distance = the ltrb offset from the cell centre, in grid units), convert to an xyxy pixel box, then greedy IoU NMS. The demo runs a full YOLOv8 forward on ONE image, decodes + NMS, DRAWS the surviving boxes as colored rectangle outlines, writes yolo_detect.ppm, and prints the (class, score, xyxy) list. It loads the committed pico fixture (tests/fixtures/tiny_yolo.safetensors, a tiny random yolov8) by default so it runs fully OFFLINE; pass a real ultralytics yolov8 checkpoint + config.json (+ optional score/IoU thresholds) for real detections through the same path. Because the fixture has random weights the "detections" are not meaningful objects — the demo self-reports (asserts no NaN/Inf and finite box coords), exercising the full DFL-decode + NMS + draw pipeline end to end. Importer parity is asserted to max |diff| < 1e-4 vs a numpy float64 oracle over the raw head (TestYoloObjectDetectionParity), the decode is verified against a manual DFL + sigmoid (TestYoloDetectionDecode), fixture generator tools/yolo_tiny_fixture.py. Pure CPU, <1 s on the fixture.
• OWL-ViT open-vocabulary object detection — the demo for BuildOwlViTFromSafeTensors (neuralpretrained.pas), an open-vocabulary (zero-shot, text-conditioned) object-detection importer (model_type owlvit/owlv2: google/owlvit-base-patch32 and siblings). Unlike DETR (open but closed-vocabulary — a fixed label set), OWL-ViT (Minderer et al. 2022) matches every image patch against arbitrary free-text query embeddings by cosine similarity, so the "classes" are whatever text you encode, decided at inference time. It REUSES the CLIP towers verbatim (the checkpoint tensor names are CLIP's under an owlvit. prefix): the CLIP ViT image tower (via the reusable BuildClipVisionTower, here returning the post-layernorm (num_patches+1,1,hidden) hidden states) and the CLIP text tower (causal, argmax-of-ids EOS pooling, text_projection). The new code is the OWL-ViT detection head built on top of the post-LN vision states: the CLS merge (image_embeds[p] = post_ln[1+p] * post_ln[0], the class token broadcast over the patch rows via TNNetChannelMulByLayer, then a final layer_norm); the class head dense0 (hidden → text_dim, L2-normalized and matched by cosine against the L2-normalized query embeddings) with a learnable per-patch logit_shift and ELU-gated logit_scale; and the box head (3-layer exact-erf-GELU MLP → raw cxcywh). The cosine match + (logit + shift)*scale and the box sigmoid(box_raw + grid_box_bias) are finished by DecodeOwlViTDetections (the grid box-bias depends on the patch index and is added there); query embeddings are pooled+normalized by OwlViTQueryEmbedding. The demo loads the committed pico fixture (tests/fixtures/tiny_owlvit.safetensors), runs one tiny image, embeds a few free-text query token sequences, and prints the best-matching patch (sigmoid score + cxcywh box) per query — offline, reproducing the parity test's numbers. Parity is asserted to max |diff| < 1e-4 vs the real transformers float64 forward over the per-patch/per-query class logits AND the boxes (TestOwlViTOpenVocabDetectionParity), the decode is verified for ordering/range (TestOwlViTDetectionDecode), fixture generator tools/make_pico_owlvit_fixture.py. Pure CPU, <1 s on the fixture.
• Mask R-CNN instance segmentation (per-object mask) — the demo for BuildMaskRCNNFromSafeTensors (neuralpretrained.pas), the FIRST instance-segmentation importer (model_type maskrcnn: torchvision maskrcnn_resnet50_fpn). Unlike SegFormer (one dense class map over the whole image) Mask R-CNN (He et al. 2017) emits a separate binary mask per OBJECT. SCOPE v1 (bounded, per the tasklist): INFERENCE with externally supplied proposal boxes — the RPN / anchor generator is SKIPPED, the backbone FPN-input feature maps are fed directly (the ResNet-50 backbone's C4/C5 taps in a real run, via the already-landed BuildResNetFromSafeTensors), and one fixed proposal box is pooled. The importer builds the FPN top-down pyramid (lateral $1 \times 1$ convs + nearest-$2 \times$ upsample via TNNetDeMaxPool + $3 \times 3$ smoothing convs), the RoIAlign crop of the proposal from the chosen pyramid level (the already-landed TNNetRoIAlign, torchvision aligned=True half-pixel offset, sampling_ratio from config) at both the box pool size (7) and mask pool size (14), the box head (fc6 → ReLU → fc7 → ReLU → parallel cls_score + bbox_pred; the fc6 input columns are PERMUTED from PyTorch channel-major to CAI depth-major), and the small mask head ($4 \times$ $3 \times 3$ conv+ReLU → ConvTranspose2d(2,stride2)+ReLU → $1 \times 1$ conv to per-class H×H mask logits). RunMaskRCNN rewrites the proposal box on the RoIAlign layers and returns the class logits, box deltas and per-class mask logits for one proposal. The demo loads the committed pico fixture (tests/fixtures/tiny_maskrcnn.safetensors, a tiny random Mask R-CNN) by default so it runs fully OFFLINE, runs the single fixture proposal, picks the best-scoring class, sigmoids that class's mask, OVERLAYS it (red) on a tiny synthetic image and writes instance_segmentation.ppm. Because the fixture has random weights the "mask" is not a meaningful object — the demo self-reports (asserts no NaN/Inf and that the thresholded mask covers a sane pixel fraction), exercising the full FPN + RoIAlign + box/mask-head pipeline end to end. Importer parity is asserted to max |diff| < 1e-4 vs a self-contained numpy float64 oracle over the mask-head logits AND the class logits + box deltas (TestMaskRCNNParity; torchvision is not in the venv so the oracle is hand-written, the same stance as the ResNet fixture), fixture generator tools/make_pico_maskrcnn_fixture.py. Pure CPU, <1 s on the fixture.
• Mask2Former universal segmentation (mask-classification set prediction) — the demo for BuildMask2FormerFromSafeTensors + RunMask2FormerSemantic + DecodeMask2FormerSemantic (neuralpretrained.pas), the FIRST universal-segmentation importer (model_type mask2former: facebook/mask2former-swin-tiny-*-semantic). Distinct from both SegFormer (per-PIXEL argmax) and Mask R-CNN (RoIAlign on region proposals): Mask2Former (Cheng et al. 2022, arXiv:2112.01527) does mask-classification set prediction — a fixed set of learned object queries, each predicting ONE binary mask + a class, unifying semantic/instance/panoptic in a SINGLE head, with NO proposals and NO per-pixel classifier. The conceptual core is MASKED ATTENTION: each decoder layer's cross-attention is restricted to the FOREGROUND of the previous layer's predicted mask (sigmoid(mask) >= 0.5 keys allowed, background keys get -1e9, with the HF "attend-to-nothing → unmask-all" fallback), realised over an explicit TNNetDotProducts score matrix + additive-bias TNNetSum (there is no off-the-shelf masked-cross-attention leaf) in the Mask2Former-specific cross → self → FFN post-norm order; the packed nn.MultiheadAttention in_proj is row-sliced into q/k/v. Because masked attention has a DYNAMIC feedback loop (layer L's mask comes from layer L−1's prediction), the decoder is built as one sub-net per layer and driven layer-by-layer by RunMask2FormerSemantic, which recomputes the mask bias (bilinear-downsampled to each round-robin level) between layers; DecodeMask2FormerSemantic then folds the per-query class+mask logits into a per-pixel label map (softmax classes, drop the no-object slot, sigmoid masks, class-weighted argmax) exactly like HF post_process_semantic_segmentation. SCOPE v1: SEMANTIC inference, decoder + heads only — the pixel-decoder outputs (mask_features + the 3 multi-scale memory levels) are fed as PRECOMPUTED inputs (mirroring how Mask R-CNN v1 took FPN feature maps directly); wiring the full Swin backbone + FPN pixel decoder into one forward is a documented tasklist.md follow-up. The demo loads the committed pico fixture (tests/fixtures/tiny_mask2former.safetensors) + its precomputed feature maps fully OFFLINE, runs the masked-attention decoder, prints the label map as a colored ASCII palette and writes segmentation.ppm. Random pico weights → wiring/throughput smoke (one class wins everywhere, matching the HF reference map), not a trained segmenter. Importer parity is asserted to max |diff| < 1e-4 (actual ~1.2e-7) vs the REAL transformers float64 Mask2Former forward over BOTH the per-query mask logits AND class logits (TestMask2FormerParity, fixture generator tools/make_pico_mask2former_fixture.py). Pure CPU, <1 s on the fixture.
• Image colorization (L → ab) — a self-supervised generative-vision task: predict the chroma of an image from its luminance alone, so any grayscale photo can be auto-colorized. Each CIFAR-10 RGB image is converted to CIELAB with the existing TNNetVolume.RgbToLab/LabToRgb helpers (standard sRGB → linear → XYZ(D65) → Lab and the exact inverse, now regression-covered by TestVolumeLabRoundTrip: RGB → Lab → RGB max‖diff‖ < 1 of 255). The L channel (1 channel, /100 normalized) is the network INPUT and the a*,b* channels (2 channels, /110 normalized) are the regression TARGET — no labels needed, the supervision is the image's own color. A small CPU-friendly conv encoder-decoder built with the reusable TNNet.AddUNet(Depth=2, BaseFeatures=16, OutputChannels=2, …) (the same builder as UNetSegmentation) is trained with plain per-pixel L2 (MSE) regression on the a*,b* channels, capped by a TNNetHardTanh so the output stays in the normalized chroma range. After training, a handful of validation images are colorized — the predicted a*,b* are recombined with the TRUE L, mapped back to RGB and written to disk as sampleN_gray.png / sampleN_color.png pairs (gray-in vs color-out) under ./colorized/. The classic quantized-bin classification head (Zhang et al. 2016) would give more vivid output; this uses the simpler regression head. Reuses existing conv layers + the existing Lab helper — the new code is the L-in/ab-out data pipeline and the recombination. Pure CPU; the default SMOKE run (2000 train images, 10 epochs) finishes in a few minutes. Needs the CIFAR-10 binary batches in the working dir (data_batch_1.bin … data_batch_5.bin, test_batch.bin; see the link printed if missing).

Training dynamics & optimization

• Optimizer bake-off — trains the same tiny MLP on the same fixed hypotenuse toy four times, changing only the optimizer (plain SGD, SGD+momentum, Adam, and RMSProp via Adam with Beta1=0); holds seed/data/architecture/LR fixed across arms and prints a loss-vs-epoch table, an epochs-to-converge summary, and an ASCII chart.
• Learning-rate range test — Leslie Smith's LR-range-test in pure Pascal: trains a tiny MLP for ~100 mini-batches sweeping LR exponentially from 1e-6 to 1e+1, prints a 1-column ASCII chart of log10(LR) vs smoothed loss with the steepest-descent row marked, and writes a CSV side-output for downstream tooling.
• Scheduler Compare - Trains the same tiny MLP four times under different learning-rate schedules (constant / Step / Cosine / WarmupCosine) from the neuralscheduler unit, driving the LR manually per epoch. Prints an ASCII chart of each LR curve plus a final-loss table — you can see constant stay flat, Step drop in stairs, and Cosine/WarmupCosine anneal. Pure CPU, ~1s.
• Batch-size sweep — trains the SAME tiny MLP on the SAME synthetic hypotenuse task y = sqrt(X^2 + Y^2) across batch sizes {1, 8, 32, 128} (net, data, and RNG seed all held fixed so batch size is the only variable) and prints how the knob trades wall-clock-per-epoch against epochs-to-converge. A beginner-oriented companion to the activation/optimizer bake-offs. Pure CPU, well under a minute.
• Mixup Augmentation - Mixup data augmentation (Zhang et al. 2018): CreateMixedVolumePairList (in neuralvolume) forms synthetic training pairs by convex-combining two real pairs x_mix = λ·x_i + (1-λ)·x_j, y_mix = λ·y_i + (1-λ)·y_j with λ ~ Beta(α,α) (built-in Beta/Gamma sampler; Beta(1,1)=Uniform fast path). Trains a tiny classifier with vs without mixup. Pure CPU, ~1s.
• CutMix Augmentation - CutMix data augmentation (Yun et al. 2019): CreateCutMixVolumePairList (in neuralvolume) pastes a random rectangle of a permuted partner image into the input (box = (r·W)×(r·H), r = sqrt(1-λ), clamped) and mixes the targets by the TRUE pasted-area fraction λ_adj = 1 - box_area/(W·H) with λ ~ Beta(α,α) (shares Mixup's Beta/Gamma sampler). ComputeCutMixBox exposes the rand_bbox geometry for deterministic use. Trains a tiny conv classifier with vs without cutmix on a synthetic image toy. Pure CPU, ~1s.
• AutoAugment (RandAugment / TrivialAugment) — automatic single-image augmentation policy (in neuraldatasets): a fixed op bank (autocontrast, equalize, rotate, shear-x/y, translate-x/y, posterize, solarize, color/contrast/brightness/sharpness) over a TNNetVolume, plus RandAugment (NeuralRandAugment, N ops at fixed magnitude M; Cubuk et al. 2020), the parameter-free TrivialAugment (NeuralTrivialAugment, one op, magnitude drawn uniformly; Müller & Hutter 2021) and RandomErasing / Cutout (NeuralRandomErasing; Zhong et al. 2020). Magnitudes follow the torchvision transforms-v2 _AUGMENTATION_SPACE ranges on a 0..30 integer scale (M=0 ≈ identity; rotate(0)/shear(0)/translate(0) are bit-identity). TNeuralAugmentationPolicy bundles a policy + RandomErasing into the new opt-in TNeuralImageFit.ImageAugmentationFn hook, applied AFTER the built-in flip+crop so existing CIFAR examples can opt in unchanged. Demo trains a tiny conv classifier without vs with the policy on a synthetic toy. Pure CPU, ~3 s. Expected to give a small consistent top-1 lift on real CIFAR-10.
• Shake-Shake regularization bake-off — the headline regularisation win for the TNNet.AddShakeShakeBlock builder / TNNetShakeShakeMerge layer (Gastaldi 2017, Shake-Shake regularization): two parallel residual branches are merged with a STOCHASTIC convex weight (alpha forward, an independent beta backward, both resampled per pass; eval is the deterministic 0.5/0.5 mean). The demo contrasts a shake-shake stack against an otherwise-identical deterministic two-branch (0.5/0.5) residual on a deliberately noisy, over-parameterised toy (40-D input, only 2 informative dims, 64 training samples with 25% label noise) so the over-sized net memorises the noise and the generalisation gap surfaces. Both arms hit 100% train accuracy, but Shake-Shake generalises better on every metric — val accuracy 74.5% vs 70.2%, val loss 0.619 vs 0.659, and a narrower train/val accuracy gap (25.5 vs 29.8). Pure CPU, ~7 s.
• Pre-norm vs post-norm residual bake-off — wires the same 12-block deepish residual MLP three ways via the TNNet.AddPreNormResidual / AddRMSNormResidual / AddPostNormResidual builders and trains each on the hypotenuse toy at matched seed/LR/epochs; surfaces the textbook stability gap (the pre-norm arms drop to near-zero error while post-norm oscillates and ends ~10x higher) and handles any diverging arm cleanly.
• ReZero vs gated-residual depth ablation — trains the same deepish residual MLP on the hypotenuse toy two ways at matched seed/arch/LR/epochs, differing only in the residual gate: a scalar TNNetReZero (one learnable weight, init 0) vs the per-channel TNNet.AddGatedResidual builder (one gate per channel, init 0). Dumps the learned gates per block as an ASCII chart so the per-channel gate's UNEVEN opening (many channels stay exactly 0, a few grow both signs) is visible against the uniformly-opening ReZero scalar. Runs in ~15 s on CPU.
• Highway depth-trainability sweep — the headline trainability-at-depth win for the new TNNetHighway, the input-dependent learned-gate Highway layer (Srivastava, Greff & Schmidhuber 2015): y = T(x)⊙H(x) + (1-T(x))⊙x with T(x)=sigmoid(W_T·x+b_T) computed FROM THE INPUT each forward pass and an explicit (1-T)·x identity carry. Negative gate-bias init (-1.5) starts a fresh deep stack near the identity. On an "identity + small residual" target, plain tanh-MLP stacks and Highway stacks are trained head-to-head at depths {2,5,10,20,40}: the plain stack's test MSE degrades monotonically (≈16× from depth 2→10, ≈44× by depth 20) while the Highway stack stays essentially flat through depth 10 and ≈8× better at depth 20, because the carry keeps gradients flowing to the early layers. Also reports the mean learned gate T per layer. Distinct from TNNetReZero (scalar gate), AddGatedResidual (per-channel constant gate) and TNNetGLU/SwiGLU (no identity carry). Pure CPU, ≈40 s.
• DropPath / Stochastic-Depth ablation — sweeps the DropPath probability p in {0.0, 0.1, 0.2} on the same small 6-block ResNet-style classifier (each block y = x + TNNetDropPath(p)(ReLU(PointwiseConvLinear(x))), the stochastic-depth layer on the residual BRANCH) at matched seed/data/epochs, and prints a per-p table of final train/test loss and accuracy. Honest read: on this tiny easy 3-way toy DropPath does NOT raise test accuracy (all arms tie), but the held-out cross-entropy falls monotonically with p (3.03→2.80→2.37) — a real over-confidence-reducing regularisation signal. The gate asserts only true invariants (every arm trains, nothing diverges, p=0.0 reproduces the no-drop baseline); eval forces EnableDropouts(false) so inference is the deterministic identity. Pure CPU, ~12 s.
• Noise-layer train/inference delta sweep — sweeps the drop probability p in {0.0, 0.1, 0.2, 0.4} for all four stochastic noise layers (TNNetDropout, TNNetDropPath, TNNetSpatialDropout1D, TNNetSpatialDropout2D) on the same tiny 4-block ResNet-style classifier (each layer on the residual BRANCH, y = x + Noise_p(ReLU(PointwiseConvLinear(x)))), and for every arm reports the train loss with the noise ON (EnableDropouts(true), stochastic mask), the train loss with the noise OFF (EnableDropouts(false), inference identity), the held-out val loss and the train/val gap. The point is the train-vs-inference distinction: a loss is only comparable across train and val if the noise is the deterministic identity at inference. Built-in checks (Halt(1) on failure): every inference pass is bit-for-bit deterministic (each probe run twice), and at p=0.0 train(ON)==train(OFF) exactly (the layer is the identity in both regimes). The Spatial1D/2D arms are numerically identical on a SizeY=1 tensor (same channel masks, same RNG) — documented honestly. Pure CPU, single-threaded, ~63 s.
• Weight Standardization + GroupNorm vs BatchNorm — the headline use case for the new TNNetWeightStandardizationConv (the convolution sibling of the dense TNNetWeightStandardization, Qiao et al. 2019, Micro-Batch Training with Batch-Channel Normalization and Weight Standardization): each conv FILTER's weights are standardized to zero-mean/unit-std BEFORE the convolution (exact per-output-channel Jacobian in backward), which smooths the loss landscape INDEPENDENTLY of the batch — the regime where BatchNorm's noisy per-batch statistics hurt. A two-arm bake-off trains one matched backbone at a deliberately SMALL batch, swapping only the normalization: bn (plain conv filters + AddMovingNorm BatchNorm + ReLU) vs ws_gn (weight-standardized conv filters + TNNetGroupNorm + ReLU, the Qiao recipe). Graded checks (Halt(1) on failure): both arms train and reach a healthy classifier, and WS+GroupNorm is COMPETITIVE — its test accuracy is within tolerance of (or beats) BatchNorm. The head-to-head is reported honestly, not assumed (on this easy synthetic toy the two MATCH within tolerance). Pure CPU, ~65 s.
• Sharpness-Aware Minimization (SAM) — a hand-rolled two-pass SAM (Foret et al. 2021) on a noisy-label 2D-blob toy: ascent-perturb the weights by rho * g/||g|| using the global gradient norm, take a SECOND forward+backward at the perturbed point, then restore the original weights (whole-net SaveDataToString snapshot) and apply the perturbed gradient with plain SGD. Contrasts SAM vs plain SGD and prints the flatness via TNNet.LossLandscapeProbe for both. Two built-in invariants hold: rho=0 reproduces plain SGD bit-for-bit, and the LossLandscapeProbe sharpness falls as rho grows. Pure CPU, ~8 s.
• Muon optimizer (Newton-Schulz orthogonalized momentum) — a hand-rolled gradient-surgery demo of the Muon optimizer (Jordan et al. 2024) in the SharpnessAwareMinimization idiom (NOT a core optimizer rewrite). Per step under NN.SetBatchUpdate(True) (so Neurons[].Delta is populated), each TNNetFullConnectLinear weight matrix gets a momentum buffer M <- mu*M + G, then M is replaced by its nearest semi-orthogonal matrix via 5 fixed quintic Newton-Schulz iterations (coeffs 3.4445/-4.7750/2.0315, all via TNNetVolume matmuls), and W <- W - lr*sqrt(max(rows,cols))*O. Bakes Muon off against SGD-momentum and Adam on a tiny regressor (all converge comparably, ~1 s). Headline check (Halt(1) on failure): the published 5-step quintic is a deliberately approximate orthogonalizer, so the singular values of O land in the Muon band ~[0.7,1.3] (||O^T O - I||_F bounded, top sigma cross-checked with EstimateSpectralNorm) — semi-orthogonal, not strictly sigma=1. README distinguishes it from the forward-weight reparametrizers TNNetWeightNormLinear / TNNetWeightStandardization (Muon normalizes the update, not the weights). Pure CPU.
• SoftCapping logit stability — micro-experiment showing that a TNNetSoftCapping(c) layer before the final SoftMax tames logit blow-ups under a deliberately-aggressive learning rate. The same tiny 4-class blob classifier is trained twice at LR=5.0, differing only in whether TNNetSoftCapping(8.0) is inserted before SoftMax: the uncapped arm's logits explode (~1e4, accuracy collapses to chance) on every epoch while the capped arm pins every logit to [-8, 8], logs zero exp-overflow events, and reaches 100% accuracy. Pure CPU, ~2 s.
• Domain-adversarial NN — smallest possible DANN (Ganin et al. 2015) demo around the new TNNetGradientReversal layer: two 2D-blob domains with 90-degree-rotated class-conditional means, shared trunk feeding a label head + a domain head behind a Gradient Reversal Layer. With GRL enabled the label head reaches ~99% on both domains while the domain head collapses to chance (~0.500); toggle a constant to confirm the contrast. Runs in ~2s on CPU.
• Forward-Forward — reproduces Geoffrey Hinton's 2022 Forward-Forward algorithm on a pure-CPU toy using only existing in-tree layers. It does NOT learn by end-to-end backpropagation: instead of forming one global loss and backpropagating it through the whole stack, FF trains each layer locally with separate positive/negative forward passes and a per-layer goodness objective.
• Predictive Coding — a backprop-free Predictive Coding Network (Rao & Ballard 1999), a SECOND biologically-plausible learning paradigm distinct from Forward-Forward. Each layer carries explicit value nodes and local prediction errors e_l = x_l - W_l*act(x_{l+1}); training alternates a Phase-1 inference relaxation that settles the value nodes to minimise the summed squared prediction-error energy with a Phase-2 purely local Hebbian weight step dW = e (x) act — no global loss, no backward pass chained across layers. The energy/value math is done directly with TNNetVolume (no new core layer). Prints the per-sweep energy descent and a side-by-side accuracy vs a same-shape backprop MLP (PCN ~0.873 vs backprop ~0.970 on 3 Gaussian blobs, ~3s on CPU).

Vision & convolutional models

• Image classifier with SELU — a 32x32x3 image classifier built around the SELU activation function (self-normalizing networks): conv + max-pool stack with SELU activations and SELU-friendly initialization, a worked example of wiring TNNetSELU into a CNN.
• Simple image classifier (parallel) — a CIFAR-10 image classifier (THistoricalNets + TNeuralImageFit) configured for parallel/multi-threaded training; a parallel-training companion to the simple CIFAR-10 classifier examples.
• DenseNet Fashion-MNIST — CAI-optimized DenseNet image classifier trained on the Fashion-MNIST dataset (T-shirt/Trouser/Pullover/… 10 classes), a clothing-image companion to the CIFAR-10 DenseNet example.
• CAI-optimized DenseNet (CIFAR-10) — command-line tool that trains a CAI-optimized DenseNet on CIFAR-10 (inspired by liuzhuang13/DenseNet) with configurable bottleneck, inner-conv count, neuron count, and optional separable convolutions. A full image-classification training program rather than a toy.
• Pooling-head bake-off — same tiny conv classifier on a synthetic blob-quadrant task, swapping only the pooling head across TNNetAvgPool / TNNetMaxPool / TNNetLpPool (sweep p in {1,2,4,8}) / TNNetSoftPool (sweep beta in {0.5,1,2,8}) / TNNetStochasticPool (sample-at-train, expectation-at-inference). The task is built so the class-mean is invariant and only energy CONCENTRATION discriminates, so AvgPool sits at chance while MaxPool solves it and LpPool's p / SoftPool's beta trace the average→max interpolation. Reports per-arm train/val accuracy and the train/val gap to probe whether StochasticPool's regularization narrows it.
• Adaptive-pool resolution invariance — the headline property of adaptive pooling: ONE fully-convolutional stack (Conv8(3x3,pad1)+ReLU x2 -> MaxPool(2) -> [adaptive head]) accepts inputs of DIFFERENT spatial sizes (16x16 and 24x24) and still emits a FIXED-size head, the classic adaptive global pool -> FC classifier pattern. It exercises both TNNetAdaptiveAvgPool and TNNetAdaptiveMaxPool (global Create(1) and Create(2) heads), prints input -> post-conv -> adaptive -> output shapes per resolution to make "variable in, fixed out" explicit, asserts two built-in degeneracies (Create(1) == global pooling, Create(N) == identity when N equals the post-conv size), and trains a global-head classifier at 16x16 then runs inference at the unseen 24x24 (via a weight-shared sibling net) to show the trained fixed-size head transfers across resolutions. Pure CPU, ~1.5 s.
• CoordConv spiral — minimal demo of the new TNNetCoordConv layer (Liu et al. 2018). Two identical 1x1-conv + global-avg-pool nets are trained to regress the (x, y) coordinate of the single bright pixel in an 8x8 image. The plain net is translation-invariant by construction and gets stuck near the constant-prediction baseline (MSE ~0.84); a single TNNetCoordConv at the front collapses MSE by ~400x to ~0.002. Runs in ~25s on CPU.
• CBAM Attention - Convolutional Block Attention Module (TNNet.AddCBAM, Woo et al. 2018): channel attention (dual avg+max pooling) followed by spatial attention, dropped into a small conv classifier. Pure CPU.
• Sub-Pixel Super Resolution - A tiny TNNetPixelShuffle (depth-to-space) head learns a 2x upscaling on synthetic tiles (test PSNR 4 dB → 39 dB), pure CPU
• Spatial Transformer Network — learned geometric canonicalization — the headline demo for the new TNNetAffineGridSample (Jaderberg et al. 2015, Spatial Transformer Networks), an op that warps a feature map by a CONTINUOUS, input-conditioned geometric transform (every existing spatial op — conv, pool, upsample, deconv — resamples on a FIXED integer grid). The parameter-free two-input layer reads an image source and a caller-wired 6-value affine theta (a 2×3 matrix from a small localization head), back-warping each output pixel (x',y') = theta·[x_n,y_n,1] in normalized [-1,1] coords and bilinearly interpolating the 4 nearest source pixels; both backward paths (d/d-source scatter and the headline d/d-theta sampler partials) are finite-difference checked. The example prepends a conv localization head → FullConnectLinear(6) BIAS-INITIALISED to the identity affine [1,0,0,0,1,0] (weights zeroed) → the sampler in front of a small shared classifier, trained end-to-end on RANDOMLY ROTATED/TRANSLATED synthetic glyphs (no data download): the plain classifier sits at 0.879 while the CNN+STN front-end learns to de-rotate/re-center the input and recovers to 1.000 (+12.1 pp). Concat-style serialization (theta source index in the structure string), no persisted weights. Pure CPU, ~40 s.
• Glimpse Downsampler — learned scale+translate "hard" attention glimpse — the scale+translate-restricted follow-up to Spatial Transformer. Reuses the same TNNetAffineGridSample sampler but constrains the warp to a learned CROP/ZOOM via the new parameter-free TNNetScatterToAffine, which scatters a localization head's 4 outputs (s_x, s_y, t_x, t_y) into the 2×3 affine [s_x,0,t_x; 0,s_y,t_y] with the rotation/shear slots held at a HARD zero (no parameter ever lands in them, so they can never drift). A small bright glyph is dropped at a random offset on a larger CLUTTERED canvas; the localization head learns WHERE and how much to zoom, reading a small 14×14 canonical patch from the 28×28 input (the sampler's warped view is avg-pooled to the patch size — output strictly smaller than input). Contrasted against a FIXED center-crop of the same patch size feeding the same classifier: the learned glimpse tracks the roaming glyph (1.000) while the blind center-crop misses off-centre glyphs (0.906, +9.4 pp). Pure CPU, ~1 min.
• Deformable convolution (content-adaptive sampling grid) — the headline demo for the new TNNetDeformableConv (Dai et al. 2017, Deformable Convolutional Networks), a conv whose KxK sampling positions are PREDICTED rather than fixed. A zero-initialized "offset head" (an ordinary conv) emits 2*K*K per-location offset maps; each tap is then gathered from the input by BILINEAR interpolation at (base_position + offset) before the usual weighted sum, so the receptive field bends to follow content instead of sitting on a rigid axis-aligned window (distinct from dilated conv's fixed dilation and group conv's rotated copies). Backward propagates into the conv weights, the input, AND the offsets — the offset gradient flows through the four bilinear corner coefficients (dSample/dpx, dSample/dpy), the interesting/hard part, and is finite-difference checked. Zero-init means the layer starts identical to a plain conv. The demo asks both arms to reproduce a field translated by (+3,+3) whose answer lies OUTSIDE a 3×3 window: the rigid TNNetConvolutionLinear (9 weights) structurally cannot reach and floors at val-MSE ~0.040, while TNNetDeformableConv learns the +3 sampling offset and reaches ~1e-6 (~40000× lower). Honest caveat (README): the offset head diverges to NaN at high LR (unbounded offsets × pixel-scaled bilinear gradient), so the example uses a small LR. Pure CPU, ~40 s.
• Video action recognition with 3-D convolution — the headline demo for the new TNNetConvolution3D (spatiotemporal / volumetric) convolution, a conv that mixes information across time as well as space. Each sample is a short grayscale CLIP (T frames of a cGrid×cGrid image) in which a bright blob slides in one of FOUR directions, and the task is to classify the MOTION direction — something a single frame is ambiguous about, so the network must integrate across time. The (T, H, W, C) clip is packed for TNNetVolume's three axes by laying the T frames contiguously along the Depth axis (Depth = T·C, SizeX=W, SizeY=H); the layer slides a (FeatureSizeT × FeatureSizeXY × FeatureSizeXY) kernel over the spatial grid (padded/strided like the 2-D conv) AND over the time blocks within the depth axis, with each frame's C channels contiguous so every kernel tap is a contiguous-depth AVX dot product. The net stacks two Convolution3D(F=8,T=3,K=3) blocks (each shrinking OutputT by 2) → FullConnectLinear(4) → SoftMax. A BASELINE on the identical stack but with FeatureSizeT=1 (a per-frame 2-D conv shared across frames, no cross-frame coupling) is trained alongside to make the temporal-mixing axis explicit. Layer covered by TestConvolution3DInputGradientCheck / TestConvolution3DWeightGradientCheck / TestConvolution3DSerializationRoundTrip. Pure CPU, ~1 min.
• Video classification with an imported VideoMAE transformer — the end-to-end demo for BuildVideoMAEFromSafeTensors, a video-classification importer (a clip of T frames → an action label) and the pay-off of the TNNetConvolution3D layer. It loads the committed pico checkpoint tests/fixtures/tiny_videomae.safetensors (the very fixture the float64 parity test TestVideoMAEClassificationParity pins to HF) and runs the full VideoMAE forward on CPU: a non-overlapping tubelet 3-D conv (kernel = stride = (tubelet_size, patch, patch)) splits the (T,H,W,C) clip into T′×H′×W′ space-time tokens, a fixed sin-cos 3-D position table (rebuilt exactly from HF's get_sinusoid_encoding_table, not learned) is added, the stock pre-LN transformer encoder applies joint space-time attention (every token attends to every token), and the finetuned head mean-pools the tokens → fc_norm → a linear classifier → action logits. The (T,H,W,C) clip is packed for TNNetVolume's three axes with frame t's C channels contiguous at depth [t·C, t·C+C) (the TNNetConvolution3D convention); the temporal tubelet stride is realized by selecting the non-overlapping output frames with TNNetSplitChannels + TNNetDeepConcat. The program synthesizes two sliding-blob clips, classifies each, and prints the per-class logits + argmax (the pico weights are random, so the label is not semantic — the point is the importer + forward path runs). Pure CPU, < 1 s.
• ImageNet top-1 / top-5 accuracy eval — the end-to-end demo for EvaluateImageNet / ImageNetReport (neural/neuralimagemetrics.pas), the import-VERIFICATION backstop for the landed classifier importers (ResNet / ViT / Swin / DINOv2 / MobileNetV3 / VGG / Inception-v3 / EfficientNet) — the CV analogue of MMLUEval / PerplexityEval on the LLM side. Each importer's parity test only compares raw logits on one or two tensors, which catches a transposed weight but not a wrong preprocessing pipeline (resize / center-crop / normalize) or a label permutation; running a folder of labelled ImageNet-val images through the real transform (neuraldatasets.PreprocessImageForVisionModel: shorter-side resize → center-crop → (x/255 − csImageNetMean)/csImageNetStd) and the real net, then checking top-1 / top-5 against the published numbers, is exactly that backstop. The harness takes already-preprocessed TNNetImageNetSample volumes + gold labels, runs NN.Compute, forms top-1 and top-K via TopKIndices (first-max tie-break), tallies top-1 (argmax == gold) and top-K (gold anywhere in the top-K) accuracy, and retains up to MaxConfusion top-1 misses for a human-readable confusion sample (each flagged top-K-hit vs top-K-miss). ImageNetReport formats the top-1 / top-5 lines + the confusion sample. To stay self-contained (no network fetch, no multi-GB download, no real ImageNet) the default SMOKE builds a small CNN, trains it for 30 epochs on a tiny DETERMINISTIC synthetic 6-class coloured-pattern set (fixed RandSeed) rendered large and pushed through the SAME PreprocessImageForVisionModel transform the real path uses, then evaluates with the harness — exercising the real transform + real harness end to end in seconds under the 3 GB ulimit (the cleanly-separable synthetic classes report 1.0000 / 1.0000; the point is the harness + transform mechanics, not a real number). The scoring path is checkpoint-agnostic: swap the smoke CNN for a BuildResNetFromSafeTensors (or any classifier importer), feed LoadImageForVisionModel-produced volumes into the same TNNetImageNetSample records, and the harness is unchanged. The --full <dir> flag prints the documented real-ImageNet-val recipe (folder layout: labels.txt of <filename> <class_index> lines + the JPEGs; import a backbone, LoadImageForVisionModel with its ImageSize + ImageNet mean/std, EvaluateImageNet(NN, Samples, 1000, 5)); wiring a real checkpoint + the full 50 k-image ImageNet-val run is a documented follow-up. Pure CPU.
• Capsule reconstruction & pose perturbation — the reconstruction-decoder stretch demo for TNNetCapsuleRouting (Sabour et al. 2017, Dynamic Routing Between Capsules): a CapsNet is trained with the paper's MARGIN loss on capsule lengths plus a masked-reconstruction MSE — a small decoder (Input(16) → ReLU(64) → Sigmoid(144)) is fed the TRUE-class capsule's pose vector and its input gradient is chained back into the encoder. Headline: after training, the program sweeps each pose dimension over [-0.25,+0.25], picks the most responsive one, and shows that varying that single dimension smoothly and MONOTONICALLY varies an interpretable visual factor (stroke intensity/thickness: total ink 32.6 → 24.9) — a genuine, visible disentanglement, not faked. Uses a tiny synthetic 12×12 two-class "bars" set (controllable thickness/position pose factors, chosen over a real MNIST parse to stay in budget). CapsNet and a param-matched MLP both hit 100% on the easy 2-class task (the capsule advantage here is the interpretable pose vector, not raw accuracy). Pure CPU, deterministic, ~18 s.

Generative image models

• DDPM diffusion digit generation — a generative-by-diffusion example (Ho et al. 2020): a small time-conditioned U-Net learns to denoise 28×28 MNIST digits and then generates fresh ones from pure Gaussian noise. Forward process is the standard LINEAR beta schedule (beta_1=1e-4 .. beta_T=0.02, T=200); the training objective is epsilon-prediction MSE (x_t = sqrt(alpha_bar_t)·x_0 + sqrt(1-alpha_bar_t)·eps, predict eps); generation is the ancestral (reverse) DDPM sampling loop from x_T~N(0,I) down to x_0, written out as a PNG grid. It USES TNNetSinusoidalTimeEmbedding (scalar timestep → sinusoidal vector) and injects the timestep into every U-Net block as a per-channel scale/shift via TNNet.AddFiLMConditioned/TNNetFiLM; skip connections reuse TNNetDeepConcat and the decoder upsamples with TNNetUpsample (all layers pre-existing — the new content is the schedule + sampling loop + tiny U-Net builder). The eps-MSE starts at the ~1.0 trivial baseline (variance of N(0,1) noise) and falls below it within ~100 steps, confirming real denoising; a from-scratch diffusion net spikes early so the loop clips the update magnitude. It also supports CLASS-CONDITIONAL generation (pick a digit) via classifier-free guidance (eps = eps_uncond + s·(eps_cond − eps_uncond), a label embedding added to the time embedding, trained with ~10% label dropout for a null/unconditional token) plus a deterministic DDIM fast sampler (10–50 steps vs the full ancestral loop). Default SMOKE mode (a few hundred steps + a 10×10 class-conditional grid where each row is a requested digit 0–9, sampled via DDIM+CFG) runs in ~2 min on CPU; --full for sharper digits. Needs the standard MNIST idx-ubyte files in the working directory.
• Flow Matching / Rectified Flow digit generation — the modern ODE/transport alternative to DDPM (Lipman et al. 2023, Liu et al. 2023): the same tiny time-conditioned U-Net as DiffusionMNIST, but trained to regress a velocity field along STRAIGHT interpolation paths. With noise x0~N(0,I), data x1, and continuous t~U(0,1), it builds the linear interpolant x_t = (1−t)·x0 + t·x1 and minimises MSE(v_theta(x_t,t), x1−x0) — no beta schedule, no alpha_bar, no noise/score parameterisation. Generation is a deterministic forward Euler ODE x_{t+dt} = x_t + dt·v_theta(x_t,t) from noise to t=1 in just ~25 steps (far fewer than DDPM's ancestral loop). Continuous t is rescaled by 1000 before TNNetSinusoidalTimeEmbedding so the embedding sees the integer-like range it was designed for. Reuses the DiffusionMNIST MNIST idx files (../DiffusionMNIST/..., no copy). Default SMOKE mode runs in ~2 min on CPU (velocity MSE ~2.8→~0.35) and writes an 8×8 Euler-sampled grid; --full for sharper digits.
• Class-conditional diffusion + classifier-free guidance — a generative image example that lets you pick the digit (VisualGAN, DiffusionMNIST in its plain form, and FlowMatching all draw UNCONDITIONAL samples). It trains the same tiny time-conditioned U-Net as DiffusionMNIST, but conditions on the class label y∈0..9 by mapping it through a learned TNNetEmbedding and adding that vector to the sinusoidal TNNetSinusoidalTimeEmbedding before a shared cond MLP, so a single TNNetFiLM cond vector carries both how much noise (t) and which digit (y). The CFG mechanism comes from LABEL DROPOUT: index 10 is a dedicated NULL/unconditional token and, with ~10% probability per training example, the real label is swapped for it, so the one network learns both the conditional score eps(x_t,t,y) and the unconditional eps(x_t,t,null) (Ho & Salimans 2022). The example is built around ONE question — what does the guidance weight w do? — and answers it QUANTITATIVELY: a tiny side MNIST classifier is trained purely to score class fidelity, and the denoiser is sampled over a sweep w∈{0,1,2,4,8} reporting per-w class-fidelity (classifier agreement with the requested digit; the textbook CFG effect is for it to RISE with w) and a diversity proxy (mean per-pixel std across same-class samples, expected to FALL with w as the model collapses onto the class mode) — the classic CFG fidelity/diversity trade-off. All noising (AddNoise), the reverse DDIM trajectory (Sample) and the CFG mix (ApplyCFG) reuse the model-agnostic neuraldiffusion.pas (TNNetDiffusionScheduler); nothing is hand-rolled. Writes a PNG grid whose ROWS are increasing w and COLUMNS are independent samples of one chosen digit, and asserts no NaN/Inf. The clean trade-off needs a well-trained denoiser (--full); the default SMOKE run (short denoiser + classifier + a small sweep, ~3 min on CPU) is intentionally undertrained — like the DiffusionMNIST smoke its samples are noisy, so the run mainly validates the pipeline and self-reports the observed metric direction. Reuses the DiffusionMNIST MNIST idx files (../DiffusionMNIST/..., no copy).
• Consistency-model few-step distillation — a FAST-SAMPLING generative example (Song et al. 2023): the landed diffusion examples (DiffusionMNIST, ConditionalDiffusion, FlowMatching) all generate in 16–200 reverse steps, whereas this one DISTILS a multi-step DDPM teacher into a CONSISTENCY MODEL that samples a digit in 1, 2 or 4 steps. A consistency function f(x_t,t) maps ANY point on a probability-flow trajectory directly to that trajectory's origin x_0, so a single network call is already a (rough) sample. The boundary condition f(x,t→0)=x is enforced EXACTLY by the Karras skip/out parameterisation f = c_skip(t)·x + c_out(t)·F_theta(x,t) with c_skip = σ_data²/(σ(t)²+σ_data²), c_out = σ(t)·σ_data/√(σ(t)²+σ_data²), and σ(t)=√((1−ᾱ_t)/ᾱ_t) read from the scheduler's AlphaBar — pure example-side arithmetic, no new layer. Distillation: forward-noise x_0 to a sub-grid timestep t_{n+1}, take ONE deterministic teacher DDIM ODE step down to t_n, then minimise ‖f_theta(x_{t_{n+1}},t_{n+1}) − f_target(x_{t_n},t_n)‖²; the target net f_target is the repo's TNNetEMAWrapper shadow (stop-grad EMA of the student), Update() per step. The c_out(t_{n+1}) factor is folded into the backprop target so the raw network head trains on the right residual. Both teacher (eps-prediction) and student F_theta reuse the DiffusionMNIST tiny time-conditioned U-Net wholesale (TNNetSinusoidalTimeEmbedding → shared cond MLP → TNNetFiLM per block, TNNetDeepConcat skips, TNNetUpsample decoder). The few-step sampler starts from σ(T)·N(0,I) on the scaled view and, for K>1, re-noises the x_0 estimate to the next lower timestep and re-evaluates f. Reports per-sampler mean nearest-train-digit MSE (a fidelity proxy — lower = closer to the data manifold) + a NaN check, and writes a PNG grid whose rows are teacher-multistep / 1-step / 2-step / 4-step. Reuses the DiffusionMNIST MNIST idx files (symlinked); falls back to a SYNTHETIC bar dataset if they are absent so it still runs in CI. Default SMOKE mode (short teacher pretrain + short distillation, ~3 min on CPU) is intentionally undertrained — like the DiffusionMNIST smoke its samples are rough, so the headline is that the 1/2/4-step consistency MSE tracks the multi-step teacher at a fraction of the steps; --full for sharper output. Reuses the DiffusionMNIST U-Net + TNNetDiffusionScheduler + TNNetEMAWrapper (no new layer).
• SDEdit image-to-image / real-image editing — a LATENT-DIFFUSION EDITING example (Meng et al. 2021), structurally distinct from every other generator in the tree, which all start from PURE noise: SDEdit starts the reverse process from a REAL image that has been encoded to a latent and then only PARTIALLY noised, so the source image's coarse layout survives while a NEW conditioning prompt steers the content — the classic "edit this picture" workflow. The whole pipeline reuses ALREADY-LANDED importers with no new model and no new layer: ① the VAE encoder (BuildVaeEncoderFromSafeTensors) maps the RGB image to a clean latent z0; ② the reusable TNNetDiffusionScheduler.AddNoise noises z0 up to an intermediate timestep t_start = round(strength·T) (the strength knob in 0..1: strength 0 keeps the source, strength 1 is full noise = ordinary text→image-from-noise); ③ the landed PixArt denoiser (BuildPixArtFromSafeTensors + PixArtDenoise) runs a TRUNCATED reverse DDIM trajectory from t_start down to 0 conditioned on the new prompt's T5 states, driven by the same scheduler Step loop the other diffusion examples use; ④ the VAE decoder (BuildVaeDecoderFromSafeTensors, latent /0.18215) returns the edited RGB. The ONLY new code is the encode→partial-noise→denoise→decode driver + the strength/steps knobs; nothing is hand-rolled. Because real SD/PixArt checkpoints are far too large for CI, the default offline SMOKE run wires up the committed tiny RANDOM importer-parity fixtures (tests/fixtures/tiny_vae_encoder|decoder|pixart), generates a synthetic color-ramp source, and exercises the full driver at several strengths in seconds under a tight RAM/time budget — the nets are untrained so the "edit" is not a meaningful picture; the point is that the pipeline runs end-to-end and writes edit_before.ppm (source VAE round-trip) / edit_after.ppm (edit), asserting no NaN/Inf decoded pixels. Point --vae-encoder / --vae-decoder / --pixart (with their --*-config) at real checkpoints to run it for real. Bridges the two independent fixture grids (8×8×4 VAE latent vs 6×6×4 PixArt sample) by center-cropping the latent for the denoise loop (a no-op with matched real checkpoints). The --inpaint flag turns the same driver into a diffusion INPAINTING run (the RePaint / SD-inpaint resample trick, Lugmayr et al. 2022): a binary mask over the latent grid marks the region to regenerate (1) vs keep (0), and BEFORE each reverse step the UNMASKED latent is overwritten with the clean encoded latent RE-NOISED (AddNoise) to that step's timestep, so the kept region always carries the correct noise level for the current step while only the masked region is driven by the denoiser — only the masked region is regenerated and the rest stays pixel-faithful. It adds a mask volume + a per-step composite to the existing loop (no new model); the smoke run asserts the unmasked output latent matches the source exactly and the masked region differs, and writes edit_mask.ppm alongside before/after. The diffusion-based sibling of the GAN context-encoder Inpainting example. Pure CPU; reuses the VAE/PixArt importers + TNNetDiffusionScheduler (no new layer).
• Blended-diffusion Stable-Diffusion inpainting — latent-space SD INPAINTING on the standard 4-channel SD UNet (Avrahami et al. 2022, Blended Diffusion): regenerate ONLY a masked hole while the rest stays pixel-faithful, with no 9-channel inpaint-specialized conv_in and no retrained inpaint weights — the most-used no-retrain SD inpaint path. The sibling of ImageToImage (it shares the encode→denoise→decode skeleton) but drives the landed SD UNet (BuildSDUNetFromSafeTensors + SDUNetDenoise) instead of PixArt and uses matched VAE/UNet latent grids (8×8×4) so there is no crop. The pipeline: ① the VAE encoder (BuildVaeEncoderFromSafeTensors) maps the RGB source to a clean latent z0; ② a binary RGB mask (1 = hole to reconstruct, 0 = keep) is downsampled to a per-voxel latent mask in {0,1}; ③ the new reusable driver SDUNetDenoiseInpaint (neuralpretrained.pas) runs a full reverse DDIM trajectory and at EVERY step blends the known region back in — latents = mask·denoised + (1−mask)·AddNoise(z0,t) (the visible region always carries the right noise level for t via the reusable TNNetDiffusionScheduler.AddNoise forward q_sample, so only the hole follows the denoiser; a final composite at t=0 pins the kept region to z0 exactly); ④ the VAE decoder returns the inpainted RGB. The ONLY new code vs SDEdit is the per-step latent blend + the latent mask — both inside SDUNetDenoiseInpaint; the 4-channel UNet, VAE encoder/decoder and scheduler are reused untouched. Because real SD checkpoints are far too large for CI, the default offline SMOKE run wires up the committed tiny RANDOM importer-parity fixtures (tests/fixtures/tiny_vae_encoder|decoder + tiny_sd_unet), generates a synthetic color-ramp source + a rectangular right-half hole, runs the blended loop in seconds under a tight RAM/time budget (the nets are untrained so the inpaint is not a meaningful picture), and writes masked_input.ppm (source round-trip with the hole greyed) / inpainted.ppm, asserting no NaN/Inf decoded pixels. TestDiffusionInpaintSmoke (TestNeuralPretrained.pas) asserts the KEPT latent region equals z0 exactly (max|diff| < 1e-4) while the MASKED hole was changed by the denoiser. Point --vae-encoder / --vae-decoder / --unet (with their --*-config) at real config-compatible checkpoints to run it for real. DEFERRED follow-ups: the 9-channel inpaint-specialized UNet (conv_in widened to latent|mask|masked-latent, the diffusers stable-diffusion-inpainting weights) and the non-deterministic reparameterized VAE sampling head. Pure CPU; reuses the VAE/SD-UNet importers + TNNetDiffusionScheduler (one new reusable driver, no new layer).
• Pix2Pix conditional image-to-image translation — a PAIRED image-to-image translation (Isola et al. 2017), distinct from the UNCONDITIONAL VisualGAN (noise → CIFAR): the generator is conditioned on an input image and emits a deterministic translation of it. On a synthetic generated-in-code grayscale → color task (random filled circles/rectangles on a dark background; the colorization rule is geometry-dependent — circles→red, rectangles→green, background→dark blue — so the net must both reconstruct the silhouette from grayscale AND infer each shape's color from its round-vs-straight edges, a genuine conditional translation, not a per-pixel lookup). The generator is a U-Net built by ONE TNNet.AddUNet call (the same builder as UNetSegmentation) with a Tanh output in [-1,1]; the loss is an L1 reconstruction term + an adversarial loss; the discriminator is a PatchGAN — a small fully-convolutional net scoring a grid of NxN overlapping patches as real/fake (composed from existing conv layers, NOT a new leaf class) rather than one global logit. Trains the standard alternating D/G GAN loop (reusing the VisualGAN wiring). Reports per-epoch held-out L1 and a color-accuracy proxy and renders an ASCII (input | target | generated) triplet plus a PPM. Pure CPU; the default SMOKE run (16×16, depth 2, 30 epochs, ~3.5 min) drives held-out L1 from ≈0.69 → ≈0.13 and color-accuracy to ≈0.88, with a --full flag for sharper output. Foundational conditional-generation recipe; reuses AddUNet + existing convs (no new layer).
• CycleGAN unpaired image-to-image translation — an UNPAIRED image-to-image translation (Zhu et al. 2017), the unpaired sibling of Pix2Pix: the two image domains are sampled from separate independent random draws, so no per-sample target exists and an L1-to-target loss is impossible. On a synthetic generated-in-code red shapes ↔ green shapes task (the same kind of filled circles/rectangles drawn in red for domain A and green for domain B, from independent seeds so a sample of A and a sample of B share no geometry — the learnable mapping is a content-preserving recoloring). Two generators G:A→B / F:B→A are each ONE TNNet.AddUNet call (the same builder as UNetSegmentation / Pix2Pix) with a Tanh output in [-1,1]; two PatchGAN discriminators (image alone, no condition stacked in — the adversary judges domain membership, not input/output consistency) trained with the least-squares GAN objective. The crux is the cycle-consistency loss λ_cyc·(|F(G(a))−a| + |G(F(b))−b|) (round-trip) plus an identity colour anchor λ_id·(|G(b)−b| + |F(a)−a|): the cycle term genuinely backprops through the composed F∘G / G∘F — F.Backpropagate on the cycle gradient updates F AND, via EnableErrorCollection on F's input, leaves d(cycle)/d(g) in F.Layers[0].OutputError, which is then fed into G's output error (the mirror for the backward cycle); both directions accumulate per step under SetBatchUpdate(true) before a single weight update. No simplification of the objective vs canonical CycleGAN (L1 cycle/identity via sign(·) sub-gradient, LSGAN — itself a standard CycleGAN choice); only the scale is toy. Reports per-eval cycle-reconstruction error (cycA/cycB), a translation-colour score, and the two LSGAN losses, and renders an ASCII (a|G(a)|F(G(a)) / b|F(b)|G(F(b))) panel plus a PPM. Pure CPU; the default SMOKE run (16×16, depth 2, 64 train / 32 test, 10 epochs, ~3.5 min) drives held-out cycle error from ≈0.68 → ≈0.05 and both translation scores above ≈0.93, with a --full flag for sharper output. Reuses AddUNet + existing convs (no new layer).
• VQ-GAN train-from-scratch (adversarial discrete autoencoder) — the GENERATIVE-CV upgrade of the reconstruction-only VQVAE: trains the SAME discrete autoencoder (conv encoder → TNNetVectorQuantizer codebook → conv decoder) but with the VQ-GAN objective of Esser et al. 2021 (Taming Transformers) — reconstruction + a perceptual / feature-matching term + the codebook commitment term + a PatchGAN adversarial term switched on only after a warmup. The adversarial + perceptual pressure is what turns blurry L2 VQ-VAE reconstructions into SHARP ones with crisp edges (the whole point of VQ-GAN over a plain-L2 VQ-VAE). Self-contained on a synthetic shapes task (filled circles + rectangles, grayscale in [-1,1], the same family as Pix2Pix); it trains BOTH a plain-L2 VQ-VAE and a VQ-GAN on the IDENTICAL data/architecture and directly compares a high-frequency edge-sharpness metric to demonstrate the payoff. Reuses TNNetVectorQuantizer (straight-through + commitment/codebook gradients), the PatchGAN + LSGAN loop + gradient-surgery trick from Pix2Pix/CycleGAN, and the LPIPS perceptual primitives from neuralpretrained.pas (feature-matching fallback over the discriminator's own hidden maps — no VGG download). Reports codebook usage + perplexity and renders an ASCII (original | L2-VQVAE | VQ-GAN) panel plus a PPM. No new layer class (the new code is the loss SCHEDULE + commitment balance). Pure CPU.
• Image inpainting (context encoder, free-form hole completion) — a FREE-FORM MASK COMPLETION example (Pathak et al. 2016, Context Encoders): fill a hole in an image from its surroundings. Distinct from both image-translation siblings — the UNCONDITIONAL VisualGAN (noise → image) and the PAIRED Pix2Pix (grayscale → color) — here the input and target are the SAME image: a random rectangular region is zeroed out and the network must hallucinate the missing pixels from the visible context. Self-contained, download-free synthetic colored-shapes scenes (a red circle + optional green rectangle over a blue gradient, $16 \times 16 \times 3$ RGB in [-1,1]; the CIFAR loader would drop in unchanged but a synthetic scene keeps the smoke run tiny and offline). The two pieces of new code are (1) a random rectangular-hole mask generator — the network input is the masked RGB stacked with the binary mask on the depth axis (4 channels [maskedRGB | mask], the standard context-encoder convention so the net knows exactly which pixels are missing), and (2) a masked-region-WEIGHTED reconstruction loss: L1 + (1−SSIM) (reusing the landed neuralimagemetrics.ComputeSSIMLossAndGradient helper exactly as FrameInterpolation does, per RGB channel) with pixels inside the hole weighted $6 \times$ vs $1 \times$ outside — the hole is the only part the net cannot trivially copy, so up-weighting it focuses learning where it matters (Pathak et al. use 10×; gentler here because the masked input zeroed the hole too). The custom per-pixel gradient is injected through the standard TNNet.Backpropagate path via the pseudo-target identity Desired = Output − GradOut. The network itself is a stock conv encoder-decoder + skip connections built by ONE TNNet.AddUNet call (same builder as Pix2Pix/UNetSegmentation) with a Tanh output — the skips carry visible context across the bottleneck so capacity is spent on the hole. An OPTIONAL adversarial term (--adv) adds a small PatchGAN discriminator (LSGAN) scoring the COMPLETED image (visible context composited with the generated hole), reusing the exact hand-rolled GAN gradient loop from Pix2Pix; it is off by default so the smoke run stays fast. Reports held-out L1/SSIM for the WHOLE image AND for the HOLE interior separately, renders an ASCII masked | reconstructed | original panel and writes the same triplet to inpainting_sample.ppm. Pure CPU (no LCL/image deps); the default SMOKE run (200 train / 40 test, 10 epochs) finishes in ~60 s and drives hole L1 ≈0.73 → ≈0.33 (SSIM ≈−0.02 → ≈0.44), with --adv and --full flags. The diffusion-based inpainting sibling (re-noise only the masked latent region, the RePaint/SDEdit trick — see ImageToImage) is a SEPARATE tracked task. Reuses AddUNet + the SSIM helper (no new layer).
• Neural Style Transfer — an optimise-the-pixels demo (Gatys, Ecker & Bethge, CVPR 2016): a frozen VGG-16 feature extractor (imported with BuildVGGFromSafeTensors) scores a CONTENT match (MSE between a deep tap's activations) and a STYLE match (MSE between Gram matrices of five taps relu1_2..relu5_x), and the canvas image is optimised by gradient DESCENT on the INPUT pixels — the perceptual-loss flip-side of GradientAscent. The new code is a plain ComputeGram helper (G = F·Fᵀ / (C·H·W) over a feature map) plus the style-gradient dL/dF it implies; no new layer class. Gradients are injected at the tap layers and a single manual Backpropagate() from the truncated net's last layer carries them to the input (the net is FROZEN via SetBatchUpdate(true) + never calling UpdateWeights, so only the pixels move). Self-contained and CI-runnable: with no --vgg/--content/--style it synthesises tiny 64×64 images and uses the committed tiny VGG fixture (pipeline demo — style loss 9.82→0.68 in 40 steps, ~0.1 s on CPU), and writes a stylized PNG; point --vgg/--config at real torchvision VGG-16 safetensors and pass real images for genuine artistic output. README documents the importer change that lets backprop reach the input. Pure CPU.
• AdaIN Style Transfer — fast arbitrary style transfer (Huang & Belongie, ICCV 2017): the feed-forward counterpart to the optimise-the-pixels StyleTransfer demo. The new code is the parameter-free two-source TNNetAdaIN layer — TNNetAdaIN.Create(ContentFeatures, StyleFeatures) instance-normalizes the content feature map per channel and re-scales/re-shifts it by the per-channel mean and std of the style feature map (out = style_std·(content − content_mean)/content_std + style_mean); its full input gradient flows to both sources and is numerically gradient-checked in the test suite. The example wires a shared shallow conv encoder → AdaIN → conv decoder, trains the decoder for a handful of iterations on a toy AdaIN-reconstruction objective, then stylizes a synthetic content image with a synthetic style image in a single forward pass (no per-image optimisation loop). Self-contained, no external weights or images; runs in well under a second on CPU under ulimit -v 3000000. Pure CPU.
• TinyNeRF differentiable volume renderer — a differentiable VOLUME RENDERER (Mildenhall et al. 2020, NeRF): an output modality distinct from every landed image generator (diffusion/GAN/VQ) and from the 2-D implicit-function fits (SIREN, FourierFeaturesSpectralBias) — those are 2-D pixel regressions, this is 3-D ray integration. A tiny coordinate MLP F(x,y,z) -> (r,g,b,sigma) (Input(3) -> TNNetFourierFeatures(M,sigma) positional encoding — the SAME primitive the Fourier example uses — -> FullConnectReLU(W) -> FullConnectReLU(W) -> FullConnectLinear(4)) is rendered into an image by casting rays from a pinhole camera pose, sampling points along each ray (fixed near/far linspace), evaluating F at every sample, and alpha-compositing: C = sum_i T_i (1 - exp(-sigma_i·delta_i)) c_i with transmittance T_i = exp(-sum_{j<i} sigma_j·delta_j). The new code is this compositing step and its hand-derived backward (the analytic gradient of the composite + transmittance + sigmoid-rgb/softplus-sigma activations w.r.t. each sample's raw MLP outputs, computed in an O(N) far-to-near suffix-sum sweep), fed to the last linear layer's OutputError and driven through the standard Backpropagate() path (last-layer IncDepartingBranchesCnt() once before the loop, ResetBackpropCallCurrCnt() per sample). It is plain-array math in the example driver — no new layer class. The scene is a synthetic analytic coloured sphere ray-marched with a KNOWN emission/density field to generate the posed ground-truth views deterministically inside the program (NO dataset download, NO committed binary images); the MLP learns to reproduce a handful of training poses, then renders a HELD-OUT pose it never saw and writes tinynerf_gt.ppm / tinynerf_pred.ppm (P6), reporting held-out PSNR before vs after training to prove the renderer learns (a flat output would mean the compositing gradient is wrong). Default SMOKE run (24×24 render, 5 poses, 8 samples/ray, ~1000 ray batches, ~1.5 min on CPU under ulimit -v 3000000) lifts held-out PSNR well above the untrained baseline; scale ImgRes/poses/samples/iters up for a sharper view. Establishes the ray-marching + alpha-compositing primitive future 3-D / view-synthesis work would build on. Pure CPU.

Attention & transformers

• Attention copy task — smallest possible end-to-end attention training demo: a single TNNetScaledDotProductAttention head learns to copy a 16-token input (vocab 8) to its output. Embedding + sinusoidal positional encoding + Q/K/V projection + SDPA + per-position softmax readout; trains on the fly in ~6 s CPU and reports 100% per-token accuracy on fresh probes.
• Causal-mask sanity (does the unmasked model cheat?) — trains the SAME tiny TNNetScaledDotProductAttention next-token model twice on a synthetic 2nd-order recurrence (tok[t]=(tok[t-1]+2*tok[t-2]+1) mod 6, target = the next token), differing only by SDPA's CausalMask flag (the strictly-upper-triangle -1e9 fill, the SDPA-internal equivalent of TNNetMaskedFill). The unmasked arm can peek at the future token it is asked to predict, so it drives train cross-entropy to ~0 (0.0002 vs the masked 0.92) and scores 100% teacher-forced — but under true autoregressive evaluation (future positions blanked) the cheat evaporates (26%) and the masked model wins (70%). Three built-in gates (Halt(1) on failure): unmasked train loss < masked (cheating confirmed), masked causal-eval accuracy > unmasked (it actually generalizes), and the unmasked head puts real attention mass on future keys (0.84) while the masked one puts ~0. Pure CPU, deterministic, fast.
• Positional-encoding demo — forward-only ASCII-heatmap inspection of the two additive position-encoding layers that ship with neural-api: TNNetSinusoidalPositionalEmbedding (Vaswani et al. sin/cos table) and TNNetAddPositionalEmbedding (despite the name, also a fixed sin/cos table). Builds a tiny (SeqLen=16, 1, Depth=16) model around each, feeds an all-zero volume so the output IS the encoding table, and renders it. Pure CPU.
• Position-encoding bake-off — trains the same tiny causal attention model four times on a predict-the-previous-token task, switching only the position scheme: (a) none, (b) sinusoidal TNNetAddPositionalEmbedding, (c) RoPE TNNetRotaryEmbedding, (d) ALiBi TNNetALiBi. Prints a final-loss table plus one sample prediction per scheme. Sinusoidal and RoPE drive the loss to ~0; the no-position arm is worst and ALiBi (a single-head recency bias, no positional content in the values) sits just above it — showing ALiBi targets locality, not fixed-offset addressing. Pure CPU, all four arms in ~15 s.
• CumSum position encoding — Part 1 is a forward-only demo showing that TNNetCumSum applied to a constant [1, 1, ..., 1] depth channel produces a strict linear position ramp [1, 2, 3, ...], ready to be concatenated alongside real features via TNNetConcat. Part 2 is a train-time bake-off on a permutation-invariant "find the marker's position" task: the SAME order-agnostic model (PointwiseConvReLU -> MaxChannel -> FullConnect) is trained twice, differing only in whether the CumSum position ramp is concatenated to the input — the no-position arm sits at chance (~13%) while the CumSum arm solves it (~89%), proving the feature carries the positional signal. Pure CPU, ~19 s.
• RoPE base-frequency sweep — empirically interrogates the cargo-culted 10000 in RoPE's per-pair frequency theta_i = base^(-2i/d). Trains the same tiny single-head causal attention model once per base ∈ {1e2, 1e3, 1e4, 1e5} (the only knob is TNNetRotaryEmbedding.Create(base)) on a position-sensitive copy-the-token-3-steps-back task and prints a train/val-loss + accuracy table. Every arm solves the task (100% val accuracy), but the final loss is monotone in base — the smaller base resolves this short-range offset more sharply and 10000 is beaten by 100 and 1000 — showing the base is a tunable inductive-bias knob, not a sacred constant. Pure CPU, all four arms in ~49 s, no downloads.
• ALiBi slope-base sweep — empirically interrogates the cargo-culted 8 in ALiBi's per-head slope 2^(-Base*(h+1)/H). Trains the same tiny single-head causal attention model once per Base ∈ {4, 6, 8, 12} (the only knob is TNNetALiBi.Create(Base)) on a recency-sensitive copy-the-most-recent-vowel task and prints a train/val-loss + accuracy table. On this task the trend is monotone — flatter slopes (smaller Base) win and 8 is beaten by 4 and 6 — showing the slope base is a tunable inductive-bias knob, not a sacred constant. Pure CPU, all four arms in ~17 s, no downloads.
• Causal-mask + logit SoftCapping interaction study — sweeps the logit soft-cap c ∈ {5, 10, 20, 30, ∞} (the ∞ arm omits the TNNetSoftCapping layer entirely) on the SAME tiny single-head causal next-token model (shared seed/arch/data across arms), asking what soft-capping c*tanh(x/c) actually does to a softmax's input. It reports TWO logit norms because measuring one is misleading: the EFFECTIVE post-cap norm is always ≤ c (that is the bound SoftCapping buys), but because tanh saturates the net responds by INFLATING its RAW pre-cap logits (raw-norm balloons to ~51 at c=5 vs ~5 uncapped) — the cap conditions the softmax input without taming the underlying projection. Headline lesson: "logit norm" depends on where you measure. Pure CPU, all 5 arms in ~10 s.
• Cosine-attention learnable scale — turns the scale scalar of TNNetCosineSimilarityAttention into a single learnable parameter (the ReZero single-weight pattern; init 1.0) and trains it on a tiny synthetic next-token task to check whether training drives it toward the cargo-culted 1/τ temperatures (scale ≈ 10..20) that cosine-attention papers hard-code. On a task that rewards a sharp softmax (each query must copy exactly one key) the learned scale climbs 1.0 → ~6.7 (still rising, loss → 0); on a smooth task where sharpening does not help it instead collapses toward 0 (uniform attention) — so the learnable scale adapts to the data rather than to a fixed constant. Drives the layer directly (Compute/Backpropagate), pure CPU, no downloads.
• Linear attention scaling probe — wall-clock scaling probe for the new softmax-free TNNetLinearAttention layer (Katharopoulos et al. 2020, Transformers are RNNs). It replaces softmax(QKᵀ)V with a positive feature map φ(x)=elu(x)+1 and exploits associativity to accumulate a d_k×d_v key-value matrix once, so cost is O(SeqLen·d_k·d_v) — linear, with no SeqLen×SeqLen score matrix ever formed. Times one forward pass at SeqLen ∈ {16,32,64,128,256} and prints a table showing ~2× (linear), not ~4× (quadratic), growth per sequence-length doubling.
• Linformer low-rank attention — the headline demo for the new TNNetLinformerAttention layer, Linformer self-attention (Wang et al. 2020, Linformer: Self-Attention with Linear Complexity, arXiv:2006.04768). Single-head self-attention that first projects the Key and Value sequences DOWN along the sequence axis from SeqLen to a small fixed rank k ≪ SeqLen with two LEARNABLE projection matrices E, F (each k×SeqLen): K' = E·K, V' = F·V, then Attn = softmax(Q·K'ᵀ / √d_k) (a SeqLen×k score matrix, not SeqLen×SeqLen) and Out = Attn·V', making attention O(SeqLen·k) instead of O(SeqLen²). Same Q|K|V input contract as TNNetScaledDotProductAttention/TNNetLinearAttention (SizeY=1, input depth 3·d_k, output depth d_v=d_k); E,F are stored as two trainable neurons with exact finite-difference-checked input AND weight gradients. Because E,F carry a fixed SeqLen dimension the layer requires a FIXED SeqLen (asserted in SetPrevLayer), the standard Linformer constraint. Distinct from the kernel/feature-map linear-attention family (TNNetLinearAttention φ(x)=elu(x)+1, TNNetGatedLinearAttention, TNNetDeltaNet, TNNetWKV): Linformer keeps the softmax and instead low-rank-projects the sequence. The demo contrasts the Linformer arm against a full quadratic TNNetScaledDotProductAttention arm at the SAME SeqLen on a majority-value classification task (predict the most frequent label — a global aggregate, the regime where attention is approximately low-rank): both arms share a $1 \times 1$ Q|K|V projection → attention → $1 \times 1$ readout → per-position softmax, differing ONLY in the attention layer. Headline: at SeqLen 17 the Linformer arm learns the task well above chance (20%) using a score matrix of just 23.5% the size of full SDPA ($17 \times 4$ vs $17 \times 17$) — the classic Linformer trade of a little accuracy for LINEAR attention cost. Pure CPU, ~5 s.
• Performer / FAVOR+ random-feature attention — the headline demo for the new TNNetPerformerAttention layer, Performer self-attention (Choromanski et al. 2020, Rethinking Attention with Performers, arXiv:2009.14794). Unlike TNNetLinearAttention (deterministic φ(x)=elu(x)+1, a different kernel) Performer uses positive random features (FAVOR+) to give an unbiased, low-variance estimate of the actual softmax kernel exp(q·k) at linear cost: for an m×d_k FROZEN (non-trainable) random projection W, φ(x)=exp(W·x−‖x‖²/2)/√m so that E[φ(q)·φ(k)]=exp(q·k). Attention then reassociates like the kernel family — S=Σ_s φ(K_s)⊗V_s (m×d_v), Z=Σ_s φ(K_s), Out_t=(φ(Q_t)·S)/(φ(Q_t)·Z) — at O(SeqLen·m·d_v) with no SeqLen×SeqLen score matrix. W's rows are i.i.d. N(0,1) and, when m≥d_k, orthogonalized block-by-block (Gram–Schmidt + chi-norm rescale) for the lower-variance "+" in FAVOR+. Same Q|K|V contract as the sibling attention layers (SizeY=1, input depth 3·d_k, output d_v=d_k). W is frozen (no weight gradient) but dL/dQ, dL/dK ARE backpropagated through φ (chaining both the W·x and −‖x‖²/2 terms — input gradient finite-difference checked); d_k, m and the RNG seed round-trip via FStruct[] so the frozen W reloads bit-identically (verified by a save/load round-trip test). Part 1 reproduces the headline FAVOR+ claim: averaged over many random seeds, the RMS error between Performer and full TNNetScaledDotProductAttention SHRINKS as m grows (≈0.118 at m=4 → ≈0.060 at m=128, with Q/K pre-scaled by d_k^{−1/4} so both arms use the same kernel). Part 2 trains the layer as a drop-in attention block on a majority-value task, reaching 55.7% (chance 20%). Distinct from TNNetLinformerAttention (keeps softmax, low-rank-projects the sequence axis) and the deterministic kernel family. Pure CPU, <1 min. Covered by TestPerformerAttentionInputGradientCheck/TestPerformerAttentionSerializationRoundTrip.
• Latent Attention - Multi-head Latent Attention (MLA, DeepSeek-V2, Liu et al. 2024) via TNNet.AddMultiHeadLatentAttention: unlike GQA (which shares K/V across query-head groups) MLA low-rank-factors the K/V projection through a tiny shared latent c_KV := x·W_DKV (width d_c << d_model), then reconstructs per-head K/V by up-projections — so the cacheable per-token state shrinks from 2·d_model to just d_c, a saving d_c/(2·d_model) independent of head count. A three-arm next-token copy bake-off (NoPE MLA, decoupled-RoPE MLA via the builder's RopeDim parameter — per-head rotated rope-Q plus ONE rope-K projection shared across all heads, with NO absolute positional embedding so position enters only through the rope slice — and a param-matched AddMultiHeadSelfAttention) shows MLA solving the task at a comparable/lower weight budget. Then the headline KV-cache incremental-decode win is demonstrated and measured: (a) token-at-a-time decode through the SDPA BeginIncrementalDecode cache machinery (RoPE arm uses TNNetRotaryEmbedding.PositionOffset := t per step) and (b) a TRUE latent-only cache whose per-token state is just d_c floats (re-running the up-projections over the cached latents per step) — both match the full re-encode to < 1e-5, and the printed table compares cache bytes/token (32 B latent vs 192 B MHA K+V at d_model=24). Pure CPU, ~35s.
• Grouped-Query Attention - GQA/MQA via TNNet.AddMultiHeadGroupedQueryAttention(QueryHeads, KVHeads, CausalMask) (Ainslie et al. 2023, arXiv:2305.13245): QueryHeads query heads share only KVHeads key/value projection heads -- the K and V token-wise projections shrink from QueryHeads*d_k to KVHeads*d_k channels, a factor QueryHeads/KVHeads fewer K/V projection parameters. KVHeads=QueryHeads degenerates to plain MHA (numerically identical to AddMultiHeadSelfAttention given the same projection weights); KVHeads=1 is Multi-Query Attention (Shazeer 2019, arXiv:1911.02150). The demo trains three arms differing ONLY in KVHeads (4=MHA, 2=GQA, 1=MQA) on the SAME content-based key->value recall stream and prints the per-arm parameter counts plus held-out recall MSE / exact-recall accuracy, reproducing the GQA paper's headline on a toy: recall quality stays competitive as the K/V heads are shared away. GQA nets stream through TNNetStreamingDecoder unchanged (one cached SDPA per query head). Pure CPU, well under a minute. Covered by TestMultiHeadGroupedQueryAttention* and TestStreamingDecoderGQAMatchesFullForward in the test suite.
• Sink-attention stability — attention-SINK stability micro-experiment for TNNetSinkAttention, the "attention sink" idea from StreamingLLM (Xiao et al. 2023). Softmax attention must distribute a full unit of probability mass over the keys; a learnable sink token absorbs the leftover mass so the real keys are not forced to soak up noise. Pure CPU.
• Sliding-window vs full-causal attention bake-off - The same tiny causal next-token model trained three times, swapping only the mask layer: TNNetSlidingWindowMaskedFill(2), (4), and full-causal TNNetMaskedFill. On a content-gated copy task whose answer lives in a width-2 window, both sliding arms match FULL's ~0 loss while inspecting far fewer keys per query (mean 1.92/3.50 vs 6.50) — charting the long-context cost/quality trade. Graded (PASS/FAIL), pure CPU
• Differential-attention noise cancellation — the headline noise-cancellation micro-experiment for TNNetDifferentialAttention (Differential Transformer, Ye et al. 2024). Softmax attention must spend a full unit of probability mass on the keys even when none is relevant; differential attention subtracts two softmax maps to cancel that common-mode noise. Shows the attention noise on an all-keys-irrelevant query position falling versus plain attention. Pure CPU.
• Induction Heads - A tiny 2-layer causal attention transformer spontaneously forms an induction head that does in-context copying (Olsson et al. 2022). On a sequence repeated against itself the model learns content-based prefix-matching: second-half (repeated) accuracy ~100% vs first-half ~chance, and the layer-2 attention matrix shows the induction stripe directly as an ASCII heatmap. Pure CPU, ~17s.
• Tiny Transformer FFN - The feed-forward half of a transformer block (no attention): a stack of AddRMSNormResidual([ AddSwiGLUFeedForward(...) ]) pre-norm residual blocks on a per-token denoising toy. Shows the SwiGLU + RMSNorm + residual FFN composes into a stable deeper stack and reconstructs a clean nonlinear target from a noised input (~6x over the do-nothing baseline). Pure CPU, ~15s.
• Transformer Decoder Block - Wires an encoder memory into a full encoder-decoder decoder block (TNNet.AddTransformerDecoderBlock): causal self-attention + cross-attention + SwiGLU FFN residuals. A tiny end-to-end forward/backward demo on the decoder grid. Pure CPU.
• TinyGPT — char-level GPT, end-to-end — the capstone decoder-only (GPT-style) transformer demo, built only from existing library blocks: one-hot char Input(24,1,128) -> PointwiseConvLinear(64) token projection -> AddPositionalEmbedding -> 2x AddTransformerEncoderBlock(Heads=4, d_ff=64, CausalMask=true) (causal multi-head self-attention + SwiGLU FFN, pre-norm residuals) -> FullConnectLinear(128) -> SoftMax (43 layers, ~188k weights). A GPT block is just causal self-attention + FFN, so the causal-masked encoder block builder is exactly a decoder block (the repo's AddTransformerDecoderBlock is an encoder-decoder block needing cross-attention, which a GPT does not). Trains next-char prediction on a tiny embedded "quick brown fox / lazy dog" corpus (no downloads), streams the dropping loss (accuracy ~0.01 -> ~0.29, loss ~5.2 -> ~3.5 over 8192 examples / 8 epochs, ~1.3 min on a free CPU; the loss falls monotonically and keeps dropping past ~1.0 with more epochs), then autoregressively generates corpus-style continuations from seed prompts with TNNetSamplerTopP (the 8-epoch samples already reproduce corpus n-grams like og/ox/fog/for). Honest headline: it memorizes the tiny corpus (the expected, acceptable outcome for a sub-5-minute pure-CPU char-LM). The pure-transformer sibling of SimpleNLP (conv/transformer char-LM from a file). After training it also prints a teacher-forced corpus perplexity via PerplexityFromChars from neural/neuralnlpmetrics.pas (perplexity / mean NLL / bits-per-char over the 8 unique sentences, EOS targets excluded; 128 = uniform/untrained, ~1 = perfect memorization).
• Sequence packing for LM pretraining — the headline demo for the new TNNetSequencePacker (neural/neuraldatasets.pas): instead of padding every short document to the context length, documents are concatenated (each followed by one separator token 1) and cut into full ContextLen windows — the GPT-2/GPT-3 pretraining recipe and a large throughput win on pad-heavy corpora. Three layouts: pmSplitAcrossWindows (stream packing, documents may split across window boundaries, only the final partial window padded), pmNoSplitGreedy (greedy bin fill, documents never cross a boundary) and pmOneDocPerWindow (the classic padded baseline). The packer also produces the per-position loss mask: GetTrainingPair leaves pad-target rows all-zero and ApplyLossMask copies the actual output into the desired output at masked positions, so with the framework's e = Output - Desired convention exactly zero gradient flows from padding (verified to 0 in tests/TestNeuralPacking.pas). The demo trains the SAME tiny causal RoPE transformer (per-position softmax head, TNNetDyT norms so no cross-token statistics leak) three times for the SAME 400 optimizer steps on a templated word corpus with held-out sentences, re-shuffling + re-packing every epoch (fixed document order lets the model memorize the stream): padded 33.3% window utilization -> held-out PPL 5.41; no-split packed 73.3% -> 5.22; stream-packed 100% utilization -> PPL 4.35 (neuralnlpmetrics.Perplexity, same wall-clock per step). Attention may cross document boundaries inside a packed window (no per-sample dynamic SDPA masks; standard GPT-2/3 behaviour, noted follow-up in tasklist.md). Pure CPU, ~1 s.
• Sequence reverse (attention learns a permutation) — the classic "copy/reverse" probe: a tiny single-block self-attention model learns to output the REVERSE of its input token sequence ([a..h] → [h..a]), showing that attention can learn a pure POSITIONAL permutation. Stack: TNNetEmbedding → TNNetSinusoidalPositionalEmbedding (the position signal reversal needs) → one non-causal block (PointwiseConvLinear(3*d_k) per-token Q|K|V → TNNetScaledDotProductAttention(CausalMask=False) → PointwiseConvLinear(d_model) → residual TNNetSum) → per-position readout PointwiseConvLinear(Vocab) → TNNetPointwiseSoftMax(1), trained with a hand-rolled per-position cross-entropy loop. Reaches 100% per-token AND 100% exact-sequence accuracy on 1000 held-out sequences (chance = 8.33%); CE collapses 2.58 → <0.001. Per-token projections use PointwiseConvLinear (not FullConnect, which would flatten/mix the sequence). Pure CPU, ~2 s.
• Token Merging (ToMe) — drop half the tokens, keep the accuracy — the headline demo for the new weightless TNNetTokenMerging layer (Token Merging, Bolya et al. 2023, ICLR, arXiv:2210.09461) + the AddToMeTransformerBlock builder. ToMe shortens a (SeqLen,1,Depth) token sequence to a static (SeqLen-R,1,Depth) by bipartite soft matching: split tokens into alternating A/B sets, score each A-token's cosine similarity to its best B-token, merge the top-R A→B pairs by size-weighted averaging (the paper's proportional bookkeeping), and pass the rest through — with zero trainable parameters (like TNNetSinkhorn). It is distinct from every other sequence reducer in tree: AddAttentionPooling/AddPerceiverEncoder learn fixed query slots, AddMixtureOfDepths routes/skips tokens, and the pooling layers collapse the whole axis — ToMe instead fuses redundant tokens. Backward is the plain weighted-average adjoint; the top-R selection is frozen per forward pass (like MaxPool's argmax). A small SDPA classifier with one ToMe layer inserted before the deep block drops 50% of its tokens (64 → 32) while accuracy holds (66.3% → 74.5%) and wall-clock falls (~62 s → ~36 s), at no extra parameter cost. Pure CPU.
• Span-corruption pretraining (T5/BART objective, from scratch) — the natural end-to-end demo for neuraldatasets.TNNetSpanCorruptionCollator (the T5/SpanBERT contiguous-span masking collator): a tiny encoder-decoder pretrained FROM SCRATCH on the span-corruption objective. The collator masks contiguous WORD-level spans, collapses each masked span in the ENCODER input to one unique sentinel id (<extra_id_0>, <extra_id_1>, … at the TOP of the vocabulary, descending), and the DECODER is trained to emit the dropped spans as the sentinel/span stream <0> span0 <1> span1 … <final>. Both branches live in ONE TNNet (two TNNetInput layers, fed together with the array form of TNNet.Compute) so the loss back-propagates end-to-end through the cross-attention into the encoder — the from-scratch pretraining graph, not the importer's hand-filled-encoder-states convention. Architecture: TNNetTokenAndPositionalEmbedding → 2× AddTransformerEncoderBlock (bidirectional) → LayerNorm = encoder states; decoder TNNetTokenAndPositionalEmbedding → 2× AddTransformerDecoderBlock (causal self-attn + cross-attn over the encoder states) → LayerNorm → per-token PointwiseConvLinear(Vocab) → TNNetPointwiseSoftMax(1) LM head. Teacher forcing with a right-shifted decoder; the padded decoder tail is loss-masked by copying the model's own output into those FDesired rows (zero seed, the ApplyLossMask idiom). The corpus is a small strongly deterministic lexicon (the fox chases the rabbit at dawn, the owl hunts the mouse at night, …) where each subject fixes its verb/object, so a masked word-span is recoverable from context — exactly the regularity span corruption is meant to capture (char-level masking of an ambiguous corpus, by contrast, collapses to a constant output — the wrong demo). The inlined greedy seq2seq decode (the DecodeSeq2SeqGreedy convention: argmax of the per-token logits, autoregressive, padded-causal) reconstructs the held-out masked spans: per-token CE falls ~2.0 → ~0.04, and the reconstructions are clearly INPUT-DEPENDENT (fox→chases, owl→hunts, whale→swims/ocean, …) hitting exact-match on ~10/12 demo lines (~95% per-token target accuracy) — the few misses are honest near-ties (an ambiguous time preposition at vs in). ~223 K weights; the whole build + 12000 train steps + decode demo runs in ~2 min of pure CPU inside ulimit -v 3000000.
• Conformer (convolution-augmented transformer) — the headline demo for the new TNNet.AddConformerBlock(Heads, d_ff, ConvKernelSize) builder, the convolution-augmented transformer block of Conformer (Gulati et al. 2020, Conformer: Convolution-augmented Transformer for Speech Recognition, arXiv:2005.08100). A "macaron" block that sandwiches a multi-head self-attention module (GLOBAL mixing) and a convolution module (LOCAL mixing) between two HALF-step feed-forward modules, every sub-module a pre-norm residual, with a final LayerNorm: x += 0.5·FFN(x); x += MHSA(x); x += Conv(x); x += 0.5·FFN(x); x := LayerNorm(x). It is a builder composed entirely from existing serializable primitives (TNNetLayerNorm, TNNetPointwiseConvLinear per-token projections, AddMultiHeadSelfAttention, TNNetGLU conv gating, TNNetCausalConv1D 1-D conv over the time axis, TNNetSwish/SiLU, TNNetSum residuals, TNNetMulByConstant(0.5) macaron scaling), so it needs no new leaf class and round-trips through SaveToString/LoadFromString; shape-preserving over (SeqLen,1,d_model) so blocks stack. (The paper's per-channel depthwise conv has no 1-D-over-sequence primitive in tree, so TNNetCausalConv1D — a channel-mixing 1-D sequence conv — is the documented stand-in.) The toy is a per-token tag = (adjacent bigram (3,4) at t) XOR (far token S[0]==1): the bigram is a LOCAL pattern only the conv can localise, S[0] is a LONG-RANGE bit only attention can route, and the XOR forces the model to use both pathways (no pooling — a per-token softmax keeps every position's gradient alive). Headline (asserted, gate > 90%): one block reaches 97% overall with 93% at local-bigram positions (conv pathway) and 84% on global-bit-set sequences (attention pathway). Covered by TestAddConformerBlockShape/TestAddConformerBlockSerializationRoundTrip/TestAddConformerBlockGradientFlow in the test suite. Pure CPU, ~8 s.

Generation & decoding strategies

• Beam-Search Decoding — sequence-level deterministic decoding (neuraldecode unit: DecodeGreedy / DecodeBeamSearch / DecodeBeamSearchAll), the missing counterpart to the per-token stochastic TNNetSamplerGreedy/TopK/TopP family. Beam search keeps the B highest cumulative-log-probability partial sequences (log-space, summed — never multiply raw probs), so it can RECOVER from a locally-greedy first-token mistake that single argmax locks in forever. The demo trains a tiny char-level next-token model on a SYNTHETIC corpus deliberately built so greedy dead-ends (the locally-likeliest first token opens a high-entropy branch; a slightly-less-likely token opens a near-deterministic high-probability tail), then prints a Greedy-vs-Beam(B=2,4,8) total-log-prob table, the Wu et al. 2016 length-penalty α contrast (α=0 short-biased vs α>0), and a diversity contrast against the TopK/TopP samplers. Pure CPU.
• Speculative Decoding - Speculative sampling (Leviathan et al. 2023 / Chen et al. 2023) on a tiny pure-CPU draft/target pair: a small fast DRAFT proposes a block of K tokens, the big TARGET scores them in one batched pass, and each token is ACCEPTED with prob min(1, p_target/p_draft) or, on first rejection, resampled from the renormalised residual max(0, p_target - p_draft). The committed distribution is provably the target's, exactly — pinned bit-for-bit in the degenerate draft==target case (mandatory Halt(1) gate) and confirmed empirically with a real imperfect draft (histogram TV ~0.02). A draft-quality sweep shows the accept rate (hence big-model calls saved, ~70→80%) rising with draft/target agreement. Forward-only (recomputes the prefix each pass; KV-cache is the follow-up). Pure CPU, ~70s.
• Multi-Token Prediction (MTP) — the headline demo for the new TNNet.AddMultiTokenPrediction builder, Multi-Token Prediction (Gloeckle et al. 2024, Better & Faster LLMs via Multi-token Prediction; scaled up by DeepSeek-V3). Instead of predicting only the next token, the builder taps the shared trunk hidden state (SeqLen,1,d_model) and attaches NumFuture parallel per-token heads, head h forecasting the token at t+1+h — each head is PointwiseConvLinear(Vocab) → TNNetPointwiseSoftMax (token-wise 1×1 convs, NOT FullConnect, so the sequence axis survives), and the heads are TNNetDeepConcat'd into one (SeqLen,1,NumFuture*Vocab) output where slab h is the t+1+h distribution. Supervise it with a matching target whose slab h at position t is the one-hot of token t+1+h; the framework's default (output−target) seed gives the standard per-head cross-entropy, so plain Backpropagate trains all heads at once — densifying the training signal (every position now carries NumFuture losses, not one). It is a builder composed from existing primitives (no new leaf class), distinct from the two-net draft+verify SpeculativeDecoding and from the decode-time repetition penalty TNNetTokenHistoryPenalty — MTP is ONE net with parallel future heads. The toy rule is a deterministic arithmetic progression token[t]=(start+t·step) mod V (step∈{1,2,3,4}); two arms share an identical causal trunk and the same data/eval streams per seed, differing only in the MTP arm's extra t+2/t+3 heads. Headline (asserted): at an EARLY checkpoint averaged over 8 seeds the MTP arm reaches higher next-token (t+1) accuracy (mean ≈75% vs ≈70%, ahead in 7/8 seeds) — the auxiliary future losses accelerate the primary head (the win is about convergence SPEED; train to convergence and the easy rule saturates for both). At inference the extra heads are reusable for self-speculative decoding (draft NumFuture tokens in one pass, verify next pass — no separate draft net). Pure CPU, ~12 s.
• Self-Speculative Decoding — speculative decoding from one model: the extra future heads of a TNNet.AddMultiTokenPrediction model act as their own draft, dropping SpeculativeDecoding's second draft net entirely (the MTP-as-draft trick deployed by DeepSeek-V3). Each pass forwards the committed prefix plus the pending drafts ONCE: head-0's greedy argmax at row r verifies the draft at position r+1 (accept the longest prefix of matches), the first mismatch is corrected from the same row (a rejection still commits one token), a full accept yields a bonus token, and heads 1..NumFuture-1 at the last committed row draft the next block — so each forward commits 1..NumFuture tokens. Greedy speculative decoding is exact: every committed token is head-0's argmax over fully-committed causal context, so the output is asserted IDENTICAL to plain one-token-per-forward greedy decoding on every run (Halt(1) gate). On a heavily-overfit char-level toy (vocab 25, NumFuture=4): per-distance accept rates 68.5% / 79.5% / 89.7% (t+2/t+3/t+4; later rates are conditional on earlier accepts), 2.63 tokens committed per pass, 60.9% of forward passes saved (225 vs 576), 2.04× wall-clock (forward-ratio 2.56×). Deterministic smoke test in tests/TestNeuralNumerical.pas (TestSelfSpeculativeDecodeGreedyExactness). Forward-only; composing with the SDPA KV-cache (IncrementalDecode) is a separate open task. Pure CPU, ~75 s.
• Early-exit / self-speculative decoding (LayerSkip / CALM) — self-speculative decoding from one model with no second checkpoint and no separate prediction head: the model becomes its own draft by reading logits at an intermediate layer through its own LM head (the frozen-body LogitLens splice — snapshot the exit layer's activation, copy it into the head-input slot, recompute ONLY the head sub-stack → p_exit). The library routine DecodeEarlyExitSelfSpeculative (neural/neuraldecode.pas) drafts argmax(p_exit) when the early exit is confident (max p_exit ≥ Confidence, the LayerSkip/CALM static gate) and verifies it against the full-depth argmax (exact-greedy verify, exactly as SpeculativeDecoding does for a separate draft net). The emitted token is always the full-depth argmax, so the accepted sequence is bit-identical to plain greedy — the early exit only changes how much tail-layer work a cached decoder could skip, never the output. Distinct from SelfSpeculativeDecoding (drafts from MTP prediction HEADS) and EarlyExitNetwork (a BranchyNet CLASSIFICATION demo with trained auxiliary heads): here the draft is the model's own intermediate-layer readout. The example trains a small constant-width char-level LM on a deterministic corpus, asserts the self-speculative continuation equals plain greedy bit-for-bit (Halt(1) gate on mismatch), and reports the accept/reject counters, acceptance rate, and tokens/sec for both paths. Counters from one run: steps=16 drafts=16 accepted=2 rejected=14 → 12.5% acceptance, full-depth ≈1500 tok/s vs self-speculative ≈1330 tok/s (v1 has no cached tail-skip yet, so it does the full forward PLUS a head-only splice and is intentionally a touch slower — the acceptance rate is the speed signal: with the cached tail-skip each accepted token would skip the tail layers, turning acceptance directly into speedup at bit-identical output). Regression test TestEarlyExitMatchesGreedyBitIdentical (+ …HighConfidenceMatchesGreedy, …AcceptCountsAreConsistent) in tests/TestNeuralDecode.pas. Open follow-up: per-token-adaptive exit + the cached tail-skip. Pure CPU, ~1 min.
• Constrained (structured) decoding — the TNNetTokenConstraint "allowed next tokens" hook (neural/neuraldecode.pas) that the streamed loop applies to the post-softmax row before the sampler. The same tiny untrained char-level net is decoded three ways: free (character soup), JSON mode (TNNetJSONConstraint — a char-level JSON pushdown automaton makes even an untrained model emit ONLY valid JSON, EOS gated until a complete value stands), and forced sequence (TNNetForcedSequenceConstraint multiple-choice over yes/no/maybe). Every JSON sample is re-checked through a fresh automaton. Pure CPU, seconds.
• Structured output (schema-constrained / function-calling decoding) — extends ConstrainedDecoding from a hardcoded free-form JSON grammar to a user-supplied JSON Schema. CompileJSONSchemaToGBNF (neural/neuraldecode.pas) compiles a JSON Schema into a GBNF grammar the existing TNNetGrammar consumes, and CreateJSONSchemaConstraint wraps it in a TNNetGrammarConstraint. On a get_weather(location, days, unit) tool-call arguments schema (two required fields + an optional enum, additionalProperties:false) the same untrained char-level net can ONLY emit JSON that validates against the schema — right keys in declared order, right value types, nothing extra — every sample re-checked through a fresh TNNetGrammar machine and parsed as JSON. Covers the practical subset (object/array/string+enum+pattern/number/integer/boolean/null, anyOf/oneOf, $ref/$defs recursion). Pure CPU, seconds.
• Decode-Efficiency Features Bakeoff — SimpleNLP/DecodeFeaturesBakeoff.lpr: a 6-phase, 270-seconds-per-phase benchmark of the decode-efficiency features on a REAL tokenized NLP workload (TinyStories, 3k-token vocab — dataset download required, see the SimpleNLP README). Each phase time-boxes training (~190 s, enough for a clear convergence clue) then benchmarks decoding with a hard token-exactness assert (Halt(1) on divergence) between the full re-encode and streamed arms: (1) SDPA KV-cache incremental decode, 20–38× ms/token; (2) MTP-heads self-speculative decode, 0.53 target-fwd/token, 1.90× wall-clock; (3) DiagonalSSM O(1)-per-step decode, 22–65× with FLAT step cost vs prefix length; (4) MLA decoupled-RoPE cached decode, ~25× plus the latent-cache economics table (160 B/token analytic = 15.6% of MHA K+V); (5) speculative decoding composed with the KV cache (TruncateCache rollback) — verification really halves target forwards (0.54), yet plain KV-cached decoding still wins wall-clock on CPU (compute-bound verify), the day's headline negative result; (6) the hybrid the other phases recommend — 2×DiagonalSSM + 1×MLA(latent 32, rope 8) with DyT + SwiGLU and token-only embedding, whose streamed loop drives BOTH mixer families at once (SSM O(1) state + SDPA caches + rope PositionOffset) — final loss 1.39 vs 2.34/2.53/2.87 for the pure SSM/MLA/transformer stacks at the same wall-clock budget, decoding at ~0.35 ms/token, ~31×, flat in prefix length. Run one phase (--phase N) or all six (run_decode_bakeoff.sh). Pure CPU, ≤270 s per phase.
• Abstractive summarization with imported BART — the demo for BuildBartFromSafeTensors (neuralpretrained.pas), the dominant pretrained encoder-decoder for summarization (model_type bart: facebook/bart-large-cnn, sshleifer/distilbart-cnn-12-6). BART is a bidirectional BERT-style encoder + GPT-2-style causal decoder with cross-attention, returned as the same TWO-net pair as the T5/Marian importers (run with RunT5; decode autoregressively with the landed DecodeSeq2SeqBeamSearch). The importer reuses the Marian POST-norm block skeleton with BART's deltas: LEARNED absolute positions with the +2 padding offset (token position p reads embed_positions row p+2), a layernorm_embedding LayerNorm after the token+position embeddings, exact-erf GELU FFN (the BERT Phi+ReGLU composition), scale_embedding off, decoder_start = eos (BART's shift), and the shared embedding tied to the lm_head plus a final_logits_bias row. End to end: the checkpoint's GPT-2 byte-level BPE tokenizer.json (already read by TNeuralHFTokenizer — no SentencePiece) encodes the article as bos … eos, token-id beam search writes the summary, and ROUGE-1/2/L F1 (neuralnlpmetrics) score it against a reference. Runs on a built-in demo article with no input files, or point -af/-rf at your own. Pico parity fixture: encoder hidden 1.5e-6 / decoder logits 3.6e-6 vs the float64 HF oracle (TestBartParity, generator tools/bart_tiny_fixture.py). Needs a real BART download for a meaningful summary; keep -enc/-dec/-beam small on CPU. Pegasus (model_type pegasus: google/pegasus-xsum, pegasus-cnn_dailymail) is the close PRE-norm cousin and rides the SAME two-net + RunT5 + DecodeSeq2SeqBeamSearch path via BuildPegasusFromSafeTensors — its deltas vs BART are pre-norm blocks (LayerNorm before each sub-layer), STATIC half-split sinusoidal positions (the Marian table builder, no +2 offset), no layernorm_embedding, a FINAL encoder and decoder layer_norm closing each pre-norm stack, and scale_embedding on. The model import is parity-verified (TestPegasusParity, generator tools/pegasus_tiny_fixture.py); end-to-end text summarization is gated on the SentencePiece/Unigram tokenizer task (Pegasus tokenizes with .spm, not the GPT-2 BPE the Summarize demo uses), but once states are in hand the decode is identical: SummaryIds := DecodeSeq2SeqBeamSearch(Enc, Dec, SourceIds, {StartTokenId=}Config.DecoderStartTokenId, {EOSTokenId=}Config.EosTokenId, {MaxNewTokens=}64, {BeamWidth=}4, {LengthPenalty=}0.8);.
• Seq2seq beam-decode + BLEU/ROUGE plumbing (OFFLINE) — a fully self-contained demo of the end-to-end seq2seq decode-and-evaluate pipeline that needs no network access and no multi-GB checkpoint: it imports the committed pico Marian fixture (tests/fixtures/tiny_marian.*, the same fixture the importer parity test uses) via BuildMarianFromSafeTensors, runs token-id beam search (DecodeSeq2SeqBeamSearch, neuraldecode.pas) over a fixed source-id sequence, and scores the decoded ids against a reference with corpus BLEU + ROUGE-1/2/L (neuralnlpmetrics, using their token-id overloads so no detokenizer is needed). Because the pico fixture is randomly initialized, the "translation" is gibberish and the decoded-vs-reference scores are meaningless as quality — this is deliberately a PLUMBING demo (import → beam decode → BLEU/ROUGE numbers print), not a quality demo. It asserts the two things that ARE meaningful on random weights: the beam decode is deterministic (an identical re-run reproduces the ids exactly) and the metrics are well-formed (a reference scored against itself gives BLEU = ROUGE-1 F1 = 1.0); the program halts non-zero if either fails. To translate real text, point the same DecodeSeq2SeqBeamSearch call at a real Helsinki-NLP/opus-mt-* checkpoint and feed ids from its tokenizer.json (see Summarize for the BART text path). Pure CPU, runs in seconds inside ulimit -v 3000000.

Multimodal & vision-language

• CLIP zero-shot classification — the demo for BuildClipFromSafeTensors (neuralpretrained.pas), a vision-language importer and ViT (model_type clip: openai/clip-vit-base-patch32 and siblings). CLIP is a contrastive dual encoder returned as TWO independent nets (the T5/Marian two-net convention, but as peers — no cross-attention): a causal pre-LN TEXT tower (token embedding + learned positions, biased q/k/v/out, quick_gelu x*sigmoid(1.702x) = TNNetSwishLearnable(1.702), final_layer_norm, bias-free text_projection per token, pooled at the eot position via ClipTextEosPosition — both modeling_clip branches: the legacy eos_token_id=2 ARGMAX-of-ids rule of every published OpenAI CLIP and the fixed first-eos rule) and a bidirectional ViT VISION tower (bias-free patch conv with kernel = stride = patch_size, the learned class token folded into row 0 of the position table over a zero-padded class slot — exact, no new layer — pre_layrnorm, post_layernorm + bias-free visual_projection per token, image embedding = row 0; factored as the reusable BuildClipVisionTower for future ViT/DINO/SigLIP imports). The demo embeds one deterministic test image and N class-prompt token sequences, scores HF-style exp(logit_scale) * cosine (ClipExtractEmbedding + ClipSimilarity) and softmaxes — offline on the committed pico fixture (its two reference logits reproduce HF's logits_per_image exactly; parity ~7e-7 vs the float64 oracle, TestClipParity), or on a real checkpoint passed as argument. Pure CPU, <1 s on the fixture.
• CLIPScore (reference-free text↔image alignment metric) — the standard reference-free generative-quality metric (Hessel et al. 2021, arXiv:2104.08718) for text-to-image / image-captioning, reusing the landed CLIP dual encoder BuildClipFromSafeTensors (neuralpretrained.pas). It runs the image through the vision tower and the prompt through the text tower, L2-normalizes both pooled embeddings and returns w · max(0, cos(image, text)) with the paper's w = 2.5 — a semantic image↔text score that complements the image-only FID / IS / KID (neuralimagemetrics.pas); the max(0, ·) clips the rare negative cosines to 0. New helpers (a metric/helper, NOT a new layer): ClipScore (end-to-end — runs both towers, pools vision row 0 + text at the eot row via ClipTextEosPosition, L2-normalizes), ClipScoreFromEmbeddings (from two pre-extracted unit-L2 embeddings) and RefClipScoreFromEmbeddings — the captioning RefCLIPScore variant (harmonic mean of CLIPScore with the candidate↔reference-caption cosine). The demo scores one image against three prompts and shows the mismatched prompt scores lower (its negative cosine clips CLIPScore to exactly 0) while the matching prompt scores higher, then prints a RefCLIPScore — offline on the committed tiny_clip pico fixture. Parity (TestClipScore): the torch float64 oracle already ships in the fixture (logits_per_image = exp(logit_scale)·cosine, so cosine = logit/exp(logit_scale) and CLIPScore = max(0, 2.5·cosine)); ClipScore reproduces it to < 1e-4 and RefClipScoreFromEmbeddings matches the harmonic-mean formula to < 1e-5. Pure CPU, <1 s on the fixture.
• SigLIP zero-shot classification (the sigmoid-loss dual encoder) — the demo for BuildSigLIPFromSafeTensors (neuralpretrained.pas), the de-facto vision tower of modern open VLMs (model_type siglip/siglip2: google/siglip-base-patch16-224 and siblings). SigLIP is a sigmoid-pairwise-loss image-text dual encoder returned as TWO independent nets and reuses CLIP's pre-LN encoder block, but is architecturally DISTINCT from CLIP and is NOT force-fit onto the CLIP path: (a) the score has a learnable logit_scale AND logit_bias and a per-pair SIGMOID (SigLIPLogit = exp(logit_scale)*cosine + logit_bias), so each class is an independent yes/no — not CLIP's softmax-over-classes; (b) the TEXT tower is BIDIRECTIONAL (no causal mask) and pools the LAST token through a biased head (text_model.head) — not CLIP's eos-argmax + bias-free projection; the VISION tower pools via a Multihead Attention Pooling head (MAP: one learnable probe query cross-attends over the patch tokens via TNNetSoftPrompt + TNNetCrossAttention, then out = attn + mlp(LayerNorm(attn)), row 0) — not a CLS token; (c) the MLP activation is gelu_pytorch_tanh (the tanh-approx GELU); (d) the patch conv is biased, there is NO class token, and the position table covers exactly num_patches rows. The reusable BuildSigLIPVisionTower offers a pVisionFeatures skip-pooling / select-hidden-layer mode for future LLaVA/VLM consumption. The demo embeds one deterministic test image and N class-prompt token sequences and prints both the native per-pair sigmoid match probabilities and a softmax ranking — offline on the committed pico fixture (its reference logits reproduce HF's logits_per_image exactly; parity < 1e-4 vs the float64 oracle, TestSigLIPParity, generator tools/siglip_tiny_fixture.py), or on a real checkpoint passed as argument. NaFlex / variable-resolution siglip2 is an explicit follow-up. Pure CPU, <1 s on the fixture.
• BLIP image captioning — the demo for BuildBlipForCaptioningFromSafeTensors (neuralpretrained.pas), a GENERATIVE vision-language importer of the encoder-decoder kind (model_type blip: Salesforce/blip-image-captioning-base). Unlike the CLIP/SigLIP dual encoders (contrastive, no generation) and unlike a future decoder-only-with-projector LLaVA, BLIP captioning is architecturally a two-net encoder-decoder: a ViT image encoder (the ViT-importer tower — biased patch conv, class token folded into position row 0, pre-LN encoder blocks, post_layernorm; emits all num_patches+1 last_hidden_state rows, no pooling) feeds a BERT-style causal text DECODER through cross-attention (TNNetCrossAttention, the same rectangular Q-from-text / K|V-from-image wiring as the T5/Marian/Pegasus decoders, plus the T5EncoderStatesInput second-TNNetInput convention). Each decoder block is the Marian POST-norm skeleton with BERT's deltas: causal self-attention (attention.self.query/key/value + output.dense/output.LayerNorm), then cross-attention to the image (crossattention.*), then an exact-erf GELU FFN (intermediate.dense + output.dense, the Phi+ReGLU composition), then the BERT LM head (cls.predictions: transform LN(GELU(dense(x))) then the vocab decoder). The vision attention loads from BLIP's fused self_attn.qkv slab ([Q\|K\|V] over all heads) + self_attn.projection. The image is encoded ONCE and DecodeBlipCaptionGreedy rolls out the caption autoregressively from bos_token_id, stopping at sep/eos. The demo greedily captions the committed pico fixture's deterministic test image — offline, printing the generated token ids (decode with the BLIP WordPiece tokenizer for text). Pico parity: per-position next-token logits < 1e-4 vs the float64 HF oracle and the greedy caption ids match HF generate() exactly (TestBlipCaptioningParity / TestBlipCaptionGreedy, generator tools/make_pico_blip_fixture.py). A real caption needs a real BLIP download + the WordPiece tokenizer (decode-to-text is the follow-up). Pure CPU, <1 s on the fixture.
• LLaVA image captioning — the demo for BuildLlavaFromSafeTensors (neuralpretrained.pas), a GENERATIVE vision-language importer of the decoder-only-with-projector kind (model_type llava: llava-hf/llava-interleave-qwen-0.5b-hf and siblings) — the classic LLaVA recipe. Unlike the CLIP/SigLIP dual encoders (contrastive, cannot generate) and unlike the BLIP/TrOCR encoder-decoder-with-cross-attention captioners, LLaVA is a decoder-only LM whose token-embedding sequence is spliced with projected visual tokens — no cross-attention, just ordinary causal self-attention over [text-embeds | visual tokens | text-embeds]. Returned as THREE nets: (a) a ViT vision tower in vision-feature mode (BuildSigLIPVisionTower/BuildClipVisionTower with pVisionFeatures — vision_feature_layer = -1 runs every encoder block but skips post_layernorm, since HF captures hidden_states[-1] BEFORE the post-norm; SelectHiddenLayer = num_layers + feature_layer + 1 selects an earlier layer like CLIP's -2), (b) a 2-layer MLP projector (multi_modal_projector.linear_1 → gelu → linear_2, biased, mapping vision_hidden → text_hidden — BuildLlavaProjector), and (c) the stock Llama/Qwen2 decoder (BuildLlamaFromSafeTensors, its TNNetEmbedding fed externally). The new plumbing: LlavaProjectImage runs the tower + projector once; LlavaAssembleEmbeddings looks up each text token's embedding row and splices the projected visual tokens at the image_token_index placeholder slots; LlavaRunLogits injects the assembled (SeqLen,1,d) embedding sequence into the decoder's embedding-layer output and runs the decoder from the next layer onward (skipping the token lookup so the splice survives) — the embedding-injection convention, the decoder-only sibling of RunT5's external-states feed. The multimodal cfLlava chat template (neuralchat.pas: the llava_v1 vicuna preamble + "USER: <image>\n… ASSISTANT:" turns) renders the prompt. The demo greedily captions the committed pico fixture's deterministic test image — offline (the pico LLaVA is randomly initialized, so the caption ids are gibberish: the demo exercises the image→text PLUMBING). Pico parity: the projected visual tokens AND the mixed image+text next-token logits both < 1e-4 vs the float64 HF oracle (TestLlavaVisualTokenParity / TestLlavaNextTokenLogitsParity, generator tools/llava_tiny_fixture.py); cfLlava round-trips in TestLlavaMultimodalTemplate. A real caption needs a real LLaVA download + its tokenizer + the SigLIP/CLIP image preprocessing (ReadClipImageProcessorConfig + ClipPreprocessImage); the KV-cache fast decode path (the demo re-runs the full prompt per token) is the follow-up. Opens the door to Qwen-VL / PaliGemma. Pure CPU, <1 s on the fixture; cap memory with ulimit -v on a real checkpoint.
• PaliGemma captioning — the demo for BuildPaliGemmaFromSafeTensors (neuralpretrained.pas), a PREFIX-LM vision-language importer (model_type paligemma: google/paligemma-3b-mix-224 and siblings). Structurally PaliGemma is LLaVA-with-a-twist and reuses almost everything: the SigLIP vision tower (BuildSigLIPVisionTower in feature mode, but — unlike LLaVA's vision_feature_layer = -1 — it uses the SigLIP last_hidden_state WITH post_layernorm, so SelectHiddenLayer = 0), the LLaVA splice (LlavaAssembleEmbeddings: text ids look up the √d-scaled Gemma embedding rows, image slots receive the projected visual tokens), and the stock Gemma decoder (BuildLlamaFromTensorReaderWithConfig with the gemma model_type — √hidden embedding scale, +1 RMSNorm gain, decoupled head_dim). The NEW pieces are (a) a SINGLE biased linear multimodal projector (multi_modal_projector.linear, no gelu, no second layer — BuildPaliGemmaProjector), and (b) the PREFIX-LM attention mask: the image tokens AND the prompt tokens (the "prefix", token_type_id 0) attend to all prefix positions with FULL BIDIRECTIONAL attention, while ONLY the generated suffix (token_type_id 1) is causal. This is wired with the new transient TNNetScaledDotProductAttention.PrefixLen knob (TNNet.SetAttentionPrefixLen): for query i/key j, j is attendable iff causal-allowed (j ≤ i) OR both i,j are in the prefix block [0..PrefixLen-1]. PaliGemmaRunLogits sets PrefixLen for the duration of the forward then restores pure causal; PrefixLen stays FIXED at the image+prompt length while the suffix grows. The demo greedily captions the committed pico fixture's deterministic test image — offline (the pico PaliGemma is randomly initialized, so the caption ids are gibberish: the demo exercises the PREFIX-LM image→text PLUMBING). Pico parity: the mixed image+text next-token logits < 1e-4 vs the float64 HF oracle — and the test proves the bidirectional prefix mask is load-bearing by re-running with a causal-everywhere mask (PrefixLen = 0, the LLaVA mask) and asserting the logits differ (TestPaliGemmaLogitParity, generator tools/make_pico_paligemma_fixture.py). Follow-ups: multi-image prompts, a real-checkpoint slicer + the Gemma tokenizer's <image> expansion, 448/896 resolutions, the M-RoPE Qwen2-VL path. Pure CPU, <1 s on the fixture; cap memory with ulimit -v on a real checkpoint.
• TrOCR optical-character recognition — the demo for BuildTrOCRFromSafeTensors (neuralpretrained.pas), an OCR / image-to-text vertical (model_type vision-encoder-decoder: microsoft/trocr-small-printed, trocr-small-handwritten and siblings) — a cropped text-line image → a transcribed string. Structurally a two-net encoder-decoder seq2seq with a VISION encoder, riding the same T5EncoderStatesInput two-net + RunT5/DecodeSeq2Seq* convention as T5/Marian/Pegasus/BART/BLIP. The DeiT/ViT image encoder is the ViT-importer tower with two OCR-specific traits: a BIASED patch conv (unlike CLIP's bias-free conv), and two prepended special tokens — a class token AND a distillation token — folded into position rows 0 and 1 over zero-padded slots (position_embeddings has num_patches+2 rows); pre-LN blocks (layernorm_before/layernorm_after, separate q/k/v/o_proj loaded into the fused Q|K|V slab, mlp.fc1/fc2, exact-erf GELU) then a final layernorm; it emits all num_patches+2 last_hidden_state rows (cls + distillation + patches), no pooling. The decoder cross-attends to every one of those rows (TNNetCrossAttention). The TrOCR decoder is a BART decoder reusing the landed BART/Marian post-norm block builder (BuildBartStackBlocks + LoadMarianStack): LEARNED absolute positions with the +2 padding offset, a layernorm_embedding, post-norm blocks, exact-erf GELU FFN, all q/k/v/out/fc Linears biased, scale_embedding (√d_model on the token embeddings), and the output_projection tied to the token embeddings with no final_logits_bias. The image is encoded ONCE and DecodeTrOCRGreedy rolls out the transcription autoregressively from decoder_start_token_id, stopping at eos. The demo greedily transcribes the committed pico fixture's deterministic test image — offline, printing the generated token ids (decode with the TrOCR GPT-2 byte-level BPE tokenizer for text). Pico parity: per-position next-token logits < 1e-4 vs the float64 HF oracle (TestTrOCRParity, generator tools/make_pico_trocr_fixture.py). A real transcription needs a real TrOCR download + the GPT-2 BPE tokenizer (decode-to-text is the follow-up). Pure CPU, <1 s on the fixture.
• Florence-2 unified vision — the demo for BuildFlorence2FromSafeTensors (neuralpretrained.pas), the repo's first "spatial-output-as-text" VLM (model_type florence2: microsoft/Florence-2-base/-large) — a structurally DISTINCT vision-language model that does captioning, detection, segmentation AND OCR through ONE task-prompted seq2seq head, unlike the single-task PaliGemma (prefix-LM caption) or LLaVA (causal chat). The input is an image + a short TASK TOKEN (<CAPTION>, <OD>, ...) and a BART-style decoder emits a text/coordinate token stream parsed per task. The genuinely new idea: boxes and polygons are emitted as quantized LOCATION tokens <loc_0..loc_999> in the vocabulary — spatial outputs as text — via Florence2QuantizeCoord/Florence2DequantizeCoord (a normalized coordinate ↔ <loc_> token id round-trip). Architecture, riding the landed seq2seq two-net + T5EncoderStatesInput convention: the ENCODER is a TEXT BART encoder fed a VISUAL-TOKEN PREFIX — the multimodal projector turns the DaViT vision-tower feature map into visual tokens (a learned 2D absolute position embedding = row table + column table concatenated per grid cell, added to the feature map; flatten H·W; a fixed cosine 1D temporal embed; the visual tokens = [spatial-mean ; per-cell tokens]; a bias-free image_projection Linear + a biased image_proj_norm LayerNorm), those visual tokens are prepended to the embedded+√d-scaled task-prompt text, and the whole [visual; text] sequence runs through the BART encoder (BuildBartStackBlocks REUSE). The BART decoder cross-attends to it (RunFlorence2Logits, the encoder-states feed). The whole projector + visual-prefix encoder + decoder are pinned to the REAL HF Florence2ForConditionalGeneration float64 oracle: pico parity < 1e-4 on the decoder logits (TestFlorence2Parity) and the location-token round-trip (TestFlorence2LocationTokens), generator tools/make_pico_florence2_fixture.py (its self-checks assert the projector means/positions, the visual prefix, the cross-attn and the +2 positions all move the reference). The demo runs ONE greedy decoder step (the caption argmax) and prints a sample detection box encoded as <loc_> tokens and decoded back — offline on the committed pico fixture (random weights → wiring smoke, not a trained caption). Scope v1: <CAPTION> + <OD> inference; the DaViT vision tower is the DEFERRED gap — the importer takes the tower's last_hidden_state feature map as a precomputed input (mirroring the tracked Qwen2-VL "merged visual tokens as input v1"). A real run needs a real Florence-2 download + its tokenizer + the DaViT tower (depthwise convs + window attention + grouped channel attention + 4-stage conv-embed). Pure CPU, <1 s on the fixture; cap memory with ulimit -v on a real checkpoint.
• Qwen2-Audio audio understanding — the demo for BuildQwen2AudioFromSafeTensors (neuralpretrained.pas), the AUDIO analogue of LLaVA/PaliGemma (model_type qwen2_audio: Qwen/Qwen2-Audio-7B-Instruct and siblings) — a clip-in / text-out demo (transcribe / caption / answer a question about an audio clip). Like LLaVA it is a decoder-only LM whose token-embedding sequence is spliced with projected modality tokens — here the modality is audio, not vision. Almost everything is pure REUSE: the audio tower is the Whisper-style log-mel conv+transformer ENCODER (the landed BuildWhisperStackBlocks / LoadWhisperStack / LoadWhisperConv1D machinery under the audio_tower. key spelling — Conv1d+GELU×2 frontend whose stride-2 conv2 halves the mel frames, fixed-table positions, pre-norm blocks, bias-free k_proj, exact-erf GELU), and the text side is the stock Qwen2 decoder (BuildLlamaFromTensorReaderWithConfig, its TNNetEmbedding fed externally). The NEW pieces are (a) the Qwen2-Audio encoder TAIL — after the encoder blocks, an AvgPool1d(2, stride 2) over the frame axis (halves the frames a second time) then a final LayerNorm (Qwen2AudioProjectAudio does the frame-pair average explicitly, since TNNetAvgPool's 2-D poolsize² divisor is wrong on a (frames,1,d) grid), so the mel input length 2*max_source_positions → max_source_positions frames after conv2 → max_source_positions//2 audio tokens after the pool; (b) a SINGLE biased linear multimodal projector (multi_modal_projector.linear, audio d_model → text hidden — Qwen2AudioBuildProjector); and (c) the embed SPLICE that replaces the <|AUDIO|> placeholder rows with the projected audio frames — the SAME splice the vision-LLM importers use, LlavaAssembleEmbeddings reused verbatim (audio frames in place of visual tokens). Qwen2AudioRunLogits runs the tower+pool+norm+projector once, splices, then injects the assembled embedding sequence into the decoder and runs it causally (the embedding-injection convention). The demo greedily answers about the committed pico fixture's deterministic test mel — offline (the pico Qwen2-Audio is randomly initialized, so the answer ids are gibberish: the demo exercises the audio→text PLUMBING). Pico parity: BOTH the projected audio tokens AND the mixed audio+text next-token logits < 1e-4 vs the float64 HF Qwen2AudioForConditionalGeneration oracle (TestQwen2AudioParity, generator tools/qwen2audio_tiny_fixture.py). Scope v1: a SINGLE full-length clip + text, single-turn — padded/batched audio (feature_attention_mask, the legacy per-audio expand), the real log-mel frontend from a waveform, and the KV-cache fast decode path are follow-ups. Pure CPU, <1 s on the fixture; cap memory with ulimit -v on a real checkpoint.
• BLIP-2 Q-Former bridge — the demo for BuildBlip2FromSafeTensors / BuildBlip2QFormerFromSafeTensors (neuralpretrained.pas), a querying-transformer vision-language bridge (model_type blip-2: Salesforce/blip2-flan-t5-xl and siblings). Unlike the LLaVA/PaliGemma/SigLIP projectors — simple linear/MLP heads from a frozen vision tower into an LLM — BLIP-2's Q-Former is a DIFFERENT bridging module: a small BERT-style transformer fed a fixed set of LEARNED query tokens (query_tokens, e.g. 32) that, in each block, SELF-attend among themselves AND CROSS-attend (crossattention) into the FROZEN ViT patch features, distilling the image into 32 query embeddings, which a language_projection linear then maps into the LLM token space (spliced ahead of the prompt for the FLAN-T5 decode). The new code is the interleaved self/cross-attention Q-Former block: BERT POST-LN sub-blocks (q := LN(q + attn.output.dense(MHA(q)))) plus, on layers where layer_idx mod cross_attention_frequency = 0, a RECTANGULAR cross-attention — the query stream is Q, the (NumPatches, encoder_hidden) ViT features (fed as the net's SECOND TNNetInput, the T5EncoderStatesInput two-source convention) supply K|V, so the scores are NumQuery × NumPatches and encoder_hidden may differ from hidden (per-head leaf = TNNetCrossAttention, exactly the T5/Marian cross-attn wiring) — then the query-specific FFN (intermediate_query/output_query), all in exact-erf gelu (the same ReGLU(Phi|x) composition the BERT importer uses). A model-level layernorm normalizes the query embeddings first. BuildBlip2FromSafeTensors returns the Q-Former net + the learned query_tokens (input0) + the language_projection net; BuildBlip2QFormerFromSafeTensors builds the standalone Q-Former alone. The FROZEN ViT tower (BuildClipVisionTower, the EVA/CLIP-style ViT) and the FLAN-T5 decode tail (BuildT5FromSafeTensors via T5EncoderStatesInput) are REUSE — documented, not built by this v1. The real checkpoints are large / not obtainable offline, so — like the CLIPSeg/LLaVA pico fixtures — it falls back to the committed config-faithful random pico BLIP-2 (tests/fixtures/tiny_blip2_full.*, built by tools/blip2_qformer_tiny_fixture.py from the real HF Blip2QFormerModel float64 oracle), parity-checked < 1e-4 (TestBlip2QFormerParity — the Q-Former query embeddings, measured ~2.5e-7; TestBlip2FullBridgeParity — the projected query embeddings through language_projection). The fixture self-checks assert the cross-attention, the query tokens AND the exact-vs-tanh gelu all genuinely move the reference, so the gate is not vacuous. The demo builds the Q-Former + projection, feeds the learned query_tokens and a deterministic synthetic ViT patch grid, runs the bridge and prints the projected query embeddings that splice into the LLM — offline (random pico weights → wiring/throughput smoke, not a trained caption). Scope v1: image-conditioned query distillation with the FLAN-T5 tail reused; deferred are the text-grounded use_qformer_text_input=true ITC/ITM path, blip2-opt (no OPT importer in-tree), InstructBLIP, and the full ViT→Q-Former→FLAN-T5 caption on a real download. Establishes the Q-Former primitive shared by InstructBLIP and many later VLMs. Pure CPU, <1 s on the fixture; cap memory with ulimit -v on a real checkpoint.

Audio, speech & music

• Speech Commands keyword spotting — the audio analogue of the simple image classifier and a from-scratch (no pretrained-model import) audio TRAINING example: it proves the log-mel frontend in neural/neuralaudio.pas (ComputeWhisperLogMel — the same feature extractor that drives the Whisper / Wav2Vec2 importers) is usable for ordinary supervised training, not only checkpoint replay. Pipeline: 16 kHz mono waveform → ComputeWhisperLogMel (the REAL frontend; (NumFrames,1,NumMelBins) = time along SizeX, mel bins along Depth) → a small conv stack over the time axis (ConvolutionReLU(24,5)/MaxPool(4) → ConvolutionReLU(32,3)/MaxPool(4) → ConvolutionReLU(48,3)/MaxPool(2)) → FullConnectReLU(64) → Dropout → FullConnectLinear + SoftMax, trained with TNeuralFit (ClassCompare argmax accuracy). The default smoke needs no network: a deterministic synthetic ten-keyword set (fixed RandSeed = 1234) chosen to be genuinely CONFUSABLE rather than trivially separable — three closely-spaced pure tones (430/470/510 Hz, ~9% apart), two two-tone chords that overlap one of those tones, a 7-9 Hz AM (tremolo) tone, fast up/down chirps, and two colored-noise textures differing only in spectral tilt — each with seeded pitch jitter and a comparatively LOW SNR (noiseAmp ≈ 0.18). It is generated in-process and passed through the real frontend, so the whole frontend+training path runs end to end on CPU in under a minute; because the task is hard, validation climbs GRADUALLY (≈0.30 at epoch 1 → 0.65 at 6 → 0.91 at 10 → 0.99 at 16) and only hits the 100% early-stop at epoch 17 rather than saturating at epoch 2, finishing at ≈99% held-out test accuracy (observed 98.93%) far above the 1/10 = 10% chance line. An optional --full <dir> path loads real Google Speech Commands v2 WAVs (one subfolder per label) via LoadWavResampledToVolume (the new windowed-sinc resampler in neural/neuralaudio.pas accepts ANY sample rate — a 16 kHz file passes through bit-identically); a tiny downloader (scripts/download_speech_commands.sh, NOT run by the smoke) fetches a few keyword folders. Pure CPU, reproducible.
• Whisper speech-to-text (imported openai/whisper-tiny) — an audio demo: transcribes a 16 kHz WAV to text with a real openai/whisper-tiny checkpoint, pure CAI Pascal end to end. WAV reader + the exact HF WhisperFeatureExtractor log-mel frontend (400-pt periodic-hann STFT / hop 160 / 80 slaney mel bins / log10 / global max-8 clamp, in the new neural/neuralaudio.pas; parity ~1e-5 vs the float64 oracle) feeds BuildWhisperFromSafeTensors (neuralpretrained.pas, an encoder-decoder import on the same two-net RunT5 convention): a Conv1d+GELU×2 frontend whose stride-2 conv halves 3000 mel frames to the 1500-state encoder grid, fixed sinusoidal encoder positions, learned decoder positions, pre-norm blocks with final stack norms, bias-free k_proj, exact erf GELU, rectangular TNNetCrossAttention, tied head. Greedy decode from the <|startoftranscript|><|en|><|transcribe|><|notimestamps|> prologue, byte-level BPE detokenization. Verified on whisper.cpp's jfk.wav: "And so my fellow Americans ask not what your country can do for you ask what you can do for your country." Pico parity fixture: encoder hidden 2.7e-6 / decoder logits 4.8e-6 vs HF (TestWhisperParity). Needs the ~151 MB download; ~5 min and ~2.5 GB on CPU. Pass --word-timestamps to also emit per-word [start - end] spans with an alignment confidence: the curated "alignment heads" cross-attention is averaged into a (token×frame) matrix, median-filtered (kernel 7, openai-whisper default) to kill single-frame spikes, then DTW'd into a monotonic path (WhisperWordTimestamps, single-30 s-window; TestWhisperWordTimestamps asserts the score/DTW parity, that kernel 1 ≡ no smoothing, kernel-7 monotonic boundaries, and in-range confidence).
• Wav2Vec2 / HuBERT speech-to-text (imported facebook/wav2vec2-base-960h) — an audio demo and a self-supervised-encoder ASR: transcribes a 16 kHz WAV with a real facebook/wav2vec2-base-960h (or facebook/hubert-large-ls960-ft) CTC checkpoint, pure CAI Pascal end to end. Architecturally distinct from Whisper (a mel-spectrogram encoder-decoder that autoregresses tokens): Wav2Vec2/HuBERT is a RAW-WAVEFORM conv feature extractor → transformer encoder → linear CTC head, no decoder, decoded in ONE forward pass. LoadWav16ToVolume (neural/neuralaudio.pas) reads the WAV to mono floats in [-1,1), optionally zero-mean/unit-var normalized (the base-960h do_normalize), then BuildWav2Vec2FromSafeTensors (neuralpretrained.pas, model_type wav2vec2/hubert) builds: a multi-layer strided 1-D conv feature extractor (TNNetConvolutionLinear on the (T,1,C) grid, bias-free; first conv → TNNetGroupNorm(channels) per-channel-over-time GroupNorm → GELU, the rest conv → GELU), a feature projection (TNNetTokenLayerNorm + Linear), a conv-based relative positional embedding (a grouped conv1d, kernel num_conv_pos_embeddings, weight-norm-parametrized — the effective weight reconstructed from original0/original1 with dim=2; an even kernel makes a TNNetCrop SamePad drop the extra frame) added to the projected features then encoder.layer_norm, a post-LN bidirectional transformer encoder (the SAME BERT block math: LN(x+Attn(x)) / final_LN(x+FFN(x)), exact erf GELU), and a linear CTC head. DecodeCTCGreedy (neuraldecode.pas) collapses the per-frame argmax (blank = last vocab id); a tiny vocab.json char map renders the ids (the | delimiter → space; no SentencePiece). HuBERT shares the EXACT topology and CTC head — the SAME importer with the hubert flag. Runs a self-contained pico smoke test (synthetic tone on the committed fixture) with no arguments, or transcribes a real WAV with a downloaded checkpoint. Pico parity fixture: encoder hidden 2.3e-6 / CTC logits 1.2e-7 (wav2vec2) and 2.6e-6 / 2.0e-7 (hubert) vs the float64 HF Wav2Vec2ForCTC/HubertForCTC oracle (TestWav2Vec2CTCParity / TestHubertCTCParity, generator tools/wav2vec2_tiny_fixture.py). The -large/robust LayerNorm-everywhere + pre-norm variant is a documented follow-up. Needs the ~360 MB download for a real transcription; CPU-only.
• Speaker diarization — "who speaks when" (pyannote/segmentation-3.0 shape) — a speaker-diarization model and a distinct voice vertical: frame-level multi-speaker activity ("who was speaking when"), deliberately not transcription/CTC like the landed Whisper/Wav2Vec2 paths. BuildPyannoteSegmentationFromSafeTensors[Ex] (neuralpretrained.pas, model_type pyannote) imports the pyannote/segmentation-3.0 SHAPE: a SincNet learnable band-pass front-end — the new leaf layer TNNetSincConv1D, whose every filter is materialized from just two scalars (low_freq, band) as a Hamming-windowed difference of ideal low-pass sincs — over the raw (T,1,1) waveform, then abs → MaxPool → TokenLayerNorm, a standard conv block (Conv1d + ReLU → MaxPool → TokenLayerNorm), a bidirectional minimal-LSTM temporal trunk (TNNetMinLSTM forward + time-reversed via TNNetFlipX, concatenated along Depth — REUSE of the landed LSTM cell), a linear layer, and a per-frame POWERSET multilabel head (the 3.0 model emits the 7 powerset classes covering every subset of ≤ 3 concurrent speakers; PyannotePowersetDecode turns the per-frame argmax back into a per-speaker binary activity matrix). The smoke synthesizes a short two-tone "two speaker" 4 kHz waveform, runs the net on the committed pico fixture, prints a per-frame speaker-activity timeline and one RTTM line per speaker turn, and saves the clip via SaveVolumeToWav16 (neural/neuralaudio.pas). Pairs naturally with WhisperTranscribe for "who said what". Pico parity (tools/make_pico_pyannote_fixture.py → tests/fixtures/tiny_pyannote*): the per-frame powerset logits match a hand-written numpy float64 forward oracle (SincNet kernel materialization, conv/pool/LayerNorm, BiLSTM, powerset head) to < 1e-4 (TestPyannoteParity, observed ~2e-7). Scope v1: inference-only CPU, raw-waveform in, the pyannote SHAPE on a re-randomized pico fixture (the pyannote.audio python package is not installed here; a real-checkpoint key-mapping + a vanilla-LSTM trunk are documented follow-ups). The SincNet front-end is also reusable for raw-waveform speaker-verification / keyword-spotting fronts later. Pure CPU, a fraction of a second.
• Speaker verification — utterance → embedding → cosine score (ECAPA-TDNN shape) — the companion to SpeakerDiarization and a distinct voice vertical: a discriminative utterance → fixed-length speaker embedding encoder so two clips can be COMPARED (the core of speaker verification, "are these the same person?", and the cross-window clustering a full diarization pipeline needs). BuildEcapaTdnnFromSafeTensors[Ex] + TEcapaTdnnConfig (neuralpretrained.pas, the speechbrain/spkrec-ecapa-voxceleb SHAPE) imports the ECAPA-TDNN architecture over a (T,1,NumMel) log-mel frame sequence: a conv_pre dilated TDNN conv + ReLU, then 3× SE-Res2Block (TNNet.AddSERes2Block) — each a Res2Net hierarchical-residual dilated TDNN cascade built on the new leaf TNNetTDNNConv1D (a non-causal centred "SAME" dilated channel-mixing conv whose receptive field grows within the block) with squeeze-excitation channel gating (REUSE of the landed AddSEBlock) — then multi-layer feature aggregation (concat the 3 block outputs + $1 \times 1$ conv), then the new leaf TNNetAttentiveStatsPooling (a per-frame attention head whose softmax weights give a context-weighted mean AND standard-deviation over time, concatenated — distinct from the parameter-free AvgChannel/MaxChannel and from AttentionPooling which computes neither a weighted std nor the mean‖std concat), and finally a linear → the 192-d-style embedding (AAM-softmax is training-only; inference reads the embedding directly). Speaker verification is exposed as EcapaCosineScore (cosine between two embeddings). The smoke synthesizes three tiny log-mel clips — two phrasings of speaker A and one of speaker B — embeds all three on the committed pico fixture, and prints the same-speaker vs different-speaker cosine scores, showing the same-speaker pair scores higher (observed 0.97 vs 0.75). Pico parity (tools/make_pico_ecapa_fixture.py → tests/fixtures/tiny_ecapa*): the embedding matches a hand-written numpy float64 forward oracle (TDNN convs, the full SE-Res2Block cascade incl. the TNNetAvgChannel /N² scaling quirk, attentive statistics pooling, embedding linear) to < 1e-4, and the verification cosine scores match too (TestEcapaParity). Scope v1: inference-only CPU, the ECAPA SHAPE on a re-randomized pico fixture (speechbrain is not installed here; a real-checkpoint key-mapping + a VoxCeleb EER smoke are network/RAM-gated follow-ups). Pure CPU, a fraction of a second.
• Moonshine streaming-ASR encoder (imported UsefulSensors/moonshine-tiny) — a speech-to-text architecture deliberately distinct from Whisper. Whisper pads every clip to a fixed 30 s log-mel spectrogram (so a 1 s utterance costs the same as a 30 s one); Moonshine has NO mel frontend — it convolves RoPE-positioned features directly off the raw 16 kHz waveform with a small strided-conv stem, so the encoder compute scales with the actual audio length. BuildMoonshineFromSafeTensors[Ex] (neuralpretrained.pas, model_type moonshine) imports the encoder (the v1 parity surface): a raw-waveform conv stem (TNNetConvolutionLinear conv1 1→hidden k=127 s=64 bias-free → tanh; TNNetGroupNorm(1) over the whole (T,C) block; conv2 hidden→2·hidden k=7 s=3 → erf-GELU; conv3 2·hidden→hidden k=3 s=2 → erf-GELU), then a PRE-norm BIDIRECTIONAL transformer encoder with partial RoPE (partial_rotary_factor, rotates the first int(head_dim·factor) channels of each q/k head, the tail passes through — the Phi-3 partial-rotary slice wiring reused verbatim), bias-free q/k/v/o, bias-free (gain-only) TNNetTokenLayerNorm entry/exit norms, and a standard erf-GELU fc1/fc2 MLP (the encoder MLP, not SwiGLU — the decoder is the SwiGLU tower). The decoder is now imported too: BuildMoonshineEncoderDecoderFromSafeTensors[WithConfig] returns the encoder net and an autoregressive RoPE + SwiGLU transformer decoder that cross-attends the encoder states (the per-block layout is x += self_attn_causal_RoPE(input_layernorm(x)) → x += cross_attn(post_attention_layernorm(x), enc_states) with no RoPE on cross-attn → x += swiglu_mlp(final_layernorm(x)), closed by a final gain-only LayerNorm + a tied bias-free LM head; the SwiGLU fc1 packs [up|gate] and TNNetSwiGLU computes up·SiLU(gate)). The decoder is the standard two-net TNNetInput-pair shape — Layers[0] = decoder token ids, the second TNNetInput holds the encoder hidden states, filled before each Compute via T5EncoderStatesInput/Seq2SeqEncoderStatesInput (the landed convention shared with T5/Marian/Pegasus/Whisper) — so DecodeSeq2SeqGreedy/DecodeSeq2SeqBeamSearch drive token-id seq2seq decoding (the audio encoder takes a raw waveform, so the example instead calls DecodeMoonshineGreedyCached — a KV-cache incremental decode that caches the self-attn K/V across steps and re-reads the constant cross-attn encoder states each step, running a length-L transcript in O(L) instead of the O(L²) re-encode-the-whole-prefix loop; the streamed argmax is bit-identical to that loop, TestMoonshineKVCacheDecodeParity). Runs a self-contained deterministic smoke (committed random fixture tests/fixtures/tiny_moonshine.*, a synthetic tone at two lengths) with no arguments — pure CPU, a fraction of a second — printing the per-length frame count + latency so the length-proportional-compute contrast vs Whisper's fixed cost is visible; with a real moonshine-tiny/moonshine-base checkpoint dir (it ships a tokenizer.json, loaded via TNeuralHFTokenizer) it imports encoder+decoder and greedily transcribes to real text. Pico parity (tools/make_pico_moonshine_fixture.py): the encoder hidden states match the HF MoonshineModel float64 oracle to < 1e-4 (TestMoonshineEncoderParity), and the decoder next-token logit row matches the HF MoonshineForConditionalGeneration float64 oracle to < 1e-4 (TestMoonshineDecoderLogitParity).
• EnCodec neural audio codec round-trip (imported facebook/encodec_24khz) — an audio-generative demo: a neural codec that compresses a waveform into a stack of discrete codes and reconstructs it (waveform → codes → waveform), the inverse of the analysis-only Whisper / Wav2Vec2 / HuBERT path. The new building block is Residual Vector Quantization (RVQ): a cascade of num_codebooks codebooks where each successive one quantizes the residual left by the previous (the single-codebook TNNetVectorQuantizer used by VQ-VAE / MaskGIT is exactly the one-stage special case), so the latent is a grid of codes (one row per codebook, one column per latent frame). BuildEnCodecFromSafeTensors (neuralpretrained.pas, model_type encodec, the causal weight-norm facebook/encodec_24khz family) builds a self-contained TEnCodecModel holder (EncodeAudioToCodes / DecodeCodesToAudio / Reconstruct): a streaming conv ENCODER (causal weight-norm Conv1d with reflect left-pad + ELU + resnet blocks + strided downsample convs + a residual 2-layer LSTM bottleneck) → RVQ encode (per frame, argmin-L2 over codebook 0, subtract, argmin over codebook 1, …) → RVQ decode (sum of chosen codebook vectors) → a mirror conv DECODER (ConvTranspose1d upsamplers + LSTM + resnet). Conv weights are weight_norm-parametrized in the checkpoint (original0 = g, original1 = v); the importer reconstructs w[o] = g[o]·v[o]/‖v[o]‖ (dim 0). Runs a self-contained pico smoke test (committed random fixture + a synthesized tone) with no arguments — pure CPU, a couple of seconds — or round-trips a real downloaded encodec_24khz. Pico parity (TestEnCodecRoundTripParity, generator tools/encodec_tiny_fixture.py): RVQ codes match the HF EncodecModel oracle exactly (integer argmin) and the reconstructed waveform to < 1e-4. The 48 kHz stereo normalize=true variant and the chunked-streaming long-audio path are documented follow-ups; this is the audio decoder the MusicGen / Bark text-to-audio follow-ups build on.
• Mimi streaming neural audio codec round-trip (imported kyutai/mimi) — the 12.5 Hz neural audio tokenizer behind Moshi / Kyutai-TTS / Sesame CSM. Mimi extends the EnCodec SEANet + RVQ design with two new pieces: a small causal RoPE transformer bottleneck (pre-norm attention + GELU MLP with per-channel LayerScale residuals) inserted after the conv encoder and before the conv decoder — with a strided downsample Conv1d / grouped upsample ConvTranspose1d to step between the EnCodec frame rate and 12.5 Hz — and a split residual vector quantizer: a semantic RVQ (the first num_semantic_quantizers codebooks, distilled at train time, here a nearest-centroid lookup) concatenated with an acoustic RVQ cascade, each owning its 1×1 input_proj/output_proj convs and storing each codebook as embed_sum + cluster_usage (effective centroid embed_sum / clamp(cluster_usage, eps)). BuildMimiFromSafeTensors (neuralpretrained.pas, model_type mimi) builds a self-contained channel-major TNNetMimi holder (Encode / Decode / Reconstruct): causal Conv1d (zero/constant left-pad, plain .conv.weight — no weight_norm) + ELU + resnet blocks → RoPE transformer → downsample → split-RVQ encode → … → upsample → RoPE transformer → mirror conv decoder. The SEANet conv blocks and RVQ codebook math are reused from the EnCodec primitives; the transformer + split-VQ glue is new. The holder carries its signal in double precision (weights stay F32) so the deep conv + high-gain transformer + conv pipeline stays inside the parity gate; the transformer's RoPE uses the Llama rotate_half convention and an exact erf-GELU MLP. Runs a self-contained pico smoke test (committed random fixture tests/fixtures/tiny_mimi.* + a synthesized tone) with no arguments — pure CPU, a couple of seconds — and writes the resynthesized clip to a 16-bit WAV via SaveVolumeToWav16; or round-trips a real downloaded kyutai/mimi. Pico parity (TestMimiParity, generator tools/mimi_tiny_fixture.py): the split-VQ code stack matches the HF MimiModel oracle exactly and the reconstructed waveform to < 1e-4. Real-checkpoint parity and the streaming chunk-at-a-time padding-cache / KV-cache decode path are documented follow-ups.
• DAC (Descript Audio Codec) neural audio codec round-trip (imported descript/dac_*) — the RVQGAN-lineage neural audio codec (model_type dac, the HF DacModel), the third codec landed here after EnCodec and Mimi and the codec the Parler-TTS importer builds on. DAC keeps the conv-encoder → RVQ → conv-decoder shape but differs from EnCodec/Mimi in three ways the importer reproduces exactly: (1) Snake activations everywhere — x + (1/(α+1e-9))·sin(α·x)² with a learnable per-channel α of shape (1, C, 1) (the in-tree TNNetSnake is a parameter-free scalar, so the holder applies the per-channel snake math directly); (2) symmetric, non-causal conv padding (padding=pad on both sides), unlike EnCodec's causal reflect-left-pad / Mimi's causal constant-left-pad; (3) a factorized, L2-normalized RVQ: each quantizer projects the latent down to a small codebook_dim with a 1×1 in_proj conv, L2-normalizes both the projected latent and the codebook rows, picks the nearest entry (argmax cosine = argmin L2 of the unit vectors), looks up the raw codebook embedding, projects it back up with a 1×1 out_proj conv, adds it to the running quantized sum and subtracts it from the residual in the full hidden space. BuildDACFromSafeTensors (neuralpretrained.pas) builds a self-contained channel-major TNNetDAC holder (Encode / Decode / Reconstruct) carrying its signal in double precision (weights stay F32). Each residual unit is snake1 → conv1(k=7, dilated) → snake2 → conv2(k=1) with the input center-cropped before the skip add; conv weights load from a fused .weight or weight_norm (parametrizations.weight.original0/1 / legacy weight_g/weight_v, folded w = g·v/‖v‖); the decoder ends with Tanh. Runs a self-contained pico smoke test (committed random fixture tests/fixtures/tiny_dac.* + a synthesized tone) with no arguments — pure CPU, a couple of seconds — and writes the resynthesized clip to a 16-bit WAV via SaveVolumeToWav16; or round-trips a real downloaded descript/dac_44khz / dac_16khz. Pico parity (TestDACRoundTripParity, generator tools/make_pico_dac_fixture.py): the factorized RVQ code stack matches the HF DacModel float64 oracle exactly (max code diff 0) and the reconstructed waveform to max |diff| ≈ 9.1e-10 < 1e-4. Real 44 kHz / 16 kHz checkpoint parity is a documented follow-up.
• MusicGen text-to-music delay-pattern generation (imported facebook/musicgen-small) — a text-to-audio generative importer: the MusicGen LM decoder predicts a stack of EnCodec codes autoregressively, the inverse of the analysis-only audio path and the audio analogue of the VQ-Model → image-LM pipeline. It composes three landed pieces — a T5 text encoder (BuildT5FromSafeTensors) for conditioning, a single-stage transformer decoder that emits the code stack, and the EnCodec decoder (BuildEnCodecFromSafeTensors) to synthesize the waveform. The new building block is the delay pattern: each of the K = num_codebooks codebooks is offset by one decode step (codebook k's frame f lives at sequence position f+k, padded before it appears), so a single set of K LM heads can predict all codebooks causally; MusicGenDelayInterleave / MusicGenDelayDeinterleave (neuralpretrained.pas) match HF build_delay_pattern_mask exactly. BuildMusicGenFromSafeTensors (model_type musicgen) builds a self-contained TMusicGenModel holder whose decoder is the PRE-norm cross-attention block skeleton (the Pegasus path, not post-norm BART) with K summed code-embedding tables, HF cat([cos, sin]) half-split sinusoidal positions, bias-free q/k/v/out + fc, a final decoder LayerNorm, and K untied LM heads, plus a biased enc_to_dec_proj mapping the T5 hidden size to the decoder hidden size before cross-attention. Generate greedily decodes a [K][frames] code stack through the delay pattern. Runs a self-contained pico smoke test (committed random fixture, a fixed pseudo-encoder-state tensor standing in for the T5 encoder) with no arguments — pure CPU, a fraction of a second. Pico parity (tools/musicgen_tiny_fixture.py): TestMusicGenDecoderParity matches the HF float64 next-token logits (K × T × vocab) to 0.0 < 1e-4 and TestMusicGenDelayPattern matches the HF delay-pattern interleave exactly + round-trips. Stereo (audio_channels=2), the full text-prompt → waveform pipeline, KV-cache incremental decode, and top-k/temperature sampling are documented follow-ups.
• End-to-end TEXT-CONDITIONED MusicGen generation (prompt → T5 → music → WAV) — the successor to MusicGenSmoke: it wires the REAL T5 text encoder that the smoke test stubbed with a fixed pseudo-encoder-state tensor, so a free-text prompt (its token ids) genuinely steers the generated music. Full pipeline, no new layer types — pure example wiring composing three landed importers: prompt token ids → BuildT5FromSafeTensors T5 encoder → its final hidden states (EncSeq × text_d_model) → MusicGen enc_to_dec_proj → cross-attention conditioning → TMusicGenModel.Generate greedy delay-pattern decode → a [K][frames] EnCodec code stack → BuildEnCodecFromSafeTensors EnCodec decoder (DecodeCodesToAudio) → mono waveform → SaveVolumeToWav16 (neuralaudio.pas) → a short .wav clip. The T5 hidden states feed the exact slot MusicGenSmoke filled by hand; the only difference is they now come from a genuine encoder run, so changing the prompt changes the codes. Runs a self-contained pico demo on committed random fixtures (tests/fixtures/tiny_musicgen_t5enc.*, tiny_musicgen_encodec.*, generated by the extended tools/musicgen_tiny_fixture.py at the matched text_d_model/codebook_size) with no arguments — pure CPU, a fraction of a second, writes musicgen_text_demo.wav; weights are untrained random so the clip is noise, not music. Regression test TestMusicGenTextWiring asserts the pipeline runs, is deterministic (same prompt → identical codes + waveform), and that the text conditioning is live (a different prompt steers the codes). Classifier-free guidance is now landed: TMusicGenModel.GenerateCFG(EncStates, UncondStates, NumFrames, GuidanceScale, Codes) runs the decoder TWICE per step — conditional + unconditional — and blends the per-codebook logits as guided = uncond + scale·(cond − uncond) before the argmax (MusicGen's default scale = 3.0; the null branch is a ZEROED text condition matching HF). When GuidanceScale ≤ 1.0 or UncondStates = nil it is bit-identical to plain Generate. Pass --guidance N to the example to enable it. Regression test TestMusicGenCFG pins the scale-1.0/nil-uncond equivalence and that a scale-3.0 pass shifts at least one emitted code. KV-cache incremental decode and top-k/temperature sampling are now landed via TMusicGenModel.GenerateEx(EncStates, UncondStates, NumFrames, GuidanceScale, UseCache, Sampler, Temperature, Codes): with UseCache the self-attention heads run on a lazily-built width-1 twin decoder (BeginIncrementalDecode, weights copied from the full decoder) and the loop feeds one delayed frame per step instead of re-running the whole prefix — bit-identical to the full re-encode greedy loop (cross-attention re-reads the fixed encoder states; the per-frame sinusoidal position is baked into the fed embedding). A non-nil Sampler (any neuralvolume TNNetSampler*, e.g. TNNetSamplerWeightedTopK) draws each codebook token from softmax(logits / Temperature); Sampler = nil is the exact argmax. The example defaults to the KV-cache greedy path and accepts --topk N, --temperature N, and --no-cache. Regression test TestMusicGenGenerateEx asserts the cached greedy decode equals the full re-encode loop, that a weighted top-k=1 sampler equals greedy on both paths, and that the temperature/top-k path is reproducible at a fixed RandSeed. In --download mode the example now seeds its sampling recipe from the checkpoint's generation_config.json when one ships beside the weights, via the new ReadGenerationDefaultsFromDir / ReadGenerationDefaultsFromJSONFile reader (neuralpretrained.pas, parsing top_k/top_p/temperature/do_sample/guidance_scale/max_length; a missing or unparsable file is graceful — Found=False, no exception). facebook/musicgen-small pins do_sample=true, top_k=250, temperature=1.0, guidance_scale=3.0, which the file now supplies (so a non-standard MusicGen variant with different decode defaults is honored automatically); the hardcoded top_k=250 fallback still applies when the file is absent, and explicit --flags always override the file. The reader generalizes beyond MusicGen to any imported generative LM that ships a generation_config.json (TestGenerationDefaultsFromJSONFile). Stereo (the 2K-codebook layout) and a real tokenizer + large downloaded checkpoint remain documented follow-ups.
• MELODY-conditioned MusicGen generation (melody → chroma → music → WAV) — the melody-conditioned sibling of MusicGenText (facebook/musicgen-melody, model_type "musicgen_melody"): generation is steered by a reference melody (a 12-bin chromagram) instead of (or alongside) text. It reuses the landed text-MusicGen path for the EnCodec codec, the delay-pattern interleaving, the K embedding tables / K LM heads, and the sinusoidal positions; two pieces are genuinely new. (1) The chroma front-end ComputeMusicgenMelodyChroma (neuralaudio.pas) matches the HF MusicgenMelodyFeatureExtractor bit-for-bit: a power spectrogram (n_fft=16384, hop=4096, periodic Hann window, center=True reflect pad, normalized by the window L2 energy sqrt(Σwindow²) — exactly torchaudio.transforms.Spectrogram(normalized=True), power=2) projected through a librosa-style chroma_filter_bank (BuildChromaFilterBank, tuning=0/power=2/weighting=(5,2)/start_at_c_chroma) onto 12 pitch classes, per-frame inf-norm normalized, then argmax one-hot. (2) A decoder-only architecture: unlike text-MusicGen's cross-attention decoder, the melody decoder is a causal self-attention LM — the conditioning is PREPENDED to the decoder sequence as concat([audio_enc_to_dec_proj(chroma), enc_to_dec_proj(text)]) (chroma first), the chroma part repeat-tiled/truncated to chroma_length, with sinusoidal positions over the whole sequence and logits read at the decoder-frame positions. BuildMusicGenMelodyFromSafeTensors[Ex] (neuralpretrained.pas) builds a self-contained TMusicGenMelodyModel holder (BuildConditioningPrefix / ComputeLogits / Generate); the self-attention-only blocks ride the Pegasus block skeleton via a new pSelfAttnOnly flag (no encoder_attn sub-block). Runs a self-contained pico smoke demo (committed random fixtures tests/fixtures/tiny_musicgen_melody.* + the matched tiny_musicgen_encodec.*) with no arguments — pure CPU, a fraction of a second: it synthesizes a deterministic A4+E5 reference melody, extracts its chroma (every frame resolves to pitch class 9 = A), prepends the chroma + a fixed text condition, greedily delay-pattern decodes a [K][frames] code stack, decodes it to audio with the EnCodec decoder, and writes musicgen_melody_demo.wav (untrained weights → noise, exercising the wiring). --no-text runs chroma-only conditioning; --frames N sets the decoder frame count. Pico parity (tools/make_pico_musicgen_melody_fixture.py, TestMusicGenMelodyParity): the chroma extractor matches the HF float64 MusicgenMelodyForConditionalGeneration oracle exactly (one-hot, max |diff| = 0) and ONE decoder forward step (chroma + text prepended) matches the HF logits to < 1e-4. A --download real-checkpoint mode for facebook/musicgen-melody (and a real-melody-conditioned smoke clip) is a documented follow-up.
• VITS / MMS-TTS text-to-speech (imported facebook/mms-tts-eng, kakao-enterprise/vits-ljs) — a text-to-speech model: VITS synthesizes a raw waveform end-to-end from token ids. BuildVitsFromSafeTensors[Ex] (model_type vits, neuralpretrained.pas) builds a self-contained TNNetVits holder doing the inference pipeline directly on channel-major arrays: a relative-position transformer text encoder (HF VitsAttention with emb_rel_k/emb_rel_v windowed bias) → per-token prior mean/log-variance; a deterministic duration predictor (use_stochastic_duration_prediction=false, the MMS-TTS default) → frames-per-token; a monotonic length regulator that expands the prior along time; prior_latents = mean + z·exp(logvar)·noise_scale; a normalizing FLOW run in REVERSE — the conditional prior IS RealNVP/Glow additive coupling (VITS's log_stddev≡0, so each layer is a pure shift second_half -= mean(first_half) with the mean from a conv_pre → WaveNet → conv_post stack, channel-flip between layers); and the HiFi-GAN decoder — the SAME generator as BuildHiFiGANFromSafeTensors (TNNetHiFiGAN), reused under the decoder. key prefix (bias-free conv_post) → tanh → waveform written with SaveVolumeToWav16 (neuralaudio.pas). WaveNet/coupling convs fold weight_norm g/v at import. VITS sampling injects prior noise, so z is an explicit input to Synthesize for a deterministic result. Runs a self-contained pico smoke test (committed random fixture tests/fixtures/tiny_vits.*, fixed ids + fixed noise) with no arguments — pure CPU, a fraction of a second, writes /tmp/tts_smoke.wav; the weights are untrained random so the output is noise, not speech. Pico parity (tools/make_pico_vits_fixture.py, TestVitsSynthesisParity): the text-encoder prior stats, the deterministic durations, the flow run in reverse, AND the end-to-end waveform all match the HF VitsModel float64 oracle to < 1e-4 (the oracle's z fed explicitly). A STRING can now be synthesized directly: TNNetVitsTokenizer is the char-level HF VitsTokenizer — it loads a char→id vocab.json plus the add_blank/normalize flags from tokenizer_config.json (LoadFromFiles/LoadFromDir) and Encode(text) reproduces HF's exact id sequence (per-char lowercasing, out-of-vocab drop, and the blank/pad id 0 interleaved between/around every char when add_blank=true), cross-checked against the real transformers.VitsTokenizer in TestVitsTokenizerParity. Running TextToSpeech "hello world" tokenizes the string (pico vocab) and synthesizes from the derived ids; with no argument the fixed-id smoke test still runs. The uroman romanization and espeak phonemizer front-ends are out of scope (the tokenizer rejects is_uroman=true/phonemize=true loudly) — feed already-romanized lowercase text. The stochastic duration predictor and multi-speaker / global conditioning remain documented follow-ups; a real downloaded checkpoint synthesizes a sentence within the ~5 min / ulimit -v 3000000 budget.
• Kokoro / StyleTTS2 text-to-speech (phonemes → waveform smoke) — a text-to-speech model in the StyleTTS2 family (hexgrad/Kokoro-82M, Apache-2.0), genuinely DISTINCT from the VITS path. BuildKokoroFromSafeTensors[Ex] / BuildKokoroFromSafeTensorsWithConfig (model_type kokoro, neuralpretrained.pas) builds a self-contained channel-major TNNetKokoro holder running the deterministic phonemes+style→waveform forward graph directly, wiring the three StyleTTS2 pieces that distinguish it from VITS: (1) style-vector conditioning — a style_dim-d voice/style vector (an explicit input in v1) is split into a prosody half s_pred = style[0..H-1] and an acoustic/decoder half s_dec = style[H..], each AdaIN/affine-injected as AdaIN1d(x,s) = gamma(s)·InstanceNorm_ch(x) + beta(s) (per-channel instance norm over time; [gamma;beta] = fc(s)) into the duration / F0 / energy predictors and the decoder — the new conditioning math vs VITS's WaveNet cond convs; (2) an iSTFTNet decoder — the generator predicts a magnitude exp(conv) + phase sin(conv) spectrogram and runs an inverse STFT to the waveform, reusing the landed ISTFTOverlapAdd(mag, phase, …) overlap-add primitive in neuralaudio.pas rather than HiFi-GAN's transposed-conv upsampling; (3) a prosody/duration stack — a style-conditioned duration predictor → round(exp(log_dur)·speed) frames per token → a monotonic length regulator expanding the per-token text encoding along time, then style-conditioned F0 and energy (N) predictors feed the decoder. Every conv reuses THiFiGANConv / RunHiFiGANConv. The grapheme→phoneme (misaki/espeak) front-end is out of scope — pre-phonemized integer ids are the input and language/g2p/phonemizer config is rejected loudly. Runs a self-contained pico smoke test (committed random fixture tests/fixtures/tiny_kokoro.*) with no arguments — pure CPU, a fraction of a second — writing the synthesized clip to a 16-bit WAV via SaveVolumeToWav16; the weights are untrained random so the audio is noise, not speech. Pico parity (tools/make_pico_kokoro_fixture.py, a self-contained numpy float64 oracle — the kokoro/HF packages are not installed, so the generator defines a faithful StyleTTS2-shaped graph in pure numpy and the holder reimplements it; TestKokoroSynthesisParity): every stage (text-encoder hidden, style-conditioned log-durations + integer durations, length-regulator expansion, F0/energy curves, iSTFTNet magnitude+phase, end-to-end waveform) matches to < 1e-4, with the reference style vector fed explicitly. The real hexgrad/Kokoro-82M checkpoint key mapping (ProsodyPredictor LSTMs, AdaINResBlock1 stacks, harmonic source module), multi-speaker / per-voice voices/*.pt reference vectors, and the g2p front-end are documented follow-ups.
• Bark text-to-speech GPT cascade (model_type bark, suno/bark) — a text-to-speech generative importer: the autoregressive GPT-style TTS family, genuinely distinct from the VITS / StyleTTS2 (Kokoro) flow-and-decoder paths. Bark chains THREE stacked GPT-2-style decoders then the landed EnCodec decoder (reused verbatim from the MusicGen path) to a waveform: a SEMANTIC model (BarkCausalModel: text+semantic tokens → semantic tokens), a COARSE acoustic model (BarkCausalModel: semantic tokens → coarse EnCodec codebooks), and a FINE acoustic model (BarkFineModel) that predicts the remaining EnCodec codebooks NON-causally over the codebook axis. BuildBarkFromSafeTensors[Ex] + TBarkConfig / ReadBarkConfigFromJSONFile (neuralpretrained.pas, model_type bark) builds a self-contained TBarkModel holder of three TBarkSubModels. No new leaf layer — each sub-model is a pre-norm GPT-2 block stack (x + out_proj(MHA(ln1(x))), x + out_proj(gelu(in_proj(ln2(x))))) reusing TNNetLearnedPositionalEmbedding + AddMultiHeadSelfAttention + TNNetTokenLayerNorm with exact-erf nn.GELU (the MulByConstant·Erf·…·ReGLU composition). The genuinely new wiring: (1) Bark uses nn.Linear ([out,in], a new transpose-free LoadLinearWeights) for the fused att_proj q|k|v / out_proj / MLP, with biases gated by config.bias and bias-free lm_head(s), unlike GPT-2's HF Conv1D ([in,out]); (2) the fine model's merged input embedding — one embedding table per codebook (n_codes_total tables, n_codes_total - n_codes_given lm_heads), where for target codebook idx the input is the sum of codebook embeddings 0..idx and the trunk runs with bidirectional time attention (TBarkSubModel.ComputeFineLogits does the codebook-sum + per-head selection in Pascal, matching HF BarkFineModel.forward). Runs a self-contained pico smoke (committed random fixtures tests/fixtures/tiny_bark_*.safetensors + the matched tiny_musicgen_encodec.* codec) with no arguments — pure CPU, a fraction of a second: runs the SEMANTIC → COARSE → FINE cascade end-to-end, EnCodec-decodes the 6-codebook stack, and writes a short bark_tts_demo.wav via SaveVolumeToWav16 (untrained random weights → noise, not speech). Pico parity (tools/make_pico_bark_fixture.py, TestBarkParity): all three sub-models' forward logits match the HF float64 oracle (BarkSemanticModel / BarkCoarseModel / BarkFineModel in .double()) to max |diff| ≈ 0 < 1e-4 — the semantic + coarse next-token logits over fixed id sequences and the fine model's codebook-conditioned non-causal logits per (sequence, codebook_idx) case; the generator's self-checks prove each quirk (positional embedding, fine codebook conditioning, fine bidirectional-over-time attention) moves the oracle. Real suno/bark key-mapping (one nested checkpoint with semantic / coarse_acoustics / fine_acoustics prefixes), a real tokenizer + voice/history prompt, and full autoregressive sampling are documented follow-ups; this v1 imports the three sub-models from separate files and pins the deterministic forward cascade.
• F5-TTS flow-matching voice clone (model_type f5tts, SWivid/F5-TTS) — a NON-autoregressive, NON-GAN text-to-speech voice cloner, genuinely DISTINCT from the VITS (GAN), Kokoro / StyleTTS2 (adversarial) and Bark / Parler (codec-LM) paths: F5-TTS (Chen et al. 2024) regresses a mel-spectrogram by integrating a conditional-flow-matching ODE through a DiT velocity field, conditioned in-context on a masked reference mel + an embedded character sequence (raw chars / pinyin, no phonemizer), so the model "infills" the target speech in the reference speaker's voice. BuildF5TTSFromSafeTensors[Ex] + TF5Config / ReadF5ConfigFromJSONFile (neuralpretrained.pas, model_type f5tts) builds the DiT velocity field as a four-input TNNet (x_t, reference cond mel, character ids, scalar time t) with NO new leaf layer — pure composition: (1) a text branch of a char embedding + ConvNeXt-V2 1-D blocks (TNNetDepthwiseConv1D k=7 → affine-free TNNetTokenLayerNorm → pointwise expand → exact-erf TNNetGELUErf → TNNetGRN global-response-norm → pointwise project, residual); (2) the genuinely new in-context conditioning — TNNetDeepConcat([x_t, cond, text_emb]) → Linear(dim) + a depthwise conv-positional residual (the F5 InputEmbedding); (3) a time branch — TNNetSinusoidalTimeEmbedding(t·1000) → SiLU-MLP → conditioning vector c; (4) a DiT trunk of adaLN-zero blocks (the landed DiTModCond / TNNetFiLM modulation, $[\text{shift},\text{scale},\text{gate}] \times 2$ chunks) with RoPE SDPA self-attention (AddMultiHeadSelfAttention(..., UseRoPE), the q/k slab loaded with the rotate_half→interleaved permute via LoadLlamaLinearWeights' RotaryHeadDim), then an adaLN norm-out + proj_out → the mel-width velocity field. The mel is produced by the flow-matching Euler ODE driver (the same machinery as FlowMatching / neuraldiffusion.pas): start at x_0 ~ N(0,I), integrate x_{t+dt} = x_t + dt·v_theta(x_t, cond, text, t) from t=0 to t=1. NON-autoregressive — there is no KV-cache (every ODE step is a full parallel forward over the whole sequence, so a cache would not help). Runs a self-contained pico smoke (committed random fixture tests/fixtures/tiny_f5.*) with no arguments — pure CPU, a fraction of a second: builds the velocity field, runs 8 Euler steps and prints the sampled mel statistics (untrained random weights → the mel is noise, not speech). SCOPE v1: imports the DiT velocity field (the genuinely new importable piece) and outputs a MEL — pair it with an already-landed vocoder (Vocos / HiFi-GAN) to reach a waveform. Pico parity (tools/f5_tiny_fixture.py, a self-contained numpy float64 oracle — F5-TTS is not in transformers, so the generator re-implements the official model/backbones/dit.py forward in pure numpy and the importer mirrors it; TestF5TTSParity): the velocity field matches the float64 oracle to < 1e-4 (the ConvNeXt-V2 text embed, in-context concat input embedding, RoPE adaLN-zero DiT trunk and adaLN norm-out). Real SWivid/F5-TTS checkpoint parity (offline / RAM-gated), the E2-TTS flat-UNet variant, non-default rope_theta, and a classifier-free-guidance cond/uncond sweep are documented follow-ups.
• Parler-TTS description-conditioned codec-LM decode (model_type parler_tts, parler-tts/parler-tts-mini-v1) — a description-conditioned text-to-speech model (Lyth & King 2024) in the codec-LM family alongside MusicGen / Bark, genuinely distinct in that it is steered by a free-text STYLE DESCRIPTION ("a female speaker with a slightly low-pitched voice, very clear audio") in addition to the transcript to speak. Parler composes three landed pieces — a (By)T5 text ENCODER (BuildT5FromSafeTensors) encoding the description for cross-attention conditioning, a codec-LM DECODER that autoregressively predicts the DELAY-PATTERNED multi-codebook DAC code stack (architecturally the MusicGen decoder: PRE-norm cross-attention blocks reusing the Pegasus skeleton, BIAS-FREE q/k/v/out + fc1/fc2, K embedding tables summed at the input, the half-split sinusoidal position table, a final decoder LayerNorm, K untied LM heads, the standard MusicGen delay pattern reusing MusicGenDelayInterleave/Deinterleave), and the landed DAC decoder (BuildDACFromSafeTensors) for the waveform — and adds ONE genuinely new wiring step: the DUAL PROMPT. The transcript prompt token ids (what to SAY) are embedded by a separate learned table (embed_prompts) and PREPENDED on the sequence axis before the codec frames, so the decoder self-attends over [transcript_prefix | codec_frames] while ALSO cross-attending the description; the K LM heads are read only at the codec-frame positions (the prefix is pure conditioning context, like MusicGen-Melody's chroma prefix but a learned text-token embedding). BuildParlerTTSFromSafeTensors[Ex] + TParlerConfig / ReadParlerConfigFromJSONFile (neuralpretrained.pas, model_type parler_tts) builds a TParlerTTSModel holder (ProjectEncoderStates / ComputeLogits / Generate) — NO new leaf layer, the codec decoder reuses the shared BuildMusicGenDecoderNet. The autoregressive DAC-code decode uses the SDPA KV-CACHE incremental-decode machinery (the same width-1-twin path as MusicGen; cross-attention re-reads the fixed description states each step, self-attention runs BeginIncrementalDecode/EndIncrementalDecode), and is bit-identical to the full re-encode loop under greedy decoding (gated by the test). Runs a self-contained pico smoke (committed fixture tests/fixtures/tiny_parler.*) with no arguments — pure CPU, a fraction of a second: feeds a FIXED description-encoder hidden-state tensor + a FIXED transcript prompt, autoregressively generates the DAC code stack with KV-cache, and prints the per-codebook codes (untrained random weights → the codes are not real speech). Pico parity (tools/parler_tiny_fixture.py, a self-contained numpy float64 oracle — parler_tts is not installed, so the generator re-implements the published Parler decoder step in pure numpy and the importer mirrors it, the SAME random state dict saved to safetensors; TestParlerTTSParity): the description-conditioned, prefix-prepended codec decoder's next-codebook logits match the float64 oracle to < 1e-4, and the KV-cache decode equals the full re-encode decode. SCOPE v1: imports the description-conditioned codec DECODER (the new importable piece) and drives it standalone with a fixed encoder-state tensor — pairing it with the real (By)T5 encoder + DAC decoder to a waveform (all three already importable in-tree), the real parler-tts-mini-v1 checkpoint key mapping, and classifier-free guidance are documented follow-ups.
• Demucs music source separation (imported facebook/demucs / htdemucs time branch) — an audio source-separation model with an audio output modality: one MIXED stereo track in, FOUR stems out (drums / bass / other / vocals). BuildDemucsFromSafeTensors[Ex] (neuralpretrained.pas, model_type demucs / htdemucs) builds a self-contained TNNetDemucs holder doing the time-domain (waveform) Demucs (Defossez et al. 2019, arXiv:1911.13254) U-Net directly on channel-major arrays — a symmetric 1-D conv stack, reusing THiFiGANConv / RunHiFiGANConv for every conv (NO new leaf layer): an encoder of depth blocks ($\text{strided} \text{Conv1d} → \text{ReLU} → 1 \times 1 \text{Conv1d} → \text{GLU}$, channels doubling, each block saving a U-Net skip) → a bi-LSTM bottleneck (lstm_layers stacked bidirectional cells run inline in the holder — there is no bidirectional-LSTM leaf layer — then Linear(2C → C)) → a decoder of depth blocks (skip-add (Demucs center_trim) → Conv1d → GLU → ConvTranspose1d → ReLU except the last) emitting sources·audio_channels channels reshaped to the 4 stems, center-trimmed to the input length. Conv/nn.LSTM weights load as plain folded tensors (encoder.i.{0,2}, lstm.weight_ih_l*/*_reverse, lstm_linear, decoder.i.{0,2}). Runs a self-contained pico smoke test (committed random fixture tests/fixtures/tiny_demucs.*, a synthesized stereo two-tone mix) with no arguments — pure CPU, a fraction of a second — writing each separated stem to a 16-bit WAV via SaveVolumeToWav16 (neuralaudio.pas, stem channels averaged to mono for the mono writer); the weights are untrained random so the stems are noise, not instruments. Pico parity (tools/make_pico_demucs_fixture.py, a self-contained numpy float64 oracle — transformers has no Demucs — built from the published architecture math; TestDemucsSeparationParity): the four separated stems match to < 1e-4. The hybrid time+spectral HTDemucs spectral branch, the v3 cross-domain transformer bottleneck, input/output normalization and a real downloaded checkpoint are documented follow-ups; this v1 lands the time-domain U-Net import.
• CLAP zero-shot audio tagging — the demo for BuildClapFromSafeTensors (neuralpretrained.pas), the audio-domain analogue of CLIP (model_type clap: laion/clap-htsat-unfused and siblings) for zero-shot audio classification and audio↔text retrieval. CLAP shares CLIP's contrastive head — two towers → projection → L2-normalize → exp(logit_scale_a) * cosine (reusing ClipExtractEmbedding + ClipSimilarity, plus ClapSimilarityMatrix for the audio×text grid) — but is genuinely NOT a near-duplicate. The image tower is replaced by an HTS-AT audio tower: a Swin hierarchical windowed-attention transformer over a log-mel spectrogram, which is pure REUSE of the landed Swin machinery (per (head, window) TNNetWindowAttention with relative-position bias + cyclic-shift mask, TNNetGatherTokens window partition/reverse, SwinBuildWindowLayout/SwinSetWindowBias, the patch-merge reorder) under the clap_audio_model key spelling (attention.self.{query,key,value} + attention.output.dense, intermediate.dense/output.dense, attention.self.relative_position_bias_table) with a Conv2d patch-embed + LayerNorm front, a final LayerNorm + token mean-pool, and a 2-layer ClapProjectionLayer (linear1 → ReLU → linear2). The text tower is RoBERTa (clap_text_model): token + learned positions offset past pad_token_id + a token-type row, post-LN bidirectional blocks (built inline exactly like BuildBertFromSafeTensors, exact-erf GELU FFN), a BERT-style pooler (dense → tanh on token 0), then the same 2-layer projection. The HF encoder's BatchNorm2d over the mel axis + the reshape_mel2img freq↔time transpose are applied up front by ClapBatchNormMelImage (the caller supplies the batch-normed, transposed (time, mel, 1) image, exactly as CLIP supplies normalized pixels). The demo embeds one synthetic log-mel clip and N text prompts, prints the cosine-similarity matrix and the top-1 zero-shot label — offline on the committed pico fixture (parity < 1e-4 on BOTH embeddings vs the float64 HF ClapModel oracle, TestClapParity, generator tools/clap_tiny_fixture.py), or on a real clap-htsat-unfused checkpoint passed as argument. Scope v1: freq_ratio = 1 (spec_size = num_mel_bins) and enable_fusion = false only — the real laion freq_ratio = 4 mel2img/group-CNN glue and the "fused" windowing are loudly rejected and tracked as follow-ups. Pure CPU, <1 s on the fixture (random pico weights → wiring/parity smoke, not a trained model).
• MERT music-representation embedding — the demo for BuildMERTFromSafeTensors[Ex] + TMERTConfig (neuralpretrained.pas, model_type mert_model / music2vec: m-a-p/MERT-v1-95M), the first MUSIC understanding encoder — the audio analogue of a frozen vision backbone, distinct from the Wav2Vec2 / HuBERT speech encoders only in its pretraining target (music) and its read-out. With the released MERT-v1-95M config (feature_extractor_cqt=false, attention_relax=-1.0, deepnorm=false, do_stable_layer_norm=false) the forward is architecturally identical to HuBERT, so the importer is pure REUSE of the landed Wav2Vec2 / HuBERT path: a raw-waveform strided 1-D conv feature extractor (first conv → TNNetGroupNorm(channels) → exact-erf GELU, the rest conv → GELU, bias-free), a feature_projection (TNNetTokenLayerNorm + biased Linear), a weight_norm-parametrized grouped conv positional embedding added to the projected features + encoder LayerNorm, then POST-LN bidirectional transformer blocks (x := LN(x + Attn(x)); x := final_LN(x + FFN(x)), the exact LoadWav2Vec2FeatureConv / LoadWav2Vec2PosConv / LoadLlamaLinearWeights / LoadLayerNormWeights loaders) — no new leaf layer. The deltas vs the speech path are: tensors at the TOP level (no hubert./wav2vec2. prefix — the MERTModel is the backbone), NO CTC head, and the MERT-specific WEIGHTED-LAYER-SUM music embedding — the deep weighted sum over all num_hidden_layers+1 transformer hidden states (the encoder input after pos-conv+LayerNorm, then each block's output, the HF output_hidden_states order) with a learned per-layer softmax weight vector (HF use_weighted_layer_sum / the *ForSequenceClassification layer_weights head, kept in TMERTConfig.LayerWeights since the base MERTModel ships none — default uniform). The builder records the N+1 hidden-state layers in an out array; MERTWeightedLayerSum pools them after a Compute() into the fixed (1,1,hidden) music embedding (per-frame weighted sum then a mean over the time frames, the HF pooled_output). The demo embeds two synthetic raw clips, prints the embeddings and their CosineSimilarity (the same cosine the CLAP / ClipSimilarity examples use) — offline on the committed pico fixture (parity < 1e-4 on the last_hidden_state, EACH raw transformer hidden state, AND the weighted-layer-sum embedding vs the float64 HF HubertModel-with-weighted-layer-sum oracle, TestMERTParity, generator tools/make_pico_mert_fixture.py), or on a real m-a-p/MERT-v1-95M checkpoint passed as argument. Scope v1: the GroupNorm base path only — the MERT-v1-330M CQT-fused front-end (feature_extractor_cqt=true), the relaxed-attention (attention_relax > 0) and DeepNorm variants are loudly rejected and tracked as follow-ups (along with downloading the real m-a-p/MERT-v1-95M pytorch_model.bin). Pure CPU, <1 s on the fixture (random pico weights → wiring/parity smoke, not a trained model).

Sequence models, recurrence & SSMs

• String manipulation (next-char) — the smallest "predict the next character in a string" demo: a tiny in-memory 3-string dataset ('happy good morning.', etc.) trains a char-level next-character model so the full embedding → sequence → readout pipeline is visible end to end. Pure CPU.
• Sentiment analysis (SST2) — learns sentiment classification on the SST2 dataset (Stanford Sentiment Treebank, via HuggingFace) with a char/token context model, an end-to-end NLP training program with autosave.
• Byte-processing relation table — a tiny, readable peek inside the symbolic byte engine (TEasyLearnAndPredictClass) that powers TNNetByteProcessing. Unlike the gradient layers, this engine INDUCES discrete cause→effect rules ("relations") mapping an input byte pattern to an output byte. The demo drives it exactly as the layer does internally (Predict() then newStateFound()), then prints the learned rules with printRelationTable. Pure CPU.
• Bit-processing shows its work — a hybrid net built on TNNetBitProcessing (1:1 affine-quantize-each-scalar-to-a-byte sibling of TNNetByteProcessing) learns y = a - b on a,b in [0,10], then is asked to EXTRAPOLATE to the unseen box a,b in [10,20]. Because the layer reduces the inputs to a discrete affine CODE that a tiny TNNetFullConnectLinear(1) readout combines, the pipeline is scale-free and barely degrades out of range (RMSE 0.099 → 0.114), while a same-size dense tanh baseline overfits its training box (0.232 → 0.579, ~5x worse extrapolation). The payoff: the symbolic engine PRINTS the human-readable rule it induced (fE[B] := (A[0] - A[1]), the engine's native csSub, f=1) via a directly-driven TEasyLearnAndPredictClass mirror — a "neural net that shows its work" the dense baseline cannot match. Deterministic, pure CPU, <1 s.
• TokenShift Baseline - Head-to-head bake-off of the attention-free TNNetTokenShift (RWKV-style t-1 token mixing) against real self-attention (TNNet.AddMultiHeadSelfAttention) on the same tiny char-level next-token copy task. TokenShift fully solves the lag-1 region for ~3x fewer weights but is stuck at chance on the long-range region that attention routes to — the structural cost/capability trade made concrete. Pure CPU.
• Echo State Network (reservoir computing) — the Jaeger 2001 Echo State Network recipe, a training paradigm different from everything else in tree: the recurrent core is fixed and random and only a single linear readout is trained, so there is no backprop-through-time. A sparse random reservoir (N=100, leak 0.3) is rescaled to a target spectral radius via the existing TNNet.EstimateSpectralNorm power-iteration helper; the leaky-integrator state h_t = (1-a)h_{t-1} + a*tanh(W_in*x_t + W*h_{t-1}) is run forward over sin(0.2t)+0.3*sin(0.31t), the states are collected, and only a TNNetFullConnectLinear(1) readout is fit on (h_t -> next value) pairs. It then free-runs autonomously, feeding its own prediction back, and renders predicted-vs-true as an ASCII plot. Built-in gates (Halt(1) on failure): teacher-forced one-step NRMSE 0.0161 beats a persistence baseline 0.2136, and a rho>1 ablation diverges — proving the spectral-radius<1 echo-state property is what makes it work. Contrasts with DiagonalSSM (which trains its recurrence; the ESN freezes it). Pure CPU, ~4 s.
• Spatial Gating Unit (attention-free token mixing) — the gMLP Spatial Gating Unit (TNNetSpatialGatingUnit / TNNet.AddgMLPBlock, Liu et al. 2021, Pay Attention to MLPs): an attention-FREE sequence mixer with no queries/keys/values and no per-pair dot product — it gates one channel half against a single learned, content-independent SeqLen×SeqLen spatial projection of the other half (W fixed after training). A two-arm bake-off trains a gMLP block vs a same-parameter-budget single-head attention baseline on a long-range first-token-broadcast toy; both reach 100% per-token accuracy, demonstrating attention-free token mixing actually learns the dependency. The gMLP-paper LayerNorms that bound the multiplicative gate live in the block builder, not the pure-primitive layer. Pure CPU, ~13 s.
• MLP-Mixer block (all-MLP sequence mixer) — the attention-free MLP-Mixer block (TNNet.AddMLPMixerBlock, Tolstikhin et al. 2021, MLP-Mixer: An all-MLP Architecture for Vision): replaces self-attention over a (Tokens,1,Channels) sequence with TWO pre-LayerNorm residual MLPs — a TOKEN-mixing MLP that mixes across token positions (shared over channels, implemented by transposing the token/channel axes via TNNetTransposeXD and running a pointwise MLP over the new Depth=Tokens axis) and a CHANNEL-mixing per-token pointwise FFN. A small "which-half" token-classification toy plants a spike at one random position and asks whether it landed in the FIRST or SECOND half of the sequence — solvable ONLY by mixing information across tokens. A 2-block all-MLP Mixer stack (2320 weights) reaches 100% train/test accuracy with a cleanly converging cross-entropy loss. Pure CPU, ~6 s.
• Tiny Vision Transformer (ViT patch embedding) — a from-scratch ViT image classifier built on the reusable TNNet.AddPatchEmbedding(PatchSize, EmbedDim, AddClassToken, AddPositionalEmbedding) builder (Dosovitskiy et al. 2021, An Image is Worth 16x16 Words). The builder turns a 2D image into a (SeqLen[+1],1,EmbedDim) token sequence in one call: a kernel=stride=PatchSize conv patchify (TNNetConvolutionLinear) → flatten the patch grid (TNNetReshape) → an optional learnable [CLS] token prepended at position 0 (TNNetSoftPrompt) → an optional learnable positional embedding (TNNetLearnedPositionalEmbedding) — replacing the conv-stride-then-reshape boilerplate patch-tokenizing examples used to hand-roll inline. The demo is a synthetic "which-quadrant" task (an $8 \times 8$ image with one bright spike; classify which quadrant it lands in — needs comparing token positions across the patch grid): AddPatchEmbedding(2,16,classtoken) → $\text{AddTransformerEncoderBlock} \times 2$ → classify from the [CLS] token, converging to 100% train/test accuracy. Pure CPU, <1 min.
• Hyena operator (attention-free implicit long convolution) — the order-2 Hyena operator (TNNetImplicitLongConv / TNNet.AddHyenaOperator, Poli et al. 2023, Hyena Hierarchy): a sub-quadratic sequence mixer whose per-channel causal filter spans the WHOLE sequence yet is generated IMPLICITLY by a tiny shared MLP over positional features (times a learnable exponential-decay window), so its weight count does NOT grow with SeqLen. A two-arm bake-off trains a Hyena operator vs a single-head self-attention baseline on a long-range recall toy (copy an early random payload to the last position). Hyena's global receptive field wins clearly while using FEWER weights (e.g. recall MSE ~0.0014 with 152 weights vs ~0.090 with 688 for attention). Pure CPU, ~11 s.
• Liquid CfC cell (closed-form continuous-time recurrence) — the CfC "liquid" recurrent cell (TNNetClosedFormContinuous, Hasani et al. 2022, Closed-form continuous-time neural networks, Nature MI): a sequence mixer that updates a hidden state with the ANALYTIC closed-form solution of a liquid time-constant ODE (no ODE solver), gating between a fast tanh input pathway and the previous state with an INPUT-DEPENDENT, per-channel continuous-time constant — distinct from the numerically-integrated AddNeuralODEBlock and the fixed-decay TNNetRetention / TNNetDiagonalSSM. A two-arm remember-then-recall toy (a cue at position 0 must be reproduced at the last position) trains the CfC cell vs a single SDPA head at MATCHED parameter count (~220 weights each); both reach 100% last-position recall, the CfC by carrying the cue in its liquid state and attention by looking back. Pure CPU, ~3 s.
• Liquid CfC vs fixed-decay diagonal SSM (last-write-wins) — the sibling of the LiquidCfC toy, isolating the one thing the CfC liquid cell (TNNetClosedFormContinuous) has and a fixed-decay diagonal SSM (TNNetDiagonalSSM) does NOT: an INPUT-DEPENDENT time constant. A longer-horizon (SeqLen=16) multi-cue last-write-wins task writes 2–4 cues at random positions (each flagged by a WRITE marker channel) and asks for the MOST-RECENTLY written cue at the last position — which requires input-dependent forgetting: each write must OVERWRITE the previous value. Both models share an embed -> mixer -> LayerNorm -> readout skeleton; only the mixer differs, and the diagonal SSM is given a wider width (so MORE total weights) since it is cheap per channel (4·d vs the CfC's 2·d²+2·d). The CfC's gate reacts to the WRITE pulse and resets its state; the fixed-decay SSM can only blend, so earlier cues bleed in. Result (seed-dependent): CfC 100% (475 weights) vs SSM ~94% (639 weights) — the liquid cell wins despite FEWER parameters. Pure CPU, ~10 s.
• sLSTM vs CfC vs diagonal SSM (xLSTM exp-gated recurrence) — the headline toy for the new TNNetSLSTMCell (TNNet.AddSLSTM), the scalar xLSTM cell (Beck et al. 2024, xLSTM: Extended Long Short-Term Memory) and a classic-LSTM-style multiplicative-gate recurrence (every other mixer — TNNetClosedFormContinuous, TNNetDiagonalSSM, TNNetRetention — is linear-state or fixed/learned decay, with no input/forget/output gates). The sLSTM's distinguishing machinery: EXPONENTIAL input/forget gates i_t=exp(...), f_t=exp(...) (sharper storage revision than sigmoid) made trainable by a running-max STABILIZER state m_t=max(log f_t+m_{t-1}, log i_t) that renormalizes the unbounded exp gates so they never overflow, plus a normalizer n_t with hidden h_t=o_t*(c_t/n_t). A SeqLen=24 copy/state-reset task with an explicit "clear memory" pulse channel contrasts sLSTM against CfC and the diagonal SSM at matched-ish budgets; all three reach 100% last-position recall (sLSTM 950, CfC 330, SSM 150 weights) — honest framing: the toy is learnable by every gated/decay mixer, the point is sLSTM matches them via a fundamentally different exp-gate-plus-stabilizer mechanism. Pure CPU, ~16 s.
• Minimal parallelizable RNNs: minGRU & minLSTM — the headline demo for the two new TNNetMinGRU and TNNetMinLSTM cells, the minimal recurrent units of Feng, Tung, Hassani, Hamarneh & Ravanbakhsh 2024 (Were RNNs all we needed?, arXiv:2410.01201). Both strip a classic GRU/LSTM down to the one recurrence that matters but make every gate a function of x_t ONLY (no h_{t-1} feed): minGRU is z_t=sigmoid(W_z x_t), ht~=W_h x_t, h_t=(1−z_t)⊙h_{t-1}+z_t⊙ht~; minLSTM adds an input gate and NORMALIZES the two gates to sum to one per channel — f'_t=f_t/(f_t+i_t), i'_t=i_t/(f_t+i_t), h_t=f'_t⊙h_{t-1}+i'_t⊙ht~. Because the gates no longer depend on the previous hidden state the recurrence becomes a linear scan h_t=a_t⊙h_{t-1}+b_t that a parallel prefix-scan can solve — distinct from the xLSTM family (TNNetSLSTMCell/TNNetMLSTMCell), whose gates DO read h_{t-1} (and which add exp gates + a running-max stabilizer). v1 keeps the simple non-log-space parameterization and a sequential scan forward (parallel-scan is a noted follow-up); the backward is exact BPTT — for minLSTM the f/(f+i) normalization couples the two gate gradients and is differentiated exactly (input + all weight tensors finite-difference checked, max-abs err ≤ 0.003). The demo is a selective-copy recall task: one of several value tokens is flagged, and a final query token asks the net to reproduce the marked value — solving it requires latching the marked token into the hidden state and holding it across distractors. Contrasted against a param-comparable memoryless per-token MLP (no cross-time state): both minimal recurrent arms reach near-100% exact recall (minGRU 100% / MSE 0.006, minLSTM 99.3% / MSE 0.009) while the MLP is stuck at the 16.7% chance floor. NOTE: BPTT here is momentum-sensitive, so the recurrent arms train with plain SGD (momentum 0). Pure CPU, ~2 s.
• Next-frame video prediction (ConvLSTM) — the headline demo for the new TNNetConvLSTMCell, a Convolutional LSTM (Shi, Chen, Wang, Yeung, Wong & Woo 2015, Convolutional LSTM Network). It is the spatial, image-state analogue of the dense recurrent cells (TNNetMinLSTM/TNNetSLSTMCell/TNNetMLSTMCell): instead of a vector hidden state it carries (H,W,HiddenC) cell/hidden feature maps and replaces every gate matrix-multiply with a K×K same-padding convolution over the channel-concatenation z=[x_t ; h_{t-1}] — i/f/o=σ(W∗z+b), g=tanh(W∗z+b), c_t=f_t⊙c_{t-1}+i_t⊙g_t, h_t=o_t⊙tanh(c_t) — so the recurrence can track where a moving object is and where it is heading without flattening away the spatial layout (which a dense LSTM would). The task is Moving-MNIST-style next-frame prediction on a self-contained, download-free synthetic dataset: a small bright blob translates across a 12×12 grid at constant integer velocity, bouncing off the walls; the model watches the first N=4 frames and predicts the 5th. The input packs the N frames stacked on the X axis as (N·H, W, 1) (the layout the cell expects), the ConvLSTM emits the N per-step hidden maps (N·H, W, HiddenC), a TNNetCrop keeps only the last timestep's map (the post-sequence summary), and a 3×3 TNNetConvolutionLinear + Tanh head projects it to the predicted next frame in [-1,1]. Training is plain supervised regression (Compute(frames); Backpropagate(nextFrame), MSE); backward is exact BPTT carrying both the dL/dc_t and dL/dh_t spatial maps right-to-left (the h_{t-1} gate feed couples adjacent timesteps — the classic ConvLSTM BPTT bug if truncated; input + all eight weight tensors finite-difference checked, max-abs err ≤ 0.0015). Reports per-epoch held-out MSE/MAE and renders an ASCII (input frames | predicted | ground-truth) panel plus a PPM (inputs gray, prediction green, truth red). Pure CPU; the default SMOKE run (240 train / 40 test clips, hidden 8, 6 epochs, ~28 s) drives held-out MSE from ≈1.20 → ≈0.095 (a ≈92% drop), with a --full flag (hidden 16, 800/80 clips, 16 epochs) for a sharper prediction.
• Video frame interpolation (direct synthesis vs. flow warping) — the headline demo for the new TNNetFlowWarp dense per-pixel backward-warp primitive, and a video task structurally distinct from VideoPrediction: rather than EXTRAPOLATING the next frame, it interpolates the unseen MIDDLE frame that sits between two endpoints (the RIFE/FILM task). It reuses the same self-contained, download-free Moving-MNIST-style blob world (a bright blob translating across a 16×16 grid, bouncing off the walls); from each 3-frame clip the model sees the two endpoints t and t+2 (stacked as the two CHANNELS of one $16 \times 16 \times 2$ image) and is supervised on the hidden middle frame t+1. The reconstruction loss is pixel L1 + (1−SSIM) using the landed neuralimagemetrics.ComputeSSIMLossAndGradient helper (SSIM's 11×11 window is why the grid is 16); the custom per-pixel gradient is injected through the standard TNNet.Backpropagate path via the pseudo-target identity Desired = Output − GradOut (since the library's last-layer rule is OutputError = Output − Desired). Two model variants are trained and compared: (a) a direct conv encoder-decoder that hallucinates the middle frame's pixels from scratch, and (b) a flow path where the SAME encoder predicts, per pixel, a dense optical-flow field for each endpoint (F0 = mid→t, F1 = mid→t+2), TNNetFlowWarp backward-warps each endpoint frame along its flow, and the two warps are averaged (symmetric 0.5/0.5 blend). The flow model never has to invent texture — only learn WHERE each pixel moved, a far better-posed problem — so on this rigid-motion data it reaches a lower held-out error than direct synthesis (typical smoke run: flow L1 ≈0.040 / SSIM ≈0.81 vs direct L1 ≈0.056 / SSIM ≈0.49, ~25% lower L1), the textbook illustration of why warping beats direct synthesis for motion. Reports per-epoch train loss and held-out L1/SSIM for both models, renders ASCII before | predicted | truth | after panels, and dumps a before | middle(green=pred, red=truth) | after PPM triplet for each. Pure CPU (no LCL/image deps); the default SMOKE run (200 train / 40 test clips, 8 epochs) finishes in ~7 s, with a --full flag for a sharper result.
• Optical flow with RAFT (correlation volume + ConvGRU refinement) — an optical-flow, two-image-in / dense-(dx,dy)-out demo exercising the new RAFT importer (Teed & Deng 2020, RAFT: Recurrent All-Pairs Field Transforms for Optical Flow) end to end. It loads the committed pico raft_small fixture (tests/fixtures/tiny_raft) with BuildRaftFromSafeTensors and runs the full forward: a shared feature encoder over both frames → the all-pairs TNNetCorrelationVolume (dot-products between EVERY pair of feature locations — the new primitive) → an iterative TNNetConvGRUCell update operator that, via a local TNNetCorrelationLookup around the current flow, refines the (dx,dy) field over a fixed small number of steps. The data is self-contained and download-free: a bright textured square on a dark field, with frame-2 a known integer translation of frame-1 (so the true flow is a constant shift — a sanity target). Because the fixture weights are random (this is a forward / plumbing demonstration, not a trained model; point the importer at a real torchvision raft_small export for real flow), the example focuses on the pipeline and the two visualisations it writes: opticalflow_field.ppm color-codes the predicted flow the standard Middlebury way (hue = direction, brightness = magnitude), and opticalflow_warp.ppm shows frame-1 | frame-1 warped toward frame-2 by the predicted flow (TNNetFlowWarp) | frame-2 side by side at the /4 flow grid — closing the loop with the landed dense-warp primitive (FrameInterpolation's TNNetFlowWarp). Pure CPU, inference-only, ~1 s.
• Retention dual form (parallel == recurrent), with learnable gamma — the headline toy for TNNetRetention (TNNet.AddRetention, RetNet, Sun et al. 2023, Retentive Network): a softmax-FREE sequence mixer whose only score weighting is a FIXED exponential-decay causal mask D[n,m]=gamma^(n-m) (n>=m, else 0). The demo trains the O(n^2) PARALLEL form on a tiny fixed-offset char-copy task, then runs the SAME trained weights through a hand-rolled O(1)-state-per-step RECURRENT loop (S_n = gamma*S_{n-1} + K_n^T V_n, out_n = Q_n S_n) and gates (Halt(1)) that the two forward passes agree token-for-token to fp tolerance — RetNet's headline dual-form property. A second LEARNABLE-GAMMA arm exercises the new follow-up: TNNetRetention.Create(..., LearnGamma:=true) stores an UNCONSTRAINED raw scalar (1 neuron weight) and uses gamma=sigmoid(raw) so the effective decay is always in (0,1) under plain SGD; backward accumulates dL/dgamma through D[n,m] (d/dgamma gamma^k = k*gamma^(k-1)) and chains the sigmoid. The arm deliberately inits gamma WRONG (0.50, below the rewarded ~0.90 schedule) and gates that gradient moves it back UP (observed 0.50 -> ~0.78). Pure CPU, ~4 s.
• Neural ODE - Continuous-depth residual trunk (TNNet.AddNeuralODEBlock, Chen et al. 2018): one shared function f integrated over Steps explicit-Euler updates y := y + h·f(y) (h = 1/Steps), so depth becomes a time axis and the parameter count is independent of Steps. Trains a tiny classifier whose only trunk is the ODE block and shows accuracy stays high and roughly flat as Steps in {1,2,4} at constant weight count — the "depth for free" property. Pure CPU, ~7s.
• Deep Equilibrium - Implicit fixed-point trunk (TNNet.AddDeepEquilibriumBlock, Bai/Kolter/Koltun 2019): one weight-tied transform f iterated z := f(z + x) to its fixed point z* = f(z*; x), so the effective depth adapts to the input at a parameter count independent of the iteration count. Trains a tiny classifier whose only trunk is the DEQ block, reports per-epoch accuracy plus the mean forward iteration-count-to-convergence (the adaptive-depth signal), and runs a param-matched AddNeuralODEBlock side-by-side (explicit unroll vs implicit fixed point). The backward is the jacobian-free phantom gradient (Geng et al. 2021), honestly disclosed. Pure CPU, ~35s.
• Hamiltonian pendulum (symplectic, energy-conserving dynamics) — the headline toy for the new TNNetHamiltonianCell, a structure-preserving (symplectic) learned-dynamics cell (Hamiltonian Neural Networks, Greydanus, Dzamba & Yosinski 2019, Hamiltonian Neural Networks). Unlike every other continuous-dynamics layer in tree — which regresses the time-derivative field directly and conserves nothing (AddNeuralODEBlock = unconstrained field, TNNetClosedFormContinuous = liquid gate, TNNetDiagonalSSM/TNNetSelectiveSSM = linear state space, TNNetKalmanFilterCell = uncertainty) — this layer parameterizes a scalar learned Hamiltonian H_theta(q,p) with a small inner MLP and takes a symplectic step from its gradient (dq=+dH/dp, dp=-dH/dq), so energy is conserved by construction. The forward already needs dH/dz (one backward sweep through the inner MLP), so the training backward differentiates through that gradient — a Hessian-vector product of H done as a second tape pass (no Hessian materialized). The demo trains on noisy (q,p) samples of an ideal pendulum and contrasts a long autoregressive rollout against an unconstrained NeuralODE-style residual MLP field of identical width. Headline: both fit the one-step transition equally well (down to the noise floor), yet over 800 rollout steps the HNN conserves energy ~4× better (max |dE| ≈ 0.16 vs ≈ 0.63) — the free field drifts off its level set while the symplectic cell stays on it. Pure CPU, ~30 s.
• DeltaNet delta-rule associative recall — the headline demo for the new TNNetDeltaNet, the delta-rule linear-attention recurrence (Yang et al. 2024, Parallelizing Linear Transformers with the Delta Rule over Sequence Length, arXiv:2406.06484). It carries a (d×d) matrix memory S and updates it per timestep with the classic delta (Widrow–Hoff) rule: it first READS the current value prediction S_{t-1}ᵀ k_t, measures the error against the target v_t, and writes back ONLY the correction scaled by a per-token write strength β_t = sigmoid(…) ∈ (0,1) — S_t = S_{t-1} + β_t·k_t ⊗ (v_t − S_{t-1}ᵀ k_t), read-out y_t = S_tᵀ q_t. This removes-then-adds associations (a true editable associative memory), unlike TNNetRetention (fixed/learned exponential decay), TNNetMLSTMCell (unbounded outer-product accumulation) or TNNetSLSTMCell (scalar exp-gated) — none of which do error-correcting writes. Keys are L2-normalized for stability and the exact dL/dS is carried right-to-left through the rank-1 write (into k, v, β AND S_{t-1}) and the read-out (input+weight gradients finite-difference checked). The demo is an overwrite key→value recall task: a key is written, then RE-written with a new value, then queried for its MOST RECENT value — retrieving it requires erasing the stale association, exactly the delta rule's job. Headline: at a matched parameter budget the delta-rule arm reaches 100% exact recall (MSE 0.006) vs the fixed-decay TNNetRetention baseline's 93% (MSE 0.023), which blends the stale and fresh values. Pure CPU, ~2 s.
• Legendre Memory Unit delayed-signal reconstruction — the headline demo for the new TNNetLegendreMemoryUnit, the HiPPO-LegS Legendre Memory Unit (Voelker, Kajić & Eliasmith 2019, Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks, NeurIPS 2019). Each input channel carries an order-N memory vector m_t holding the coefficients of an orthogonal shifted-Legendre-polynomial projection of a sliding window of that channel's signal, driven by the dense, structured, NON-diagonal HiPPO-LegS transition matrix A_ij = (2i+1)·(−1 if i<j else (−1)^{i−j+1}), B_i = (2i+1)·(−1)^i, discretized once at build time (Euler, Ā = I + (1/θ)A, B̄ = (1/θ)B) and run as the linear recurrence m_t = Ā·m_{t−1} + B̄·u_t. A trainable per-channel read-out collapses the N coefficients to one value per step. The fixed Ā/B̄ need NO gradient; only the read-out trains, and backward is a clean right-to-left adjoint scan (dL/dm_{t−1} = Āᵀ dL/dm_t, input picks up B̄ᵀ dL/dm_t) with the read-out weight gradient — both finite-difference checked (max-abs err ≈2e-4), plus a brute-force discretization smoke check. Memory cost is N numbers regardless of window length, distinct from the diagonal/complex-diagonal/matrix-memory mixers in tree (TNNetDiagonalSSM, TNNetLRU, TNNetDeltaNet, TNNetGatedLinearAttention): the state mixes ALL N coefficients through a fixed polynomial-projection operator. The demo is a pure-delay reconstruction task — a smooth random 1-D signal must be reproduced delayed by D steps — exactly what an orthogonal sliding-window memory makes trivial and a leaky scalar accumulator cannot. Headline (asserted): at a smaller parameter budget the LMU arm reaches MSE 0.0008 vs a state-matched TNNetDiagonalSSM arm's 0.0034 (40 vs 192 params). Pure CPU, ~10 s. Noted follow-up: a learnable window length θ. Covered by TestLMU* in the test suite. θ is a fixed build-time constant in v1.
• RWKV WKV time-mixing recurrence — the headline demo for the new TNNetWKV layer + TNNet.AddRWKVTimeMix builder, the weighted key-value (WKV) time-mixing operator that defines RWKV-4 (Peng et al. 2023, RWKV: Reinventing RNNs for the Transformer Era, arXiv:2305.13048) — a softmax-free, attention-free sequence mixer. Over a k|v-split (SeqLen,1,2C) sequence it computes the numerically-stabilized exponential-decay KV average wkv_t = (a_{t-1} + e^{u+k_t} v_t)/(b_{t-1} + e^{u+k_t}) with running accumulators a_t = e^{-w}a_{t-1} + e^{k_t}v_t, b_t = e^{-w}b_{t-1} + e^{k_t}, a learnable per-channel positive decay w = softplus(w_raw) and a per-channel "bonus" u that up-weights the current token (the running max is carried in log-space so e^{k} never overflows). This is distinct from every other linear-attention layer in tree: TNNetSelectiveSSM (input-dependent SSM), TNNetDeltaNet (error-correcting delta rule), TNNetDiagonalSSM (LTI), TNNetRetention (fixed-decay outer-product) — WKV is the exponential-decay KV average with a current-token bonus. Exact BPTT runs two coupled right-to-left adjoint scans over the cached accumulators (input/w/u gradients finite-difference checked). The AddRWKVTimeMix builder composes the leaf with TNNetTokenShift + {r,k,v} pointwise projections + a sigmoid receptance gate + output projection. The demo contrasts the WKV time-mix arm against a TNNetDeltaNet arm on an associative-recall task: both reach 100% exact recall (WKV MSE ≈0.006, DeltaNet ≈0.003) well above the 16.7% chance floor. Pure CPU, ~30 s.
• RWKVDecode flat-memory recurrent decoding — the constant-memory autoregressive decode headline of RWKV (the point of BuildRWKVFromSafeTensors), exercised on the core TNNetWKV recurrence via its new incremental state-carry API: BeginIncrementalDecode / Compute (one token) / ResetState / ResetCache / EndIncrementalDecode plus CaptureState/RestoreState session fork — names that mirror TNNetDiagonalSSM and the SDPA KV-cache so a decoder drives every recurrent layer type the same way. Where the ordinary Compute() re-runs a left-to-right scan over the WHOLE sequence each call, ComputeIncremental() advances one token in O(1): it resumes from the persisted RWKV-v4 running numerator/denominator state (A,B,Q) (a fixed 3·C-float triple, independent of position) instead of restarting at (0,0,-∞), applying the EXACT same log-space-stable single-step update. The demo shows two facts. (A) Bit-exact equivalence: feeding a 24-token sequence token-by-token through the incremental path reproduces the full-sequence/prefill Compute() output with max abs error 0.0 (< 1e-5). (B) Flat work/memory: after warming to position 64, four timed chunks of 128 decode steps each show us/step that does NOT trend up with position (constant 3·C-float state carried regardless of context length) — the constant-memory contrast against a transformer KV cache whose per-step attention work grows ~linearly with context. Self-contained and offline (tiny random-init TNNetWKV, no checkpoint). Token-shift block-level decode integration into TNNetStreamingDecoder is a noted follow-up (the layer-level incremental API lands here). Pure CPU, ~1 s.
• MambaDecode flat-memory recurrent decoding — the constant-memory autoregressive decode headline of Mamba (the point of BuildMambaFromSafeTensors), exercised on the core Mamba/S6 selective scan TNNetSelectiveSSM via its new incremental state-carry API — the direct sibling of RWKVDecode. The same uniform vocabulary: BeginIncrementalDecode / Compute (one token) / ResetState / ResetCache / EndIncrementalDecode plus CaptureState/RestoreState session fork, mirroring TNNetWKV, TNNetDiagonalSSM and the SDPA KV-cache so a decoder drives every recurrent layer type the same way. Where the ordinary Compute() re-runs the left-to-right selective scan over the WHOLE sequence each call, ComputeIncremental() advances one token in O(1): it resumes from the persisted [d_inner × d_state] hidden state h (a fixed Depth·DState-float matrix, independent of position) instead of restarting at h=0, applying the EXACT same single-step recurrence h_t = exp(-Δ·exp(A))(*)h_{t-1} + Δ·B_t·x_t, y_t = C_t·h_t + D·x_t (with the input-dependent Δ/B/C — the "selective" part). All three layer modes (legacy DState=1, multi-state real-Mamba DState>1, Jamba inner-norm) share the path. The demo shows two facts. (A) Bit-exact equivalence: feeding a 24-token sequence token-by-token through the incremental path reproduces the full-sequence/prefill Compute() output with max abs error 0.0 (< 1e-5). (B) Flat work/memory: after warming to position 64, four timed chunks of 128 decode steps each show us/step that does NOT trend up with position (constant Depth·DState-float state carried regardless of context length) — the constant-memory contrast against a transformer KV cache whose per-step attention work grows ~linearly with context. Self-contained and offline (tiny random-init TNNetSelectiveSSM, no checkpoint). Full Mamba-block conv-state decode (the causal conv1d ring buffer) + TNNetStreamingDecoder wiring is a noted follow-up (the SSM-leaf incremental API lands here). Pure CPU, ~1 s.
• RWKVGenerate end-to-end flat-memory RWKV block decoding — the follow-up RWKVDecode/MambaDecode deferred: driving a whole RWKV block token-by-token, not just the bare TNNetWKV leaf. The missing piece was the other stateful layer in an RWKV block — TNNetTokenShift, the per-channel time-shift that mixes x_t with x_{t-1} — which now gets the same incremental state-carry API as TNNetWKV/TNNetSelectiveSSM: BeginIncrementalDecode / Compute (one token) / ResetState / ResetCache / EndIncrementalDecode plus CaptureState/RestoreState. Its single-step output is the EXACT algebraic equivalent of one step of the full-sequence shift y[t,c]=mix[c]·x[t,c]+(1-mix[c])·x[t-1,c] (x[-1,c]=0), resuming x_{t-1} from a persisted Depth-long previous-token buffer (the entire carried state, independent of position) instead of re-scanning. A new net-wide driver — TNNet.BeginIncrementalDecode / ResetIncrementalDecode / EndIncrementalDecode — loops the layers and switches every zero-arg recurrent leaf (TNNetTokenShift, TNNetWKV, TNNetSelectiveSSM, TNNetDiagonalSSM) onto its O(1)-per-step path together, so a complete AddRWKVBlock (time-mix TokenShift→r/k/v projections→WKV→gate→out-proj plus a channel-mix sub-block with its own two TokenShifts) decodes one token at a time with all stateful layers advancing in lockstep (the stateless per-token pointwise/sum/concat/norm layers need no state). The demo builds a small COMPLETE RWKV LM net (embedding→two AddRWKVBlock→norm→vocab logits) and shows (A) bit-exact equivalence: the driver reports 8 recurrent leaves switched on (2 blocks × (time-mix TokenShift+WKV + 2 channel-mix TokenShift)), and decoding 20 tokens one-at-a-time reproduces the full-sequence next-token logits with max abs error 0.0 (< 1e-5) and the identical greedy argmax at every step (0/20 mismatches); (B) flat work/memory: after warming to position 64, four timed chunks show us/step that does NOT trend up with position (fixed-size state regardless of context). Attention KV-cache layers (which take a MaxContext budget) stay on the TNNetStreamingDecoder path; this driver targets the zero-arg recurrent leaves. Self-contained and offline (tiny random-init RWKV net, no checkpoint). Pure CPU, ~1 s.
• CrossWKV two-source external-memory recall — the headline demo for the new TNNetCrossWKV layer, a two-source variant of the RWKV-4 WKV time-mixing recurrence (Peng et al. 2023, arXiv:2305.13048). Where TNNetWKV splits its OWN input into the k|v pair driving its state — so the memory it accumulates and the stream that reads it are ONE sequence — TNNetCrossWKV reads the key|value stream from a SEPARATE source than the receptance/query stream, exactly as TNNetCrossAttention generalises self-attention's packed Q|K|V to two sources. Per channel/timestep it runs the EXACT log-space-stable RWKV-v4 kernel (wkv_t = (a_{t-1}+e^{u+k_t}v_t)/(b_{t-1}+e^{u+k_t}), a_t=e^{-w}a_{t-1}+e^{k_t}v_t, per-channel w=softplus(w_raw) + bonus u, running-max stabiliser) with k,v drawn from the key|value source and a sigmoid(r_t) receptance gate read from the query source: y_t = sigmoid(r_t)·wkv_t. The key|value source index is serialized like TNNetConcat/TNNetCrossAttention (round-trips through SaveToString/LoadFromString); exact coupled-BPTT folds dL/dk,dL/dv into the key|value source, dL/dr into the receptance source, and dL/dw,dL/du per channel (input grads into BOTH sources + weight grads finite-difference checked). The layer offers two seqlen contracts via the pAsymmetric constructor flag (FStruct[1], serialized): (a) the default symmetric/v1 contract — equal length on both sources, read-out at t uses the state accumulated over the kv source up to t; and (b) the asymmetric/full-context cross (Create(KV, pAsymmetric=true)) — a rectangular QSeqLen × KVSeqLen shape exactly like TNNetCrossAttention, where the kv memory is summarised once by a single decay scan over ALL key|value positions and EVERY query position reads that SAME full-context summary wkv = A/B gated by its own receptance (y_i = sigmoid(r_i)·(A/B)), so the query stream may be a DIFFERENT length than the memory — true permuted associative recall, not a position-aligned copy. In the asymmetric path the per-query bonus u is unused (no current-token term) so only the decay w carries weight gradient; the rectangular shape and flag round-trip through SaveToString/LoadFromString, and input grads into both sources + the w_raw grad are finite-difference checked with QSeqLen ≠ KVSeqLen. The demo is a cross-copy task — a memory sequence carries a re-randomised value stream, a SEPARATE query sequence carries read pulses, and the read-out must reproduce the memory value at each position (a value living ONLY in the memory tensor). Headline: the two-source arm reaches 100% exact recall (MSE 0.005) while a memory-blind single-source TNNetWKV that only sees the query stream stays at the 16.7% chance floor — the capability the two-source layer adds. Pure CPU, ~20 s.
• Gated Linear Attention per-channel forget gate — the headline demo for the new TNNetGatedLinearAttention layer, Gated Linear Attention (GLA) (Yang et al. 2023, Gated Linear Attention Transformers with Hardware-Efficient Training, arXiv:2312.06635). It carries a (d×d) matrix memory S and updates it per timestep with a data-dependent, per-channel (vector) diagonal forget gate α_t = sigmoid(W_a x_t): S_t[d,e] = α_t[d]·S_{t-1}[d,e] + k_t[d]·v_t[e], read-out y_t = Sᵀ_t q_t. Each key channel d decays its own slice of memory by its own input-dependent factor α_t[d] ∈ (0,1) before the new outer-product write. This is the only layer in tree with a data-dependent VECTOR forget gate on a 2-D state, distinct from TNNetWKV (FIXED-learned per-channel exp decay, not input-dependent), TNNetRetention (a single SCALAR γ), TNNetMLSTMCell (scalar exp gates + running-max) and TNNetDeltaNet (scalar WRITE gate, no multiplicative forget). Keys are L2-normalized; the exact dL/dS is carried right-to-left through the gated write (into α, k, v and the per-row α_t-scaled carry) and the read-out (input + all four weight sets + the gate bias finite-difference checked). The demo is the same overwrite key→value recall task as DeltaNet/RWKV (a key is written, RE-written with a new value, then queried for its MOST RECENT value): at a comparable parameter budget the GLA arm reaches 100% exact recall (MSE 0.009) — matching the delta-rule TNNetDeltaNet (100%, MSE 0.008) and clearly ahead of the single-scalar fixed-decay TNNetRetention baseline (96.3%, MSE 0.019), which blends the stale and fresh values. A fourth arm wires the mixer via the new TNNet.AddGatedLinearAttention builder (token-shift + per-token projection into the leaf + sigmoid receptance gate + out-projection) to show the drop-in time-mixing block; chunked/parallel forward remains a noted follow-up. Pure CPU, ~3 s.
• Gated Linear Attention block tower — the headline demo for the new TNNet.AddGatedLinearAttentionBlock builder, a full transformer-style block that wraps the gated-linear-attention time mixer (TNNet.AddGatedLinearAttention, around the TNNetGatedLinearAttention leaf — Yang et al. 2023, arXiv:2312.06635) in a pre-norm residual + token-wise SwiGLU FFN structure, mirroring AddTransformerEncoderBlock but swapping the multi-head self-attention arm for gated linear attention: x := x + GLA(LayerNorm(x)) then x := x + FFN(LayerNorm(x)) with FFN = PointwiseConvLinear(d_model)∘SwiGLU∘PointwiseConvLinear(2·d_ff). It is shape-preserving over a (SeqLen,1,d_model) sequence so blocks stack into a deep tower; PreNorm=False moves the norm after each residual sum and NormClass swaps the norm class (default TNNetLayerNorm). Pure builder (no new leaf class, no new save/load format). The demo runs two arms on the SAME overwrite key→value recall task (a key is written, RE-written with a new value, then queried for its MOST RECENT value): a single bare AddGatedLinearAttention mixer vs a 3-block tower of AddGatedLinearAttentionBlock, both between $1 \times 1$ projections. Headline (asserted): the residual+FFN tower trains stably and reaches 100% exact recall (MSE 0.007) vs the bare mixer's 81.0% (MSE 0.033) — the LayerNorm/residual/FFN scaffolding is what makes deep stacking of the mixer pay off. Pure CPU, ~8 s. The example is covered by TestAddGatedLinearAttentionBlockShape/TestAddGatedLinearAttentionBlockSmokeTrain in the test suite.
• LRU block tower (Linear Recurrent Unit) — the headline demo for the new TNNet.AddLRU builder, the full transformer-style block wrapping the stable complex-diagonal TNNetLRU recurrence (Linear Recurrent Unit, Orvieto et al. 2023, Resurrecting Recurrent Neural Networks for Long Sequences, arXiv:2303.06349). AddLRUMixer builds the shape-preserving D→D arm — input PointwiseConvLinear(D) projection → TNNetLRU complex-diagonal scan → a GLU non-linearity (PointwiseConvLinear(2D) → TNNetSwiGLU, i.e. (Wx)⊙SiLU(Vx)) → output PointwiseConvLinear(D) — and AddLRU(d_ff, PreNorm, NormClass) wraps it in a pre-norm residual + token-wise SwiGLU FFN exactly like AddGatedLinearAttentionBlock, so blocks stack into a deep tower over a (SeqLen,1,d_model) sequence (PreNorm=False moves the norm after each residual sum; NormClass defaults to TNNetLayerNorm). Pure builder — no new leaf class, no new save/load format. The demo trains a 2-block AddLRU tower on a 24-step delayed-recall task (a symbol presented early must be reproduced after a long delay): held-out recall climbs from chance (16.7%) to 96.2% (recall CE 4.29 → 0.26). Pure CPU, ~18 s. Covered by TestAddLRUBlockShape/TestAddLRUBlockSerializationRoundTrip/TestAddLRUBlockSmokeTrain in the test suite.
• Titans test-time neural long-term memory — the headline demo for the new TNNetTitansMemory layer, the Memory-as-Context (MAC) leaf-layer variant of Titans (Behrouz et al. 2024, Titans: Learning to Memorize at Test Time, arXiv:2501.00663). The hidden state is a small inner MLP M(z)=W2·GeLU(W1·z) whose weights are gradient-descended AT INFERENCE on the per-token associative loss ½‖M(k_t)−v_t‖², with the two mechanisms that separate Titans from plain Test-Time Training: (a) a momentum / "surprise" state S_t = η⊙S_{t−1} − θ⊙∇_t so a surprising store token keeps writing for several steps, and (b) a data-dependent forget gate M_t = (1−α_t)⊙M_{t−1} + S_t (α_t = sigmoid(α_raw + W_α x_t)) that leaves a stored association untouched while bland distractor tokens stream past. Read-out y_t = M_t(q_t). The outer training is exact second-order BPTT (a GeLU Hessian-vector product) carrying dL/dW1, dL/dW2, dL/dS right-to-left through the coupled momentum/forget adjoint scans — input + all weight sets numerically gradient-checked (max-abs err ≈5e-4). The demo is a long-context associative-recall task: store cNumPairs key→value pairs up front, then a long span of cDistractor=24 random distractor tokens, then query one stored key. Two param-matched arms train on the SAME stream: the Titans arm vs a plain FIXED-decay linear-attention baseline (TNNetRetention, gamma NOT learnable). Headline (asserted): across the long distractor span the test-time neural memory wins on BOTH metrics — recall MSE 0.16 / exact-recall 33.3% vs the fixed-decay baseline's MSE 0.20 / 19.8%, which unavoidably bleeds every stored value toward zero at its constant decay rate. Pure CPU, ~18 s. Noted follow-ups: a gated-DeltaNet-style chunked parallel scan and an AddTitansMemory MAC-residual builder.
• Linear Recurrent Unit long-range integration — the headline demo for the new TNNetLRU layer, the Linear Recurrent Unit (Orvieto, Smith, Gu, Fernando, Gulcehre, Pascanu & De 2023, Resurrecting Recurrent Neural Networks for Long Sequences, arXiv:2303.06349) — a stable, diagonal, COMPLEX-valued linear recurrence over the time axis. It is distinct from the existing real-diagonal TNNetDiagonalSSM (h_t = a·h_{t-1} + b·x with a REAL a = sigmoid(a_raw) ∈ (0,1)): the LRU's two defining features are (1) a complex eigenvalue λ = exp(−exp(ν) + i·exp(θ)) whose magnitude |λ| = exp(−exp(ν)) < 1 is stable BY CONSTRUCTION (the exp-of-exp parameterisation keeps it strictly inside the unit disk for any real ν, so a channel can sit arbitrarily close to |λ|=1 — a leak-free accumulator — without ever going unstable), and (2) a normalisation factor γ = sqrt(1−|λ|²) on the input drive so the hidden-state variance stays constant as |λ|→1. Per channel the COMPLEX state is h_t = λ·h_{t-1} + γ·B·x_t (carried as real & imaginary parts) read out as y_t = Re(C·h_t) + D·x_t; the complex λ lets a channel encode a damped oscillation (a rotation by exp(i·exp(θ)) per step) that a real-only diagonal SSM cannot. Six per-channel real params (ν, θ, B, Cre, Cim, D); forward is a left-to-right scan and backward is exact BPTT carrying dL/d(Re h), dL/d(Im h) right-to-left, chaining through |λ|=exp(−exp(ν)), angle=exp(θ) and γ=sqrt(1−|λ|²) (input + all six weight groups finite-difference checked, max-abs err ≈6e-4 input / ≈1.5e-3 weights). The demo is a causal prefix-sum / long-range integration task: a signal channel (plus irrelevant noise channels) must be integrated into a running cumulative sum at every position, evaluated at the FINAL position which integrates the whole 24-step window. Warm-started toward the long-range regime (|λ|≈0.999, γ·B≈1 so each channel is a true running sum, exactly the paper's "init |λ| near 1" advice), the LRU arm drives the integration MSE to 0.00003 while a param-comparable memoryless per-token MLP — which can only see the current token — is stuck at MSE 0.66, ~20000× worse. Per-channel scalar scan (scalar is fine, no AVX). Pure CPU, ~3 s.
• PonderNet adaptive computation time — the headline demo for the new TNNetPonderHalting halting head + TNNet.AddPonderNetBlock builder + TNNetPonderCostLoss regularizer (PonderNet, Banino, Balaguer & Blundell 2021, PonderNet: Learning to Ponder, arXiv:2107.05407): a learned PROBABILISTIC halting paradigm, distinct from the implicit fixed-point Deep Equilibrium block and from any fixed-depth stack. A weight-tied step function f is applied up to MaxSteps times (h_n = h_{n-1} + f(h_{n-1}), parameter count independent of MaxSteps via TNNetDeepEquilibriumSharedConv weight sharing); a shared tiny halting head emits λ_n = sigmoid(...) ∈ (0,1) per step, giving the geometric halting distribution p_n = λ_n·∏_{k<n}(1−λ_k) (last step forced to λ=1 so the p_n sum to 1). The block output is the smooth p_n-weighted sum of the per-step states — no hard argmax, so the whole block is differentiable end-to-end (input gradient finite-difference checked at 4e-5). The TNNetPonderCostLoss head adds KL(p ‖ truncated-geometric(prior_λ)), a regularizer that pulls the expected step count toward the prior's mean and so makes the model pay to ponder — it only spends extra steps where the task forces it. The toy is parity of a variable-length bit string (difficulty L = number of active leading bits — a textbook "harder needs more sequential XOR steps" problem); both the task cross-entropy and the KL ponder cost train in one backward pass by DeepConcat-ing the two heads into a single output. Headline (asserted): the network's expected ponder steps E[n]=Σ(n+1)·p_n RISE monotonically with difficulty — ≈2.85 at L=1 up to ≈3.51 at L=6 — adaptive computation time emerging from the learned halting distribution. Inference always unrolls MaxSteps (static shapes; a true cumulative-p_n early-exit would need dynamic shapes the unrolled-graph API lacks, documented). Pure CPU, ~13 s.
• Time-series forecasting (causal-conv stack) — a one-screen forecasting demo on a SYNTHETIC seasonal+trend+noise series (generated in-code, no downloads). A causal 1-D conv stack — TNNetCausalConv1D (incl. a dilation-2 layer for a longer receptive field) + TNNetPointwiseConvLinear + an FC head, ~5k weights — predicts the next value from a sliding window, then is rolled out auto-regressively for 24 steps. Shows the train/val MSE curve dropping (1.01 → 0.013 over 40 epochs) and a final horizon error of MAE 0.22 / RMSE 0.27 vs a naive-persistence baseline of MAE 1.00 (~5× better). Pure CPU, ~1-2 min.
• Growing Neural Cellular Automata — a tiny pure-CPU reproduction of Growing Neural Cellular Automata (Mordvintsev, Randazzo, Niklasson & Levin 2020, distill.pub/2020/growing-ca) built with no new layer classes — the headline use of TNNetConvolutionSharedWeights. A $16 \times 16$ grid of cells carries Ch=12 channels (4 visible RGBA + 8 hidden scratch); one CA rule step is a shared-weight residual conv stack applied to every cell in place — learned $3 \times 3$ perceive conv → TNNetPointwiseConvReLU → TNNetPointwiseConvLinear update added residually, then clamp via a bounded leaky TNNetReLUL. The rule is unrolled T=32 times sharing ONE set of weights (so the ~4.3k trainable params are independent of T); because the steps are ordinary chained layers, TNNet.Backpropagate does exact BPTT through the whole recurrence for free, with the SetBatchUpdate(True) idiom and gradient-norm clipping. Trained with an L2 loss to a target glyph (a chunky letter "A") at the final step, the net grows the glyph from a single live seed pixel — ASCII renders show diffuse noise at step 4, the "A" emerging by step 16, and a clean glyph at step 32 (L2 0.9 → ~0.002). Stability needs three standard NCA guards (zero-init update head, state clamp, grad clip) or the 32-deep residual recurrence NaNs in one update. The paper's stochastic update mask, sample-replacement pool and damage-regeneration are dropped to stay deterministic and in-budget (documented honestly in the README). Full BPTT through all 32 steps fit the budget — pure CPU, ~61 s on two cores.
• Normalizing flow (exact-likelihood density) — the headline demo for the new TNNetAffineCoupling layer, an exact-likelihood normalizing-flow primitive: an invertible RealNVP/Glow-style affine coupling (Dinh et al. 2016, Density estimation using Real NVP, arXiv:1605.08803; Kingma & Dhariwal 2018, Glow, arXiv:1807.03039). It splits the Depth axis into two halves, passes one half unchanged, and applies an affine map y_b = x_b·exp(s) + t to the other, where the per-channel log-scale s and shift t come from a tiny conditioner reading the unchanged half; s is tanh-clamped (Glow's stability trick). The map is analytically invertible (x_b = (y_b − t)·exp(−s), exposed via the pInverse constructor flag for sampling) and its Jacobian log-determinant is just sum(s), surfaced as the public LogDetJacobian property — so a stack of couplings trains by exact maximum likelihood under a unit-Gaussian base (loss = ½‖z‖² − Σ couplings log-det). This is distinct from the memory-saving AddReversibleBlock (a RevNet recompute trick with NO tractable Jacobian) and from the TNNetMixtureDensity head (a density head that is not invertible). The pTransformSecond flag alternates which half is transformed across stacked layers so every channel is updated; the -Σs (negative log-det) gradient is folded into each layer's backward pass (LogDetLossWeight, default 1) so the whole flow trains end-to-end by injecting only the data-loss gradient dL/dz = z. Both the forward transform AND the conditioner weights are finite-difference gradient-checked, and the negative-log-det term is checked separately; the forward∘inverse round-trip reconstructs the input to ~1e-7 (exact bijection). The demo fits a 2-D two-moons density with 6 alternating couplings, prints the mean log-likelihood climbing (≈−2.6 → −1.5), then draws z ~ N(0,I) and pushes it through the inverse flow z → x to generate new points on the data manifold. The demo also interleaves the coupling layers with TNNetInvertible1x1Conv (Glow's learnable invertible 1×1 convolution, a trainable generalization of the fixed channel permutation, with the cheap LU-parametrized Σ log|s| log-det) — the real Glow step — and shows the learnable mixing reaching a higher mean log-likelihood (≈ −1.16) than the fixed-permute baseline (≈ −1.56). Pure CPU, ~35 s (baseline arm) / ~150 s (Glow arm).

Embeddings & metric learning

• Char tokenizer + embedding lookup — smallest possible TNNetEmbedding demo: builds a unique-char vocabulary from a hard-coded in-memory pangram corpus, trains a single-token char -> next-char classifier through a 16-d embedding lookup, then prints the top-5 nearest characters by cosine similarity on the embedding rows for a few probe chars ('q', 'e', 't', space). Pure CPU, finishes in under one second, no external data.
• Word2Vec skip-gram — from-scratch word2vec skip-gram with negative sampling (SGNS, Mikolov et al. 2013) on a tiny built-in (~400-token, 70-word) corpus, no download. The canonical two-matrix design is realised as two Input + TNNetEmbedding(vocab, d) networks (an input/center matrix W and an output/context matrix W'); positive (center, context) pairs are formed within each sentence and K negatives are drawn from a unigram^0.75 table per positive. The sigmoid/BCE SGNS loss L = -log σ(v_c·u_w) - Σ log σ(-v_c·u_neg) and its analytic gradient are computed in Pascal and seeded into each TNNetEmbedding via the "target = output - grad" trick (frequent function words are kept in the vocabulary but excluded as center/context/negative, the role of frequent-word subsampling). It then prints cosine nearest-neighbour lists (king↔queen, dog↔cat, cat→kitten, boy→girl, ...) and the textbook analogy arithmetic — king - man + woman → queen (0.74), prince - man + woman → princess (0.85), puppy - dog + cat → kitten (0.87) all land; a thinly-attested seniority analogy honestly does not. The unsupervised distributional-semantics counterpart to SimpleNLP (char-LM) and CharTokenizer (tokenisation). Pure CPU, deterministic (fixed RandSeed), finishes in well under ten seconds.
• Cosine-Embedding Siamese - A shared-weight (siamese) embedding MLP trained with the TNNetCosineEmbeddingLoss head on a synthetic "same vs different class" pair task. Both members of a pair pass through identical pointwise-conv weights, are L2-normalized, and reshaped into the head's a|b|y depth layout (the per-pair label y is depth-concatenated, not an external target). Prints learned cosine-similarity histograms: same-class pairs collapse onto cos≈+1 while different-class pairs are driven below the margin. Pure CPU, ~1s.
• Triplet embedding (metric learning) — learns a 3-D unit-sphere embedding of a synthetic 4-class 2D-Gaussian-blob dataset with the TNNetTripletLoss head and TNNetL2Normalize, using one weight-shared siamese network: a triplet (anchor, positive, negative) is fed as three X positions of a (3,1,2) input, a pointwise-conv MLP (featuresize=1, so the same weights apply at every position) embeds each point, TNNetL2Normalize projects each onto the unit sphere, and a TNNetReshape(1,1,3*embed_dim) lays the three embeddings out as the anchor|positive|negative depth layout the loss head consumes (no external target — supervision is implicit in that layout). After training it prints a per-class mean pairwise cosine-similarity matrix (within-class ~0.99, cross-class low) and dumps the learned embeddings to embeddings.csv for plotting. Pure CPU, deterministic (fixed RandSeed), finishes in under a second.
• InfoNCE contrastive embedding — learns a unit-sphere embedding of a synthetic multi-class task with the TNNetInfoNCELoss head and TNNetL2Normalize, using one weight-shared encoder: a sample packs a query, its positive (another augmented view of the same class), and K-1 negatives (views of other classes) as the K+1 X positions of a (K+1,1,in_dim) input; a pointwise-conv MLP embeds each view, TNNetL2Normalize projects them onto the unit sphere, and a TNNetReshape(1,1,(K+1)*embed_dim) lays them out as the q | k_0(+) | k_1..k_{K-1}(-) depth layout the loss head consumes (no external target — the head seeds its own gradient via a temperature-scaled softmax L = -s_0 + logsumexp_j(s_j)). Unlike TNNetTripletLoss (a margin/hinge loss with a single negative), InfoNCE contrasts the positive against K negatives at once. Reports the positive-vs-negative cosine gap and the Wang & Isola (2020) alignment/uniformity before vs after training (gap widens, alignment drops, loss falls ~50x). Pure CPU, deterministic (fixed RandSeed), finishes in seconds.
• Center loss + softmax (joint) — reproduces the headline result of Wen et al. 2016, A Discriminative Feature Learning Approach for Deep Face Recognition: softmax cross-entropy alone makes the classes separable but leaves their features spread out, while adding the center-loss penalty jointly (lambda-weighted) pulls each sample toward its learned class center, giving visibly tighter intra-class clusters at the same accuracy. Two arms train on the same architecture / seed / data — ARM A softmax-only (TNNetCenterLoss off), ARM B softmax + TNNetCenterLoss(K, lambda=0.30) joint — and the demo reports, per arm, final accuracy, an intra-class tightness metric (mean within-class feature radius), the inter-class mean separation, the intra/inter ratio (smaller = cleaner), and an ASCII scatter of the 2-D embedding so the tightening is visible with no external plotting. Both arms reach 1.0 accuracy, but ARM B's intra-class radius drops ~17x (1.97 → 0.11) and its intra/inter ratio ~5x (0.141 → 0.029) — the Wen et al. headline. Joint wiring: a shared 2-D PointwiseConvLinear embedding feeds two consumers — a PointwiseConvLinear(K) logits head AND a DeepConcat([emb, label]) -> TNNetCenterLoss penalty head — rejoined at a final DeepConcat([logits, center]) so a single Backpropagate walks back through both and the embedding accumulates gradients from both heads (the departing-branch counter). The cross-entropy gradient is seeded on the logits region (target = output - (softmax - onehot)) while the center region's seeded residual is ignored (the center head self-generates its own lambda*(x - center_c) pull). Pairs with FeatureSeparability (which measures cluster geometry) and ArcFaceEmbedding (the angular-margin alternative). Pure CPU, single-threaded, ~2 s.
• Matryoshka embedding (nested-prefix representation learning) — trains ONE encoder whose single d=64 embedding has nested PREFIXES {8,16,32,64} that are EACH independently usable (Kusupati et al., NeurIPS 2022). The shared MLP feeds K classifier heads, each reading only the first p channels via TNNetSplitChannels(0, p) -> TNNetFullConnectLinear -> TNNetSoftMax; all heads are TNNetDeepConcat'd and the target is K stacked one-hots, so the softmax+CE backward sums the per-prefix losses (coordinate 0 gets gradient from all 4 heads, coordinate 63 only from the widest — the Matryoshka nesting). Unlike the fixed-width embedding examples (Triplet/InfoNCE/ArcFace/Cosine), one run yields a whole accuracy-vs-width curve and the vector can be truncated at retrieval time for free. Prints an ASCII accuracy-vs-width curve and contrasts the 8/16-dim prefixes against separately-trained DEDICATED fixed-8/fixed-16 baselines (observed: prefixes within ~1% of the dedicated models). README documents the prefix-loss-weighting pitfall (small prefix must not dominate the gradient). Synthetic 8-class Gaussian-blob task (MNIST would blow the budget given 3 models are trained), pure CPU, fixed RandSeed, ~6.5s.
• Semantic search with imported MiniLM sentence embeddings — an end-to-end USE of an imported encoder checkpoint (GPT2Import/LlamaImport cover decoders): embeds a small corpus with a real sentence-transformers/all-MiniLM-L6-v2 snapshot (BuildBertFromSafeTensors + the WordPiece tokenizer.json support in neuralhftokenizer.pas) and ranks it by cosine similarity for a query — the paraphrase ("kitty on the window ledge" vs "cat on the windowsill") wins at 0.72 over distractors below 0.34. The sentence vector is the sentence-transformers recipe via the neuralpretrained.pas SENTENCE EMBEDDINGS helpers: BertTokenizeSentence ([CLS] ids [SEP]) -> encoder hidden states -> BertPoolSentenceEmbedding (attention-mask-aware MEAN pooling over the real tokens only — deliberately NOT TNNetAvgChannel, which would average pad rows too — then L2 normalize, so cosine = dot product). Since the imported encoder has no attention padding mask, the demo never pads: it caches one inference-only net per distinct token length, which keeps the Pascal embeddings at cosine = 0.9999999 vs HuggingFace per sentence (compare_st_embeddings.py, bar > 0.999). Needs the ~90 MB checkpoint download; embeds 9 sentences in a few seconds on CPU.
• E5/BGE pooling-mode + instruction-prefix retrieval — the self-contained sibling of SemanticSearch: where that demo needs a ~90 MB MiniLM download and only covers the mean-pool / no-prefix case, this one exercises the two pieces that distinguish the published E5 / BGE / GTE retrievers — the pooling-mode selector (PoolSentenceEmbedding + TNNetEmbedPooling: epCLS for BGE, epMean for E5/GTE, epLastToken for e5-mistral) and the instruction-prefix table (EmbedInstructionPrefix/ApplyEmbedInstruction + TNNetEmbedInstruction: E5's "query: "/"passage: ", BGE's "Represent this sentence …" query instruction, the gte-Qwen2 "Instruct: …\nQuery: " template) — entirely on the committed pico fixture tests/fixtures/tiny_e5.*, so it runs in under a second on CPU with no network. The mean-pooled + L2-normalized query/passage vectors match the HF transformers float64 oracle within 1e-4 (tools/e5_embed_tiny_fixture.py, TestE5EmbeddingParity); no sentence-transformers install needed (for E5/BGE its SentenceTransformer is exactly AutoModel forward → mean/CLS pool → L2 normalize in float64, reproduced by the fixture maker). The shared-body passage ranks above the unrelated one. Always built pTrainable=false; pure CPU.
• DeBERTa-v3 cross-encoder reranking (RAG second stage) — the canonical retrieval-augmented-generation reranker demo on an imported DeBERTa-v3 *ForSequenceClassification checkpoint (the ms-marco family: cross-encoder/ms-marco-..., naver/trecdl...), using BuildDebertaV2FromSafeTensorsEx(..., pSeqClsHead=true) (neuralpretrained.pas) and a disentangled-attention encoder (TNNetDisentangledAttention: content-to-content plus content-to-position plus position-to-content, the position terms projecting a separate relative-position embedding table gathered by clamped log-bucketed relative distance — distinct from the scalar avT5RelPosBias). A cross-encoder concatenates the query and a candidate passage into one sequence ([CLS] query [SEP] passage [SEP]) and reads the row-0 classifier logit as the relevance score, jointly attending every query token to every passage token (far more accurate than independent bi-encoder cosine, the RAG second stage after a fast retriever returns the top-N). Text mode encodes with the DeBERTa-v3 Unigram tokenizer.json (the landed TNeuralHFTokenizer Unigram reader) when one sits beside the checkpoint; a raw-token-id mode runs offline on the committed pico seq-cls fixture (tests/fixtures/tiny_debertav2_seqcls.*, whose float64 oracle logits the importer reproduces to <1e-4, TestDebertaV2SeqClassificationParity). Ranks the candidates descending by score. Always built pTrainable=false; pure CPU.
• ColBERT late-interaction retrieval (MaxSim, RAG third paradigm) — completes the bi-encoder / cross-encoder / late-interaction retrieval trio (after SemanticSearch and DebertaReranker) on an imported colbert-ir/colbertv2.0-class checkpoint. ColBERT sits between the other two and is different: it keeps the per-token contextual embeddings of query and document (NO pooling), projects each token to a small dim (128) and L2-normalizes via the ColBERT linear head (a bias-free [hidden → 128] dense), then scores a (query, doc) pair by the MaxSim late-interaction sum score = Σ_{q∈query} max_{d∈doc} ⟨E_q, E_d⟩ — every query token matched to its single best document token, then summed (cross-encoder-grade accuracy at bi-encoder cost: docs are pre-encoded once). New neuralpretrained.pas helpers: BuildColBERTFromSafeTensors[Ex] (the stock BuildBertFromSafeTensors encoder + the linear projection head), ColBERTBuildInput (the [Q]/[D] marker convention with query [MASK] augmentation), ColBERTEmbedTokens (the per-token L2-normalized projected matrix, skipping the pooling the bi-encoder helper forces), ColBERTMaxSimScore, and ColBERTRetrievalReport (Recall@k + nDCG@10 driven by MaxSim instead of cosine). The ColBERT forward (encoder + bias-free projection + per-row L2 norm + MaxSim) is pinned against an HF float64 oracle on a synthesized pico checkpoint (tools/colbert_tiny_fixture.py, TestColBERTParity) to <1e-4. Always built pTrainable=false; pure CPU.
• Two-stage retrieve-then-rerank RAG pipeline — the full retrieve-then-rerank pipeline that production RAG runs, wiring the bi-encoder first stage (SemanticSearch's BertEncodeSentence + cosine) to a cross-encoder second stage that re-scores the top-K jointly with the query. The new neuralpretrained.pas CROSS-ENCODER RERANKER helpers complete the missing rung: BertTokenizePair lays out [CLS] query [SEP] passage [SEP] and the parallel segment ids (0 over the query span, 1 over the passage span — HF's token_type_ids, exercising the BERT importer's token_type_embeddings table per-position, with HF longest_first truncation); CrossEncoderScore runs one joint forward through a num_labels=1 *ForSequenceClassification net (feeding the segment ids into channel 1 of the (SeqLen,1,2) input) and returns the [CLS] relevance logit (sigmoid); RerankPassages scores a query against a candidate list (one forward each, optionally int8 via the backbone's pQuantizeInt8) and returns them most-relevant first; RerankReport quantifies the precision lift with MRR / nDCG@k before vs after reranking. Distinct from DebertaReranker (that demo scores a hand-assembled DeBERTa sequence; this one is the standard BERT-family num_labels=1 reranker driven end-to-end from a bi-encoder shortlist via the new library helpers). The pair/segment-id=1 path is pinned against the HF float64 AutoModelForSequenceClassification logit to <1e-4 on a synthesized pico reranker (tools/bert_reranker_tiny_fixture.py, TestRerankerPairLogitParity; the fixture boosts the token_type embeddings so the segment-1 path measurably moves the logit, proving the test is not vacuous); -demo runs the RerankReport lift offline (MRR 0.3333 → 1.0000) on the committed pico fixture with no download. Always built pTrainable=false; pure CPU.
• End-to-end retrieval-augmented generation (RAG) — the canonical RAG application, tying the already-landed retrieval and generation halves together with no new core code. The pipeline: chunk a small built-in knowledge base; embed each chunk and the question with the SemanticSearch sentence-embedding path (BertTokenizeSentence → BuildBertFromSafeTensors encoder → BertPoolSentenceEmbedding mean-pool + L2 normalize, one inference-only net cached per token length); retrieve the top-k chunks by cosine similarity (= dot product of the unit vectors); splice them into the canonical template Context:\n{chunks}\n\nQuestion: {q}\nAnswer:; and generate a grounded answer through the ChatTerminal chat-template + streaming-decode infra (BuildFromPretrained inference-only → EncodeChat/ApplyChatTemplate → full-recompute greedy decode streamed token-by-token). The demo turns on the headline RAG property: the default question ("the launch date of Project Halcyon") asks for an invented fact no pretrained model can know — the bare model can only guess, but once the chunk that states the fact is retrieved into the context the grounded decoder reads it off. Both models are optional CLI paths: --embed-model DIR swaps in a real sentence encoder (e.g. all-MiniLM-L6-v2) but is otherwise replaced by a deterministic built-in hashing-bag-of-words embedder, and --gen-model DIR adds the decoder — so with no arguments and no download the example still runs the chunk→embed→retrieve→splice half end-to-end (the fallback embedder already ranks the Halcyon chunk #1 for the Halcyon question) and prints the spliced prompt it would send. --selftest runs 16 offline unit checks (arg parsing, fallback cosine top-k, prompt-template shape, ChatML render of the spliced prompt) with no model files. Pure CPU; the retrieval/splice half runs within ulimit -v 3000000 in well under a minute.
• Extractive question answering (SQuAD span head) — the extractive-QA path through neuralpretrained.pas, the sibling of the sequence-classification importers (DebertaReranker/seq-cls) on the same BERT backbone but with a different head. A *ForQuestionAnswering checkpoint is the stock BERT-family encoder plus a single [hidden → 2] qa_outputs dense that emits, per token, a start logit and an end logit; the answer is argmax_{s≤e≤s+L} start[s]+end[e] over the context tokens. New helpers: BuildBertForQuestionAnsweringFromSafeTensors[Ex] (reuses TNNet.AddQuestionAnsweringHead — two per-token TNNetPointwiseConvLinear(1) projections DeepConcat'd to (SeqLen,1,2) — and loads qa_outputs row 0 onto the start projection, row 1 onto the end projection); AnswerSpan(question, context, …) (runs [CLS] question [SEP] context [SEP], masks the question/special/pad rows, picks the best valid span over context tokens, recovers the substring via EncodeWithOffsets, and returns the SQuAD2 start[CLS]+end[CLS] null-answer baseline); QAReport (SQuAD Exact-Match + macro token-F1 with the official answer normalization, mirroring STSReport/RetrievalReport); plus the exposed NormalizeSquadAnswer/SquadTokenF1 primitives. The released QA checkpoints (distilbert-base-cased-distilled-squad, deepset/roberta-base-squad2) are ~250 MB, so the importer's span-logit parity is pinned against an HF float64 oracle on a synthesized pico DistilBertForQuestionAnswering (tools/distilbert_qa_tiny_fixture.py, TestDistilBertQALogitParity) to <1e-4 (the fixture boosts the head so a start/end row swap fails loudly); the demo itself hand-wires a span head over the committed WordPiece tokenizer so the whole AnswerSpan/QAReport pipeline runs on CPU in under a second with no download. Swap in BuildBertForQuestionAnsweringFromSafeTensors for a real model — AnswerSpan/QAReport are unchanged.
• Hyperbolic tree embedding (Poincaré ball vs Euclidean) — the headline "trees embed into hyperbolic space with little distortion" win for the TNNetHyperbolicLinear Poincaré-ball layer and its companion readout head TNNetHyperbolicDistance (Nickel & Kiela 2017, Poincaré Embeddings). A complete binary tree of depth 4 (31 nodes) has ground-truth node distance = tree path length; two parameter-matched embedders (one $\text{N} \times 2$ weight matrix, no bias) map each one-hot node id to a 2-D point — Model A TNNetHyperbolicLinear(2, c) into the Poincaré ball with the curvature-c distance dist_c(a,b) = (2/√c)·atanh(√c·‖(-a) ⊕_c b‖) (the same Möbius-distance formula TNNetHyperbolicDistance computes against its prototype bank), Model B a plain TNNetFullConnectLinear(2) with Euclidean ‖a−b‖. Both regress the embedded distance to the (scaled) tree path length over sampled node pairs by a hand-rolled coupled-pair MSE loop (two forward passes per pair, seed the embedder's OutputError with the analytic dMSE/dembedding — finite-difference verified — and backprop each; SetBatchUpdate(True), gradient-clipped boundary-safe step). At the identical 2-D budget the hyperbolic embedding fits the exponentially-branching tree markedly better — held-out MSE ≈0.052 / Pearson corr ≈0.94 vs Euclidean ≈0.155 / ≈0.83 (~3× lower MSE) — because hyperbolic ball volume grows exponentially with radius (matching the tree's branching) while Euclidean space crowds the leaves. Pure CPU, ~4 s.

Efficiency, structured layers & compression

• Affine Fine-Tuning (BitFit-style) - Freeze a pretrained classifier and fine-tune only inserted per-channel affine blocks (TNNet.AddAffineBlock) on a shifted task. Adapts at a small fraction of the trainable parameters (here 68 of 388, ~18%) while recovering most of the accuracy.
• Quantization-Aware Training (QAT) — the headline demo for the TNNetFakeQuantize layer (per-tensor symmetric, observer-driven running-max-abs fake quantization; forward = dequant(quant(x)), straight-through gradient inside the clamp band, Freeze to stop the observer at inference, pQMax selects the bit-width). On a low-SNR synthetic 4-class image task it prints the three accuracy points that motivate QAT: FLOAT (full-precision baseline), PTQ (the same weights int8-quantized via TNNet.QuantizeWeightsInt8 with calibrated-but-frozen low-bit TNNetFakeQuantize activations, no retraining — observers are populated by a 2-epoch pass with every layer's LearningRate := 0), and QAT (the same quantized topology trained from scratch with the fake-quant rounding in the forward pass so the weights converge to a quantization-robust optimum through the straight-through estimator). Aggressive low-bit activations (qmax = 5) are used on purpose — at 8 bits the calibrated fake-quant is nearly lossless on this small net, leaving no PTQ gap to recover. Deterministic run (MaxThreadNum := 1, fixed RandSeed): float 94.25% → PTQ 88.00% (−6.25 pts from quantization) → QAT 91.75% (+3.75 pts recovered, near float). Includes a note on why warm-starting QAT from the float weights fails (pins it to the float net's quantization-sensitive minimum). Self-contained synthetic data, pure CPU, a few seconds under ulimit -v 3000000. Coded by Claude (AI).
• LoRA Fine-Tuning - Parameter-efficient fine-tuning with low-rank adapters (TNNet.AddLoRAAdapter, Hu et al. 2021): freeze a pretrained classifier and train only a zero-initialised rank-r bypass added to a frozen layer. Sweeps r in {1,2,4,8} and charts trainable-param count vs recovered accuracy — the textbook "most of the accuracy at a few % of the params" curve. Pure CPU.
• DPO Fine-Tuning (preference alignment) — aligns a TinyGPT-style char-level causal LM with Direct Preference Optimization (TNeuralDPOTrainer in neural/neuraldpo.pas, Rafailov et al. 2023): preference fine-tuning on (prompt, chosen, rejected) pairs with loss = -ln sigmoid(beta*((logpi_c-logref_c)-(logpi_r-logref_r))) against a frozen cloned reference — no reward model, no RL. A briefly-pretrained style-agnostic model (loss starts at exactly ln 2, margin 0) is pushed to prefer patterned over noise completions: margin 0 -> ~27, preference accuracy 50% -> 100%, loss 0.693 -> 0.0003. Each Step backpropagates the exact sigmoid(-beta*margin)-scaled (softmax - onehot) gradient (positive on chosen tokens, negative on rejected; derivation in the unit header). Pure CPU, ~3 s.
• Magnitude pruning report — runs TNNet.MagnitudePruningReport(NN, Samples [, Labels]), a forward-only no-retrain compressibility diagnostic answering "if I zero the smallest-magnitude weights, how much can I throw away before the model breaks?" — measured by actually pruning and re-running, not predicted from a proxy. It snapshots the whole net once (SaveDataToString), then for each global sparsity s in {0,10,...,90,95,99}% computes the magnitude threshold that zeros the smallest s% of |w| across all trainable layers (a single global percentile — the standard global-magnitude criterion), zeros every |w| <= threshold in place, runs one forward pass over the probe batch to read the resulting loss (and, with labels, top-1 accuracy), and restores the original weights bit-for-bit (LoadDataFromString) before the next level. It reports an accuracy-(or loss-)vs-sparsity #-bar curve, the prunability knee (max sparsity whose top-1 drop stays within Tolerance, default 1%), the per-layer pruned fraction at the knee (which layers absorb the pruning — typically the wide head), the realised-vs-requested sparsity at each level (a built-in check that the percentile threshold hit its target to within one weight), and a highly-compressible / moderate / fragile verdict; an optional PerLayer flag switches to a per-layer percentile (the uniform-per-layer baseline) so global-vs-uniform pruning is visible side by side. The example runs the SAME synthetic 3-class problem three times — an over-wide net (deep knee, highly-compressible), a tight-fit net (shallower knee), and the over-wide net again with the per-layer criterion — so the over-parameterised-is-compressible story shows up in one run. Built-in checks: s=0% reproduces the unpruned loss/accuracy exactly. Distinct from FisherImportanceReport (ranks by a Fisher proxy, never removes weights) and LayerSensitivityReport (random weight jitter, never a magnitude-thresholded zeroing). Pure forward-only — weights are restored bit-for-bit, never stepped.
• Magnitude pruning + fine-tune recovery — the persistent-mask counterpart to MagnitudePruning: instead of restoring the weights, it KEEPS them pruned and fine-tunes. TNNet.PruneWeightsByMagnitude(Sparsity [, PerLayer]) zeros the smallest s% of |w| across the net and installs a persistent keep/prune mask (one volume per neuron) that is re-enforced after every weight update — the base AfterWeightUpdate hook calls ZeroPrunedWeights, covering both the batch UpdateWeights path and the inline online TNNetFullConnect.BackpropagateCPU path, and it also zeros the pruned weights' delta / inertia / Adam-moment entries so a pruned weight can never grow back. The example trains a small 4-class classifier, measures the dense baseline, prunes to a target sparsity (50 / 80 / 90%), measures accuracy right after pruning (a drop — capacity removed), fine-tunes for a few epochs with the mask enforced, and measures accuracy after fine-tuning — printing the dense -> pruned -> fine-tuned triple per sparsity so the recovery is visible (e.g. at 80% sparsity a ~11-pt drop recovers ~+15 pts past the dense baseline). The mask survives throughout (reported pruned-weight count and realised sparsity are unchanged after fine-tuning). Companion helpers: ApplyPruneMasks / HasPruneMasks / ClearPruneMasks / CountPrunedWeights / GetPruneSparsity. Pure CPU, fixed RandSeed, well under a minute.
• Lottery Ticket — Iterative Magnitude Pruning (IMP) — a follow-up to LotteryTicket implementing the paper's ACTUAL method, ITERATIVE magnitude pruning (Frankle & Carbin 2019): loop train → prune the bottom 45% of SURVIVING weights → rewind survivors to theta_0 → retrain for 5 rounds, so (1-0.45)^5 ≈ 5% survivors = ~95% sparsity, the level where the sibling one-shot run collapsed. Prints a per-round sparsity/accuracy table plus a budget-matched head-to-head against a one-shot prune-to-95% baseline, and GRADES it with an explicit verdict. HONEST negative result: with a generous matched epoch budget the one-shot ticket does NOT collapse (the sibling collapses only because every arm got just 100 epochs), so IMP shows no advantage on this easy toy — reported plainly rather than assumed, mirroring the sibling's own 95%-collapse honesty. Pure CPU, ~140 s.
• Knowledge distillation (KL divergence) — trains a teacher MLP on a synthetic multi-class 2D-Gaussian-blob task, then distills a smaller student against the teacher's temperature-softened soft targets using the TNNetKLDivergence head (SoftMax -> TNNetKLDivergence, target = teacher distribution), and contrasts it with an identical-capacity student trained on hard one-hot labels via the standard cross-entropy path. Prints the loss curves and final test accuracy for teacher / distilled-student / hard-label-student so the soft-target effect is visible. Pure CPU, fixed RandSeed, finishes in well under a second.
• Mixture of Experts - Soft/dense Mixture-of-Experts feed-forward block (TNNet.AddMixtureOfExperts, Shazeer et al. 2017): a softmax gating network blends N parallel shape-preserving expert MLPs. Trains on a synthetic multi-task toy where different experts can specialise. Pure CPU, no dataset download. (Hard top-k routing is the TopKMoE example below.)
• Top-k Mixture of Experts (sparse routing + load balancing) — the sparse-dispatch follow-up to the soft AddMixtureOfExperts: TNNet.AddTopKMixtureOfExperts(InputLayer, NumExperts, ExpertHiddenDim, TopCnt, out AuxLossHead, AuxCoeff) routes each token through only its TopCnt highest-gated experts via the new TNNetTopKGate (keeps the top-TopCnt gate weights, zeroes the rest, renormalizes survivors to sum 1, exact fused mask+renorm Jacobian backward) and attaches a load-balancing auxiliary loss TNNetLoadBalanceLoss (Switch Transformer, Fedus et al. 2021: L_aux = coeff·E·Σ_i f_i·P_i, with f_i the stop-gradient fraction of tokens routed to expert i and P_i its mean gate prob, so the gradient flows through P_i only). The demo trains two identical networks — both deliberately seeded collapsed onto one expert — differing only in the aux-loss weight: WITHOUT it the router stays fully collapsed (held-out per-expert load E0=100%, imbalance max/mean = 4.00); WITH it the load becomes near-uniform (≈28/23/26/23%, max/mean = 1.11). Pure CPU, ~19 s. (Gumbel-noise gating and true compute-sparse dispatch are logged follow-ups.)
• Expert-Choice Mixture of Experts (transposed routing, load-balanced by construction) — the transpose of the token-choice MoE family above: instead of each token picking its top-TopCnt experts, each EXPERT picks its top-Capacity TOKENS via the new TNNetExpertChoiceGate (Expert Choice routing, Zhou et al. 2022, Mixture-of-Experts with Expert Choice Routing, arXiv:2202.09368). The gate transposes TNNetTopKGate's logic — it keeps the top-Capacity token positions ALONG the SizeX (token) axis per expert channel and zeroes the rest — so every expert processes EXACTLY Capacity tokens and load balance is guaranteed structurally, with NO Switch-style TNNetLoadBalanceLoss head required (and a token may be handled by 0, 1, or several experts). The builder TNNet.AddExpertChoiceMixtureOfExperts(InputLayer, NumExperts, ExpertHiddenDim, Capacity) reuses the per-expert MLP + SplitChannels/DeepConcat.Replicate/CellMulByCell combine of AddTopKMixtureOfExperts, swapping only the gate; the survivor gate value is the raw per-token softmax (no renorm) so the backward is a hard 0/1 straight-through mask. The demo drives both gates on the SAME skewed gate matrix and prints the per-expert token counts: token-choice top-1 → [7 3 2] (spread 5, lopsided) vs expert-choice Capacity → [4 4 4] (spread 0, uniform by construction). Pure CPU, sub-second.
• Mixture of Depths - Conditional-compute block (TNNet.AddMixtureOfDepths, Raposo et al. 2024): a per-token router sends only the top-Capacity sequence positions through a wrapped block and lets the rest bypass it via the residual path, so FLOPs drop by (SeqLen-Capacity)/SeqLen at static tensor shapes. Sweeps Capacity ∈ {SeqLen, SeqLen/2, SeqLen/4} on a tiny next-token task and charts the loss/accuracy vs processed-token-fraction trade. Pure CPU.
• Early-exit / adaptive-inference network — the BranchyNet (Teerapittayanon, McDanel & Kung 2016) "anytime inference" pattern: ONE trunk of stacked FC+ReLU blocks with an auxiliary softmax classifier head branching off after each block (wired via AddLayerAfter and all heads Concat'd into a single packed K*NumClasses output), trained JOINTLY by deep supervision — a manual loss loop seeds each head's (p - onehot) softmax-cross-entropy gradient through the Concat and a single Backpropagate (under SetBatchUpdate(true)) accumulates into the shared trunk. The synthetic task is difficulty-graded (easy well-separated blobs mixed with hard near-the-margin points). At INFERENCE a confidence-gated dynamic-compute policy walks heads shallow→deep and EXITS at the first head whose softmax max-prob exceeds a threshold tau, recording the per-sample exit depth; sweeping tau prints the accuracy-vs-average-exit-depth trade-off (split by easy/hard) as an ASCII chart, showing easy samples leave at depth ~1 while hard ones run the full depth. Two built-in invariants hold: tau=1.0 forces every sample to the final head (== plain full-depth accuracy, exactly) and average exit depth is monotone non-decreasing in tau. Distinct from the PredictionDepth example above: there the heads do not exist — a post-hoc k-NN probe measures where a FIXED single-head net "makes up its mind"; here the early heads are TRAINED and actually GATE compute. Pure CPU, runs in ~9 s.
• HyperNetwork — context-generated weights — the headline demo for the new TNNetHyperLinear (Ha, Dai & Le 2016, HyperNetworks), a layer that owns NO trainable weights of its own: its weight matrix is GENERATED on the fly by an upstream network and read from a SECOND input tensor rather than from Neurons[].Weights. Two-source wiring (like TNNetCrossAttention/TNNetAffineGridSample): the main feature vector on PrevLayer, a flat generated-weights vector (Din*Dout (+Dout) row-major matrix + optional bias) on WeightsSource; forward is y = W_gen·x (+b) and backward propagates BOTH into the main features (W_gen^T·dy) AND back into the generated-weights tensor (dy⊗x, dy), so the upstream generator trains end-to-end. The demo conditions a small generator (learned task embedding → FullConnect → FullConnectLinear(Din*Dout+Dout)) on a one-hot task id to emit the weights of ONE shared hyper layer, which then solves a FAMILY of 4 distinct 2D→2D rotate+scale tasks: per-task held-out MSE collapses to ~1e-12 (the hyper layer has zero owned weights) while a fixed shared TNNetFullConnectLinear baseline is stuck averaging the tasks at ~8e-2. Concat-style serialization (weights-source index in the structure string), save/load round-trip checked. Pure CPU, ~1 s. A spatial cousin, TNNetHyperConv (builder TNNet.AddHyperConv(InChannels, OutChannels, FeatureSize, ContextLayer)), extends the same idea to a VALID stride-1 convolution: the generator emits the whole flat conv kernel W[o,ky,kx,i] (+ optional per-output-channel bias) and the weightless HyperConv applies it to a (SizeX,SizeY,InChannels) image, with both gradient paths (into the image and back into the generated kernel) written. Memory/param trade-off: the generator must output the ENTIRE kernel in one shot, so its parameter count scales as OutChannels*K*K*InChannels — keep K and the channel counts small (chunked/tiled kernel generation is a deferred follow-up).
• BitLinear bake-off — ternary-vs-full-precision head swap, the TNNetBitLinear (BitNet) follow-up. The same tiny classifier is trained three times on the same synthetic multi-class task at fixed seed/data/init order, changing only the head type: full-precision TNNetFullConnectLinear (32 bits/weight), the BitNet-style TNNetBitLinear that quantizes its weights to {-1, 0, +1}, and the BitNet b1.58 fully-quantized path (TNNetBitLinear with its activation-quant flag ON: ternary weights plus per-token absmax-int8 activations, STE backward). Reports accuracy, effective weight memory, and a PASS/FAIL gate. Pure CPU.
• Circulant linear parameter-efficiency bake-off — the headline accuracy-per-weight win for the new TNNetCirculantLinear, a STRUCTURED-MATRIX dense layer whose n×n weight matrix is CIRCULANT (every row a cyclic shift of one learned length-n vector c), so the map is y = circular_convolution(c, x) (+ bias) and the layer stores O(n) weights instead of O(n²). On a target that is genuinely a circular convolution it fits the teacher kernel almost exactly with 2n weights where a param-matched dense TNNetFullConnectLinear needs n²+n — at n=16 that is 32 vs 272 weights (8.5× fewer) and a far higher accuracy-per-weight. Distinct from LoRA (low-rank), AddGroupedFullConnect (block-diagonal) and TNNetBitLinear (quantized): this one imposes shift-invariant Toeplitz/circulant structure. Pure CPU, <1 s. (Opt-in FFT O(n log n) fast path is a logged follow-up; the direct sum is the default.)
• Monarch structured-linear parameter/accuracy bake-off — the headline "structured = fewer params, comparable accuracy" win for the new TNNetMonarchLinear, a SUB-QUADRATIC structured dense layer that factorises an n×n map as y = Pᵀ(L(P(R·x))) where R and L are BLOCK-DIAGONAL (b blocks of size m, n=b·m) and P is a fixed reshape-transpose permutation, so the dense n×n matrix is never formed — forward and backward are all block-local m×m matmuls and the layer stores only 2·n·√n weights instead of n². The square map INFERS n from the previous layer (constructor is just Create(pSuppressBias), no N arg). Three square 64→64 mixers are trained on the same random linear-then-tanh regression teacher behind an IDENTICAL tiny linear read-out head: Monarch (b=m=8, 1088 mixing weights) lands at a comparable training MSE to the dense TNNetFullConnectLinear(64) (4096 weights, 3.8× more) while TNNetCirculantLinear(64) (128 taps) is an even leaner structured point on the params-vs-accuracy curve. Distinct from TNNetCirculantLinear (shift-invariant circulant), TNNetHouseholderLinear (exactly orthogonal) and AddGroupedFullConnect (single block-diagonal): Monarch is a TWO-factor butterfly with an interleaving permutation between them. NOTE: the asked-for DFT-init sub-check is intentionally SKIPPED — the layer exposes no DFT-init path (InitDefault is random), documented in the program and README rather than invented. Pure CPU, ≈3 s.
• Kronecker structured-linear parameter/accuracy bake-off — the headline "structured = far fewer params, comparable accuracy" win for the new TNNetKroneckerLinear, a SUB-QUADRATIC structured dense layer whose n×n weight is a single KRONECKER PRODUCT W = A ⊗ B of two small learned factors A (p×p) and B (q×q) with n = p·q. The dense n×n matrix is never formed: x is reshaped to a q×p matrix X (X[i,j] = x[i·p+j]) and the matvec is two small GEMMs Y = B·X·Aᵀ (O(n^1.5)), with y = vec(B·X·Aᵀ) = (A⊗B)·x under the row-major vec convention; backward is the exact transpose chain dX = Bᵀ·dY·A, dA = dYᵀ·(B·X), dB = dY·(X·Aᵀ)ᵀ (all gradient-checked). Stores only p²+q² ≈ 2n factor weights instead of n². The square map INFERS n from the previous layer (constructor Create(pSuppressBias, pP), pP=0 auto-picks p = round(√n)). Three square 256→256 mixers are trained on a tiny MNIST-shaped 10-class task (16×16 prototype+noise images) behind an IDENTICAL ReLU→linear(10)→softmax head: Kronecker (p=q=16, 768 mixing weights) reaches the SAME 100% test accuracy as the dense TNNetFullConnectLinear(256) (65536 weights, 85× more) and TNNetMonarchLinear (8448 weights) is an intermediate O(n^1.5) structured point. Distinct from TNNetCirculantLinear (single cyclic kernel), TNNetHouseholderLinear (exactly orthogonal), TNNetMonarchLinear (two block-diagonal factors + permutation) and LoRA (low-rank): Kronecker is a single tensor-product factorisation. Pure CPU, a few seconds.
• Tensor-Train structured-linear parameter/accuracy bake-off — the "structured = fewer params, comparable accuracy" win for the new TNNetTensorTrain, the next rung of the structured/sub-quadratic weight family after TNNetMonarchLinear and TNNetKroneckerLinear. It factors the n×n map as a Tensor-Train (Matrix-Product-State / MPO) — a CHAIN of d small 4-D cores G_k ∈ R^{r_{k-1}×m_k×n_k×r_k} with boundary TT-ranks r_0=r_d=1 and a tunable interior rank r (default d=2) — and contracts them left→right via the exact MPO-vector sweep, so the dense n×n matrix is never materialized (params ~ d·m·n·r² instead of n²). The square map INFERS n from the previous layer (no N arg, like Monarch); backward is the standard "freeze the other cores, contract" reverse sweep (numerically gradient-checked, both cores). The bake-off trains a TensorTrain mixer (576 weights) against a dense TNNetFullConnectLinear (4096 weights, 7.1× more) at comparable MSE, with a TNNetKroneckerLinear arm as an even-leaner structured point. Distinct from Kronecker (single 2-factor A⊗B) and Monarch (two block-diagonal factors + permutation): a d-core chain with a tunable internal rank gives a smooth compression↔capacity dial. Pure CPU, a few seconds.
• Householder exactly-orthogonal dense stack — the gradient-stability win for the new TNNetHouseholderLinear, an EXACTLY-orthogonal dense layer whose n×n weight is a product of K Householder reflections Q = H_1·…·H_K (H_i = I − 2·v_iv_iᵀ/v_iᵀv_i), so Q is orthogonal for ANY reflection vectors v_i — no constrained optimization or re-projection, yet exactly norm-preserving and invertible. The demo builds deep plain linear stacks (no activation/norm/residual) at depths 1…32 and prints the input-side gradient norm for a unit gradient planted at the top: the Householder stack holds it at exactly 1.0 at every depth (an orthogonal Jacobian is an isometry) while a param-matched unconstrained TNNetFullConnectLinear stack explodes geometrically (≈1047 at depth 32). A second sweep varies K ∈ {1, n/2, n} to show K trades cost (O(K·n)/layer) and expressivity for representational reach but NOT gradient stability (every K is exactly orthogonal). Distinct from TNNetSpectralNorm (bounds only σ_1), the structured-matrix layers (constrain the matrix form) and Muon (orthogonalizes the update). Builder TNNet.AddHouseholderLinear(N, NumReflections, UseBias). Pure CPU, well under a minute.
• Complex linear parameter-efficiency bake-off — the structured-weight-sharing win for the new TNNetComplexLinear, the 2-dimensional base rung of the same Cayley–Dickson ladder as TNNetQuaternionLinear (4D) and TNNetOctonionLinear (8D). It reinterprets the input/output Depth (a multiple of 2) as packed complex numbers (group g holds Re = chan[2g], Im = chan[2g+1]) and learns an (OutC × InC) grid of complex weights w = a + b·i, each driving a full 2×2 complex-multiply block [[a,-b],[b,a]] (Re' = a·Re − b·Im, Im' = a·Im + b·Re) — so one complex's 2 reals steer a whole 2×2 output block and the layer stores ~1/2 the weights of an equal-width dense TNNetFullConnectLinear while still mixing the real and imaginary parts. Backward propagates into BOTH the input (transpose block) and the two real weight components per block (numerically gradient-checked). A param-matched bake-off on a true complex-product task (left-multiply by a fixed 50° phase rotation + gain plus cross-complex coupling): the complex layer essentially solves it (val-MSE ≈ 0) at 8 weights (+4 bias) while a param-matched dense bottleneck and block-diagonal grouped head plateau around 0.4 — vs the 16 weights of a full 4×4 dense layer. It then verifies the algebraic guarantee the layer rests on, |w·X| = |w|·|X| and arg(w·X) = arg(w) + arg(X), to ~1e-7. Pure CPU, well under a minute.
• Quaternion linear parameter-efficiency bake-off — the headline structured-weight-sharing win for the new TNNetQuaternionLinear, a hypercomplex layer (Parcollet et al. 2019; Gaudet & Maida 2018). It reinterprets the input Depth (a multiple of 4) as packed quaternions and learns an (OutQ × InQ) grid of quaternion weights q = r + xi + yj + zk, each driving a full 4×4 Hamilton-product block [[r,-x,-y,-z],[x,r,-z,y],[y,z,r,-x],[z,-y,x,r]] — so one quaternion's 4 reals steer a whole 4×4 output block and the layer stores ~1/4 the weights of an equal-width dense TNNetFullConnectLinear while still mixing all four components (cross-channel coupling a block-diagonal AddGroupedFullConnect cannot express). Backward propagates into BOTH the input (conjugate/transpose block) and the four real weight components per block (numerically gradient-checked). A param-matched bake-off on a true quaternion-rotation task: the quaternion layer essentially solves it (val-MSE ≈ 0) at 80 weights while a dense bottleneck (128 weights) and a block-diagonal grouped head (64 weights) plateau around 0.2. Pure CPU, ~3 s.
• Quaternion convolution parameter-efficiency bake-off — the spatial sibling of the dense quaternion layer above: the structured-weight-sharing win for the new TNNetQuaternionConv. Both the input Depth and the filter count are multiples of 4; with InQ = inDepth/4 and OutQ = features/4 the layer learns, per kernel tap, an (OutQ × InQ) grid of quaternion weights q = r + xi + yj + zk applied by the same trusted 4×4 Hamilton-product block as TNNetQuaternionLinear, accumulated over the kernel window — so it stores ~1/4 the weights of a real conv of equal input/output width while coupling the four channel components across space. Backward propagates into BOTH the input (transpose/conjugate block) and the four real weight components of every tap (numerically gradient-checked, the classic one-component sign error is exactly what the test catches). A param-matched bake-off on a true "colour-rotation + 3×3 blur" image task ($8 \times 8$, 4 channels = one quaternion/pixel; target = a fixed unit-quaternion rotation of a fixed 3×3 spatial average, which is EXACTLY a quaternion convolution): the quaternion conv essentially solves it (val-MSE ≈ 0) at 40 weights while a param-matched real-conv 4→1→4 bottleneck (72 weights) plateaus around 0.03. Padding/stride follow the usual convolution semantics. Pure CPU, well under a minute.
• Octonion linear parameter-efficiency bake-off — the 8-dimensional sibling of the quaternion layer: the structured-weight-sharing win for the new TNNetOctonionLinear. It reinterprets the input/output Depth (both multiples of 8) as packed octonions and learns an (OutO × InO) grid of octonion weights w = o0 + o1·e1 + ... + o7·e7, each driving a full 8×8 Cayley–Dickson block M(W)[i][j] = SGN[i][j]·W[i xor j] — so one octonion's 8 reals steer a whole 8×8 output block and the layer stores ~1/8 the weights of an equal-width dense TNNetFullConnectLinear while still mixing all eight components. The hard-coded multiplication table is derived from the standard octonion = pair-of-quaternions doubling (a,b)(c,d) = (a·c − d*·b, d·a + b·c*) and verified independently by norm multiplicativity |W·X| = |W|·|X| in the tests; backward propagates into BOTH the input (transpose block) and the eight real weight components per block (numerically gradient-checked). A param-matched bake-off on a true octonion-product task: the octonion layer essentially solves it (val-MSE ≈ 0) at 32 weights (+16 bias) while a dense bottleneck (64 weights) and a block-diagonal grouped head (128 weights) plateau around 0.1–0.3 — vs the 256 weights of a full 16×16 dense layer. Pure CPU, well under a minute.
• Octonion convolution parameter-efficiency bake-off — the spatial sibling of the dense octonion layer above and the 8-dimensional analogue of TNNetQuaternionConv: the structured-weight-sharing win for the new TNNetOctonionConv. Both the input Depth and the filter count are multiples of 8; with InO = inDepth/8 and OutO = features/8 the layer learns, per kernel tap, an (OutO × InO) grid of octonion weights w = o0 + o1·e1 + ... + o7·e7 applied by the same trusted 8×8 Cayley–Dickson block M(W)[i][j] = SGN[i][j]·W[i xor j] as TNNetOctonionLinear (same hard-coded, norm-multiplicativity-verified multiplication table), accumulated over the kernel window — so it stores ~1/8 the weights of a real conv of equal input/output width while coupling all eight channel components across space. Backward propagates into BOTH the input (transpose block) and the eight real weight components of every tap (numerically gradient-checked, the classic one-component sign error is exactly what the test catches). A param-matched bake-off on a true "octonion left-mul + 3×3 blur" image task ($8 \times 8$, 8 channels = one octonion/pixel; target = a fixed unit-octonion left-multiplication of a fixed 3×3 spatial average, which is EXACTLY an octonion convolution): the octonion conv essentially solves it (val-MSE ≈ 0) at 80 weights while a param-matched real-conv 8→1→8 bottleneck (144 weights) plateaus around 0.036 — vs the 576 weights of a full 8×8-channel 3×3 real conv. Padding/stride follow the usual convolution semantics. Pure CPU, well under a minute.
• CondConv dynamic-convolution bake-off — the headline "match a wider conv at single-conv inference cost" win for the new TNNetCondConv, a CONDITIONALLY-PARAMETERIZED ("dynamic") convolution (Yang et al. 2019, NeurIPS, arXiv:1904.04971). The layer owns a bank of K expert kernels W_1..W_K (each a normal Features × FeatureSize × FeatureSize × InChannels kernel) plus a tiny per-sample routing head (global-average-pool → FullConnect → sigmoid) emitting K mixing coefficients alpha_k PER INPUT SAMPLE; the effective kernel is the per-sample blend W_eff = sum_k alpha_k · W_k applied as ONE ordinary convolution — so inference cost stays that of a single conv regardless of K while capacity grows with the bank. Backward routes dL/dW_k = alpha_k · dL/dW_eff, sends dL/dalpha_k = <dL/dW_eff, W_k> back through the sigmoid + FC + pool into the input, and propagates the standard conv input gradient through W_eff (all three — input, expert-bank weights, routing head — are numerically gradient-checked; K/Features/FeatureSize/Padding/Stride round-trip via FStruct). DISTINCT from siblings: TNNetHyperConv GENERATES the whole kernel from a second tensor in one shot; AddMixtureOfExperts mixes K expert OUTPUTS post-hoc (K forward passes); CondConv mixes K kernels BEFORE the conv (one forward pass). The demo is an input-dependent filtering task on $8 \times 8$ single-channel fields where the SIGN of each sample's global mean selects which of two ground-truth 3×3 filters (edge vs blur) made the target — so the right kernel DEPENDS on the input. A K=2 CondConv (22 weights, single-conv inference) drives val-MSE to ≈0.02 while a single plain conv (9 weights) is stuck at ≈1.18 (it can't switch behaviour) and a much WIDER plain conv (8 maps + 1×1, 80 weights, ~2× the inference wall-clock) only reaches ≈1.96. Pure CPU, well under a minute.
• p4 group-equivariant CNN vs param-matched plain CNN — the headline "rotation robustness BY CONSTRUCTION, not from augmentation" win for the new TNNetGroupConvP4 + TNNetGroupPoolP4 (Cohen & Welling 2016, Group Equivariant Convolutional Networks, arXiv:1602.07576) — a layer that is rotation-equivariant by construction rather than only measured after the fact by TNNet.EquivarianceReport. TNNetGroupConvP4 is the lifting rung for the C4 group of 90° rotations: ONE learned K×K kernel bank is shared across the four rotations, the layer convolves the input with the four rot-{0,90,180,270} copies of the same filter and stacks the responses along a new 4-fold orientation sub-axis (Depth = 4·FeaturesCount, channel = co·4 + r), so a 90° input rotation only cyclically permutes those orientation channels (and rotates the spatial map) — it never scrambles them. The three extra rotated filters are index views of the one trained bank materialised at forward time (FRotMap), and the backward folds the four orientation gradients back onto the single shared kernel (rotation-tied weight-gradient sum — the place a silent reduction bug would hide, so it is gradient-checked hard alongside the input gradient and an EXACT-equivariance forward test that reads 0.0 error to machine precision). TNNetGroupPoolP4 then max- (or mean-) reduces over the four orientation channels of each feature to collapse the C4 field back to a rotation-invariant (SizeX, SizeY, FeaturesCount) map; followed by a global spatial average pool the whole stack is exactly C4-invariant. DISTINCT from TNNetFlipX/FlipY/TransposeXD (fixed parameter-free involutions — data-augmentation primitives, not weight-shared equivariant maps) and from CondConv/Quaternion/Octonion (which share weights across a different algebra, none across a spatial symmetry group). Features/FeatureSize/Padding/Stride/SuppressBias (and the pool's reduce mode) round-trip via FStruct. The demo trains BOTH a tiny p4-CNN and a parameter-matched plain CNN (≈320 weights each, same pooling/classifier tail) on UPRIGHT-only chiral 8×8 glyphs (the classic Cohen–Welling setting, no rotation augmentation) and evaluates on a 90/180/270°-rotated test set: the p4 net more than DOUBLES the plain net's rotated accuracy at equal weights (≈0.92 vs ≈0.41) and its measured C4-invariance error is ≈1e-6 vs ≈0.64 — and it prints TNNet.EquivarianceReport on both to close the loop with the existing diagnostic. v1 is the lifting conv only; the full p4m group (reflections), steerable/SO(2) harmonics, and a field→field p4 group conv are noted as follow-ups. Pure CPU, ≈45 s.
• Tropical morphology hypothesis-class demo — the headline "different algebra, different hypothesis class" win for the new TNNetTropicalLinear, a max-plus / min-plus morphological dense layer computing in the TROPICAL semiring instead of the usual multiply-accumulate ring: y_i = max_j (x_j + W[i,j]) (a morphological DILATION) with a paired ERODE mode y_i = min_j (x_j + W[i,j]) selected by a constructor flag (round-trips via FStruct[6]). The weights are learnable additive thresholds and the combine op is max/min, so the layer learns piecewise-linear convex (dilation) / concave (erosion) functions and tropical polynomials — different from TNNetFullConnect* / the structured-linear family (all sum_j W·x) and from parameterless max/min pooling. Backward is the same hard arg-max/arg-min subgradient as TNNetMaxPool (route dy_i to the single winning j*). The demo fits a convex piecewise-linear envelope (the upper envelope of three lines) with an affine-feature bank → TNNetTropicalLinear dilation stack, versus a SAME-width single linear layer: the tropical stack drives the MSE to ≈0 while the linear baseline plateaus at the best-fitting single straight line (≈1.39 MSE — it provably cannot bend), and the min-plus ERODE sibling does the same for a concave target. Forward is O(Din·Dout); input AND weight gradients are numerically gradient-checked away from the tie kink. Pure CPU, a few seconds. A second program in the same directory (TropicalMorphologyConv.lpr) exercises the SPATIAL conv sibling TNNetTropicalConv (subclass of TNNetConvolutionLinear): it slides a learnable additive structuring element over a 12×12 binary glyph and takes a per-cell max (DILATION → thicker strokes) or min (ERODE → thinner strokes), constructor Create(NumFeatures, FeatureSize, Padding, Stride, Erode). Trained against the glyph's classical 3×3 morphological dilation/erosion versus a same-size linear TNNetConvolutionLinear, the tropical conv wins on the trained per-pixel MSE on BOTH targets — dilation is an exact MSE-0 / 100%-pixel win, while the linear sum-of-products can't match the per-pixel max; erosion is also won on MSE but stops just short of 0 because the hard arg-min subgradient (one-hot, like TNNetMaxPool) leaves a few boundary taps un-pressured. Runs in ≈1 s.
• Soft decision tree vs matched MLP (two-moons) — the headline "hierarchical soft routing wins AND stays interpretable" demo for the new TNNetSoftDecisionTree layer (Kontschieder et al. 2015, Deep Neural Decision Forests; Frosst & Hinton 2017, Distilling a Neural Network Into a Soft Decision Tree) — a structurally new paradigm in this repo (not factorization/attention/recurrence/kernel). A single balanced depth-3 tree (7 inner gates p_i = sigmoid(β·(wᵢ·x + bᵢ)), 8 leaf logit vectors) routes each sample to leaves with probability = product of left/right gate decisions along the root-to-leaf path; the output is the path-probability-weighted mixture of leaf logits. The exact analytic backward collapses the product-of-gates path responsibilities (the pᵢ/(1−pᵢ) divisions cancel against dpᵢ/dzᵢ = β·pᵢ·(1−pᵢ)) to dL/dzᵢ = β·(Aᵢ·(1−pᵢ) − Bᵢ·pᵢ) with Aᵢ/Bᵢ the left/right subtree responsibility sums — finite-difference verified for input, gate weights/biases AND leaf vectors (TestSoftDecisionTree*). On a non-linearly-separable two-moons toy, the tree and a plain ReLU(7) → linear(2) MLP are sized to the same 37 parameters behind an identical softmax; the tree reaches ≈99.9% held-out accuracy vs ≈95% for the matched MLP and exposes a human-readable decision path (per-node left/right p(left) and the dominant leaf logits) the MLP cannot. Pure CPU, ~15 s.

Sets, graphs, memory & spectral / operator methods

• Deep Sets (permutation-invariant set learning) — reproduces the Zaheer et al. 2017 Deep Sets recipe from existing layers only: the N elements of a set are laid along the X axis as an (N,1,1) bag, a shared per-element encoder (TNNetPointwiseConvReLU -> TNNetConvolutionLinear, featuresize=1 so every element sees identical weights) maps each one, a symmetric pool (TNNetMaxChannel, (N,1,F)->(1,1,F)) collapses the set, and a TNNetFullConnectReLU -> TNNetFullConnectLinear(1) rho head regresses the set's MAX. The headline is the architectural invariant: the trained output is bit-for-bit unchanged (max|dy| = 0) under 200 random permutations of the elements yet shifts sharply when an element's value is edited, and the same weights generalize to an unseen set size (train N=5, test N=8). README explains why a flatten->dense net cannot be permutation-invariant and contrasts with self-attention. Pure CPU, deterministic, ~3 s. (Note: on an (N,1,F) bag TNNetAvgChannel divides by N^2, not N; TNNetMaxChannel is exact.)

• Set Transformer (ISAB + PMA) — exercises the two new permutation-invariant Set-Transformer primitives (Lee et al. 2019): TNNetInducedSetAttention (ISAB — replaces O(N^2) self-attention with an O(N*M) bottleneck through M learnable inducing points via two stacked cross-attentions H=MAB(I,X), Y=MAB(X,H)) and TNNetAttentionPooling (PMA — a learnable, content-addressed pooler that collapses a set (N,1,d) to a fixed (k,1,d) by letting k learnable seed vectors cross-attend over the inputs; k=1 is a learned-query weighted-sum pool, categorically unlike the parameter-free TNNetAvgChannel/TNNetMaxChannel). Three parts: (1) a tiny ISAB->PMA(k=1) net gives bit-for-bit invariant pooled output (max|dy| ~ 1e-8) on a shuffled vs original bag before training (every softmax-over-inputs is symmetric in the rows); (2) on max-of-set regression ISAB+PMA(k=1) beats a mean-pool baseline (MSE 0.014 vs 0.046) because the attention pool can concentrate its softmax mass on the largest element; (3) prints the N×M (ISAB) vs N×N (full self-attention) score-matrix sizes. v1 MABs are single-head with identity Q/K/V projections (only the inducing/seed bank is learnable — keeps the two-stage softmax-Jacobian backward exact and gradient-checkable). Pure CPU, deterministic, runs in seconds.

• Set Attention Block (SAB) — the TNNet.AddSAB(InducingPoints, Heads, DFF) builder, the Set Transformer's Set Attention Block (Lee et al. 2019): it wraps the multi-head MAB (built exactly like AddInducedSetAttention — Heads heads, each a per-token input projection feeding a single-head TNNetInducedSetAttention with its own inducing bank, DeepConcat, per-token out-projection) in two post-norm residual sublayers H=LayerNorm(X+MAB(X,X)), out=LayerNorm(H+FFN(H)), with a token-wise FFN (TNNetPointwiseConvReLU(DFF)->TNNetPointwiseConvLinear(d_model)). Every op is a 1×1 (pointwise) conv over Depth except the symmetric softmax, so the block is shape-preserving (N,1,d)->(N,1,d) and permutation-equivariant (shuffle the rows, the output rows follow). Two parts: (1) the SAB stack's output rows track the input permutation to max|dy| < 1e-6 before training; (2) on a per-element above-the-set-mean classification task — which REQUIRES cross-element interaction (the mean depends on every element) — a 2-block SAB stack reaches ~97% while a per-element MLP baseline (no cross-element path) is stuck near its ~86% ceiling because it cannot compute the set mean. Pure CPU, deterministic, runs in seconds.

• Perceiver latent-bottleneck encoder — the TNNet.AddPerceiverEncoder(NumLatents, d_latent, Heads, Depth) builder, the Perceiver latent bottleneck (Jaegle et al. 2021, Perceiver: General Perception with Iterative Attention; Perceiver IO 2021). A small, fixed-size learnable latent array Z of NumLatents rows (NumLatents << InputSeqLen) (1) cross-attends ONCE to the (possibly huge) input to absorb it into a (NumLatents,1,d_latent) summary — the only step touching the input, cost LINEAR in length — then (2) is refined by a stack of Depth self-attention + FFN blocks acting only over the NumLatents rows (O(NumLatents^2) per block, independent of length). Composes already-landed pieces: a token-wise input projection, AddAttentionPooling(NumLatents, Heads) as the latent cross-attention read (its seed bank IS the latent array Z), and AddTransformerEncoderBlock × Depth for the tower. Output length = NumLatents regardless of input — the missing third mode vs the Set-Transformer builders (InducedSetAttention projects BACK to n input rows; AttentionPooling is a single pool with no refinement). Headline: the demo builds the SAME net on SEQLEN and 2*SEQLEN and prints identical weight counts (doubling the input adds ZERO weights — the cost lives in the latent tower), then classifies a deliberately long 256-token input climbing from chance (~25%) to 100%. Pure CPU, ≈55 s.

• Product-Key Memory (sparse key->value retrieval) — the TNNetProductKeyMemory layer / TNNet.AddProductKeyMemory(NumKeys, ValueDim, TopK, Heads) builder (Lample et al. 2019, Large Memory Layers with Product Keys): a large, sparsely-accessed key→value memory that factorizes |K| keys into the Cartesian product of two small half-key banks K1, K2 (each sqrt(|K|) keys of half the query dim), so a query is scored against the product K1 x K2 in O(sqrt(|K|)) work — top-TopK per half, re-score the TopK x TopK combinations, pick the global top-TopK, softmax, and gate a sparse weighted sum over only the touched value rows. The demo learns NumPairs=24 random (query, value) associations and trains the product-key memory against a same-capacity flat (dense softmax over all NumKeys) baseline on identical data: the product-key memory matches retrieval accuracy (MSE 0.000000 vs 0.000019) while touching only TopK=4 value rows per query instead of all 64. Prints a per-slot read-count histogram (49/64 distinct slots used here) to expose the classic key-usage collapse failure mode, with a README note on the paper's batch-norm-on-query fix that spreads reads. v1 is single-head (Heads=1; multi-head is a documented follow-up). Pure CPU, deterministic, ~4 s.

• Hopfield Retrieval - A modern Hopfield network as attention (Ramsauer et al. 2020, Hopfield Networks is All You Need): store K patterns and recover a corrupted query in a SINGLE softmax-attention step softmax(β·Xq)X. Sweeps the inverse-temperature β to show the blurry-average → clean-snap transition, and the corruption level to find the capacity edge. Forward-only, pure CPU, ~1s.

• Modern Hopfield associative memory (one-shot recall) — the continuous modern-Hopfield layer (TNNetModernHopfield / TNNet.AddModernHopfieldRetrieval, Ramsauer et al. 2020, Hopfield Networks is All You Need): an ENERGY-BASED associative memory over a learnable bank of stored patterns X that ITERATES the softmax retrieval xi := X^T · softmax(beta·X·xi) to a fixed point — distinct from the single-pass attention layers because K>1 update steps SHARPEN toward the single nearest stored memory. The demo stores four binary 8×8 pixel patterns directly in the bank, presents each as a bottom-half-masked + noise-flipped query, and contrasts K=1 (one-pass attention, a blurry blend of patterns) against K=3 (iterated retrieval) — iterating cleanly completes the pattern (e.g. total Hamming 8→0 and L2 14.4→7.5 over the four queries) where one pass does not. Pure CPU, <1 s.

• Neural Turing Machine (writable external memory, COPY task) — the headline demo for TNNetNTMMemory, a writable differentiable external-memory layer (Neural Turing Machine, Graves et al. 2014, arXiv:1410.5401). Unlike the read-only associative memories (TNNetModernHopfield iterated recall, TNNetProductKeyMemory sparse lookup), an NTM carries a persistent memory matrix M (NumSlots × SlotWidth) the layer both reads and writes as it sweeps the time axis of an (T,1,InputDim) input → emits the per-step read vectors (T,1,SlotWidth). Per step the input is projected to a content key (cosine-addressed against every slot, softplus-beta-sharpened softmax over slots → weights w, read r=w^T·M) plus a sigmoid erase e and add a that update M[i] := M[i]·(1−w[i]·e)+w[i]·a; the four projection matrices are the only trainables and backprop is full BPTT through the recurrent M update (both dL/dM and dL/dw chain backward across steps). The classic COPY task presents a random binary sequence + delimiter then asks the net to reproduce it from memory: the NTM matches a param-matched TNNetSLSTMCell arm's recall (87.5% bit-accuracy) at ~2.6× fewer weights (138 vs 366). v1 is content-addressing only, single read+write head (location-based addressing + DNC temporal links are documented follow-ups). Pure CPU, ~0.8 s.

• Holographic Reduced Representation cleanup memory — the headline demo for the new TNNetHolographicBinding, the Holographic Reduced Representation (HRR) vector-symbolic binding layer (Plate 1995, Holographic Reduced Representations), the associative-binding sibling of the exotic-algebra family (TNNetComplexLinear/TNNetQuaternionLinear/TNNetOctonionLinear/TNNetTropicalLinear/TNNetHyperbolicLinear). It reads two equal-length Depth vectors a and b packed as adjacent halves of the input (Depth must be even =2n; first n channels = a, last n = b, the same adjacent-halves idiom TNNetComplexLinear uses for Re/Im) and outputs the n-vector circular convolution c = a ⊛ b, c[k] = Σ_j a[j]·b[(k-j) mod n] — the HRR bind operator that composes a role→filler pair into a new vector dissimilar to both operands, so many pairs can be superposed (just added) into ONE fixed-width trace. An Unbind flag switches the forward to the circular correlation c = a ⊛ involution(b) (the approximate inverse used to query a bound trace). Distinct from the FFT mixers (TNNetFourierMix/TNNetFourierMixFFT learn a spectral mix of one tensor; HRR is a bilinear bind of two). Weightless; direct O(n²) cyclic forward, exact bilinear adjoint backward (both bind and unbind input gradients numerically gradient-checked, max-abs err ≈2e-4/8e-4), Unbind round-trips via FStruct[6]. The demo (n=256) binds a growing number P of random key→value atom pairs into ONE superposed trace t = Σ_i key_i ⊛ value_i, unbinds each key, and snaps the noisy result to the nearest codebook value (cosine cleanup): single-pair sanity confirms the algebra (trace dissimilar to its own filler cos ≈ 0, correct-key unbind recovers the value cos ≈ 0.72 → right nearest neighbour), and the HRR capacity curve degrades GRACEFULLY — 100% recall at P=1..9 down to 88.5% at P=24 (the textbook superposition tradeoff, not a catastrophic cliff). Noted follow-ups: an FFT O(n log n) path, a learnable per-channel "protect" permutation, and a TNNet.AddHRRMemory builder pairing binding with a TNNetVectorQuantizer codebook. Pure CPU, no training, ≈2 s.

• Graph node classification (spectral GCN) — the headline demo for the new TNNetGraphConvolution (Kipf & Welling 2017, Semi-Supervised Classification with Graph Convolutional Networks), a layer that does MESSAGE PASSING over an arbitrary node graph rather than a grid (image) or 1-D sequence. The input is a (NumNodes,1,FeatureDim) volume; SetAdjacency(A) takes the raw 0/1 adjacency and builds the symmetric-normalized Ahat = D^-1/2 (A+I) D^-1/2 internally, and forward is H' = Ahat·(H·W)(+bias) — a per-node pointwise linear map over the feature axis (nodes never mixed by W, reusing the PointwiseConvLinear weight layout) followed by a constant-Ahat neighbour aggregation (its backward just left-multiplies the error by the symmetric Ahat). Two stacked GCN layers + SoftMax do semi-supervised TRANSDUCTIVE node classification on a synthetic two-community stochastic-block-model graph from only a handful of labelled nodes: the GCN reaches 100% held-out accuracy while a param-matched features-only MLP baseline (identity adjacency) sits at 50% chance — the message passing is what carries the signal. The adjacency is caller-supplied and NOT serialized (re-SetAdjacency after load). Pure CPU, ~2 s.

• Graph attention vs spectral GCN — the headline demo for the new TNNetGraphAttention (Veličković et al. 2018, Graph Attention Networks), the ATTENTIONAL counterpart to GraphNodeClassification's fixed-weight TNNetGraphConvolution. Same (NumNodes,1,FeatureDim) layout and caller-supplied SetAdjacency 0/1 mask, but instead of a constant symmetric-normalized Ahat, each edge gets a LEARNED coefficient: e[i,j] = LeakyReLU(a_src·Z[i] + a_dst·Z[j]) (slope 0.2) over the transformed features Z = H·W, masked to the graph's edges and softmax-normalized per node's neighbourhood, then Y[i] = Σ alpha[i,j]·Z[j]. The two arms share data + seed on an SBM graph with injected NOISY/HETEROPHILOUS cross-community edges; GAT down-weights the bad edges the GCN is forced to average in, reaching 90% held-out vs the GCN's 85% (+5 pp). Honest caveat (in output + README): attention only wins in this noisy-edge + class-indicative-feature regime — on a clean graph with weak features the GCN's symmetric averaging already wins. Single head; the attention vector serializes per-neuron, the adjacency is caller-supplied and re-SetAdjacency after load. Also demos MULTI-HEAD GAT via the TNNet.AddMultiHeadGraphAttention builder (K independent heads CONCAT in the hidden layer, AVERAGED at the output — paper eq. 5/6; 4 heads beats 1 head by ~7.5 pp) and the paper's ATTENTION-DROPOUT regulariser on the normalized per-edge coefficients (training-time only, deterministic at inference; ~+5 pp on the noisy graph). Pure CPU, a couple of seconds.

• Fourier features and spectral bias — the headline Tancik et al. 2020 spectral-bias demo for TNNetFourierFeatures: the SAME small ReLU coordinate-MLP fits a high-frequency target y = sin(20x) + 0.5*sin(53x) once on the raw scalar x and once behind a fixed random Fourier-feature front-end (the raw MLP cannot fit the high frequencies, the Fourier one nails them — ~700x lower MSE), then sweeps sigma in {0.5, 2, 8, 32} to show the single-knob bandwidth U-curve.

• Random Fourier Features RBF-kernel demo — the headline "a single FROZEN random-feature layer + linear head = an RBF-kernel machine" win for the new TNNetRandomFourierFeatures (Rahimi & Recht 2007). It maps a Din-vector x to a 2·D-vector phi_k(x) = sqrt(1/D)·[cos(w_k·x), sin(w_k·x)] with the projection rows w_k (D×Din) drawn once i.i.d. from N(0, 1/sigma²) and frozen by default, so that <phi(x),phi(y)> → exp(-‖x-y‖²/(2·sigma²)) (the RBF / Gaussian kernel) as D grows and a plain linear head over phi(x) approximates a kernel SVM — without forming the N×N Gram matrix. This is mathematically DISTINCT from the learnable FFT layers (TNNetFourierMixFFT, TNNetSpectralConv1D/2D, the TNNetCirculantLinear FFT path): RFF is a FIXED random Gaussian projection of a shift-invariant kernel, not a transform along a signal axis. An optional constructor flag makes W trainable ("deep kernel learning"; sigma stays fixed). D/seed/trainable round-trip via FStruct[0,5,6], sigma via FFloatSt[0], and W reloads identically. The demo classifies concentric rings (not linearly separable): the frozen RFF(D=256) → FullConnectLinear → SoftMax model hits ≈1.0 test accuracy (matching a ReLU MLP) while the same linear classifier on the raw (x,y) is stuck at ≈0.47 (chance — one straight boundary). Pure CPU, ≈5 s.

• FNet Fourier token mixer bake-off — the FNet expressiveness-vs-cost trade for the new TNNetFourierMix, the parameter-free token mixer of Lee-Thorp et al. 2021 (FNet). Over a (SeqLen, 1, d) sequence it replaces self-attention with an UNPARAMETERISED 2D real DFT across the sequence and hidden axes, y = Re(DFT_seq(DFT_hidden(x))), so the mixer holds zero trainable weights. Because Re(DFT) is a fixed self-adjoint real linear operator, the exact input gradient is the same DFT applied to dL/dy (verified against finite differences). On a tiny $8 \times 8$ global token-mixing task the FNet arm (Fourier mix + per-token MLP) is benchmarked head-to-head against an attention arm (AddMultiHeadSelfAttention + the same MLP): the Fourier mixer drops the entire learned Q|K|V|out projection (≈51% fewer parameters) and trades only a modest accuracy loss — the paper's point that on short sequences the mix is nearly free. Distinct from TNNetTokenShift (RWKV t-1 shift), TNNetCirculantLinear (learned circular conv) and attention (learned mix): this one is a fixed, weightless spectral mix. Opt-in radix-2 FFT fast path (UseFFT, default off, checked vs the direct path to <1e-5). Pure CPU, ≈13 s.

• Fourier Neural Operator resolution-invariance demo — the headline resolution-invariance win for the new TNNetSpectralConv1D, the core layer of the Fourier Neural Operator (Li et al. 2021, arXiv:2010.08895) and a layer with learnable complex spectral weights (distinct from the parameter-free TNNetFourierMix and the fixed-random TNNetFourierFeatures). Over a (SeqLen, 1, InDepth) sequence it takes a real radix-2 FFT along SeqLen (reusing the proven FourierMixFFT helper), truncates to the lowest Modes frequencies (a spectral low-pass), applies a learnable per-(in,out)-channel complex weight R[m] per kept mode (an InDepth×OutDepth complex matmul packed via the same 2×2 complex-multiply idiom as TNNetQuaternionLinear/TNNetOctonionLinear), then inverse-FFTs back. Because the learned weights live in mode space, not grid space, the SAME weights describe the SAME continuous operator at ANY resolution. The demo learns the 1-D antiderivative operator (u' = f, a Poisson/diffusion-step solution operator) on a coarse 32-point grid, then evaluates with no retraining on a finer 64-point grid it never saw (weights copied with CopyWeights): the FNO keeps essentially the same held-out relative-L2 error across resolutions (≈7.9% → 7.9%) while a param-matched local TNNetCausalConv1D baseline — whose taps encode a fixed grid spacing — degrades sharply (≈63% → 86%). Backward is the exact real adjoint of the FFT → complex-matmul → IFFT pipeline; both the input AND the complex (real+imag) spectral-weight gradients are numerically gradient-checked. Pure CPU, ≈90 s.

• 2-D Fourier Neural Operator resolution-invariance demo — the 2-D headline for the new TNNetSpectralConv2D, the 2-D spectral convolution of the Fourier Neural Operator (Li et al. 2021, arXiv:2010.08895) and the natural 2-D sibling of TNNetSpectralConv1D. Over a (SizeX, SizeY, InDepth) image it takes a 2-D FFT (real radix-2 FFT along X for every row, then along Y for every column — both reuse the proven FourierMixFFT helper), truncates to the lowest ModesX × ModesY 2-D modes (a 2-D spectral low-pass), applies a learnable per-(in,out)-channel complex weight R[mx,my] per kept 2-D mode (an InDepth×OutDepth complex matmul packed via the same 2×2 complex-multiply idiom as TNNetQuaternionLinear/TNNetOctonionLinear), then inverse-2D-FFTs back and takes the real part. Because the learned weights live in 2-D mode space, not grid space, the SAME weights describe the SAME continuous operator at ANY resolution. The demo learns a smooth 2-D low-pass diffusion operator (each 2-D mode scaled by 1/(1+c·(kx²+ky²))) on a coarse $16 \times 16$ grid, then evaluates with no retraining on a finer $32 \times 32$ grid it never saw (weights copied with CopyWeights): the 2-D FNO keeps essentially zero held-out relative-L2 error across resolutions (≈0.01% → 0.01%) while a param-matched local TNNetConvolutionReLU 3×3 baseline — whose taps encode a fixed grid spacing — degrades sharply (≈29% → 58%). Backward is the exact real adjoint of the 2-D FFT → complex-matmul → 2-D IFFT pipeline; both the input AND the complex (real+imag) spectral-weight gradients are numerically gradient-checked. OutDepth/ModesX/ModesY round-trip via FStruct[0..2]. Pure CPU, small grids, ≈2 min.

• Darcy-flow PDE surrogate (FNO coefficient→solution map) — the headline FNO use case applied end-to-end with the TNNet.AddFourierNeuralOperator2D builder: learn a parametric-PDE coefficient→solution operator G : a(x,y) → u(x,y) in ONE forward pass, replacing an iterative numerical solve. The data is generated in pure Pascal at startup — a smooth strictly-positive permeability a = exp(band-limited random field) and the deterministic SOLUTION u of the periodic Poisson problem -Δu = (a-1) (the linear, FNO-tractable member of the Darcy-flow family), obtained by Jacobi sweeps of the 5-point finite-difference Laplacian with the grid spacing h² folded into the source so the SAME continuous operator is reproduced at any resolution. The surrogate ($\text{lift} → \text{AddFourierNeuralOperator2D}(\text{width}=8, 6 \times 6 \text{modes}, \text{no} \text{activation}) → \text{project}$) trains with MSE on a $16 \times 16$ grid: held-out relative-L2 error drops ≈0.64 → 0.025 over 150 epochs (HEADLINE 1, the FNO learns the operator). HEADLINE 2 (resolution invariance): the SAME trained weights are evaluated with NO retraining on a finer $32 \times 32$ grid drawn from the same continuous operator (weights copied with CopyWeights) — error stays bounded (0.0251 at $16 \times 16$ → 0.0284 at the unseen $32 \times 32$) because the spectral weights live in resolution-independent mode space. README documents honestly that the fully NONLINEAR -div(a grad u)=f Darcy operator (non-diagonal in mode space) does not converge under plain SGD within the CPU budget, so the linear Poisson member is used. Power-of-two grids only (the separable radix-2 FFT requires it). Pure CPU, ≈3 min.

• Wavelet shrinkage denoising — the headline "multi-resolution basis beats a single-scale low-pass filter" win for the new TNNetDWT1D, the lifting-scheme single-level 1-D discrete wavelet transform. Over a (SeqLen,1,Depth) sequence one level maps (L,1,D) → (L div 2,1,2·D) (first D channels = approximation / low-pass band, next D = detail / high-pass band); forward and inverse share ONE lifting step list so it is exactly invertible for any taps (InverseChannel reconstructs a channel from packed [approx|detail] bands), with Haar / CDF-5/3 / Daubechies-4 filter selectors and an optional learnable-taps mode. The demo runs the classic Donoho-Johnstone wavelet shrinkage denoiser on the canonical piecewise-constant "Blocks" signal under Gaussian noise: a Mallat pyramid (recurse the DWT on the approximation band only) → per-level soft-threshold of the detail coefficients at the universal threshold λ = σ·sqrt(2 ln M) with a robust MAD σ estimate → inverse pyramid. Because Blocks is sparse in the wavelet domain (jump energy collapses onto a few large detail coefficients) but dense in the local-average domain, wavelet shrinkage reaches ≈21.2 dB reconstruction SNR, beating a param-free moving-average low-pass baseline (≈16.6 dB, +4.6 dB) and the noisy input (≈18.3 dB, +2.9 dB) — the single fixed scale must blur the very edges that define the signal. Also exercises the new TNNet.AddWaveletPacketTransform(Levels, Filter, Learnable) builder, which stacks Levels single-level DWTs into the full balanced wavelet-PACKET tree (every channel recursively decomposed → (SeqLen div 2^Levels, 1, (2^Levels)·Depth)). Pure CPU, no training, ≪1 s.

• Learning to sort through a doubly-stochastic relaxation — the headline "learn a DISCRETE operation through a CONTINUOUS relaxation" win for the new TNNetSinkhorn, a differentiable optimal-transport / doubly-stochastic normalization layer (Mena et al. 2018, arXiv:1802.08665). Where TNNetSoftMax / sparsemax normalize ONE axis, TNNetSinkhorn normalizes a square (N,1,N) score matrix to be doubly stochastic (every row AND every column sums to 1) by iterating Sinkhorn–Knopp in log-space — KIter alternating row/col subtract-logsumexp steps on score/tau, then exp. Doubly-stochastic matrices are the convex hull of permutation matrices, and as the temperature tau → 0 the output sharpens to a hard permutation, so a permutation becomes a smooth function of a score matrix. The demo trains Input(N) → FullConnectReLU → FullConnectLinear → Reshape(N,1,N) → TNNetSinkhorn to sort 5 scalars: the soft permutation P is applied to the input (yhat = P·x) and trained with plain MSE against the ascending sort, the loss gradient dL/dP[i,j] = (yhat[i]−sorted[i])·x[j] set by hand on the Sinkhorn output and back-propagated through the entire unrolled iteration (each step caches its input so backward replays the exact softmax-style adjoint). Annealing tau from 1.0 → 0.07 sharpens P and exact-sort accuracy climbs monotonically ≈2.5% → ≈62.5% (chance is 1/120 ≈ 0.8% for all 5 elements). No trainable params in the layer; KIter/tau round-trip via FStruct[0]/FFloatSt[0]. Pure CPU, small N and batches, ≈50 s.

• Solving a linear-assignment problem with a soft permutation — the optimal-transport / bipartite-MATCHING companion to the sort demo, on the canonical linear-assignment task (distinct from sorting). Given an N × N cost matrix C, the net emits a soft permutation matrix P through TNNetSinkhorn and is trained on the continuous OT cost of that soft assignment, L = sum_{i,j} P[i,j]·C[i,j] with the trivial hand-set gradient dL/dP[i,j] = C[i,j] back-propagated through the entire unrolled Sinkhorn iteration. The net is never told the optimal permutation — only its own soft cost — yet learns to assign workers to tasks at minimum cost. Pipeline Input(N,1,N) → PointwiseConvReLU → PointwiseConvLinear(N) → TNNetSinkhorn (token-wise score head over the row axis; FullConnect would flatten/mix rows). Annealing tau from 1.0 → 0.15 sharpens P and on held-out cost matrices the exact-match rate climbs ≈0.5% → ≈95% (chance 1/24 ≈ 4% for N=4) while the mean optimality gap shrinks ≈200× (0.32 → 0.0015). Evaluation brute-forces the true optimum to score the gap. Pure CPU, tiny N/batches, ≈18 s.

• A spiking net learns through a surrogate gradient — the headline "an EVENT-DRIVEN spiking net learns via a SURROGATE GRADIENT" win for the new TNNetLIFNeuron, a spiking leaky-integrate-and-fire neuron layer (Neftci, Mostafa & Zenke 2019; Zenke & Ganguli 2018, SuperSpike). A new computational paradigm: a stateful neuron integrates an input current into a membrane potential over a time axis and emits a binary {0,1} spike when it crosses threshold, then resets — V[t] = beta·V[t-1]·(1−S[t-1]) + I[t], S[t] = 1 if V[t] ≥ V_th (beta = exp(-1/tau) leak). The forward is exactly this hard, faithfully-binary dynamics; since the Heaviside derivative is zero almost everywhere, the backward substitutes the fast-sigmoid SuperSpike surrogate sigma'(V) = 1/(1+alpha·|V−V_th|)^2 and back-propagates through time across the T unrolled steps. No trainable params (a pointwise neuron model over an upstream linear/conv layer, like an activation); beta/V_th/alpha round-trip via FFloatSt[0..2]. The demo rate-encodes a few synthetic classes as Bernoulli spike trains and trains Input(T,1,DIN) → PointwiseConvLinear → TNNetLIFNeuron → AvgChannel(rate readout) → FullConnect → SoftMax. Headline payoff: accuracy reported alongside the spike rate — the net reaches ≈99% while its hidden neurons fire on only ≈14% of (time, neuron) sites (≈86% silent, event-driven sparsity). Honest caveat: hard-forward / smooth-backward means the surrogate is a biased gradient estimator, so it wants a gentler LR / more steps than a ReLU MLP (training dips before it climbs). Pure CPU, tiny dims, ≈2 s.

Science-of-deep-learning phenomena

• Toy models of superposition — pure-CPU reproduction of Anthropic's Toy Models of Superposition (Elhage et al. 2022) using existing layers only. An importance-weighted-MSE autoencoder Input(20) → TNNetFullConnectLinear(5) → TNNetFullConnectReLU(20) packs more sparse features than it has bottleneck dimensions, and the geometry it picks tracks feature sparsity. Sweeping sparsity S ∈ {0.0, 0.7, 0.9, 0.99} shows the monosemantic→polysemantic phase transition (superposition ratio 1.0 dense → 3.8 sparse), with per-feature represented-norm bars and a glyph-shaded interference-matrix heatmap. Runs in ~2 min.
• Double descent — pure-CPU reproduction of the model-wise "double descent" risk curve (Belkin et al. 2019; Nakkiran et al. 2020). The same tiny MLP is trained across a width sweep H ∈ {1..128} on a 60-sample regression task with 15% label noise; test error falls, then rises to a sharp peak at the interpolation threshold (params ≈ train size), then falls again in the over-parameterised regime. A label-noise on/off ablation shows the peak is noise-driven (the clean-label curve is ~monotone). Runs in ~1 min.
• Epoch-wise double descent — reproduces the THIRD axis of double descent from Nakkiran et al. 2020 (the epoch-wise / temporal figure): a FIXED mildly over-parameterised MLP is trained on a small label-noisy classification set and held-out test error is charted against training EPOCH (the only swept axis is time). Pure-CPU, in-tree layers only. Complements the model-wise DoubleDescent example.
• Random label memorization — pure-CPU reproduction of Zhang et al. (ICLR 2017), Understanding deep learning requires rethinking generalization. The same fixed over-parameterised MLP ($\text{Input} → \text{FullConnectReLU}(64) \times 2 → \text{FullConnectLinear}(\text{K}) → \text{SoftMax}$) is trained twice on the same 5-class Gaussian-blob inputs: once with true labels and once with the labels randomly shuffled. Both runs reach ~100% train accuracy (it memorises pure noise), yet only the true-label run generalises (test ≈ 100% vs ≈ chance 1/K for random labels) — train error alone says nothing about generalisation. Complements DoubleDescent (fixed capacity, true-vs-random labels). A Part 2 label-corruption-fraction sweep (p ∈ {0.0, 0.25, 0.5, 1.0}) charts the smooth interpolation between real structure and pure memorization: epochs-to-fit-train rises with p and the train/test gap widens (0% → 75%). Runs in ~12 s.
• Edge of Stability - Reproduces "progressive sharpening" under full-batch gradient descent: the top Hessian eigenvalue rises to ~2/η and hovers there (Cohen et al. 2021), measured online with TNNet.HessianCurvatureReport
• EWC continual learning — pure-CPU reproduction of catastrophic forgetting and the Elastic Weight Consolidation cure (Kirkpatrick et al., PNAS 2017). A tiny 4-class 2-D MLP is trained to convergence on Task A, then continued on Task B (the same clusters with the input coordinates rotated, a "perturbed-input" task in the spirit of permuted-MNIST). After Task A it snapshots the optimal weights w_A and the diagonal empirical Fisher F_i of every parameter, computed exactly the way TNNet.FisherImportanceReport does (accumulate squared per-parameter gradients over Task A on a frozen net; the curvature an EWC penalty consumes). Two arms then learn Task B: PLAIN sequential fine-tuning (no penalty → Task-A accuracy collapses, the forgetting) versus EWC, which after each weight step applies a decoupled, clamped pull w_i ← w_i − clamp(LR·λ·F_i,0,1)·(w_i − w_A_i) — the gradient of the quadratic penalty Σ (λ/2)·F_i·(w_i − w_A_i)² — that pins the high-Fisher (important-for-A) parameters at w_A while leaving the low-Fisher ones free to learn B (the clamp keeps the step stable despite the large λ the tiny empirical Fisher forces). The headline is a 2×2 table — on this run PLAIN drops Task A to 0.285 while learning B to 1.000, whereas EWC retains Task A at 0.535 (a +0.25 retention gain) for only a 0.016 Task-B cost. Built-in PASS/FAIL gates assert that PLAIN actually forgets A and that EWC retains A meaningfully better, so the demo cannot silently prove nothing. Manual training loop (SetBatchUpdate(true)); pure CPU, single seed, ~3 s, no download.

Diagnostics & introspection reports

Architecture & cost

• Architecture diff — demonstrates TNNet.DiffArchitecture(OtherNet) by building two near-identical classifier variants (one with an extra TNNetChannelStdNormalization, one with a swapped activation) and printing a unified-diff-style report so refactors of builder helpers are easy to verify.
• FLOPs report — builds a tiny MLP and a tiny CIFAR-style conv stack and prints TNNet.CountFLOPsPerLayer(NN): per-layer estimated forward-pass FLOPs, the layer's share of the network total, and a tally of layer classes the estimator doesn't model (so unrecognised layers are visibly accounted for). Pure structure inspection — no training, no probe batch.
• Memory-footprint report — demonstrates TNNet.MemoryFootprintReport, a static per-layer estimator of activation / parameter / gradient memory for a TNNet. Builds a tiny MLP (64 -> 256 -> 256 -> 10 + softmax) and prints its per-layer memory table. Pure structure inspection — no probe batch, no forward pass.
• Receptive-field report — a purely analytical walk (no data, no forward pass) that propagates the receptive-field recurrence through a VGG-style 3x3 stack and a stride-2 downsampling stack and prints TNNet.ReceptiveFieldReport(NN): per-layer cumulative receptive-field size, effective stride (jump), the fraction of the input plane one deepest output unit can see, the shallowest layer whose RF first covers the whole input ("the rest is global mixing" cut point), and a 1x1/pointwise-tail flag list. Tracks X and Y independently. Answers "does my stem see enough context before the global-pool head?" with no trained weights.
• Effective receptive-field report — runs TNNet.EffectiveReceptiveFieldReport(NN, Probes), the empirical (gradient-measured) counterpart of the analytical ReceptiveFieldReport: it answers "what input region does a deep output unit actually WEIGHT?" rather than "what COULD it see?". It picks the centre output unit of the final spatial layer, enables input-gradient flow via TNNet.EnableInputGradient, and over a probe batch back-propagates a one-hot output error to accumulate |d out_centre/d input| (summed over depth) into a per-(x,y) input-plane ASCII heatmap, then reports the effective RF (smallest centred square holding 90% of the gradient mass — radius, per-axis half-widths, centroid, mass-weighted spatial std) and the effective/theoretical ratio side-by-side (it calls ReceptiveFieldReport internally for the analytical number). The example sweeps kernel-size / stack-depth configs (1x3x3 .. 4x5x5) whose theoretical RF roughly doubles, prints a (config, theoretical_RF, effective_RF_diam, eff/theo) table, and writes the (radius, mass_fraction) CSV side-output per config — single large kernels weight their whole window (ratio ≈ 1.0) while stacked small kernels concentrate the gradient (ratio drops to ≈ 0.76), so the effective RF growing sub-linearly in the theoretical RF (Luo et al. 2016) is visible in one run. Distinct from SaliencyReport (per-sample attribution for a class logit, not a batch-averaged spatial-extent measurement). Forward+backward only on a frozen net; weights are never modified.

Gradient, weight & training health

• Gradient-norm report — runs a single forward+backward pass on a 12-layer ReLU MLP with and without a midpoint TNNetLayerNorm and prints TNNet.GradientNormReport(NN, Input, Target): per-layer ||dL/dx_in|| and ||dL/dW||, consecutive-layer ratio, vanishing/exploding flags, and a log10 10-bin ASCII histogram.
• Numerical-gradient eps sweep — a didactic finite-difference diagnostic: takes one well-tested net (TNNetFullConnectLinear -> TNNetHyperbolicTangent, fixed hand-set weights/input, MSE loss) and runs the exact central-difference gradient check from tests/TestNeuralNumerical.pas for eps in {1e-1 .. 1e-7}, printing per-eps max abs/rel error vs the analytic Backpropagate gradient. Reproduces the classic U-shaped error curve — large eps dominated by O(eps^2) truncation, tiny eps dominated by FP32 round-off/cancellation — with the minimum near eps ~ 1e-2..1e-3. Explains why the test suite probes at eps = 1e-4 with a ~0.01 tolerance (FP32, not FP64, sets the sweet spot). Forward+backward only, no RNG, sub-second.
• Weight-histogram report — builds a small MLP and prints TNNet.WeightHistogramReport(NN) before and after a short training run: per-trainable-layer min/max/mean/std, ||W||_2, ||W||_inf, near-zero count, and a per-layer 32-bin ASCII bar histogram over [-MaxAbs, +MaxAbs]. Lets you eyeball how the weight distribution moves away from its init.
• Weight-spectrum report — for every trainable layer prints TNNet.WeightSpectrumReport(NN) before and after a short training run: the top singular value sigma_1(W) (a few power-iteration steps via the reusable TNNet.EstimateSpectralNorm helper), ||W||_F, the stable-rank-flavoured ratio sigma_1/||W||_F (near 1 hints at rank-1 collapse), a Marchenko-Pastur baseline ratio, a network histogram, and a flag list (spectral-norm > threshold / stable-rank ≈ 1). Pure forward-only on the weight tensors — no probe batch.
• Weight spectral-tail report — runs TNNet.WeightSpectralTailReport(NN), a label-free, weights-only heavy-tailed self-regularization (HT-SR) diagnostic that predicts per-layer training quality from the weights alone — no probe batch, no labels, no test set (Martin, Mahoney & Peng 2021, Nature Communications; the WeightWatcher alpha metric). For every trainable layer it forms the smaller Gram matrix of the reshaped weight tensor (W^T W or W W^T), computes its full eigenvalue spectrum with a self-contained Double-precision cyclic Jacobi eigensolver, and fits a power law rho(lambda) ~ lambda^(-alpha) to the upper tail via the Clauset/Hill MLE (swept over lambda_min cuts, min-KS selection). It reports per layer the power-law exponent alpha (well-trained layers land in [2,4]; >6 flags under-trained / still-random-like, <2 flags over-correlated / memorising), the capacity-weighted weighted-alpha alpha*log10(lambda_max), the KS goodness-of-fit, lambda_max, the Marchenko-Pastur bulk edge, and a log10(lambda) histogram, plus the network-level average weighted-alpha (a single label-free model-quality scalar), an alpha-across-depth chart, and per-layer flags. Built-in checks: PSD non-negativity, Frobenius trace invariance, and lambda_max == EstimateSpectralNorm(W)^2. Pure weight inspection — no forward pass, weights never touched.
• Weight-drift report — trains a 6-layer ReLU MLP on a tiny hypotenuse-like task for a few epochs with ONE hidden layer's LearningRate pinned at 0, snapshots the network before and after training, and prints TNNet.WeightDriftReport(SnapA, SnapB): the frozen layer shows ~0 L2 drift and a ~1.0 frozen fraction while the surrounding layers show non-trivial drift. Pure CPU, runs in seconds.
• Activation-stats report — runs a probe batch through a small net and prints TNNet.ActivationStatsReport(NN, Probes) before and after a short training run: per-layer mean/std/min/max/|median|/|skew|/kurtosis, saturated/negative/near-zero fractions, a 16-bin ASCII histogram per layer, plus a network-level per-layer-std histogram and a flag list (near-collapsed and saturating layers) so vanishing/exploding activation patterns jump out at a glance. Pure forward-only.
• Dead-neuron report — trains the same ReLU MLP twice (sane LR vs deliberately aggressive LR) and prints TNNet.DeadNeuronReport(NN, Probes) for each: per-TNNetReLUBase-family layer dead-unit count and percentage across a probe batch, mean per-sample zero-fraction, a 10-bin ASCII histogram of dead% across layers, and the worst layer. Surfaces the dying-ReLU pathology empirically.
• Dead-ReLU diagnostic — trains the same small classifier four times, identical except the hidden activation (TNNetReLU/TNNetLeakyReLU/TNNetGELU/TNNetSwish), and prints the per-epoch fraction of dead hidden units plus a summary table comparing the four. Unlike the static Dead-neuron report, this is a per-epoch trajectory and a cross-activation comparison: under an aggressive LR plain ReLU strands ~19% of units at zero gradient while the leaky/smooth activations lose none.
• Dead-ReLU learning-rate sweep — the LR-sweep follow-up to the Dead-ReLU diagnostic: sweeps the learning rate over a grid (0.02 -> 0.5) and, for each LR, trains the same classifier with TNNetReLU/TNNetLeakyReLU/TNNetGELU/TNNetSwish, printing an LR x activation table of peak dead-unit fraction. ReLU's dead fraction climbs monotonically (10% -> 17%) as the LR grows while the leaky/smooth activations stay at 0% across the whole sweep — the cleanest "dying ReLU is a learning-rate pathology" curve.
• Neuron-correlation report — runs TNNet.NeuronCorrelationReport(NN, Probes) on a probe batch before and after a short training run on a single-ReLU-feature synthetic target (y = max(0, w.x)): for every trainable layer it computes the pairwise Pearson correlation rho_ij between neurons across the batch (along the neuron axis) and prints a 10-bin |rho_ij| histogram, the top-K most-correlated pairs (a merge/prune candidate list), an effective neuron count = participation ratio N^2 / sum_ij rho_ij^2 (in [1, N]), and per-layer flags (near-duplicate pair, collapsed layer, constant neurons). At fresh init the high-|rho| tail is empty and no near-duplicate flags fire; the trained net grows a clear |rho|>0.8 tail, raises near-duplicate flags, and its effective neuron count drops as units align onto the single useful direction — intra-layer redundancy made visible. Pure forward-only.
• Layer-sensitivity report — runs TNNet.LayerSensitivityReport(NN, Probes [, Targets]) across three model families (a tiny MLP with a loss target, a CIFAR-style conv stack, and an attention stack) on synthetic probe batches. For every trainable layer it multiplicatively jitters the weights (W *= 1 + eta, eta ~ N(0, sigma^2)), re-runs the probe batch to measure the forward output-delta L2 (and the MSE loss-delta when targets are supplied), then restores the weights exactly between trials via a whole-net SaveDataToString/LoadDataFromString snapshot. It prints per-layer mean/max output-delta, a param-count-normalised sensitivity, a 10-bin histogram, high-/low-impact flags (top/bottom 10%), and a one-line fragility verdict (max/median layer sensitivity). Pure forward-only, no backward pass.
• Gradient-conflict report — runs TNNet.GradientConflictReport(NN, Samples) on two labelled probe batches measured on the same frozen trained classifier, answering "do the samples in this batch pull the weights in compatible directions, or do they fight each other?". For each sample it runs one forward + one backward (ClearDeltas before each, never UpdateWeights) and snapshots that sample's full flattened per-parameter weight-gradient vector g_i (reusing the per-parameter gradient tensors Backpropagate already populates — no input-gradient enablement, like FisherImportanceReport), then reports the pairwise gradient cosine cos(g_i,g_j) = <g_i,g_j>/(||g_i|| ||g_j||): a 10-bin ASCII histogram over [-1,1], the conflict fraction (share of pairs with cos < 0 — gradients that actively undo each other) plus a strong-conflict fraction (cos < -0.5, the genuinely-opposed tail, since cross-class pairs of a softmax head sit just below zero by construction), the mean/median cosine, the most-conflicting sample pair (a "these two examples disagree most" pointer), and a per-class-pair mean-cosine matrix (numeric + glyph heatmap) so a class pair whose gradients systematically oppose stands out. An optional LayerIdx restricts the cosine to one trainable layer's gradient slab (the conflict is often concentrated in the classifier head). The example contrasts a clean linearly-separable 3-cluster batch (cosines clustered positive, conflict fraction ~0) against a deliberately label-noised / overlapping batch (a fat negative-cosine tail and a high conflict fraction emerge). The self-cosine cos(g_i,g_i)=1 diagonal and matrix symmetry are the built-in correctness checks. Weights are never stepped (a measurement, not training).
• Gradient noise-scale report — runs TNNet.GradientNoiseScaleReport(NN, Samples), the gradient signal-to-noise diagnostic that analytically predicts the batch-size sweep (McCandlish et al. 2018, An Empirical Model of Large-Batch Training). On a frozen net (ClearDeltas before each sample, never UpdateWeights) it snapshots each sample's full per-parameter weight-gradient vector g_i, forms the mean gradient and per-parameter variance, and reports the per-parameter gradient SNR |g_bar|/std histogram + per-layer mean, the simple noise scale B_simple = tr(Sigma)/||g_bar||^2 (the critical batch size beyond which bigger batches stop buying faster convergence), the effective-batch noise curve noise(B)=B_simple/B, and per-layer signal-/noise-dominated flags (with an optional LayerIdx restricting every statistic to one layer's gradient slab). The example contrasts a clean linearly-separable batch (high SNR, tiny B_simple) against a label-noised / overlapping batch (low SNR, large B_simple) and prints an empirical batch-size sweep so the prediction can be eyeballed against reality. Built-in checks: feeding the same sample N times drives B_simple to ~0, and a single-sample batch warns rather than dividing by zero. Weights are never stepped.
• Hessian-curvature report — runs TNNet.HessianCurvatureReport(NN, Samples), a loss-surface sharpness diagnostic built on Hessian-vector products estimated by central-differencing the gradient (Hv ~= (grad L(theta+eps*v) - grad L(theta-eps*v))/(2*eps)) — no second-order autograd, reusing the whole-batch forward+backward machinery and the SaveDataToString/LoadDataFromString snapshot/restore pattern. On a frozen net it reports the Hessian trace tr(H) via the Hutchinson estimator over Rademacher probes (mean curvature), the top eigenvalue lambda_max(H) via power iteration on the HVP operator (the canonical flat-vs-sharp-minimum metric, Keskar et al. 2017 / Foret et al. SAM 2021), the curvature-concentration ratio lambda_max/(tr(H)/N), a per-layer trace breakdown, a per-probe v^T H v histogram, and a flat/moderate/sharp verdict. The example trains the same tiny MLP into a sharp minimum (small batch + high LR) and a flat one (large batch + low LR + weight decay) so the lambda_max gap the generalization literature ties to sharpness is visible. Built-in checks: probe-count-independence of tr(H) on a linear net, and lambda_max <= tr(H) in the PSD regime. Weights are never stepped (a measurement, not training).
• Local Learning Coefficient (LLC) report — runs TNNet.LocalLearningCoefficientReport(NN, Samples [, ChainLen, Eps, Gamma]), an empirical estimate of the Local Learning Coefficient (LLC) — the Real Log Canonical Threshold (RLCT) from Singular Learning Theory (Watanabe; Lau, Murfet, Wei et al. 2023, Quantifying Degeneracy in Singular Models via the LLC). Unlike a second-order Hessian top-eigenvalue (HessianCurvatureReport), the LLC measures the volume-scaling / EFFECTIVE dimensionality of the minimum the optimizer settled into — it counts the flat, degenerate directions a sharpness metric cannot see, so a redundant / over-parameterised solution reads LLC_hat << dim(w) (far fewer effective degrees of freedom than raw weights). From the trained weights w* it runs a short tempered, anchored SGLD chain pinned to the basin by a Gaussian anchor (gamma/2)*||w - w*||^2, then forms the WBIC free-energy estimate LLC_hat = n*beta*(mean_chain[L(w)] - L(w*)) with beta = 1/ln(n) and the per-step anchored-Langevin update w <- w - (eps/2)*(n*beta*g + gamma*(w - w*)) + N(0, eps). It reuses the existing forward+backward gradient machinery (SetBatchUpdate(true), Delta = -LR*grad divided back out; the only new infrastructure is the anchored update + chain average) and is non-destructive — w* is snapshotted and restored bit-for-bit on return. The report prints LLC_hat, the raw parameter count dim(w) and the ratio LLC_hat/dim(w), with the caveat that the absolute value is calibration-dependent (it shifts with eps/gamma/chain-length) but the ordering across nets under fixed hyperparameters is the robust signal. The example contrasts a minimal trained net, an over-parameterised net with two hidden units forced redundant (duplicate-then-halve), and a random-init net: both trained nets read small LLC_hat << dim(w) while the untrained net (not a critical point) reads a large/negative LLC_hat — the honest "this is not a minimum" signal (the README documents the known pitfall and why the fine minimal-vs-over-parameterised ordering needs a longer chain than the CPU-toy budget allows). Distinct from HessianCurvatureReport (sharpness / top-eigenvalue, blind to degeneracy) and IntrinsicDimensionReport (activation-manifold geometry, not loss-basin volume). Weights are never stepped (a measurement, not training).
• Loss-landscape probe — given a trained network and a small validation batch, samples the loss along a random filter-normalised (Li et al. 2018) 1D direction in weight space at K offsets in [-R, +R], then prints an ASCII curve, a "sharpness" scalar (second central difference of the loss at the centre), and the "loss-doubling radius" (smallest |alpha| where L(alpha) > 2*L(0)). Restores the original weights at the end; uses TNNet.LossLandscapeProbe(NN, Samples, K, R).
• Mode-connectivity report — runs TNNet.ModeConnectivityReport(NN, SnapshotB, Samples), the linear mode-connectivity / loss-barrier diagnostic between two trained nets of the same architecture (Garipov et al. 2018; Frankle et al. 2020). With the live net as endpoint A and a SaveDataToString snapshot of endpoint B, it sweeps alpha in [0,1] at K+1 points, sets the live weights to the interpolation theta(alpha) = (1-alpha)*A + alpha*B via whole-net snapshot arithmetic (TNNetVolume.MulMulAdd, no per-scalar hot loop), runs one whole-batch forward over Samples at each alpha, and reports the loss curve L(alpha) as a #-bar ASCII chart, the barrier height max_alpha L(alpha) - max(L(0), L(1)) (>0 = a bump between basins; ~0 = linearly connected), the argmax-alpha where the barrier peaks, and a connected / weak barrier / separated verdict. The example trains the same tiny MLP twice — once from the same init (low barrier, same basin) and once from different inits (higher barrier) — so the contrast is visible in one run. Built-in checks: L(0)/L(1) recomputed on the path match a direct forward to <1e-5 (snapshot-arithmetic faithfulness), and B := A collapses the curve to a flat zero-barrier line. Distinct from WeightDriftReport (weight-space L2 drift, no loss along the path) and LossLandscapeProbe (one net, random direction). Endpoint-A weights are restored bit-for-bit; pure forward-only.
• Permutation-alignment report (Git Re-Basin) — runs TNNet.PermutationAlignReport(NN, SnapshotB, Samples [, ScoreMode, K]), the "Git Re-Basin" weight-space NEURON-PERMUTATION alignment diagnostic (Ainsworth, Hayase & Srinivasa 2022; Entezari et al. 2021) — the dual of ModeConnectivityReport. That report measures the linear-interpolation loss barrier between two independently-trained nets but does nothing about it; this one shows most of that barrier is an illusion of neuron-labelling: a hidden layer's units are interchangeable up to a permutation (permute the units and the next layer's matching input-weight columns and the represented function is unchanged). It walks the trainable layers front-to-back, for each hidden layer greedily solves the permutation P_L of net B's units that best aligns them to net A's — by weight-row cosine (ScoreMode=0, default) or per-unit activation correlation over Samples (ScoreMode=1) — applies P_L to B's output neurons and compensates the next layer's input columns, then re-runs ModeConnectivityReport's interpolation sweep theta(alpha) = (1-alpha)*A + alpha*P(B) (the same TNNetVolume.MulMulAdd snapshot arithmetic) and reports the loss barrier before vs after alignment as two scaled #-bar rows, the per-layer permutation churn (fraction of units moved), and a barrier collapsed / partially reduced / unchanged verdict. Three built-in PASS/FAIL checks: permutation invariance (permute+compensate leaves B.Compute bit-for-bit unchanged — the foundational identity), align-to-self (SnapshotB := A gives identity permutations and a flat zero barrier), and monotonicity (post-alignment barrier <= pre-alignment barrier). The example trains the same tiny MLP twice from different inits (so a real barrier exists) and prints a weight-matching run, an activation-matching run, and the align-to-self check — the barrier visibly shrinks (~65–75% reduction) once the permutation symmetry is quotiented out. Distinct from RepresentationSimilarityReport (compares activations for similarity, never produces a weight permutation or re-interpolates), NeuronCorrelationReport (intra-layer redundancy of one net) and WeightDriftReport (raw L2 drift, no symmetry quotient). Endpoint-A weights restored bit-for-bit; forward-only.
• Intrinsic-dimension report — runs TNNet.IntrinsicDimensionReport(NN, Probes), a forward-only representation-geometry diagnostic answering "how many effective dimensions does each layer's activation cloud actually occupy?" — the dimensionality of the data manifold the probe batch traces out at each depth. For every trainable layer it runs one forward pass over an unlabelled probe batch, flattens each sample to a row of an N x D_l matrix, and reports two complementary intrinsic-dimension estimates side by side: (1) the linear / PCA ID via the participation ratio of the activation covariance eigenspectrum PR = (sum lambda)^2 / sum lambda^2 (eigenvalues from the smaller N x N Gram / D x D covariance via the same Double-precision cyclic Jacobi eigensolver WeightSpectralTailReport ships), and (2) the TwoNN nonlinear estimator (Facco et al. 2017) read off the least-squares slope of -log(1 - F(mu)) against log(mu), with mu = r2/r1 the per-sample 2nd-to-1st nearest-neighbour distance ratio. It reports per layer both IDs, the linear-vs-nonlinear gap, a D_l-normalised compression ratio TwoNN_ID/D_l, an ID-across-depth ASCII bar chart, and expanded / compressed / near-full-rank flags. The example contrasts a fresh-init net (flat, near-input ID) against a trained net (the expand-then-contract "hunchback" of Ansuini et al. 2019) and includes a known-k-dim-subspace ground-truth recovery so the estimate is visibly correct in one run. Built-in checks: a k-dim linear subspace recovers PCA_ID ~ k and TwoNN_ID ~ k, identical samples drive both IDs to ~0, and PCA eigenvalues are non-negative. Distinct from NeuronCorrelationReport (linear redundancy among feature axes) and FeatureSeparabilityReport (label-aware class geometry). Pure forward-only — NN.Compute only, weights never touched.
• Neural-tangent-kernel report — runs TNNet.NeuralTangentKernelReport(NN, Samples [, TargetClass]), a forward-only diagnostic measuring the empirical Neural Tangent Kernel (Jacot, Gabriel & Hongler 2018) of a classifier over a small probe batch (N kept to ~8–16; the kernel is O(N^2) entries each an O(P)-param dot product). On a frozen net (SetBatchUpdate(true), ClearDeltas before each sample, never UpdateWeights) it runs one forward + one backward per probe, seeded one-hot at TargetClass (default -1 = each sample's own predicted argmax), to snapshot the per-parameter weight-gradient vector g_i of that scalar logit — reusing the same Neurons[*].Delta/FBiasDelta gradient read-out (divided back out by the layer learning rate) that FisherImportanceReport / GradientConflictReport / GradientNoiseScaleReport share; the Delta = -LR*grad sign cancels in the Gram dot products. It forms the empirical NTK Gram K_ij = <g_i, g_j> and reports the kernel as a glyph-shaded ASCII heatmap, its full eigenspectrum via the same self-contained Double-precision cyclic Jacobi eigensolver WeightSpectralTailReport / IntrinsicDimensionReport ship (no new numerical code), the condition number lambda_max/lambda_min (guarded when lambda_min<=0), the kernel-target alignment <K, yy^T>_F/(||K||_F ||yy^T||_F) (Cristianini et al. 2001; y = centered target-class indicator) — the headline number, the effective rank / participation ratio (sum lambda)^2/sum lambda^2, and a log10(lambda) histogram. The example contrasts a fresh-init vs a trained tiny MLP on a synthetic 3-class blob task so the NTK reshaping (alignment / conditioning / effective rank) is visible in one run. Built-in checks: kernel symmetry residual ~0 and a strictly-positive diagonal K_ii = ||g_i||^2. A possible follow-up (not done here) is a fresh-init-vs-trained NTK-drift contrast. Distinct from GradientConflictReport (pairwise gradient cosines, not the raw Gram + its spectrum + label alignment) and FisherImportanceReport (per-parameter curvature, not the sample-sample kernel). Forward + backward read only — weights never stepped.
• Fisher-importance report — runs TNNet.FisherImportanceReport(NN, Samples) on a labelled probe batch for a freshly-initialised classifier and the same architecture after a short training run, estimating the diagonal (empirical) Fisher information F[theta] = E_x[(d log p(y|x)/d theta)^2] of every trainable parameter by accumulating per-sample squared parameter gradients (one forward + one backward per sample, gradient taken w.r.t. the true or predicted label) on a frozen net whose weights are never stepped. Reports per-layer total Fisher mass and its network share (the "which layers can I least afford to prune" ranking), per-layer mean/max per-parameter Fisher and a near-zero count, a 10-bin ASCII log10(Fisher) histogram, the effective-parameter-count = participation ratio (sum F)^2/sum F^2 (a one-number concentration proxy), and per-layer high-importance / prunable / dead-layer flags. The README sketches how the same Fisher tensor feeds a downstream Elastic-Weight-Consolidation (EWC) two-task penalty. Forward+backward only; reuses the per-parameter gradient tensors Backpropagate already populates (no input-gradient enablement, unlike the saliency report).

Representation & interpretability

• Linear-probe report — runs TNNet.LinearProbeReport(NN, Samples [, ValSamples]) on a labelled probe batch for a freshly-initialised classifier and the same architecture after a short training run, answering "where does the model become a classifier?". For every intermediate layer it fits a closed-form ridge linear probe W = (X^T X + Lambda*I)^-1 X^T Y on that layer's flat activations (a self-contained Double-precision Gauss-Jordan solve — no SGD loop, no backward pass) and reports per-layer top-1 probe accuracy, a held-out probe accuracy (the train/val gap flags overfit probes), the one-hot regression MSE, the per-layer accuracy delta, a 10-bin ASCII bar chart of probe accuracy across depth, and Collapse / Saturation-point / near-Random flags. Over-wide layers are deterministically random-projected down to MaxFeatDim (default 256) to bound the O(D^3) solve. On the synthetic XOR + concentric-rings task (not linearly separable from the raw input) the untrained net's probe accuracy degrades with depth while the trained net's is preserved and climbs — the per-layer linear-separability reshaping made visible. Pure forward-only on the backbone, which is never modified.
• Logit-lens report — runs TNNet.LogitLensReport(NN, pInput [, HeadStartIdx]) on an unlabelled probe batch for a freshly-initialised classifier and the same architecture after a short training run, the logit-lens diagnostic (nostalgebraist 2020; cf. "Tuned Lens", Belrose et al. 2023) answering "if we read out the prediction at THIS layer using the network's OWN trained output head, what would it already say?" — the model's running, self-decoded belief at each depth, fitting zero parameters. It identifies the trailing readout head (default HeadStartIdx = the last trainable layer plus its activation/softmax tail) and, for every earlier layer whose flat activation is shape-compatible with the head's expected input, splices that activation into the head's input slot and recomputes only the head layers to get a per-layer lens distribution p_L. Reports the per-layer argmax-agreement-with-final bar chart, the crystallization depth (shallowest layer after which the lens argmax matches final and never flips again — per-batch mean + 10-bin histogram), per-layer mean top-1 confidence and lens entropy (the readout sharpens with depth), and the per-layer KL(p_L || p_final) curve (monotone decrease = the residual stream incrementally refining toward the final answer); width-incompatible layers are listed explicitly as SKIPPED (the honest constraint of the classic lens). Built-in checks: applying the lens at the head input (no substitution) reproduces p_final exactly (agreement 1.0, KL 0), and a single-layer head degenerates to the trivial "everything resolves at the last layer" profile. The constant-width body keeps every hidden layer lens-compatible so the depth profile is dense, and the input layer shows up as SKIPPED. Distinct from LinearProbeReport (which fits a fresh ridge probe per layer — the lens fits nothing and reuses the model's own trained head), ActivationPatchingReport (causal cross-input swaps) and FeatureSeparabilityReport (cluster geometry, no readout). Pure forward-only — NN.Compute plus per-head-layer recompute, weights never stepped; the live state is restored on exit.
• Tuned-lens report — runs TNNet.TunedLensReport(NN, pInput [, HeadStartIdx, TrainIters, LearningRate]) next to TNNet.LogitLensReport on the same unlabelled probe batch, the learned sibling of the logit lens (Belrose et al. 2023, "Eliciting Latent Predictions with the Tuned Lens"). Where the logit lens splices a raw hidden activation straight into the model's own frozen head, the tuned lens first passes that activation through a small per-layer learned affine translator (one TNNetFullConnectLinear of the head-input width, plus bias) trained to map the layer's residual state into the final-layer basis before the frozen head decodes it — correcting the representation drift that biases the raw logit lens at early depths. The trunk and head are frozen; for each lens-compatible layer a private throw-away mini-net Input -> Translator(identity-seeded) -> clone of the frozen head is fit by minimising KL to the model's own final distribution (distillation-to-self, no labels — with a softmax head, backpropagating p_final as the soft target is exactly that KL gradient). Prints the tuned lens' KL-to-final, entropy and agreement side by side with the raw logit-lens columns, plus a paired KL-to-final curve (logit . vs tuned #) and the aggregate mean KL-to-final for the logit lens, the untrained (identity) tuned lens and the trained tuned lens. The headline Belrose result is visible: the tuned curve sits lower at the early/middle layers (commits earlier, tracks the final answer more faithfully). Built-in PASS/FAIL checks: an untrained translator does no better than the raw lens (ties its KL-to-final — no free lunch before fitting), fitting lowers the mean KL-to-final, and at the head input the translator collapses to the identity so tuned == logit == final (max |dp| ~ 0). The model's own weights are never stepped (translators live in a private net) and the live state is restored on exit. Distinct from LogitLensReport (zero fitted params) and LinearProbeReport (a fresh label-supervised ridge probe per layer — the tuned lens is label-free and reuses the model's own frozen head). Pure CPU, ~1 s.
• Representation-similarity report — runs TNNet.RepresentationSimilarityReport(NN, Probes [, OtherNet]) on a deliberately over-deep ReLU MLP (16 -> 16 x6 -> 1) on a single-ReLU-feature target, answering "how does the representation reshape with depth, and which layers do redundant work?". It computes the linear Centered Kernel Alignment (CKA, Kornblith et al. 2019) similarity between every pair of layer activations over a shared probe batch: for each layer it flattens the per-sample activation to a row of an N x D_l matrix X_l, column-centers it, and via the N x N Gram trick computes CKA(i,j) = <K_i,K_j>_F / (||K_i||_F ||K_j||_F) — a number in [0,1], invariant to orthogonal rotation and isotropic scaling. Reports the full LxL CKA matrix as a glyph-shaded ASCII heatmap, the adjacent-layer CKA(l,l+1) vector (high = near pass-through redundant layer, sharp dip = reorganization), the single most-redundant layer pair (a merge/prune candidate), the block structure (contiguous runs of mutually-similar layers — "representational stages"), and a one-line verdict. The self-CKA diagonal is 1.0 by construction (the built-in correctness check) and the matrix is symmetric. The example contrasts a fresh-init net (adjacent CKA already high — untrained layers barely transform their input) against the same net after training (a clearer block structure emerges yet the over-deep middle layers stay near-duplicates, so "depth is wasted" lights up), then runs cross-CKA between two independent nets of the same shape ("do these two nets learn the same intermediate features?"). Pure forward-only; weights are never touched, no backward pass is run.
• Feature-separability report — runs TNNet.FeatureSeparabilityReport(NN, Samples, NumClasses) on a labelled probe batch, a label-aware class-geometry / Neural-Collapse diagnostic (Papyan, Han & Donoho 2020) answering "how tightly does each layer cluster the samples of a class, and how far apart are the classes?" — the geometry of the feature space, not its decodability. For every trainable layer it runs one forward pass over the batch and computes the Fisher-style scatter decomposition: within-class scatter tr(Sw) = mean_c mean_{i in c} ||x_i - mu_c||^2 (NC1 cluster tightness), between-class scatter tr(Sb) = mean_c ||mu_c - mu||^2, the Fisher ratio tr(Sb)/tr(Sw), the mean silhouette coefficient (fit-free cohesion-vs-separation in [-1,1]), and the class-mean pairwise-cosine matrix (numeric + glyph heatmap) with the simplex-ETF check (NC2): the mean off-diagonal cosine printed next to its collapse target -1/(NumClasses-1). It prints a per-layer Fisher-ratio-across-depth ASCII bar chart and Collapse / well-Separated / near-Random flags; over-wide layers are deterministically random-projected down to MaxFeatDim (default 256). The built-in correctness check is the scatter-decomposition identity tr(Stot) == tr(Sw) + tr(Sb) (exact on a class-balanced batch). The example contrasts well-separated vs deliberately-overlapped 2-D Gaussian blobs in one run, so the Fisher-ratio and silhouette contrast (and the off-diagonal cosines approaching the ETF target) is visible. Pure forward-only — NN.Compute only, weights never touched, no backward pass.
• Neural-collapse report — runs TNNet.NeuralCollapseReport(NN, Samples, NumClasses [, FeatureLayerIdx, MaxFeatDim]) on a class-balanced labelled probe batch, measuring the four canonical Neural-Collapse metrics (Papyan, Han & Donoho 2020, Prevalence of neural collapse during the terminal phase of deep learning training) on the penultimate-layer features (the activations feeding the final linear head; FeatureLayerIdx = -1 auto-selects). Where FeatureSeparabilityReport stops at the tr(Sw) collapse + Fisher magnitude (a partial NC1), this computes the full headline geometry. It reuses that report's class-mean / within-class scatter Sw / between-class scatter Sb machinery for NC1 = within-class variability collapse tr(Sw·Sb⁺)/C → 0 (trace-ratio surrogate). NC2 = convergence to a simplex equiangular tight frame: the centered class means become equinorm (coefficient of variation of ‖mu_c − mu‖ → 0) and equiangular (every pairwise cosine → −1/(C−1); it prints the mean cosine plus the mean and max deviation from that target, with a centered-mean cosine glyph heatmap). NC3 = self-duality, the cosine alignment between the centered class-mean matrix and the classifier weight rows — honestly skipped with a printed flag when the head is not a width-matched TNNetFullConnectLinear / TNNetPointwiseConvLinear. NC4 = the classifier collapses to a nearest-class-mean rule: the fraction of probe points whose argmax logit equals their nearest centered class mean (→ 1). The example trains a small 4-class classifier well past zero train-error (the terminal phase) on synthetic Gaussian blobs and calls the report every N epochs on a fixed probe, printing an ASCII trajectory of the mean pairwise cosine marching onto the −1/(C−1) line — the simplex assembling itself. Pure forward-only — NN.Compute only, weights read but never stepped; pure CPU, no download.
• Information-plane trajectory — reproduces the Information Bottleneck information-plane story (Tishby & Zaslavsky 2015; Shwartz-Ziv & Tishby 2017, Opening the Black Box of Deep Neural Networks via Information) for a tiny fully-connected classifier on a synthetic binary task: it tracks the mutual-information pair (I(X;T), I(T;Y)) of each hidden layer across training epochs and prints every layer's path through the 2-D information plane as an ASCII scatter — no new layer, no new gradient machinery, only a forward pass (Net.Compute) over the full dataset at each logged epoch reading each layer's Output. MI uses the original binning estimator (each neuron's activation discretized into B equal-width bins, the per-sample bin-tuple is a discrete code; I(X;T)=H(T) since the deterministic net with unique inputs has H(T|X)=0, and I(T;Y)=H(T)-H(T|Y) from per-class code histograms, reported in bits). The narrative target is the two phases — a fast fitting phase (I(T;Y) rises to its 1-bit ceiling) followed by slow compression (I(X;T) drops while I(T;Y) stays high). HONEST headline in the house "what did NOT reproduce" style: the binning estimator requires a saturating activation to show compression, so two arms ship — Arm A (tanh trunk, TNNetFullConnect, bounded [-1,1]) where the bend appears (deepest hidden layer loses ~2 bits of I(X;T) over training while keeping I(T;Y)=1), and Arm B (ReLU trunk, TNNetFullConnectReLU, unbounded → per-epoch-rescaled bins are ill-defined) where the clean bend vanishes — demonstrating the Saxe et al. 2018 controversy rather than overclaiming. Pitfalls documented: MI is upper-bounded by log2(#samples) and B^width, and absolute values are estimator-dependent — the robust signal is the shape of the trajectory and the tanh-vs-ReLU difference, not the nats. Pure CPU, deterministic, well under a minute.
• Prediction-depth report — runs TNNet.PredictionDepthReport(NN, Support, SupportLabels, Queries [, QueryLabels]) on a labelled support batch and a query batch for a freshly-initialised classifier and the same architecture after a short training run, the per-example difficulty diagnostic of prediction depth (Baldock, Maennel & Neyshabur 2021, Deep Learning Through the Lens of Example Difficulty) answering "at how deep a layer does the network actually make up its mind about THIS example?" — a per-sample resolution depth, not a per-layer aggregate. It snapshots each trainable layer's flattened activation over the support and query batches, takes a k-NN vote (default K=5, cosine distance) over the support at every layer, and defines a query's prediction depth as the index of the shallowest layer after which the k-NN vote agrees with the network's final argmax and never disagrees again. Reports a 10-bin ASCII histogram of prediction depth + mean/median, the per-layer newly-resolved count (where examples get decided), the K deepest (= hardest) query indices as a ready-made hard-example / relabel queue, and — with QueryLabels — a correctness cross-tab (mean depth of correctly vs incorrectly classified queries; the literature's headline result is depth correlates with error). Over-wide layers are deterministically random-projected down to MaxFeatDim (default 256) to bound the k-NN cost (the same projection LinearProbeReport uses). On the synthetic 4-class 2D-blob task with an ambiguous between-blob query subset, fresh-init depths pile up at the last layer (right-skewed) while after training the mass shifts shallow for the well-separated clusters and the hard subset keeps a deep tail, and the incorrect-minus-correct mean-depth gap is positive. Built-in checks: feeding the support set as its own queries gives every sample a finite depth with final-layer agreement ~1.0 (a point is its own nearest neighbour at distance 0), and a one-class support set drives every depth to 0. Distinct from LinearProbeReport (parametric per-layer accuracy via a ridge solve — this needs only distances), FeatureSeparabilityReport (per-layer aggregate cluster geometry), TopLogitMarginReport (last-layer confidence) and MCDropoutUncertaintyReport (stochastic uncertainty). Pure forward-only — NN.Compute only, weights never touched, no backward pass.
• Equivariance report — runs TNNet.EquivarianceReport(NN, Probes) on two synthetic 8x8x3 image classifiers: a plain conv net (flip-sensitive) and a global-average net (Input -> TNNetAvgChannel -> FC) that is FlipX/FlipY-invariant by construction. For a fixed menu of input-side symmetry transforms (TNNetFlipX, TNNetFlipY, TNNetReverseChannels, a 1-channel TNNetRoll) it reports the per-transform invariance error mean_x ||f(T(x)) - f(x)||_2 / ||f(x)||_2, the top-1 argmax agreement rate, a 10-bin ASCII histogram of per-sample error, and a verdict (invariant / approximately invariant / sensitive). The plain net reads sensitive on the flips while the global-average net reads ~0 (invariant) — the built-in correctness check. Pure forward-only; each T(x) is produced by a tiny Input -> Transform wrapper net and the inspected weights are never touched.
• Saliency report — runs TNNet.SaliencyReport(NN, Probe) on a trained synthetic 8x8x2 two-class image classifier and prints, for the predicted class c, three input-attribution heatmaps side by side: vanilla input-gradient saliency |d logit_c/dx| (one forward + one backward pass with a one-hot output error), SmoothGrad (the vanilla map averaged over N noisy copies of the input), and Integrated Gradients (the path integral from a zero baseline to x). Per channel it reports total attribution mass, the top-K most-attributing pixels, and the IG completeness gap |sum(IG) - (logit_c(x) - logit_c(0))| as a built-in correctness check (small relative gap == faithful integration). The discriminative blob lights up in all three maps. Forward+backward only; the trained weights are left untouched.
• LRP report — runs TNNet.LRPReport(NN, Probe) (Layer-wise Relevance Propagation; Bach et al. 2015) on a trained synthetic 6x6x1 two-class dense classifier (FullConnectReLU -> FullConnectReLU -> FullConnectLinear -> SoftMax). Unlike the gradient-based SaliencyReport/GradCAMReport, LRP is a conservation method: it seeds the explained class's relevance with R_c = logit_c and back-distributes it toward the input via the epsilon-rule R_i = sum_j (a_i w_ij)/(sum_k a_k w_kj + eps*sign(z_j)) R_j, redistributing (not differentiating) relevance. It prints the headline per-layer conservation residual |sum(R_in) - sum(R_out)| (which is O(eps) and -> 0 as eps -> 0), the top-K most-relevant input positions (channel,x,y) with signed relevance, and a per-channel ASCII relevance heatmap (the top-relevant cells cluster on the class-specific blob). A dense stack is used on purpose so the epsilon-rule has an exact closed form; the SoftMax head and any attention/normalisation layer are skipped honestly (listed as SKIPPED/passthru, never faked). Forward-only; the trained weights are restored bit-for-bit before returning.
• Activation-patching report — runs TNNet.ActivationPatchingReport(NN, CleanInput, CorruptInput [, TargetIdx]), a forward-only causal mechanistic-interpretability diagnostic (activation patching / causal tracing; Meng et al. ROME 2022, Wang et al. IOI 2022) answering "which layer's activations CARRY the information that decides this prediction?" — by intervention, not correlation. Given a (CleanInput, CorruptInput) pair the trained net maps to different argmax classes, it caches every layer's clean Output, runs a corrupt forward, then for each layer L restores only that layer's cached clean activation into the corrupt run (CopyNoChecks) and recomputes layers L+1..last, reading off the recovery r_L = (logit_c(patch_L) - logit_c(corrupt))/(logit_c(clean) - logit_c(corrupt)) (c = clean argmax class, the default TargetIdx). It prints a per-layer r_L ASCII bar chart across depth (the causal-trace curve), a per-layer un-normalised delta-logit column (so a near-zero normalisation denominator is visible), the argmax-flip layer, the peak-recovery layer, and an early/late/distributed localisation verdict. The example builds a branched net (a main ReLU branch + a raw-input skip fused by a Concat) on a synthetic XOR-of-signs task so the trace is graded by construction: patching a single main-branch layer recovers little (the corrupt input still flows on the skip) and recovery jumps to ~1 at the fusion layer — the ground-truth localisation. Built-in checks: r_0 == 1 exactly (patching the input reconstructs the full clean run), r_last == 1 exactly (the last layer's Output IS the logits), and CorruptInput == CleanInput collapses the denominator so the report WARNS rather than dividing by zero. Distinct from SaliencyReport (input-space gradient attribution) and LayerSensitivityReport (random weight jitter — never swaps real activations between two inputs). Forward-only — the only mutation is the transient activation overwrite, reverted by a final clean recompute; weights are never touched.
• Activation steering (concept vectors) — the interventional flip-side of the read-only probes (ActAdd / representation engineering, Turner et al. 2023): rather than asking "what is decodable from this layer?", it INJECTS a concept direction into a hidden activation mid-forward and shows it causally controls the output. It trains a small softmax classifier on a synthetic two-cluster task, computes a steering vector v = mean(act_k | class 1) - mean(act_k | class 0) (a diff-of-class-means direction at hidden layer k, no extra training), then for a sweep of alpha in {-3..+3} runs forward up to k, does Output_k.MulAdd(alpha, v) and recomputes layers k+1..last (reusing the ActivationPatchingReport recompute machinery), charting target-class probability vs alpha as an ASCII curve. Built-in checks: alpha = 0 reproduces the unsteered forward pass bit-for-bit, the target-class probability moves monotonically with alpha, and steering with v shifts the output far more per unit norm than an equal-norm RANDOM direction (the concept direction is special). Distinct from ActivationPatchingReport (swaps WHOLE cached activations between two inputs), SaliencyReport (input-space gradient), GradientAscent (ascends on the input image) and LinearProbeReport (only READS what a layer encodes — none ADD a direction to a hidden activation and measure the controllable output shift). Pure CPU, forward-only, weights never stepped after the classifier is trained. The sibling activation-steering depth sweep repeats the alpha sweep at EVERY hidden layer k and reports which depth gives the cleanest monotone P(target)-vs-alpha control and the smallest alpha-to-flip — the "where does a concept vector bite hardest?" question (on the toy task k=1/3/4 are perfectly monotone while k=2's diff-of-means direction is genuinely non-monotone).
• Concept bottleneck model (test-time intervention) — an interpretable-by-design Concept Bottleneck Model (Koh et al. 2020) built from existing layers only: a trunk feeds a TNNetFullConnectSigmoid(K) layer of K human-meaningful concepts which is the sole input to a TNNetFullConnectLinear label head, trained jointly with deep supervision (label cross-entropy + a concept-prediction loss on the bottleneck, the same packed-target / SetBatchUpdate(True) idiom as EarlyExitNetwork). The headline payoff is test-time concept intervention: overwrite the predicted concept vector at the bottleneck and recompute only the downstream label head (the CopyNoChecks-then-recompute machinery of ActivationSteering/ActivationPatching) — injecting the true concepts lifts label accuracy to the head's ceiling, and flipping a single concept bit deterministically flips the predicted class in the direction that concept controls. Built-in invariants: a no-op overwrite with the model's own predicted concepts reproduces the logits bit-for-bit, and setting the concept-loss weight lambda = 0 makes the bottleneck drift (mean concept accuracy collapses toward chance — the "leaky" CBM failure mode). Distinct from LinearProbeReport (post-hoc frozen probe, READS only), ActivationSteering (edits anonymous activations, no concept supervision), and DomainAdversarial (gradient reversal REMOVES info). Pure CPU, deterministic, ~10 s.
• Sparse autoencoder — a self-contained pure-CPU reproduction of the headline result of Anthropic's Towards Monosemanticity (Bricken et al. 2023): a sparse autoencoder with an OVERCOMPLETE dictionary, trained on dense POLYSEMANTIC activations, recovers the original sparse ground-truth features as MONOSEMANTIC dictionary atoms — and there is a sweet-spot sparsity weight (too little gives dense polysemantic atoms, too much kills atoms). Complements the Superposition example.
• VQ codebook-usage probe — the demonstration companion to the new TNNetVectorQuantizer codebook-usage probe (ResetCodebookUsage / ActiveCodeCount / CodebookUsageCount), which exposes the headline VQ-VAE failure mode: CODEBOOK COLLAPSE, where only a handful of codes ever win the nearest-neighbour argmin and the rest of the codebook is dead weight. A tiny VQ-VAE-style bottleneck (encoder → quantizer → decoder, MSE reconstruction) trains on synthetic well-separated Gaussian blobs with MORE codes than clusters (K=12 vs 5 clusters) so unused codes are visible; each epoch it probes the codebook over a fixed batch and prints active-code count + a per-code usage histogram, ending with a graded PASS/FAIL verdict (PASS = stays healthy, using ≥ #clusters distinct codes; Halt(1) on collapse). The probe is pure runtime bookkeeping — it does NOT touch the quantization math, any gradient, or serialization. Pure CPU, <1 s.
• VQ codebook collapse — stress test + dead-code re-init — the failure-mode counterpart to VQCodebookUsage: it deliberately DRIVES codebook collapse on TNNetVectorQuantizer and then shows a published mitigation lifting it back out. A 64-code codebook on 16 Gaussian blobs runs two arms sharing data + seed; the COLLAPSE arm contracts the encoder weights each epoch (×0.70) to shrink the latent cloud, and the active-code count (read via ResetCodebookUsage → probe pass → ActiveCodeCount) falls 23 → 5 of 64. The MITIGATED arm adds DEAD-CODE RE-INITIALIZATION every 3 epochs — codes with zero CodebookUsageCount are re-seeded to live encoder latents through the public Neurons[code].Weights accessor — recovering to ~16 active (the true mode count), ending with a graded VERDICT: PASS. Example-only (core untouched). Gotcha documented in its README: build the encoder as Create(1,1,cEmb) so latents live on the Depth axis (Create(cEmb) gives Depth=1 scalar codes). Pure CPU, ~1.5 s.
• VQ-VAE discrete-image generation — the GENERATIVE payoff of TNNetVectorQuantizer, the two diagnostic probes (VQCodebookUsage / VQCodebookCollapse) only inspect codebook health on synthetic blobs; here the same vector-quantization bottleneck is wired into a real convolutional encoder/decoder that reconstructs 28×28 MNIST digits and then an autoregressive prior synthesises brand-new ones (van den Oord et al. 2017, Neural Discrete Representation Learning). Stage 1 (representation): encoder → TNNetVectorQuantizer → decoder trained on reconstruction MSE (gradient flows through the non-differentiable argmin via the layer's built-in straight-through estimator + commitment/codebook terms), writing a reconstruction grid PNG and reporting codebook usage. Stage 2 (prior + generation): FREEZE the encoder, harvest each digit's 7×7 grid of discrete code indices via the new public LVQ.ChosenCodeIndex(X, Y) accessor (the argmin token cached on the last Compute() — this is what turns the continuous latent into a 49-token discrete sequence), fit a tiny causal transformer LM (reusing AddTransformerEncoderBlock with CausalMask=true, exactly like TinyGPT) over those sequences, then AUTOREGRESSIVELY sample new 49-token grids, look each token up in the codebook, and DECODE through the frozen decoder to write a generated-samples PNG. SMOKE mode finishes in ~2 min on one CPU (reconstruction MSE falls, codebook stays healthy, no NaN); --full for sharper digits. Mirrors the classic two-stage VQ-VAE recipe (discrete autoencoder first, autoregressive prior over its codes second). Needs the standard MNIST idx-ubyte files in the working directory. Pure CPU.
• FSQ-VAE — codebook-free Finite Scalar Quantization on MNIST — the collapse-free counterpart to the learned-codebook VQVAE / VQCodebookCollapse, built on the new layer TNNetFiniteScalarQuant (Mentzer et al. 2023, Finite Scalar Quantization: VQ-VAE Made Simple; a port of the lucidrains vector-quantize-pytorch FSQ). FSQ has NO learned codebook, NO EMA and NO commitment loss: each latent channel is squashed by a bounded tanh and ROUNDED to one of L_i integer levels (f(z)=tanh(z+shift)·half_l−offset, zhat=round(f(z)), gradient through the round via the layer's built-in straight-through estimator), so the implicit codebook (here 5⁶ = 15625 codes) cannot collapse by construction. A tiny MLP autoencoder 784→…→6 FSQ channels→…→784 is trained on a small MNIST subset with hand-rolled mini-batch SGD; a per-channel TNNetChannelStdNormalization + fixed gain keeps the latents in the informative tanh region (the documented FSQ training pitfall — latents that collapse to ~0 or saturate both stick on one level). After training it probes the bottleneck over a fresh batch and reports per-channel level utilization (climbs to ~100% — every level of every channel exercised, the collapse-free guarantee in action) and the number of DISTINCT full codes seen via the public LFSQ.CodeIndex(X,Y) accessor (the discrete token a downstream transformer prior would consume), ending with a graded VERDICT: PASS. SMOKE finishes in ~33 s on one CPU; --full for a longer run. Needs the standard MNIST idx-ubyte files in the working directory (e.g. run from ../VQVAE). Pure CPU.
• LFQ-VAE — Lookup-Free (binary) Quantization demo — the BINARY sibling of FSQVAE, built on the new layer TNNetLookupFreeQuant (Yu et al. 2023, MagViT-v2, Language Model Beats Diffusion: Tokenizer is Key to Visual Generation; a port of the lucidrains vector-quantize-pytorch LFQ). LFQ takes FSQ's per-channel quantization to the limit L_i = 2: each latent channel is just sign(z) in {-1,+1}, so the implicit codebook is the product set {-1,+1}^D of size 2^D with NO codebook lookup at all (no learned vectors, no argmin). The discrete token at a position is the bit-packed sign pattern read via the public LFQ.CodeIndex(X,Y); gradients flow through the non-differentiable sign via the straight-through estimator clipped to the |z| <= 1 band (the lucidrains LFQ math, reproduced exactly). The headline addition is LFQ's entropy auxiliary loss in its tractable factorized binary form — per channel softmax(-t·[(z+1)², (z-1)²]), then EntropyAuxLoss = PerSampleEntropy − DiversityWeight·CodebookEntropy — surfaced as PUBLIC accessors (PerSampleEntropy / CodebookEntropy / EntropyAuxLoss) on the layer; the demo READS them to show they track code usage (the layer injects NO entropy gradient, so a real tokenizer ADDS this scalar to its reconstruction loss to keep the codebook diverse — exactly like TNNetLoadBalanceLoss is added to a MoE loss). A tiny self-contained MLP autoencoder (16→…→6 LFQ channels→…→16) trains on synthetic prototype data with hand-rolled mini-batch SGD and reports, before/after training, the distinct binary-code count + the entropy terms, ending with a graded VERDICT: PASS (the binary bottleneck cleanly separates the data prototypes into distinct codes). NO external data; finishes in ~2 s on one CPU. Pure CPU.
• VAE continuous-latent generation — the continuous / Gaussian counterpart to the discrete VQVAE: a small Variational Autoencoder on MNIST (Kingma & Welling 2014, Auto-Encoding Variational Bayes) built around the new layer TNNetGaussianReparameterize. The encoder emits a packed (mu | log_var) tensor on the depth axis; the new layer splits it and draws the latent with the reparameterization trick z = mu + exp(0.5·log_var)·eps, eps~N(0,1) — sampled once per forward and frozen for the matching backward (the fixed-noise discipline of TNNetGumbelSoftmax/TNNetDropout), so the draw stays differentiable w.r.t. mu and log_var. A conv decoder reconstructs the digit and training minimises reconstruction-MSE + beta·KL(q(z|x)‖N(0,1)). KL composition: the reparameterize layer contributes ONLY the reconstruction gradient (dz/dmu=1, dz/dlog_var=0.5·sigma·eps); the KL term is a SEPARATE penalty (dKL/dmu=mu, dKL/dlog_var=0.5·(exp(log_var)−1), also packaged as the TNNetVAEKLDivergence loss head), and the example wires the two explicitly so they SUM at the (mu|log_var) fork — a reconstruction backward through the reparameterize layer followed by a KL-only backward through the encoder. Generation: sample z~N(0,1) from the prior the KL term trained toward, inject it at the latent layer and run the decoder tail, writing an 8×8 grid of fresh digits. SMOKE mode finishes in ~1–2 min on one CPU (reconstruction MSE ~0.21→~0.02, KL stays finite, no NaN); --full for sharper digits. Needs the standard MNIST idx-ubyte files in the working directory. Pure CPU.
• MaskGIT non-autoregressive image generation — the masked-token PARALLEL generator over the same discrete VQ codebook as VQVAE (Chang et al. 2022, MaskGIT: Masked Generative Image Transformer). Structurally distinct from every landed generator — autoregressive (TinyGPT, the VQVAE prior), GAN (VisualGAN, StyleGAN2Generate) and diffusion (DiffusionMNIST, ConsistencyDistill). Stage 1 reuses the VQVAE discrete autoencoder verbatim (conv encoder → TNNetVectorQuantizer → conv decoder over MNIST; reconstruction-MSE training, dead-code revival, codebook-usage reporting, reconstruction PNG) so each 28×28 digit becomes a 7×7 grid of 49 discrete code indices via LVQ.ChosenCodeIndex(X,Y). Stage 2 is the new part: a BIDIRECTIONAL transformer (AddTransformerEncoderBlock with CausalMask=FALSE — the exact opposite of the VQVAE/TinyGPT causal prior, plus a per-position TNNetPointwiseSoftMax head) is trained with a masked-token objective — a cosine-family fraction of the 49 tokens is replaced by a dedicated [MASK] symbol (id = codebook_size, an extra vocab slot) and the model predicts the ORIGINAL ids at the masked positions only (cross-entropy on masked positions; the report shows masked-token prediction accuracy rising epoch over epoch). Generation is PARALLEL ITERATIVE DECODING: start the whole grid all-[MASK], then for ~10 steps predict every position at once, sample tokens with a per-position confidence (the sampled token's probability), keep the most-confident fraction per a cosine mask schedule gamma(t/T)=cos(pi/2·t/T) and re-mask the rest, until the grid is full — ~10 forward passes for the whole image vs the prior's 49 autoregressive steps — then look each final token up in the codebook and DECODE through the frozen VQ decoder. The only new code is the [MASK] masking + the confidence-based cosine unmasking SCHEDULER (the generation loop); the transformer, VQ encode/decode and PNG I/O are all reused — NO new TNNet layer. Self-reports masked-token accuracy, generated-grid codebook usage, and a NaN/Inf check. SMOKE mode finishes in ~3.5 min on one CPU; it is intentionally undertrained (like the DiffusionMNIST smoke its digits are rough — the headline is that the parallel-decode pipeline fills a coherent grid in ~10 steps), --full for sharper output. Reuses the MNIST idx files (working dir, then ../VQVAE/../DiffusionMNIST, no copy); falls back to a SYNTHETIC bar dataset if absent so it still runs offline in CI. Pure CPU.
• VQGAN / discrete image-tokenizer import — the demo for BuildVqModelFromSafeTensors (neuralpretrained.pas), the IMPORTER for a pretrained diffusers VQModel (VQGAN), the discrete tokenizer used by autoregressive / masked image generators (MaskGIT, Parti, LlamaGen) to turn an image into a grid of codebook token IDs and back. Where VQVAE trains a VQ-VAE from scratch, this imports one: it reuses the landed VAE encoder/decoder ResNet+attention blocks (the AutoencoderKL BuildVaeEncoder/BuildVaeDecoder siblings) but wires the discrete quantizer between them — encoder → quant_conv → nearest-neighbour codebook lookup (image → token IDs via argmin squared-L2 to quantize.embedding.weight) and the inverse (token IDs → gathered embeddings → post_quant_conv → decoder → image), with NO 0.18215 latent scaling (that is AutoencoderKL-only). The new code is plain-Pascal argmin/gather on the holder class TNNetVqModel.EncodeImageToTokens / DecodeTokensToImage (no new TNNet* layer), so an imported autoregressive image LM (stock Llama/GPT path) could model these token grids end to end. The demo round-trips a tiny image through the codebook (image → 8×8 token grid → image) and reports the reconstruction error; with no arguments it loads the committed pico fixture (tests/fixtures/tiny_vqmodel.*), the same one the parity tests use — encode token IDs match a numpy float64 oracle exactly (integers, TestVqModelEncodeParity) and decoded pixels match < 1e-4 (TestVqModelDecodeParity / TestVqModelRoundTrip). Point it at a real diffusers VQModel checkpoint + config.json for the full thing. Pure CPU, <1 s on the fixture.
• StyleGAN2 generator synthesis import — the demo for BuildStyleGAN2Generator (neuralpretrained.pas), a style-based generative import (Karras et al. 2020, Analyzing and Improving the Image Quality of StyleGAN). The new primitive is the leaf layer TNNetModulatedConv2D: a WEIGHT-SPACE modulated/demodulated convolution whose owned base kernel is scaled per-forward by a per-input-channel style vector read from a SECOND input — w'_{ijk}=s_i·w_{ijk} (modulate) then w''_{ijk}=w'_{ijk}/sqrt(Σ w'^2+ε) (demodulate) — with the backward routing gradient to the kernel (through the demod normalization), the feature input AND the style branch (numerical-gradient-checked input+weight+style in TestNeuralNumerical). The importer builds the full INFERENCE-ONLY synthesis path: an 8-layer-style mapping MLP (z→w, TNNetFullConnectLinear+TNNetLeakyReLU), then a tower of resolution blocks each = nearest-x2 upsample (TNNetDeMaxPool) → [modulated conv + per-pixel noise injection (a learned TNNetReZero strength) + LeakyReLU] → a toRGB modulated 1×1 conv (no demod) summed into an upsampled RGB skip tower; each modulated conv's style comes from a small affine "A" (TNNetFullConnectLinear) off w. NO discriminator / path-length reg / training (v1). The official weights are not obtainable offline, so — exactly like the RRDBNet/VAE-decoder pico fixtures — it falls back to the committed config-faithful random pico generator (tests/fixtures/tiny_stylegan2.*, built by tools/make_pico_stylegan2_fixture.py), parity-checked < 1e-4 against a hand-written float64 numpy oracle of the exact modulated-conv+demod+noise+toRGB math (TestStyleGAN2GeneratorParity). The demo synthesizes one image from a fixed latent and writes a P6 color PPM + an ASCII intensity preview; point it at a real .safetensors (+ config.json) for your own checkpoint. Pure CPU, <1 s on the fixture.
• Real-ESRGAN / ESRGAN RRDBNet super-resolution upscale — the END-TO-END demo for BuildRRDBNetFromSafeTensors (neuralpretrained.pas), the repo's pure-convolutional, NO-diffusion super-resolution importer (Wang et al. 2021, Real-ESRGAN; xinntao/Real-ESRGAN). RRDBNet is a Residual-in-Residual Dense Block network: conv_first → num_block RRDBs (each = 3 chained ResidualDenseBlocks of 5 dense channel-concat 3×3 convs with TNNetDeepConcat skips + LeakyReLU(0.2), scaled residual x + 0.2·block(x)) → conv_body global residual → an upsample tail of $(\text{nearest}-2 \times + 3 \times 3 \text{conv} + \text{LeakyReLU})$ stages (TNNetDeMaxPool(2) for the F.interpolate(mode='nearest') 2× spacing) → conv_hr + conv_last. Both upscaling factors are wired: scale=4 = two upsample stages (conv_up1,conv_up2, RealESRGAN_x4plus), scale=2 = one stage (conv_up1 only, RealESRGAN_x2plus). Real-ESRGAN ships its weights as a .pth whose state_dict is nested under a top-level params_ema key — loaded transparently through the same BuildRRDBNetFromSafeTensors call (CreatePretrainedTensorReader dispatches .pth/.pt/.bin to TNNetTorchBinReader, which unwraps the params_ema/params/state_dict/model wrapper dict automatically). The example builds the committed pico RRDBNet (tests/fixtures/tiny_rrdbnet{,_x2}.*, built by tools/rrdbnet_tiny_fixture.py), synthesises a tiny 6×6 RGB gradient, writes it as a PNG (neuraldatasets.SaveImageFromVolumeIntoFile), reads it back (LoadImageFromFileIntoVolume), normalises 0..255→[−1,1], runs the forward upscale and writes the output PNG for both scales (6×6→24×24 at x4, 6×6→12×12 at x2). Parity is asserted < 1e-4 against a hand-written float64 numpy oracle for the safetensors x4 path, the params_ema .pth x4 path and the x2 path (TestRRDBNetParity{,Pth,Scale2}). Point it at a real RealESRGAN_x4plus.pth/.safetensors (+ config.json) to upscale your own images. Pure CPU, <1 s on the fixture (random pico weights → import + forward + PNG-I/O smoke, not a photoreal upscaler).
• NAFNet image restoration import — the demo for BuildNAFNetFromSafeTensors (neuralpretrained.pas), a non-diffusion image-to-image RESTORATION import that is NOT super-resolution (the RRDBNet/ESRGAN path is x4 upscaling only); NAFNet restores (denoise / deblur / dejpeg) at the same resolution (Chen et al. 2022, Simple Baselines for Image Restoration). The new primitive is the parameter-free leaf layer TNNetSimpleGate: split the channel axis in half and multiply the halves (out[c]=in[c]·in[C+c]) — a GLU with no activation, backward = product rule (numerical-gradient-checked in TestNeuralNumerical). Simplified Channel Attention (SCA) reuses landed pieces — global avg pool (TNNetAvgChannel) → 1×1 conv → channelwise multiply (TNNetChannelMulByLayer), no activation. The importer assembles a symmetric U-Net of NAFBlocks: each NAFBlock = per-pixel LayerNorm2d (TNNetTokenLayerNorm) → 1×1 conv → 3×3 depthwise conv → SimpleGate → SCA → 1×1 conv, residual-added with a learnable per-channel beta, then a second LayerNorm → 1×1 conv → SimpleGate → 1×1 conv, residual-added with gamma; the U-Net downsamples with a stride-2 conv (2× channels) and upsamples with a 1×1 conv + PixelShuffle (TNNetDepthToSpace), adding the encoder skip, and ends with a 3×3 conv + global input skip. The official checkpoints are large / not obtainable offline, so — exactly like the RRDBNet/VAE-decoder pico fixtures — it falls back to the committed config-faithful random pico NAFNet (tests/fixtures/tiny_nafnet.*, built by tools/nafnet_tiny_fixture.py), parity-checked < 1e-4 against a hand-written float64 numpy oracle of the exact NAFBlock math (TestNAFNetParity). The demo adds synthetic Gaussian noise to a deterministic test image, runs the restoration forward pass, writes before/after P6 PPMs + an ASCII preview and reports the round-trip RMSE; point it at a real .safetensors (+ config.json) for your own trained checkpoint. Pure CPU, <1 s on the fixture (random pico weights → wiring/throughput smoke, not a trained denoiser).
• SwinIR transformer super-resolution import — the demo for BuildSwinIRFromSafeTensors (neuralpretrained.pas), a transformer image-restoration import (classical super-resolution), architecturally distinct from the CNN-only RRDBNet/ESRGAN SR path and the SimpleGate-CNN NAFNet denoiser (Liang et al. 2021, SwinIR: Image Restoration Using Swin Transformer). The window / shifted-window attention is pure REUSE of the landed Swin building blocks — per (head, window) TNNetWindowAttention (relative-position bias + cyclic-shift attention mask) with TNNetGatherTokens window partition / reverse. The new pieces are only: a shallow conv stem (3×3 → embed_dim), the Residual Swin Transformer Blocks (RSTB = depth Swin layers — each a token-sequence W-MSA/SW-MSA + GELU MLP over the H×W grid — then a 3×3 conv, with a residual over the whole block), a final token LayerNorm + conv_after_body with a deep-feature residual onto the stem, and a pixel-shuffle upsample tail (3×3 conv → TNNetDepthToSpace(upscale) → conv_last, with a TNNetLeakyReLU(0.2)). Keys follow the official SwinIR repo state_dict (single packed attn.qkv sliced into q/k/v at load, layers.L.residual_group.blocks.M.*). The official checkpoints are large / not obtainable offline, so — exactly like the RRDBNet/NAFNet pico fixtures — it falls back to the committed config-faithful random pico SwinIR (tests/fixtures/tiny_swinir.*, built by tools/swinir_tiny_fixture.py), parity-checked < 1e-4 against a hand-written float64 numpy oracle of the exact RSTB + window-attention + pixel-shuffle math (TestSwinIRParity; the 2-layer RSTB exercises both W-MSA and SW-MSA). The demo runs the 2× SR forward pass on a deterministic synthetic image, writes input/upscaled P6 PPMs + an ASCII preview and reports the upscaled shape; point it at a real .safetensors (+ config.json) for your own trained checkpoint. Pure CPU, <1 s on the fixture (random pico weights → wiring/throughput smoke, not a trained SR model).
• RIFE video frame interpolation import — the demo for BuildRIFEFromSafeTensors (neuralpretrained.pas), a VIDEO-generative import: it synthesises an unseen intermediate frame between two input frames (t=0.5). This fills the gap left by the RAFT optical-flow import (which estimates flow but does NOT synthesise frames) and by the from-scratch toy FrameInterpolation (a trained-here TNNetFlowWarp model, not an importer). RIFE (Huang et al. 2022, Real-Time Intermediate Flow Estimation for Video Frame Interpolation; hzwer/Practical-RIFE) estimates a bidirectional intermediate flow with a coarse-to-fine stack of IFBlocks, then backward-warps both frames and blends them with a learned soft fusion mask. The new primitive is the differentiable backward-warp leaf/two-source layer TNNetBackwardWarp: out(x,y,c) = image(x+dx(x,y), y+dy(x,y), c), bilinear, flow in PIXEL units, border-clamp padding — the integer-pixel equivalent of RIFE's F.grid_sample(mode='bilinear', padding_mode='border', align_corners=True) (a sibling of TNNetFlowWarp; full forward + backward, both dL/d(image) and dL/d(flow) numerically gradient-checked in TestNeuralNumerical). Everything else reuses landed layers: the IFBlock is $\text{conv0}(3 \times 3) → \text{PReLU} → \text{conv1}(3 \times 3) → \text{PReLU} → \text{lastconv}(3 \times 3 → 5)$ (4 flow channels + 1 mask), each block adds a flow RESIDUAL (the last block's mask wins); the accumulated 4-ch flow splits into (dx,dy) for each frame (TNNetSplitChannels), each frame is warped, and the soft blend merged = warped0·m + warped1·(1−m) with m = sigmoid(mask) broadcast over RGB (TNNetSigmoid + TNNetDeepConcat.Replicate + TNNetCellMulByCell + TNNetSum). The input is the depth-concat [frame0 | frame1] (2·in_channel channels); the output is the in_channel interpolated middle frame. The official Practical-RIFE checkpoints are not obtainable offline, so — exactly like the NAFNet/SwinIR pico fixtures — it falls back to the committed config-faithful random pico RIFE (tests/fixtures/tiny_rife.*, built by tools/rife_tiny_fixture.py), parity-checked < 1e-4 against a hand-written float64 numpy oracle of the exact IFBlock + backward-warp math (TestRIFEParity). The demo makes two synthetic frames (a bright blob translating across the grid), stacks them, runs the interpolation forward pass, writes rife_frame0/middle/frame1.ppm + an ASCII frame0 | middle | frame1 preview; point it at a real .safetensors (+ config.json) for your own trained checkpoint. Scope v1: one intermediate frame at t=0.5, inference-only, a small IFNet of num_blocks full-resolution IFBlocks; real multi-scale checkpoint parity, arbitrary-t, and the recursive 2×→4× schedule are tracked follow-ups. Pure CPU, <1 s on the fixture (random pico weights → wiring/throughput smoke, not a trained interpolator).
• CLIPSeg text-prompted zero-shot segmentation import — the demo for BuildCLIPSegFromSafeTensors (neuralpretrained.pas), a "free-text prompt → dense single-channel mask" import: given an image and an arbitrary text prompt it emits one H×W logit map for "whatever the prompt names", with NO fixed label set (unlike the box/mask importers DETR/SAM/Mask2Former, which are class- or geometry-driven). CLIPSeg (Lüddecke & Ecker 2022, Image Segmentation Using Text and Image Prompts; CIDAS/clipseg-rd64-refined) is heavy REUSE: the frozen CLIP ViT image tower (BuildClipVisionTower-style pre-LN blocks) and the frozen CLIP text tower (the BuildClipFromSafeTensors text half) are the same landed CLIP encoder blocks. The new piece is the lightweight FiLM-conditioned transformer decoder: the vision tower exposes the hidden states after each tapped encoder block (config.extract_layers, default [3,6,9]); each (tokens, vision_hidden) tap is projected to reduce_dim (reduces[i]), and at the conditional_layer is FiLM-modulated by the CLIP text embedding of the prompt — film_mul(cond)·x + film_add(cond) via TNNetFiLM (γ|β broadcast over the token axis) — then refined by a post-norm CLIP-style block (relu MLP, LayerNorm after each residual). The CLS token is dropped, the patch tokens reshape to a (grid, grid, reduce_dim) map, and a single non-overlapping ConvTranspose2d(reduce_dim, 1, patch, stride=patch) upsamples to the image-resolution logit mask — realized as TNNetPointwiseConvLinear(patch²) + TNNetDepthToSpace(patch) (the channel patch_w·patch + patch_h carries the spatial offset). The importer returns three nets (vision, text, decoder); the conditional embedding is read off the text tower's eot row (ClipTextEosPosition, not L2-normalized) and the tapped vision hidden states are fed to the decoder's per-tap inputs (the T5EncoderStatesInput "fill an Input before Compute" idiom generalized to several inputs), driven end-to-end by RunCLIPSeg. The real checkpoint is large / not obtainable offline, so — like the NAFNet/SwinIR pico fixtures — it falls back to the committed config-faithful random pico CLIPSeg (tests/fixtures/tiny_clipseg.*, built by tools/clipseg_tiny_fixture.py from the real HF CLIPSegForImageSegmentation float64 oracle), parity-checked < 1e-4 (TestCLIPSegParity: both the conditional embedding and the full mask logits). The demo runs the image+prompt pipeline on a deterministic synthetic image (prompt given as token ids — the pico fixture has no tokenizer), thresholds the logits at 0 and writes a binary-mask PPM + an ASCII preview; point it at a real .safetensors (+ config.json) and real tokenizer ids for your own checkpoint. Scope v1: a single text prompt → one mask, inference-only (use_complex_transposed_convolution=false); image-prompt conditioning and the 3-stage complex upsample are deferred. Pure CPU, <1 s on the fixture (random pico weights → wiring/throughput smoke, not a trained segmenter).
• Segment Anything (SAM) promptable click→mask import — the demo for BuildSAMFromSafeTensors / BuildSAMVisionTower and the new RunSAMMaskDecoder (neuralpretrained.pas), a promptable-segmentation importer (model_type sam: facebook/sam-vit-base and siblings). SAM (Kirillov et al. 2023, Segment Anything) is built around a heavy ViT-det image encoder that produces a dense (Grid, Grid, OutCh) image embedding once per image, after which a lightweight prompt-conditioned mask decoder turns a CLICK into a binary mask cheaply — and both stages now import with float64 parity. The IMAGE ENCODER: a biased patch conv (kernel = stride = patch) to a (Grid, Grid, hidden) grid; a learned 2-D absolute pos_embed added directly (no CLS token, no flatten — via TNNetCellBias); num_hidden_layers pre-LN transformer blocks whose self-attention is the leaf TNNetSAMVisionAttention — windowed attention (SAM zero-padding window partition) for most blocks, global attention for the global_attn_indexes blocks, plus the MViTv2 DECOMPOSED relative-position bias Q·rel_pos_h + Q·rel_pos_w (query-dependent, distinct from Swin's query-independent bias table); then the neck (conv1 1×1 no-bias → LayerNorm2d → conv2 3×3 pad1 no-bias → LayerNorm2d) to output_channels; the MLP uses the exact-erf GELU (TNNetGELUErf). The MASK DECODER (v1: single point → single mask) composes the prompt encoder (point positional encoding cat(sin,cos) of 2π·((2c−1)·P) + learned point/not-a-point embeddings + the dense no-mask embedding), the two-way transformer (token↔image cross-attention over the learned IoU + mask-output tokens, with attention_downsample_rate on the cross-attentions and the layer-0 skip_first_layer_pe no-residual quirk), and the output upscaling (2× ConvTranspose2d ×2 + channels-first LayerNorm) followed by the per-mask hypernetwork-MLP dot with the upscaled embedding → low-res mask logits. The real checkpoint is large / not obtainable offline, so it falls back to the committed config-faithful random pico SAM (tests/fixtures/tiny_sam.*, built by tools/make_pico_sam_fixture.py from the real HF SamModel float64 oracle): the encoder embedding parity-checks < 1e-4 (TestSAMEncoderParity) and the single-click mask logits parity-check < 1e-4 vs HF's own forward (TestSAMMaskDecoderParity, oracle in tiny_sam_mask.json). The demo encodes a deterministic synthetic image once, then runs RunSAMMaskDecoder on one positive click (default the image centre; override via argv[2] argv[3] pixel coords) and writes a real click→mask binary PPM by thresholding the mask logits at 0 (sigmoid 0.5). Point it at a real .safetensors (+ config.json) for your own checkpoint. Multi-point/box prompts, multi-mask + IoU output, a trainable TNNet-graph decoder, and a real-checkpoint processor are documented follow-ups in tasklist.md. Pure CPU, <1 s on the fixture (random pico weights → wiring/throughput smoke, not a trained segmenter).
• End-to-end latent text-to-image (PixArt → VAE) — an end-to-end latent text-to-image pipeline: it CHAINS the already-landed generative importers into one offline CPU sampling loop, the SD3/Sora/PixArt-alpha recipe end to end. No new leaf layer — pure plumbing over landed pieces. The flow is: caller-supplied T5 text states → PixArt-alpha transformer denoiser (BuildPixArtFromSafeTensors + PixArtConditioning/PixArtDenoise, neuralpretrained.pas) driven by a multi-step DDIM / DPM-Solver++(2M) reverse loop (TNNetDiffusionScheduler, neuraldiffusion.pas) with classifier-free guidance — the cond branch uses the prompt T5 states, the uncond branch uses the null/empty-caption = ZERO T5 states (the PixArt CFG convention; mixed via TNNetDiffusionScheduler.ApplyCFG) — → the sampled (sample_size, sample_size, in_channels) latent → VAE decode (BuildVaeDecoderFromSafeTensors; the /0.18215 latent scaling lives inside the decoder's first TNNetMulByConstant layer) → RGB image → a P6 PPM. Steps 1 & 2 only: Step 3 (a real T5 encoder over a tokenized prompt + a real PixArt/VAE checkpoint) is a tracked follow-up in tasklist.md; this demo supplies deterministic synthetic T5 states and uses the committed config-faithful pico fixtures (random weights), so it is a wiring / throughput smoke — it proves the chain runs offline and produces a finite image, not photorealism. The pico pair is sized so the latent flows straight through with no reshaping: T5 states (5,1,12) → PixArt latent (6,6,4) → VAE image (12,12,3). No network access / self-contained: falls back to the committed pico PixArt (tests/fixtures/tiny_pixart.*, tools/make_pico_pixart_fixture.py, parity-checked < 1e-4 in TestPixArtParity) and a matched pico VAE decoder (tests/fixtures/tiny_vae_decoder_ltt.*, tools/vae_decoder_ltt_fixture.py — the vae_decoder_tiny_fixture.py oracle re-sized to latent_size 6 / latent_channels 4). Regression-tested by TestLatentTextToImageSmoke (TestNeuralPretrained.pas): it runs the full CFG DDIM loop + VAE decode and asserts no NaN/Inf in both the sampled latent and the decoded image. Flags: --steps N (default 4), --cfg W (default 4.0), --dpm (DPM-Solver++(2M) instead of DDIM), --unipc (the UniPC order-2 bh2 predictor-corrector few-step sampler, TNNetSamplerMethod.smUniPC in neuraldiffusion.pas — a unified predictor-corrector that REUSES the previous step's model output as a free corrector before the predictor, noticeably better quality than the DPM++(2M) predictor alone at very low step counts 5-10; Zhao et al. 2023, arXiv:2302.04867), --lcm (the Latent Consistency Model few-step sampler, TNNetLCMScheduler.LCMSample, neuraldiffusion.pas), --smoke (assert finiteness, print SMOKE OK/SMOKE FAIL, set the exit code — the build step exercises this). The --lcm path swaps the iterative CFG loop for a consistency-model few-step loop: it evaluates a learned consistency function f(x_t,t) = c_skip(t)·x_t + c_out(t)·x0_hat that maps any noised latent straight toward x0, so it needs only ~4 steps and a SINGLE conditional model pass per step — guidance is baked in (no cond/uncond double pass). The matching training recipe is LCM-distillation: a consistency loss that distills a many-step teacher (DDIM/DPM++) into this few-step student, training f so that adjacent points on the same probability-flow ODE trajectory map to the same x0 (self-consistency to the teacher). The committed pico fixtures are not LCM-distilled, so --lcm here is a wiring smoke (it proves the few-step loop runs and yields a finite image, not 4-step teacher parity); the LCM path is exercised by the same TestLatentTextToImageSmoke. Point it at your own pixart.safetensors vae.safetensors (with sibling config.json files) for a real checkpoint. Pure CPU, <1 s on the fixtures.
• End-to-end base-UNet + ControlNet single-step denoise — the offline CPU inner loop of diffusers' StableDiffusionControlNetPipeline, wired end to end over the already-landed importers. No new leaf layer — pure plumbing over the landed ControlNet importer + the new SDUNetDenoiseWithControl driver. The flow is: noisy latent + text states + a (canny-edge-style) control image → ControlNet (BuildControlNetFromSafeTensors + ControlNetResiduals, neuralpretrained.pas) produces the down_block_res_samples + mid_block_res_sample → a base SD UNet built WITH control injection (BuildSDUNet(..., pWithControl=true): each down-path skip and the mid output get an extra zero-default TNNetInput + TNNetSum, so a plain SDUNetDenoise is bit-identical to the base UNet) → SDUNetDenoiseWithControl ADDS those residuals into the decoder skip connections exactly as diffusers does — down_block_res_samples = [d + c for d, c in zip(down, controlnet_down)] and mid_block_res_sample += controlnet_mid → the predicted noise a sampler would step on. The committed pico fixtures have random weights, so this is a wiring / throughput smoke — it proves the base-UNet + ControlNet chain runs offline and produces a finite noise prediction, not photorealism. No network access / self-contained: falls back to the committed config-compatible pico SD UNet (tests/fixtures/tiny_sd_unet.*) and a matched pico ControlNet (tests/fixtures/tiny_controlnet.*) — both share block_out_channels [16,32], latent grid 8, text seq 5, cross dim 12, so the ControlNet residuals flow straight into the base UNet skips with no reshaping (latent (8,8,4) + text (5,1,12) + control (16,16,3) → 4 down residuals + 1 mid residual → predicted noise (8,8,4)). The combined forward (base UNet noise WITH the ControlNet residuals injected) is parity-checked < 1e-4 against a numpy float64 oracle (tools/controlnet_combined_fixture.py) by TestControlNetCombinedParity (TestNeuralPretrained.pas). Pass <unet.st> <unet.cfg> <controlnet.st> <controlnet.cfg> for a real (config-compatible) diffusers base UNet + ControlNet checkpoint pair. Pure CPU, well under a second on the fixtures.
• End-to-end base-UNet + T2I-Adapter single-step denoise — the offline CPU inner loop of diffusers' StableDiffusionAdapterPipeline, wired end to end over the already-landed importers. T2I-Adapter is the lighter sibling of ControlNet (its natural successor): a small conv encoder over a spatial hint (canny/sketch/depth) producing a pyramid of per-resolution feature maps that are ADDED into the SD UNet down-block hidden state — no transformer, no per-block zero-conv, just a lightweight ResNet-ish ladder. No new leaf layer — pure plumbing over existing layers (TNNetSpaceToDepth PixelUnshuffle, TNNetConvolutionLinear, TNNetReLU, TNNetAvgPool, TNNetSum) + the new SDUNetDenoiseWithAdapter driver. The flow is: a spatial hint (sketch/canny-style image) → T2I-Adapter (BuildT2IAdapterFromSafeTensors + T2IAdapterFeatures, neuralpretrained.pas): PixelUnshuffle(downscale_factor) reshapes the hint onto the latent grid (TNNetSpaceToDepth, with the conv_in input channels permuted at load from torch PixelUnshuffle order to space-to-depth order), conv_in 3×3 → channels[0], then len(channels) AdapterBlocks (block 0 keeps the grid, blocks 1.. AvgPool2d(2) first; an optional 1×1 in_conv channel change, then num_res_blocks adapter ResnetBlocks of block1 3×3 → ReLU → block2 1×1 → + identity) → one feature map per UNet down block → a base SD UNet built WITH adapter injection (BuildSDUNet(..., pWithAdapter=true): each down block gets an extra zero-default TNNetInput + TNNetSum on the main hidden-state path before its downsampler, so a plain SDUNetDenoise is bit-identical to the base UNet) → SDUNetDenoiseWithAdapter ADDS those features into sample at the end of each down block exactly as diffusers does (down_intrablock_additional_residuals) → the predicted noise a sampler would step on. The committed pico fixtures have random weights, so this is a wiring / throughput smoke — it proves the base-UNet + T2I-Adapter chain runs offline and produces a finite noise prediction, not photorealism. No network access / self-contained: falls back to the committed config-compatible pico SD UNet (tests/fixtures/tiny_sd_unet.*) and a matched pico T2I-Adapter (tests/fixtures/tiny_t2i_adapter.*) — the adapter channels [16,32] and grids (8×8, 4×4) match the pico SD UNet's two down-block stages, so the features add straight into the UNet hidden state with no reshaping (hint (16,16,3) → 2 features (16,8,8) + (32,4,4); latent (8,8,4) + text (5,1,12) → predicted noise (8,8,4)). The adapter feature pyramid is parity-checked < 1e-4 against a numpy float64 oracle (tools/t2i_adapter_tiny_fixture.py) by TestT2IAdapterParity (TestNeuralPretrained.pas). Pass <unet.st> <unet.cfg> <adapter.st> <adapter.cfg> for a real (config-compatible) diffusers base UNet + T2I-Adapter checkpoint pair. Pure CPU, well under a second on the fixtures.
• VAR next-scale image generation (coarse-to-fine → VQ decode) — the offline CPU VAR (Visual AutoRegressive, next-scale prediction; Tian et al. 2024, arXiv:2404.02905) image-generation loop, end to end over already-landed importers. No new leaf layer — pure plumbing over the landed VAR backbone + VQModel tokenizer. The flow is: class label y → class-conditional VAR transformer (BuildVARFromSafeTensors, neuralpretrained.pas) → the coarse-to-fine autoregressive SAMPLING loop (VARGenerate): for each pyramid level s = 0..K-1 it runs the forward over the partially-filled multi-scale token sequence (coarser scales already sampled, finer scales still zero), reads the next-scale logits at scale s's positions, argmax/temperature-samples the VocabSize-way tokens, and writes them back so the finer scales attend to them through the scale-block-causal mask (the new BlockCausalSegments SDPA flag) → the final scale's $\text{PatchNums}[\text{K}-1] \times \text{PatchNums}[\text{K}-1]$ token grid is a VQ token map → residual/discrete VQ decode to pixels (DecodeVARTokensToImage → TNNetVqModel.DecodeTokensToImage, the landed BuildVqModelFromSafeTensors family) → RGB image → a P6 PPM. Faithfulness note: canonical FoundationVision/var carries the cross-scale residual-VQ feature accumulation (next-scale interpolation/up-sampling of the running f_hat) inside the VQ tokenizer that produces the input embeddings; this importer's input contract is plain codebook indices embedded by word_embed, so the coarse→fine information flow here is carried purely by the transformer attention over the already-sampled coarser tokens — the full residual-VQ input embedding (and the text-conditioned Infinity variant + real-checkpoint parity) are tracked follow-ups in tasklist.md. The committed pico fixtures have random weights, so this is a wiring / throughput smoke — it proves the multi-scale loop + VQ decode run offline and produce a finite image, not a real picture. No network access / self-contained: falls back to the committed pico VAR (tests/fixtures/tiny_var.*, tools/make_pico_var_fixture.py, parity-checked < 1e-4 in TestVARParity) and a matched pico VQModel (tests/fixtures/tiny_var_vqmodel.*, tools/make_pico_var_vqmodel_fixture.py — latent grid 3 = VAR final patch_num, codebook 12 = VAR vocab, so the VAR final 3×3 token map is exactly the VQ token grid; image 6×6×3). Regression-tested by TestVARGenerateSmoke (TestNeuralPretrained.pas): it runs the full loop + VQ decode and asserts the deterministic (fixed-seed greedy) full SeqLen token sequence shape, legal token ids, and a finite $6 \times 6 \times 3$ image. Flags: --class N (default 0), --temp T (default 0 = greedy argmax; >0 = temperature softmax sampling), --seed N (default 424242), --smoke (assert finiteness, print SMOKE OK/SMOKE FAIL, set the exit code). Point it at your own var.safetensors vq.safetensors (with sibling config.json files) for a real checkpoint. Pure CPU, well under a second on the fixtures.
• Native text-to-VIDEO (CogVideoX flat-DiT → 3D-causal VAE) — the offline CPU CogVideoX (THUDM/CogVideoX-2b; Yang et al. 2024, CogVideoX, arXiv:2408.06072) native text-to-video loop, end to end over the landed importer. Architecturally distinct from AnimateDiff (which bolts a temporal module onto a frozen SD UNet): CogVideoX is a flat MMDiT-style transformer over a flattened (frame × height × width) latent token sequence with T5 text conditioning + expert adaLN-Zero modulation, with no UNet dependency. No new leaf layer — pure plumbing over landed pieces, and the two genuinely-new CogVideoX primitives are reused landed leaves: 3D RoPE over the (t,h,w)-factored video positions is TNNetMRotaryEmbedding (the Qwen2-VL M-RoPE leaf, applied to the video portion of Q/K only); the 3D causal-conv VAE is TNNetCausalConv1D over the VideoMAE space↔time view (a depth-axis causal temporal convolution — left-pad the time axis, no future-frame leakage). The flow is: caller-supplied T5 text states → CogVideoX flat-DiT denoiser (BuildCogVideoXFromSafeTensors + CogVideoXConditioning, neuralpretrained.pas) driven by a multi-step DDIM / DPM-Solver++(2M) reverse loop (TNNetDiffusionScheduler, neuraldiffusion.pas) → the sampled (NumFrames, GridH, GridW, in_channels) video latent → 3D-causal-conv VAE decode tail (BuildCogVideoXVaeDecoderFromSafeTensorsEx + DecodeCogVideoXVae: a per-spatial-cell causal temporal conv + SiLU + pointwise conv) → per-frame RGB → a sequence of P6 PPM files (frame_00.ppm, frame_01.ppm, …). The committed pico fixture has random weights, so this is a wiring / throughput smoke — it proves the chain runs offline and produces finite video frames, not real video. No network access / self-contained: falls back to the committed pico CogVideoX (tests/fixtures/tiny_cogvideox.*, tools/make_pico_cogvideox_fixture.py, parity-checked < 1e-4 on one denoiser step AND one VAE decode in TestCogVideoXParity). Flags: --steps N (default 4), --dpm (DPM-Solver++(2M) instead of DDIM), --smoke (assert finiteness, print SMOKE OK/SMOKE FAIL, set the exit code — the build step exercises this). Point it at your own cogvideox.safetensors (with a sibling config.json) for a real checkpoint. A real T5 encoder over a tokenized prompt + a real CogVideoX/VAE checkpoint is a tracked follow-up in tasklist.md; this demo supplies deterministic synthetic T5 states. Pure CPU, well under a second on the fixture.
• Attention entropy report — trains a tiny TNNetScaledDotProductAttention net on a copy task and prints TNNet.AttentionEntropyReport(NN, Probes) for every SDPA layer: per-row softmax entropy as mean ± std, a 10-bin ASCII histogram, plus per-layer "dead head" count (rows attending nearly uniformly — no routing) and "spike head" count (rows attending to one key). Uses the new public AttentionWeights accessor on SDPA.
• TracIn training-data attribution — a self-contained TracIn demo (Pruthi et al. 2020, "Estimating Training Data Influence by Tracing Gradient Descent") that attributes a prediction back to the training examples that shaped it (not to input features like SaliencyReport, nor to layers like ActivationPatchingReport): the influence of a training point on a test point is the dot product of their per-sample loss gradients < grad_loss(z_train), grad_loss(z_test) > — positive = proponent (pushed toward the prediction), negative = opponent (pushed against it). Uses the single-checkpoint "TracInLast" form (final trained weights only, no checkpoint summation). On a clearly-separable 2-D 2-class blob task it plants exactly ONE mislabelled training point (a class-0 point near the boundary, label flipped to class 1), trains a tiny MLP, then ranks every training point by TracIn influence on a corrupted test point and asserts the planted mislabel lands among the top-K most-negative opponents — the paper's headline "TracIn surfaces mislabelled data" result — printing top-K proponents/opponents and a graded PASS/FAIL line (the mislabel comes out as the single most-negative opponent, the only training point with negative influence). Per-sample weight gradients are read out via the SetBatchUpdate(true) / ClearDeltas / Compute / Backpropagate idiom (Neuron.Delta divided back out by the layer LR; never UpdateWeights) that FisherImportanceReport / GradientConflictReport share; the trained net is frozen. Cost is O(N_train) backward passes per test point (N_train kept to a few hundred). Multi-checkpoint TracIn-CP summation and a reusable TNNet.TracInReport method are noted but deferred — the single-checkpoint ranking is already clean on the toy. Pure CPU, no dataset download, well under a second.

Evaluation, calibration & uncertainty

• Confusion matrix report — trains a small MLP on a synthetic 3-cluster 2D Gaussian dataset and prints TNNet.ConfusionMatrixReport(NN, Samples, NumClasses): full CxC confusion matrix, row-normalized recall, per-class precision/recall/F1, macro/micro F1, top-1 and balanced accuracy, most-confused class pairs, and per-class hard-example indices.

• Decision-boundary report — runs TNNet.DecisionBoundaryReport(NN, Probes) on a 2-input classifier head, sweeping a Gx x Gy grid (default 41x41) over an auto-fitted bounding box of the input plane and running one forward pass per cell to render the learned function over its whole domain: an ASCII class map (argmax glyph per cell), a confidence-shaded overlay (top-1 softmax prob, or a normalised top1-top2 logit margin for linear heads), an estimated boundary length scalar (count of grid cells whose 4-neighbours disagree on argmax — a one-number proxy for how convoluted the boundary is), an optional true-class probe overlay so misclassified points stand out, and an optional x,y,argmax,top1prob CSV side-output. The example prints the class map before training (a near-constant single-class plane, boundary length 0) and after (clean separated regions), then contrasts a small net against a deliberately-overfit oversized net on a noisy two-moons set (the overfit net prints wigglier art and a larger boundary length). Guards a non-2-D input layer with a clear error message. Pure forward-only.

• Margin report — trains a small MLP on a synthetic 4-cluster 2D Gaussian dataset (one cluster deliberately wider) and prints TNNet.TopLogitMarginReport(NN, Samples, NumClasses): the per-sample top-logit margin top1_logit - top2_logit on the final-layer output as an overall 10-bin ASCII histogram, per-class mean/median margin grouped by true class (so a systematically uncertain class stands out), and the lowest-margin sample indices per class as a ready-made "hard examples" pool. Pure forward-only, one validation pass.

• Perplexity evaluation — trains tiny char-level sequence models with TNNetSoftMax and TNNetLogSoftMax heads and prints TNNet.PerplexityReport(NN, Tokens, ContextLen): per-token cross-entropy in nats and bits, perplexity, bits-per-character, top-1 / top-5 accuracy, a 10-bin ASCII histogram of per-token bits, and the K worst-predicted positions. Auto-detects log-space vs probability-space output by sniffing the final layer.

• Knowledge distillation — end-to-end demo of TNeuralKDTrainer (neural/neuralkd.pas), classic Hinton knowledge distillation L = alpha*CE(label, softmax(z_s)) + (1-alpha)*T^2*KL(softmax(z_t/T) || softmax(z_s/T)). A larger char-level next-token teacher (hidden=96) is trained with ordinary SGD on a structured synthetic 12-symbol stream, then two identical small students (hidden=12, same RNG init) are trained at matched steps / examples-seen: one WITH KD (alpha=0.3, T=3.0, soft teacher targets blended with the hard label) and one HARD-LABEL ONLY (the same trainer with alpha=1.0, where the soft term vanishes and Step() is an ordinary cross-entropy SGD step — the equivalence pinned by TestAlphaOneMatchesPlainCE). Same data order, same LR, same Step() count, so the runs differ only in the soft term. Held-out TNNet.PerplexityReport shows the KD student more than halves the hard-only student's perplexity (representative: teacher 14.06, KD student 18.71, hard-only 40.91) — the teacher's temperature-softened "dark knowledge" (relative probabilities of the wrong classes) regularises the tiny student better than a one-hot label alone. The teacher is run forward-only inside the trainer so it stays frozen (TestTeacherWeightsUnchanged). The README shows how to swap in a real imported teacher (GPT-2 / TinyStories via the safetensors / .bin importers) for users with more RAM — the only requirement is a shared vocabulary width and a TNNetFullConnectLinear(Vocab) -> SoftMax tail on both nets. Pure CPU, ~25 s, fits a 3 GB cap.

• HellaSwag-style multiple-choice eval — the end-to-end demo for EvaluateMultipleChoice (neural/neuralnlpmetrics.pas), the lm-evaluation-harness HellaSwag / ARC / PIQA scoring pattern: for each item it scores every candidate completion of a shared context with ScoreCompletion and lets the argmax win, reporting acc (gold wins by sum of completion log-probs, lm-eval acc) and acc_norm (gold wins by mean / length-normalized log-prob, lm-eval acc_norm). To stay within a tiny CPU/memory budget (no multi-GB download) it trains a small char-level next-token model on toy bigram phrases and scores four hand-written TNNetMultipleChoiceItem records — but the scoring path is checkpoint-agnostic: swap the toy model for a BuildLlamaFromSafeTensors (or any importer) and feed TNeuralHFTokenizer token ids into the same item records and the harness is unchanged. Highlights the single-next-token-head encoding gotcha (ScoreSequence uses CopyReversedNoChecksIntArr; the training loop must match it or accuracy collapses to chance). The toy model learns the bigrams perfectly so both metrics report 1.0000 (4/4). Pure CPU, well under a minute. EvaluateMultipleChoice now scores each item's candidates through ScoreCompletionsBatch, sharing the common context prefix (single-head nets skip the shared-context forwards for scores identical to the per-candidate path), and ScoreSequence/ScoreCompletion accept an optional LastWindow flag that scores over-context sequences on their trailing context-window instead of raising.

• MMLU few-shot accuracy eval — the end-to-end demo for EvaluateMMLU / MMLUReport (neural/neuralnlpmetrics.pas), the canonical 4-choice (A/B/C/D) knowledge benchmark scored the HF lm-evaluation-harness way: for each question it builds the standard k-shot prompt (Question ..\nA. ..\nB. ..\nC. ..\nD. ..\nAnswer:, with the few-shot demos drawn from the same subject) and predicts by the log-probability of the single answer-letter token that follows — not by the full-continuation perplexity that EvaluateMultipleChoice / HellaSwagEval uses (the two scoring modes are kept clearly separate). Reports per-subject accuracy plus the macro-average (mean over subjects — the headline MMLU number) and the micro-average (pooled over questions), in both 0-shot and 5-shot modes. To stay within a tiny CPU/memory budget (no network fetch, no multi-GB download) it trains a small char-level next-token model on a tiny embedded smoke subset (two toy "subjects") and runs in seconds under the 3 GB ulimit — the goal is the harness mechanics, not a real accuracy number. The scoring path is checkpoint-agnostic: swap the toy model for a BuildLlamaFromSafeTensors (or any importer), feed TNeuralHFTokenizer ids of the prompt and the four " A".." D" answer-letter tokens into the same TNNetMMLUQuestion records, and the harness is unchanged. Wiring the full cais/mmlu (hendrycks_test) splits via the venv-x datasets package is a documented follow-up.

• Calibration report — runs the forward-only neuralcalibration unit (CalibrationReport(NN, Inputs, Labels, BinCount)) on a held-out validation split of a synthetic 4-class 2D-Gaussian classifier and reports how well the model's confidence matches its accuracy: Expected Calibration Error (ECE), Maximum Calibration Error (MCE), the Brier score, and a reliability diagram (per-bin mean-confidence / accuracy / count) rendered both as an ASCII chart (accuracy # bars vs confidence - bars) and as a P2 (ASCII) PGM image (bin-accuracy bars against the y=x reference line). It then fits a single temperature-scaling scalar T via FitTemperature (a 1-D grid scan minimising validation NLL) and prints the report before and after scaling, so the ECE drop on an over-confident model is visible. Because a softmax head only exposes probabilities, temperature scaling operates on pseudo-logits z := ln(p) read off the final layer (softmax(z/T) is exactly temperature scaling on the original logits up to an additive constant); the trained backbone is never re-trained or mutated.

• Label-smoothing calibration check — tests the textbook claim (Mueller, Kornblith & Hinton, NeurIPS 2019) that label smoothing improves calibration at a small accuracy cost. Trains the same tiny softmax MLP once per smoothing strength eps ∈ {0, 0.05, 0.10, 0.20} — the only knob being TNNetLabelSmoothingLoss.Create(eps), an identity-passthrough loss head that smooths the target to t' = (1-eps)*onehot + eps/NumClasses (so eps=0 is exactly plain cross-entropy, the baseline arm) — then feeds each trained model into the forward-only neuralcalibration report and prints an eps | val-accuracy | ECE | Brier table. The task is deliberately hard so calibration actually discriminates: 6 heavily-overlapping 2D Gaussian clusters on a ring (large sigma) plus 15% random training-label noise, with calibration measured on a clean-label validation split. On this run smoothing helps with a clear sweet spot — eps=0.10 is best on both ECE (0.307 vs the 0.423 baseline) and Brier (0.944 vs 1.086) for a ~2% accuracy cost, while eps=0.20 over-smooths and regresses — so it is a calibration knob with an optimum, not a free win. Pure CPU, all four arms in ~85 s, no downloads.

• Test-time augmentation — trains a tiny synthetic 8x8x3 colored-pattern classifier (red vertical stripe / green horizontal stripe / blue checkerboard) and prints TNNet.TTAReport(NN, Probes, Labels): baseline top-1 accuracy on the untransformed inputs, per-transform top-1 accuracy with each transform applied alone (identity, TNNetFlipX, TNNetFlipY, TNNetReverseChannels, TNNetRoll(+1)), the full-ensemble TTA accuracy (argmax of the per-class average across all transforms) and its signed delta vs baseline, a per-class accuracy delta (so classes that lose under TTA are visible), the per-sample agreement rate, and a TTA helps / TTA neutral / TTA hurts verdict. Printed twice — averaging raw logits (default) and averaging post-softmax probabilities (soft voting) — so the linear-vs-geometric-mean question is visible side by side. Forward-only; each T(x) is produced by a tiny Input -> Transform wrapper net (identity feeds the raw input straight through) and the trained weights are never touched.

• MC-dropout uncertainty report — runs TNNet.MCDropoutUncertaintyReport(NN, Probes [, Labels]) on a synthetic 3-cluster 2D dropout classifier, the Monte-Carlo-Dropout epistemic uncertainty estimator (Gal & Ghahramani 2016). Unlike the rest of the family it deliberately keeps dropout active at inference (NN.EnableDropouts(true)), runs NumPasses (default 30) stochastic forward passes per probe and aggregates the per-pass softmax vectors to separate total uncertainty (predictive entropy H[mean_p]), aleatoric (expected entropy mean_t H[p_t]) and epistemic (their difference, the mutual-information / BALD score H[mean_p] - mean_t H[p_t]), plus per-sample top-class-prob variance and the pass-to-pass argmax flip rate. Across the batch it prints a 10-bin BALD histogram, the K most-uncertain sample indices (an active-learning query queue) and — with labels — a correctness cross-tab (mean entropy of correct vs incorrect predictions). The example evaluates three probe groups: the cluster cores (in-distribution — BALD ~ 0, 0% flips), an OOD band placed in the empty space between clusters (BALD ~4x higher, flip rates up to ~48% mid-band as the argmax wobbles), and a labelled validation split where the model is far more uncertain on its mistakes (0.033 vs 0.530 mean entropy) — "the model knows what it doesn't know". Spec invariants: with NumPasses=1 and dropout disabled BALD collapses to ~0, and a net with no TNNetAddNoiseBase layer emits a clear "no stochastic layers — MC sampling is a no-op" warning. The dropout-enabled flag is saved and restored; weights are never touched (forward-only).

• Deep Ensembles predictive uncertainty — the gold-standard Deep Ensembles baseline (Lakshminarayanan, Pritzel & Blundell, NeurIPS 2017) on the same synthetic 3-cluster 2D task as MCDropoutUncertainty, using only existing layers (no new layer). Trains M=5 INDEPENDENT softmax MLPs (identical architecture, different RandSeed per member), then averages the per-member post-softmax probability vectors and reports, per probe group: (a) the average single-member accuracy + ECE/Brier vs the ensemble's, both via neuralcalibration.ComputeCalibration/CalibrationReport (the ensemble-mean probabilities are fed through an Input->Identity passthrough net so the forward-only calibrator is reused, not re-implemented); (b) the predictive-entropy decomposition total H[mean_p] = aleatoric (mean_m H[p_m]) + epistemic (mutual information I = H[mean_p] - mean_m H[p_m] >= 0); and (c) an ASCII bar of per-group epistemic uncertainty. On the OOD band between clusters the epistemic/MI term spikes (~0.027 nats, ~270x the cluster cores' 0.0001) while staying ~0 on the confident cores, and the ensemble's Brier beats the average member's — the ensemble knows what it doesn't know. Built-in checks: M=1 reproduces member 0 bit-for-bit, ensemble accuracy >= mean single-member accuracy, and epistemic MI >= 0 everywhere and strictly higher on the OOD band than on the cores. Contrast: MCDropoutUncertainty samples dropout inside one net (one posterior mode) whereas an ensemble uses M independent nets (several modes); KnowledgeDistillation compresses such an ensemble teacher away, here we keep it and quantify its uncertainty; TestTimeAugmentation averages input transforms of a single model, not independent models.

• Conformal prediction — split (inductive) conformal prediction: wraps a frozen softmax MLP (trained on a synthetic 5-class overlapping-2D-Gaussian-blob task) in a guaranteed-coverage SET predictor (Vovk; Angelopoulos & Bates 2021), using only existing layers (no new layer, no library change). The data is split TRAIN / CALIBRATION / TEST; on the calibration split it scores the LAC / threshold nonconformity s = 1 - softmax[true_class], takes the conformal quantile qhat = the ceil((n+1)(1-alpha))/n empirical quantile (handling the qhat = +inf all-labels edge case), and at test time emits the set { k : 1 - softmax[k] <= qhat }. Sweeping alpha in {0.01, 0.05, 0.10, 0.20} it prints a table of alpha | target-coverage | empirical-coverage | mean-set-size | singleton% | empty%. The point is the finite-sample, distribution-free marginal guarantee P(true label in set) >= 1 - alpha that the scalar/point uncertainty examples never provide: CalibrationReport matches a single confidence to accuracy, MarginReport reports the top-1-vs-top-2 logit margin, MCDropoutUncertainty / DeepEnsembleUncertainty report stochastic/ensemble entropy — none promises anything about the true label, whereas conformal returns a variable-size LABEL SET (singleton on easy inputs, larger on hard ones) that provably covers it. Built-in checks (PASS/FAIL, Halt(1) on failure): empirical test coverage >= 1 - alpha - slack across the whole sweep, and mean set size shrinks monotonically as alpha grows. Pure CPU, single-threaded, runs in under 10 seconds.

• Adversarial-robustness report — runs TNNet.AdversarialRobustnessReport(NN, Samples, Labels, EpsList) on a small synthetic 3-class 2-D classifier, crafting FGSM (fast gradient sign method, Goodfellow et al. 2015) input perturbations x_adv = x + eps*sign(d loss/d x) at an increasing menu of epsilons (one forward + one backward per sample to get the input gradient, reusing TNNet.EnableInputGradient) and re-running a clean forward on each perturbed input. Reports a top-1 accuracy-vs-eps degradation curve (eps=0 is the clean baseline), a 10-bin histogram of the per-sample critical epsilon (smallest eps that flips the clean argmax), the mean clean-confidence of the earliest flippers vs the longest survivors, per-class accuracy at the median eps, and a robust / moderately fragile / fragile verdict. A second model trained with input-noise augmentation shows a flatter degradation curve (the expected robustness gain). The network is frozen — ClearDeltas, never UpdateWeights — so the trained weights are never modified (an evaluation of robustness, not adversarial training).

• Mahalanobis OOD Detector - Reproduces Lee et al. 2018 out-of-distribution detection: freeze a small classifier, fit per-class Gaussians with a tied covariance over the penultimate features, and score new points by negative squared Mahalanobis distance. Reports a single AUROC (rank-statistic / Mann-Whitney U form) separating in- from out-of-distribution. Pure CPU, ~2s.

• Anomaly-detection autoencoder (reconstruction-error AUROC) — the textbook reconstruction-error anomaly detector: an undercomplete autoencoder (8→16→2→16→8, ~320 weights) is trained ONLY on a synthetic NORMAL distribution (samples on a low-dimensional 2-D manifold embedded in 8-D), then scores held-out points by per-sample reconstruction MSE — anomalies off the learned manifold reconstruct poorly. Reports the mean reconstruction error of normal vs anomaly (0.95 vs 6.38) and a single rank-AUROC = 0.97 (Mann-Whitney-U form, the helper copied locally from MahalanobisOOD rather than promoted to the library) with a built-in PASS gate. Distinct from MahalanobisOOD (which scores classifier features against fitted Gaussians; this scores reconstruction error from an unsupervised generative model). Pure CPU, a few seconds.

• Masked Autoencoder (MAE) self-supervised pretraining — the self-supervised-vision example (He et al. 2022, Masked Autoencoders Are Scalable Vision Learners, arXiv:2111.06377): unlike the repo's existing full-image autoencoders (VisualAutoencoder/SparseAutoencoder/AnomalyAutoencoder/GumbelAnnealingAutoencoder), MAE drops most of the image and reconstructs only the missing patches from the few that remain, which is what forces the encoder to learn structure with no labels. Each image is cut into a 3×3-patch grid (16 tokens); 50% of the patches are randomly masked each step (replaced by a shared zero mask token), a ViT-style transformer encoder (AddTransformerEncoderBlock ×3) with learned positional embeddings processes the token sequence, a lighter transformer decoder (×1) + a per-token linear head reconstructs all patches, and the reconstruction MSE is applied ONLY to the masked patches — realised exactly without a custom layer by building the regression target so visible entries equal the prediction (zero error) while masked entries hold the true pixels (FOutputError = pred − target is the per-element MSE gradient for a linear head). The payoff: the encoder is then FROZEN and a single linear+softmax linear probe is trained on its mean-pooled features, versus the same probe on a randomly-initialised encoder of the SAME architecture — a typical run scores 0.768 vs 0.656 (an ~11-point gap, well above the 0.250 chance level), averaged over 5 probe seeds so the number is stable and reproducible. The downstream task classifies the stripe PERIOD (spatial frequency) with orientation randomised as a nuisance — a global property no single patch reveals — and the encoder width is kept deliberately narrow (a bottleneck) so a random projection has little capacity and the learned representation wins (with a wide encoder, random nonlinear features are a famously strong linear-probe baseline and the gap closes; narrowing the bottleneck is what exposes the value of pretraining). The new code versus a plain ViT is the random patch mask, the masked-only loss target, and the encoder-end feature tap the probe reads. Honest deviation from canonical MAE (a static graph can't express a per-sample variable sequence length): the encoder runs over the full token grid with masked positions held by the zero mask token, so the asymmetric encode-visible-only compute speedup is dropped — everything else (random masking, learned positions, encoder+lighter-decoder asymmetry, masked-only loss) is faithful. Pure CPU, ~10 s.

• Deep Q-Learning (DQN) — a reinforcement-learning example: a minimal but complete Deep Q-Network (Mnih et al. 2015, Human-level control through deep reinforcement learning) that learns an optimal navigation policy on a tiny, deterministic 5x5 grid-world (fixed goal +1, two terminal pits -1, a small per-step living penalty pressuring the shortest path) — the agent, the environment and the replay machinery all live in the single .lpr and the Q-network is composed from existing dense layers (no new layer class). Textbook DQN: a 25 -> ReLU(64) -> ReLU(64) -> Linear(4) Q-net (one-hot state in, one Q-value per action out), an experience-replay ring buffer of (s,a,r,s',done) transitions sampled in random minibatches, a target network re-synced periodically via CopyWeights (not LoadFromFile) to stabilise the TD target, epsilon-greedy exploration with exponential decay, and the standard single-action TD update y = r + gamma·max_a' Q_target(s',a') regressed into Q(s,a) for the taken action ONLY (the target vector is seeded with the current Q(s,·) so the gradient is exactly zero on untaken actions). Minibatch gradients are accumulated with the SetBatchUpdate(True) idiom (the manual UpdateWeights path bypasses grad clipping, so one-hot inputs + a small LR keep it stable) and applied once per batch. Headline: a moving-average learning curve climbs from random (return ~-1.06, ~17 steps, hitting pits / timing out) toward optimal (return ~0.78, the optimal 8 steps), then the greedy (epsilon=0) policy is rolled out and its ASCII trajectory reaches the goal along the shortest pit-avoiding path — 100% greedy success rate over all 21 non-terminal start cells. Pure CPU, ~32 s on two cores.

• HuggingFace Hub fetch -> import in one command — demo for the opt-in neural/neuralhfhub.pas Hub download helper, the only HTTP anywhere near the import path (the core importers in neuralpretrained.pas stay strictly offline; only programs that uses neuralhfhub link fphttpclient/OpenSSL). HubFetchModel(repo) resolves https://huggingface.co/{repo}/resolve/{rev}/{file} (redirects to the CDN handled, rev defaults to main), downloads config.json + tokenizer.json (404-tolerated) + the safetensors weights — transparently falling back to model.safetensors.index.json and every shard in its weight_map when the checkpoint is sharded — into a skip-if-present local cache (~/.cache/neural-api/hub/{repo}/{rev}/..., overridable via HubSetCacheDir / NEURAL_API_HUB_CACHE; .part-then-rename so interrupted downloads never poison it; HF_TOKEN/token param sent as a Bearer header for gated repos) and returns the snapshot directory that BuildFromPretrained accepts, so BuildFromPretrained(HubFetchModel('sentence-transformers/all-MiniLM-L6-v2')) is the whole program. Verified end to end: the MiniLM download is byte-identical to what python's huggingface_hub fetched, builds to the same 242-layer/22.5M-weight encoder, and the default no-args run pulls a ~100KB five-shard hf-internal-testing checkpoint through the index-json path (123 layers). Second run is instant and fully offline.

• Chat terminal — interactive REPL chat over any instruct checkpoint the generic importer dispatch understands (BuildFromPretrained in neural/neuralpretrained.pas: gpt2, llama, mistral, qwen2/3, gemma/2/3, phi/phi3, gpt_neo(x), gptj, cohere/cohere2, rwkv, mamba, bloom, deepseek_v2, ...): point it at a model directory (config.json + safetensors/torch-bin + tokenizer.json) and chat. The Cohere route (Command-R / Aya / Command-R7B) makes this the entry point for open multilingual chat — see the ChatTerminal README's cross-lingual session example. Multi-turn history rendered through the neuralchat.pas chat templates — the format is fingerprinted from tokenizer_config.json's chat_template (DetectChatFormatFromConfigFile) with a --format override — and each reply streams token-by-token to stdout (decoded-delta printing with a BPE/UTF-8 prefix guard, Flush per token so piping streams too). Inference knobs map onto the existing decode toolbox: --temperature / penalties run in the probability domain through a TNNetLogitsProcessorChain (TNNetTemperatureProcessor, TNNetPenaltyProcessor over TNNetTokenHistoryPenalty), --top-k/--top-p/--min-p pick the matching TNNetSampler*, and generation stops on the tokenizer EOS or the format's end-of-turn marker matched as a token-id stop sequence. Always built pTrainable=false; --int8 adds weight-only int8 (pQuantizeInt8). REPL niceties: /exit, /reset, /system <msg>; --selftest runs 31 offline unit checks (arg parsing, ChatML render parity, REPL command parsing) with no model files needed. Full forward per token (the GPT2Import convention), so it works across every imported family including the non-KV-streamable ones.

• StarCoder2 code completion — a minimal CPU code-completion demo for the StarCoder2 importer (BuildStarCoder2FromSafeTensors / BuildFromPretrained model_type starcoder2, the bigcode/starcoder2-3b/7b/15b code-LLM family — a code-specialised decoder). Point it at a model directory (config.json + safetensors/torch-bin) and its stock BPE tokenizer.json, give it a code prompt (default a Python def fibonacci(n): header), and it greedily extends the prompt one argmax token at a time and prints the decoded completion. StarCoder2 is architecturally distinct from every Llama-path importer: it pairs RoPE + GQA + (optional) sliding-window attention with three GPT-2-flavoured pieces the RMSNorm/SwiGLU importers never touch — biased nn.LayerNorm norms (NOT RMSNorm), bias=True on every linear including q/k/v AND o_proj (the exact path OLMo-2 deliberately rejects), and a plain two-matrix gelu_pytorch_tanh FFN (c_fc -> GELU -> c_proj, no SwiGLU gate). All of that lives in neuralpretrained.pas; this demo is just the generation harness (always built pTrainable=false; keep SeqLen small on the real 3B checkpoint — the full 16k context is slow on CPU). The committed pico parity fixtures (tests/fixtures/tiny_starcoder2{,_window}.*, generated by tools/starcoder2_tiny_fixture.py) pin both a full-attention and a sliding-window config to the HF float64 oracle within 1e-4 (TestStarCoder2LogitParity / TestStarCoder2WindowLogitParity).