Performance Updates - #218
Merged
Merged
Conversation
…on-free string paths neural/neuraldecode.pas, no behaviour change intended at any site. - #18: nine hand-rolled vocabulary argmax loops (DecodeGreedy fallback, DecodeBatchGreedy, DecodeEarlyExitSelfSpeculative full/exit rows, DoLa greedy + degenerate fallback, DecodeSampled, DecodePromptLookup outer and verify loops) now call TNNetVolume.MaxPos, whose ties-to-lower-index contract is the scalar loop's. SafeLogProb reuses the returned value instead of re-indexing Raw[Best]. - #20: DecodeGreedy's constrained scan tests the score before calling Constraint.TokenAllowed, so the call is paid only by a token that beats the running max. Same conjunction, same selected token. - #27: DecodeDoLa keeps the winning candidate's lens row from the JS scoring pass instead of re-splicing and re-running the whole head sub-stack for it. Candidate layers all sit below HeadInIdx, so the head recompute could not have changed that row. - #22: SelectTopBeamCandidates and the two contrastive-search top-k selections become bounded insertions - one pass over the pool plus the rare shift, instead of KeepCount sweeps over it. Every shift stops at the first score that is not strictly smaller, so Cand[0] stays the FIRST global argmax. - #13/#19: TNNetCFGProcessor.ProcessRow combines with Mul + MulAdd (two passes, not sub/scale/add) and softmaxes through ExpShiftSum; DecodeSeq2SeqSampled folds its shift into ExpShiftSum; DecodeSeq2SeqBeamSearchAll takes its row max with MaxValue and its exp-sum with ExpShiftSum over a hoisted row pointer (#12). - #23/App C: PromptLookupDraft and the Dict-side PrepareTokenHealing compare with CompareMem instead of allocating a Copy per probe; NeedleLoremFiller and NeedleSpliceAt size their result once and fill it with Moves; TNNetGrammarConstraint / TNNetJSONConstraint TokenAllowed index FTokenStr in place and bind the local string only on the rare multi-character path. - #15: the grammar packed-position stride is a power of two, so PackPos and UnpackPos are a shift and a mask rather than a multiply and two divides. Both are inlined in Release (#26). - #5: NeededNextChars and ForcedProgress hoist Length(Text); the redundant Tail copy is gone. Suite: 2840 tests green, plain and -dAVX2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
neuralgguf.pas - ReadMetaValue array branch reads through a 64 KB staging block (NeuralGGUFMetaStageBytes, writable for tests) instead of one Stream.ReadBuffer per element: a 151k-token vocab cost ~450k syscalls across tokens/scores/token_type, now a few hundred reads. The element type test is unswitched out of the loop, one loop per type (#6). - LoadTensorRowsFlat de-interleave branch: the permutation never leaves the HeadDim-row head, so one sequential read of the whole head serves every destination row in it - HeadDim seek+read pairs collapse into one contiguous read, over-read bounded by the two partial end heads (#13). - DequantizeQ4_0/Q4_1/Q5_0/Q5_1: one byte load feeds elements e and e + 16, halving the loads and dropping the per-element e < 16 test; LegacyNibble is gone with its only callers (#6/#13). - DequantizeQ6K: the low/high nibble selector becomes a hoisted shift instead of a per-element branch (#6). neuraltorchbin.pas - SETITEMS sizes the dict Keys/Vals once for the whole batch; a state_dict arrives as a single SETITEMS, so the pair-by-pair append regrew both arrays per tensor (#17). tests - TestGGUFMetaArrayStaging: string/f32/i32 arrays plus a trailing scalar key and the tensor table, read at five staging sizes down to 7 bytes, pinning the refill boundary and the restored stream position. - TestGGUFTensorDecodeParity: a mid-head row slice of the de-interleaved q_proj must match the whole-tensor de-interleaved load. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Int32Array The staging fixture declared its token_type array as `array of integer`, which does not match AddMetaInt32Array's `array of Int64` parameter, so the suite did not compile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…TFT, single-pass cycle removal neuralaudio.pas - ComputeWhisperLogMel records each slaney triangle's nonzero bin span at filter-bank build time and projects only over it, instead of a dense 201-bin dot per (frame, mel). The skipped taps are exact zeros, so the log-mel output is bit-identical (rule #3/#5). - FFTRadix2 is lifted out of ComputeMusicgenMelodyChroma to a unit-level FFTRadix2InPlace, and ISTFTOverlapAddReIm uses it for a power-of-two NFFT: the per-frame O(NFFT*NumBins) inverse DFT becomes one O(NFFT*log2 NFFT) pass over the hermitian-mirrored spectrum. The direct evaluation stays for a non-power-of-two NFFT. - ComputeMusicgenMelodyChroma drops the dead per-chroma store and the per-frame zeroing loop: SetLength zero-fills, and only the argmax cell is ever written (rule #13). - LoadWav16ToVolume unswitches the mono case out of the channel-average nest (rule #3). neuraldpo.pas - TNeuralGRPOTrainer.SampleCompletion at Tau=1 samples straight out of the layer output: the vocabulary-long copy into FProbs is dead work and the sum is one vectorized GetSum (rule #13/#19). neuralmxfp4.pas - DequantizeMXFP4 folds the block scale into a 16-entry code table once per block and walks the block with pointer advances. The scale is a power of two, so the result is bit-identical (rule #5/#11). neuralplanbuilder.pas - RemoveAllCicles no longer restarts the O(N^2) duplicate scan after every removal. RemoveSubList is applied at the highest index that still matches an earlier state, so nothing above the removal point can match afterwards and the downward scan resumes at that point; the removal order is unchanged. - TCompositePlan.ToAct drops its scan over all MaxPlans plans: ChooseBestPlanBasedOnNextStep already scored each of them with the same GetNextStep lookup, and a plan with no next step scores above any usable one, so the winner has a step whenever any plan does. The chooser now returns that step index instead of it being looked up a second time. - The plan scores are whole counts, so EvalPlan/EvalPlanBasedOnNextStep and their comparison locals move from extended to longint. - BuildPlanFn's two nested choosers share one candidate-state scratch sized once per build instead of a SetLength per call (rule #17). tests/TestNeuralAudio.pas - TestISTFTReImMatchesProductTable adds NFFT = 64 so the radix-2 inverse path is pinned against the direct product-table reference, and scales the bound to the reference's own magnitude. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… paths - TNNetGptOssGatedSwiGLU.Compute: on an AVX build the strided gate half is gathered upper-clamped and alpha-scaled into the output row, which then feeds one vectorized Sigmoid instead of HalfDepth scalar exponentials (#19, #13). The scalar loop stays for non-AVX builds, where Sigmoid has no vector path. - TNNetGptOssGatedSwiGLU.Backpropagate: same gather, using FOutputErrorDeriv as the per-row sigmoid scratch the way TNNetSwiGLU already does (#19, #17-safe: no allocation, the field is sized by SetOutputErrorSize). - TNNetSwishLearnable/TNNetMishLearnable.Backpropagate: Compute already caches dy/dx in FOutputErrorDeriv, so the whole input gradient is one MulAdd instead of a per-element recomputation behind a per-element PropagatesErr test (#13, #20). Output error is read through FData, not Raw[] (#3), and bound to a local since it is used twice per iteration (#4). - TNNetMishLearnable: 1+expVal computed once per iteration (#4). - TNNetSmish.Compute: the scalar pcr_log1pf loop between two vectorized passes becomes AddScalar(1)+Ln; sigmoid output lies in (0,1) so 1+s stays in (1,2) and log1p's small-argument compensation is not needed (#13, #19). - TNNetVAEKLDivergence.Backpropagate: both analytic halves are contiguous depth runs, so each becomes bulk passes - Move+Mul for mu, Exp+AddScalar+Mul for log_var (#13); beta*0.5 hoisted (#5). - TNNetHardSwish/TNNetHardSigmoid.Compute: the linear segment multiplies by a Single-typed 1/6 instead of dividing (#21). - GLU-family layers in this region: the X-outer/Y-inner loop pairs are swapped so the inner loop walks memory sequentially (Appendix E). 32 sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ocal-product error - TNNetPower.Compute: keep the constructor's exponent as an integer and unswitch on it (#20/#24), so exponents 2/3/4 run a multiply chain sharing the sub-product with the derivative instead of pcr_powf per element. The general pcr_powf path stays for every other exponent. - TNNetLocalProduct.BackpropagateAtOutputPos: the smoothed unit-derivative share is invariant across the window, so it is computed once and added (#5/#21); the dead SmoothErrorPropagation branch goes with it. - TNNetGridAvgPool.Compute/Backpropagate: clamp the window to the real input once per output cell and use its area as the count_include_pad=False divisor (#5/#20), removing the per-cell range tests and the whole extra counting pass in the backward pass. - tests: TestGridAvgPoolPaddedForward pins the padded corner/edge/interior divisors and depth handling; TestGridAvgPoolPaddedGradientCheck pins the clipped-window error split against central differences. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lk skip-connection error - resize/upsample family: TNNetAxisMap caches the X-axis sampling geometry per shape, so BilinearUpsample, BicubicUpsample, BilinearResize and Resize2D (nearest/bilinear/bicubic, forward and backward) look the source indices and tap weights up in a table instead of recomputing the divide, floor, clamps and cubic weights on every output row (#27). The table is rebuilt only when the axis sizes or the align_corners convention change (#17: lazily size-guarded persistent field, never a per-call allocation). - TNNetDepthwiseConv.BackpropagateCPUFast: the previous-layer coordinates only grow with the feature counters, so the lower-bound tests are vacuous and the upper bounds become a clamped feature-row range plus one test per feature column, hoisted out of the inner loop (#5/#11). The delta accumulation still covers the full window; only the previous-error write is gated. - TNNetPointwiseByteProcessing / TNNetBitProcessing.Backpropagate: with a skip connection every channel passes the straight-through gate, so unswitch the invariant flag and run the depth column as one bulk Add (#13/#20). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… Upsample and the activations - TNNetAdaIN: hoist the style mean/std column pointers out of the forward position loop and the spatial bounds out of the channel loop (#5/#8); the scalar backward branch scales by the cached inverse style deviation and by one reciprocal each of the content/style position counts instead of dividing per element (#21). - NeuralPerChannelMeanInvStd: one reciprocal of the position count feeds both channel passes (#21). - TNNetPixelShuffle forward and backward: position-outer / channel-inner nest, so the source column base and the output row base are hoisted and the source depth slot is a running sum (#5/#11/#12). - TNNetUpsample.Compute / ComputePreviousLayerError: carry both cursors across the row loop instead of two GetRawPos calls per iteration (#12). - TNNetSoftSign: one reciprocal shared by the value and its derivative (#21). - TNNetCELU: one reciprocal of alpha for the negative branch (#21). - TNNetBitProcessing.EncodeToBytes: one 255/span factor, mirroring DecodeToOutput (#21). - TNNetSignedSquareRoot family: 0.5/sqrt instead of 1/(2*sqrt). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…upling and sinc invariants TNNetEchoStateReservoir.Compute: build the whole pre-activation row, run one elementwise Tanh over it (#19), then form h_t as Move+Mul+MulAdd and copy the row to the output (#13); the recurrent term is peeled off t = 0 (#20) and the FH/FPre/FOutput and Prev row offsets are carried (#12). TNNetWKV.Backpropagate and TNNetCrossWKV.BackpropagateSym: e^{u+k} = e^u * e^k with e^u hoisted per channel, dropping one NeuralExp per (channel, step) (#5). TNNetCrossWKV.ComputeAsym: the receptance gating leaves the channel loop and runs one contiguous row per query - elementwise Sigmoid plus elementwise Mul by the cached summary row (#13, App. E) instead of a strided scalar sweep. TNNetAffineCoupling: Backpropagate binds the conditioner weight rows, delta rows and neuron objects into the per-r tables once per call instead of re-resolving them at every pixel (#9); FInverse, FLogDetLossWeight and the x_a pointer are hoisted (#8, #20); the tanh-clamp derivative divides once (#4, #21) and Compute scales by 1/clamp and a hoisted row-byte count. TNNetSincConv1D: MaterializeBank hoists the two band-edge factors and the two 2*pi*f arguments per filter, carries the tap index (#5, #6) and multiplies by 1/SampleRate; Backpropagate builds a = 2*pi*tap once per tap and passes it to both DTermDf calls, and reuses the invSR it already had. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ent and Hamiltonian cells TNNetKalmanFilterCell.Backpropagate: the two g-derivatives share the (pm+R)^2 denominator, so the step takes one reciprocal instead of two divides (#21). TNNetMLSTMCell: the read-out denominator is clamped at 1, so both Compute and Backpropagate scale by a single 1/den (and its squared negative) instead of dividing per channel (#21); the three per-step adjoint buffers are cleared with FillDWord (#13); the scalar-gate weight rows, grad rows and biases are bound above the sequence loop (#11). TNNetRGLRU.Backpropagate: d loga/d rg is fixed for a channel, and the lambda chain factor applies once to the accumulated sum rather than per step (#5). TNNetLRU.Backpropagate: keep the raw sin/cos pair from the single pcr_sincosf the scan already needs and reuse exp(nu), removing a second transcendental pair and a second exp per channel (#4). TNNetHamiltonianCell: the field and HVP helpers take their already-bound weight and delta tensors instead of re-resolving them from FNeurons at every sub-evaluation (#9); the adjoint zero loops become FillDWord and the half-width scale-copy and accumulate loops become Move+Mul and Add (#13); the half-kick scale and the half-row byte count are hoisted (#5). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ke the spiking forwards time-major TNNetHolographicBinding: both modes read the second factor with a cyclic index that wraps once, so each output (and each da/db gradient row) becomes two contiguous TNNetVolume.DotProduct calls instead of n scalar multiply-adds with a wrap test (#13). The descending cases (bind forward, unbind da) go through FBRev, a persistent reversed copy of b sized in SetPrevLayer (#17), rebuilt in O(n) per pass. The skip-if-zero error guards are dropped with the scalar loops. TNNetCirculantLinear direct path: reversing the INPUT once per forward rewrites y[i] = sum_k c[(i-k) mod n] x[k] as y[i] = sum_p c[(i+p) mod n] xr[p], so the kernel index ascends and every output is two dot products (#13); the reversal is on the input, not the kernel, so hand-edited weights stay correct with no mirror to rebuild. The input gradient gathers per input and the kernel delta gathers per tap, both as two dot products, with -FLearningRate applied once per tap (#5) and the bias delta as one vector MulAdd. Batch and per-sample updates now share one accumulation body. The per-output bias test leaves the forward loop as a single FOutput.Add (#20/#13). TNNetLIFNeuron / TNNetALIFNeuron: ComputeCPU and ComputeSurrogate ran channel outer / time inner, striding every volume by Depth. They now run time outer / channel inner, reading the t-1 row straight out of the cached FVmem/FAdapt/ FSpike, so all accesses are contiguous (App. E); row 0 is peeled since it starts from rest. Per-channel beta and V_th are materialized once per pass into Depth-sized fields allocated in SetPrevLayer, removing the EffBeta/EffVth calls from the O(T*D) body -- in ALIF EffVth was being called per element (#5/#8). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…reflection test ForwardLift, InverseChannel and BackpropagateCPU all ran sample-outer / tap-inner, so GetTap (a TapPtr() call plus a size test) and DWT1DReflect (a call with a reflection loop) fired once per (tap, sample). A lifting step reads one band and writes the other, and the transposed step reads one gradient band and writes the other, so the two loops are interchangeable: the tap value and its neighbour offset are now fetched once per tap (App. E, #5/#8), and the reflection collapses to a range test that only fails at the two edges. In the transposed steps this also keeps one tap's whole weight gradient in a register instead of doing a read-modify-write of WDelta per (tap, sample), and folds the -FLearningRate scaling into a single multiply per tap (#5). Also in BackpropagateCPU: the token-pair stride, the odd-length padding row base and the band byte count are hoisted, the three (2*i)*FDepth / i*(2*FDepth) address rebuilds become carried offsets (#12), the gradient seed and the adjoint of the final band scaling merge into one pass, and Move uses csDoubleSize (#1). InverseChannel is public and may be driven channel by channel, so its two per-call SetLength working bands become persistent fields grown once per shape (#17), and the even/odd merge carries its index. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he ToMe matching scans TNNetSinkhorn: the column normalization and its adjoint walked one column at a time, striding by N through four passes. They now run row-major over per-column accumulators sized in SetPrevLayer (App. E): the column max is one AVX MaxElements per row, the row-wise subtraction of the column log-sum-exp is one AVX MulAdd per row, and the adjoint's dotG reduction is one AVX Add per row. The per-element arithmetic of the adjoint softmax is unchanged -- it still divides by the column sum before scaling by dotG (#21). The backward column softmax is held in a grow-only matrix sized on first backward pass, so an inference-only layer never allocates it, and FSoftCol is gone with the strided loop it served. The forward row step now gets its sum from the fused ExpShiftSum kernel (#19), matching what the row adjoint already did, and both row loops carry ri*FN instead of rebuilding it (#12). TNNetTokenMerging: the matching scans tested index parity per token. Both the A scan and the B scan now step by 2 from their parity start, halving the inner scan and dropping the mod test (#20), with the b-row offset carried by two depths. Token norms are stored as reciprocals so the O(T^2/4) similarity scan multiplies (#21), and the best partner is tracked in registers, ranked on dot * 1/|b| -- 1/|a| is a positive constant over the scan, so it scales the winner once (#14). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…the truncated spectrum TNNetSpectralConv1D / TNNetSpectralConv2D: RebuildWeightPlanes runs on every forward, and it called WBase(m, ci, co) per input channel to walk the interleaved [mode][in][out][Re,Im] weights. ci is the minor axis of both the mirror and the interleaved source, so both bases are now carried -- the mirror by 1 and the source by one (out, Re/Im) pair (#12). The weight-delta scatter in both backward passes walked the same layout and gets the same treatment. The per-output spectra and the IFFT input buffers were zero-filled over the whole grid and then overwritten on the kept modes. Only the truncated part needs clearing, and it is contiguous per row, so those loops become FillChar over the tail (2-D: the tail of each kept-mode row plus the whole rows above ModesX) (#13/#5). The gradient plane and per-mode weight-row clears become plain FillChar. The grid-address rebuilds ((FSizeX*iy)+ix)*Depth+c in the 2-D pack, output write, error read and error scatter are carried by one grid row per step (#12/#5); the error scatter also stopped building its address twice (#4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…2x2 complex multiply TNNetCirculantLinear FFT path: FFT(c) was recomputed on every forward and again on every input-gradient pass, although the kernel only changes when the weights do. It is now transformed once per weight state into a cached spectrum (#27), guarded by a value snapshot of the kernel rather than an AfterWeightUpdate hook so a hand-edited weight is still honoured -- an O(n) compare against an O(n log n) transform. The forward's bias test also leaves its output loop (#20). CirculantFFT: the butterfly rebuilt i+k and i+k+half four times each per iteration; both are carried now (#12), the partner element is read once into registers (#4), and the inverse scaling multiplies by 1/N -- exact here, since the transform only accepts power-of-two N (#21). TNNetComplexLinear: forward, input gradient and weight gradient each walked a 2x2 sign/index table per term. All three are written out as the plain complex multiply, its conjugate and the outer product, the way the quaternion layer already does, so nothing is read from a table and the block base is carried (#6/#9). The two-component zero tests collapse to one condition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er copies MusicGen Melody: - ComputeLogits: the retained decoder-frame rows are the contiguous tail of the layer output, so one Move replaces the per-frame Move loop (#13, App. C). - New ComputeLogitsAtRow twin (mirrors Parler): Generate reads one frame per step, so it now fetches that row instead of the whole DecSeqLen*K*Vocab block every step. - Generate: TNNetVolume.MaxPos over the row instead of the scalar argmax scan, with the codebook slot base carried (#18, #12). - The decoder input volume is a persistent FInSeq field with a lazy ReSize instead of a Create/Free per step (#17); the fill moved into RunDecoderOn, shared by both logit readers. - BuildConditioningPrefix: the chroma tiling repeats every nIn frames, so only the first nIn rows are projected and the rest are row copies (#5, #13). Sinusoidal position tables (MusicGen, Parler, Melody): ChCnt-outer so the per-channel Exp runs once per column, with the row base carried; Angle stays the same PosCnt*Factor product, so the tables are bit-identical (#5, #12). Parler cached decode: the step logits are read straight off the borrowed last layer output; the per-step Copy into a scratch volume is gone (#13). Loaders: - LoadLinearWeights (Bark) and LoadStyleGAN2Linear: neuron j's weights are HF row j verbatim, so one Move per neuron replaces the elementwise row copy, source base carried (#13, #12). - LoadWav2Vec2PosConv: g[k]/norm[k] folded into one per-tap scale, turning the per-element divide into a multiply (#21, 1 ulp). Whisper alignment: - WhisperMedianFilterRow takes caller-owned dst/src/window buffers, so the two per-row SetLengths and the internal snapshot copy are gone (#17). - WhisperDTW keeps the backtrace table in one flat (N+1)*(M+1) block instead of a ragged array: one allocation instead of N+1, no per-probe indirection. Tests: TestMusicGenMelodyParity now pins ComputeLogitsAtRow against the whole-block ComputeLogits rows (exact match). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
neuralpretrained.pas only. Import-time weight transposes, position-table builds and a few inference decode helpers. Load path - LoadResNetConvFoldBN / LoadInceptionRectConvFoldBN / LoadMaskRCNNConv / LoadConvNeXtConv / LoadRegNetGroupedConvFoldBN / LoadDPTConvNoBias / RaftLoadConv: K=1 collapses to Move (+ TNNetVolume.Mul when the folded BN scale is not 1); K>1 carries the source index with a K*K stride (App. C, #12/#13). - LoadMaskRCNNDeconv / LoadDPTTransposeResize / RaftLoadConvGRU / LoadVideoMAETubeletConv: bind the neuron weight volume once, carry the source offset, walk the destination in order (#9/#11/#12). - LoadMaskRCNNBoxFC6: channel innermost so the CAI depth-major destination is written contiguously (App. E). - LoadConvNeXtDepthwise / LoadMobileNetDepthwiseFoldBN: carried source and destination indices instead of per-tap index arithmetic. - ReparamRepVGGBlock: the 1x1 and identity branches only touch the centre tap, so their tests leave the kernel nest (#20); the 1x1 fold is one MulAdd. - Contiguous copies promoted to Move: LoadConvNeXtChannelVector, LoadChannelBiasFromShift, LoadF5DepthwiseConv1D, LoadF5GRN, LoadGRUDirection bias split, PixArt scale_shift_table (per block and final), CogVideoX VAE conv_in/conv_out (App. C). - BuildSwinFromSafeTensors: dropped the write-only WinLayerW/WinLayerH arrays that grew one element per head. - ViTPose pos fold: Move plus a per-patch TNNetVolume.Add, no per-element mod. - BuildDetr: the spatial pos table is one constant shared by all 12 layers - fill it once and copy (#5 across calls); FillDetrSpatialPosEmbed builds the y-half per row and the x-half per column once, then two Moves per token. - BeitBuildRelPosIndex: (yj, xj) from the loop counters, no div/mod per key. - FillVideoMAEPosTable: one divide per sin/cos pair (#4); angle unchanged. - CollectRNNCells / NetHasProjectionBeforeCells: break on the single hit. Decode / inference - DecodeMask2FormerSemantic: sigmoid runs as a bulk pass per query over a 1024-pixel block (#13/#19); scratch sized once before the pixel loop. - DecodeYoloDetections: the DFL softmax and expected value share one pass, so the per-bin exp buffer is gone. - DecodeDetrDetections: the running winner lives in a local (#4). - DecodeCogVideoXVae: a cell's T frames are contiguous in both volumes, so the gather and the scatter are one Move each (App. C). - DiTDenoise / PixArtDenoise: y-outer walk with carried offsets, and a whole- grid Move when the output shape already matches (App. E). - Qwen2AudioProjectAudio: the pool folds add+halve into one MulMulAdd. - ClipPreprocessImage: carried work/destination offsets, hoisted crop bounds. Suite: 2843 tests, 0 failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uffered string builds neuralvolume.pas - new class function TNNetVolume.Sum(pSrc, N): AVXGetSum on AVX builds, scalar loop elsewhere - the pointer-and-count form of GetSum, so a single vocab row of a volume can be reduced without wrapping it (#13). neuralnlpmetrics.pas - ScorePerPositionWindow / ScoreSequence: per-row vocab sums -> TNNetVolume.Sum (#13). - PredictArgmaxAt: both argmax loops -> TNNetVolume.MaxPos (#13, ties still go to the lower index). - SelfBLEU: build each generation's n-gram maps ONCE and score every ordered pair from them (G*MaxN builds instead of 2*G*(G-1)*MaxN) via the new PairBLEUFromCounts, which is the single-pair case of CorpusBLEU evaluated on the same integers in the same order (#27). - CountNGrams: comma-joined key built through one reused buffer + write index instead of one concat per id (#23); unigrams keep the direct IntToStr. - TokenizeWithVocab: words cut with one Copy at the boundary, Ids presized and truncated once instead of grown per token (#23). - ChrF.Prep: presize + write index + truncate (#23). - PerplexityStrided: corpus stream grows geometrically instead of once per line (#23). - DecodeBIOEntities: Result presized to one span per tag, truncated once (#23). - ScoreCompletion / EvaluateLAMBADA: element copies -> Move (#13). - TopKIndices / ExtractQASpans: running best carried in a local, so the no-swap iteration costs one indirect load instead of two (#5). neuralcalibration.pas - ForwardProbsAndLogits: simplex detection -> GetSum/GetMin; softmax branch -> Move + GetMax + ExpShiftSum + Mul; pseudo-logits -> TNNetVolume.Ln (#13/#19). - NLLAtTemp: one scratch row per grid evaluation - Move + Mul(InvT) + ExpShiftSum - and the log-sum-exp identity ln(Acc) - (z_K - MaxV), which drops the label exponential and both clamps (Acc >= 1 by construction) (#13/#14/#19). FitTemperature owns the scratch row. - ComputeCalibration: Brier expanded to sum(p^2) - 2*p[true] + 1 over the vectorised dot product, floored at 0 (#13); bin clamp uses BinCountM1. - WriteReliabilityPGM: PGM rows built into a presized buffer (#23). neuralchat.pas - TJInterp: FOutput is a geometrically grown buffer written through FOutLen (Emit) and truncated once in Render (#23). - TJInterp.Tokenize: chunk stream grows geometrically, and each 'S' chunk caches its keyword/remainder split so FindMatchingEnd, ExecIf and ExecRange stop re-splitting the same chunk once per loop iteration (#23/#27). - PeekWord compares in place; ParseCompare tests characters directly instead of four Copy calls (#23). - Left-trim strips trailing whitespace by scanning back and truncating once. - RenderChatMLCore: presized buffer + Append/AppendCh write index (#23); output is byte-identical, assistant spans still index the final string. tests: TestSelfBLEU pins SelfBLEU bit-exactly (delta 0) against the naive per-ordered-pair CorpusBLEU formulation, smoothed and unsmoothed. Suite green on both the default and the -dAVX2 build: 2843 tests, 0 failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…axpys, hoist divides neuralnetwork.pas 30001-35000 sweep. Attention (decode-hot): - TNNetScaledDotProductAttention.ComputeIncremental and TNNetFusedSDPA.ComputeCachedToken split their key loops on FKVQuantInt8 and test a hoisted boolean for the soft-cap (#20); the eviction shift splits the same way, which is the guide's own worked example. - TNNetFusedSDPA.Backpropagate gains the `if A <> 0` guard its single-head twin already carries: a masked key's weight is exactly 0, so the dV MulAdd adds zero and dAttn[j] is only ever consumed multiplied by that same zero. Under a causal mask this skips about half the keys. - The same backward carries the attention-row offset and the KV group index instead of recomputing h*SeqLen and h div FGroupSize per head (#6/#12); ComputeCachedToken carries the group index too. - The prefill soft-cap test becomes a boolean local in Compute and ComputePrefillHeads. Losses and normalization: - TNNetArcFace.Backpropagate: the x term of dL/dx uses the SAME vector for every class, so its K coefficients sum into one scalar and the axpy runs once after the class loop instead of K times. - TNNetL2Normalize.BackpropagatePerChannel: the scalar depth loop becomes two depth-aligned FMAs (#13); the -invN*dot coefficient vector is built into FSumSqBuf, the forward scratch that carries nothing into backward. - TNNetMinMaxNorm.Backpropagate: sum_j gy_j*(y_j-1) = sum_j gy_j*y_j - sum_j gy_j, so both reductions become bulk ops (per-channel MulAdd+Add, full-volume DotProduct+GetSum). The full-volume comment claimed the scalar order was kept for bit-parity; this repo does not require bit-parity, and the numeric gradient checks (TestMinMaxNormGradientCheck and the per-channel twin) pass. FGYm1Buf is renamed FSumGYBuf to match what it now holds. - TNNetForgetGateBias.Compute fills the masked tail of each row with one FillDWord; its Backpropagate accumulates raw dlogit and applies -lr once. - TNNetModernHopfield normalizes its pattern-score row with one reciprocal and a bulk Mul; TestModernHopfield{Input,Weight}GradientCheck pass. - Both evidential layers share unit-level EvidentialSoftPlus/EvidentialSigmoid instead of four copies of a nested function FPC cannot inline, and their forward passes sweep the volume flat instead of nesting X/Y/channel. - Divide hoists (#21): mixture-density backward (3 divides -> 1), evidential-regression backward (6 -> 3, plus CSE of alpha+0.5, delta^2 and 2(1+nu)), evidential-classification backward (2K -> 1). - FourierMixFFT scales the inverse transform by 1/N, which is exact for a power-of-two N; the FFT path zero-fills the imaginary row with one FillChar. Tests: 2843/2843 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…nels
The forward scan ran channel-outer, so the inner timestep loop strode the
input, state caches and output by Depth and did ~10 scalar float ops per
element. The time axis is the only sequential one; the channel axis is
parallel and contiguous, so the sweep now runs timestep-outer and updates
all channels at once with Mul/MulAdd over the Depth-long run, following
TNNetDiagonalSSM.Compute. h_{t-1} is read straight out of row t-1 of the
FHre/FHim caches the sweep already writes, with a zero row for h_{-1}, so
no state copy is needed.
PrepareChannelConstants maps the six raw per-channel weights onto the
constants the kernels consume (lambda's real/imag parts, -imag for the
cross term, gamma*B, -Cim) once per forward, replacing the per-channel
prologue that ran two NeuralExp, a sincos and a Sqrt inside the old outer
loop. It is rebuilt every forward rather than cached behind a dirty flag:
TNNetLRU has no incremental-decode session, and trainers, importers and
gradient checks write Neurons[].Weights with no invalidation hook.
Measured (Release/AVX2, this box), forward only:
SeqLen=256 Depth=64 : 86 ms -> 34 ms (2.5x)
SeqLen=256 Depth=512: 783 ms -> 91 ms (8.6x)
Backpropagate is unchanged. Tests 2843/2843, including
TestLRUInputGradientCheck and TestLRUWeightGradientCheck.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…y of FOutput
The forward scan ran channel-outer, so the inner timestep loop strode Prev
by 3*Depth and evaluated three scalar transcendentals per element (two gate
sigmoids and the decay exp). Prev packs x, i_logits and a_logits as three
contiguous Depth-long blocks per timestep, so the sweep now runs
timestep-outer: TNNetVolume.Sigmoid produces both gates, TNNetVolume.Exp
the decay, and the state update is a Move plus two Mul and one MulAdd over
the Depth-long run. Only sqrt(1-a^2) stays scalar (there is no vector Sqrt
kernel), and it now walks a contiguous run. The -c*softplus(Lambda) factor
is per-channel, so it is one table built once per forward instead of a
softplus per channel inside the outer loop.
FH held exactly what FOutput holds - Compute wrote hNew to both at the same
offset, and Backpropagate only ever read h_{t-1}. It is removed; the
backward scan reads FOutput at baseC-ocStride. FOutput between this layer's
forward and backward is the same buffer every downstream layer reads as
FPrevLayer.FOutput in its own Backpropagate, so it is not written in
between. That saves a (SeqLen,1,Depth) buffer and one store per element.
Measured (Release/AVX2, this box), forward only:
SeqLen=256 Depth=64 : 377 ms -> 79 ms (4.8x)
SeqLen=256 Depth=512: 1876 ms -> 380 ms (4.9x)
Tests 2843/2843, including TestRGLRUInputGradientCheck and
TestRGLRUWeightGradientCheck.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
… the mean divide TNNetAconC.Compute and TNNetMetaAconC.Compute ran the same scalar body - d = (p1-p2)*x, y = d*sigmoid(beta*d) + p2*x - with a NeuralExp and a divide per element and three parameter loads on top. Both now call one ComputeAconMap on their common parent TNNetChannelTransformBase, which builds the two per-channel factors (p1-p2 and beta*(p1-p2)) once and then runs each pixel's contiguous depth row through Move/Mul/TNNetVolume.Sigmoid/ MulAdd, so the sigmoid is a vector kernel rather than a scalar exp per element. The two ACON layers were near-copies of each other, which is what put the map on the parent rather than in either child. Both backward bodies formed p1-p2 twice, beta*d twice and gy*xv twice per element; each is now named once. TNNetMetaAconC divided the squeeze by the pixel count per channel in the forward and again in the backward, plus once more per channel in the beta-path term: that is one reciprocal, a bulk scale, and a multiply. Measured (Release/AVX2, this box), forward only: AconC 32x32x64 : 427 ms -> 208 ms (2.1x) AconC 16x16x512 : 774 ms -> 418 ms (1.9x) MetaAconC 32x32x64 : 478 ms -> 221 ms (2.2x) Tests 2843/2843, including the AconC and MetaAconC gradient checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…window loops TNNetConvolution3D swept both directions with the output spatial bounds re-read per feature, the output depth index ot*NumFeat+f rebuilt per output pixel, the padded input coordinates oy*FStride-FPadding and ox*FStride-FPadding rebuilt per tap, and the frame/weight depth bases multiplied out per tap. Each now sits at the innermost level it is still invariant in, and the two depth bases are carried by FChannels (#6). The backward pass also formed -FLearningRate*gy twice per output pixel. TNNetCausalConv1D and TNNetTDNNConv1D rebuilt the padding span (Dilation*(Ksize-1) and Dilation*(Ksize shr 1)) on every timestep though it is fixed for the call, and their backward passes computed -FLearningRate*gy once for the bias and again for the weight delta. TNNetDepthwiseConv1D.Backpropagate rebuilt -FLearningRate*gy on every tap, read Prev and PrevErr through the [t,0,c] accessor per tap, and tested each tap for falling in the sequence. The tap range is now clamped once per timestep and one carried offset addresses both volumes. Tests 2843/2843. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…ling and log-det repeats TNNetEchoStateReservoir.Backpropagate rebuilt the FPre/FOutputError offset with a full GetRawPos per unit though only the trailing +i changes, and re-entered FWin.GetRawPtr(i,0)/FW.GetRawPtr(i,0) per unit though both row bases advance by a fixed stride - the same carries Compute already uses. The t>0 test inside the unit loop is a timestep property, so the t=0 step is peeled the way Compute peels it, leaving both halves branch-free. TNNetInvertible1x1Conv.Compute accumulated FLogDet by adding the same sum(log|s|) once per pixel; every pixel contributes the same amount and the inverse direction contributes none, so it is one product. TNNetAffineCoupling formed -FLearningRate*gspre and -FLearningRate*gt twice each per conditioner row, rebuilt the b-half slot index bOfs+r two or three times per row, and re-formed the address of the a-half of the previous layer's error twice per row though it is fixed for the pixel. Tests 2843/2843, including the affine-coupling and invertible-1x1 gradient checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…m as row kernels TNNetAffineCoupling.Compute evaluated pcr_tanhf and NeuralExp once per conditioner row inside the row loop. FS, FT, FSPre and both halves of the input are contiguous HalfB runs per pixel, so the row loop now only fills s_pre and t with its two dot products, and the clamp s = clamp*tanh(s_pre/clamp), the exponential and the transform itself (y_b = x_b*exp(s)+t, or (y_b-t)*exp(-s) for the sampling map) each run once over the whole row through Mul/Tanh/Exp/Add. exp(s) lands in one HalfB-long scratch sized in SetPrevLayer; the log-determinant sum stays a scalar add per row, which costs no transcendental. Measured (Release/AVX2, this box), forward only: 16x16x32 : 139 ms -> 83 ms (1.7x) 8x8x128 : 150 ms -> 93 ms (1.6x) Tests 2843/2843, including the affine-coupling gradient checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…eight taps TNNetCumSum: the depth-axis block-prefix scan was three passes (Move, scalar in-block prefix, carry broadcast) around a dependency chain that is sequential either way, so the blocking bought nothing. One pass reads the input column into the running sum and writes the output column; the backward pass keeps its scratch column and the vectorized span Add but loses the blocking. TNNetSparsemax: the tau numerator is already in hand when the kMax scan updates kMax, so the second cumulative-sum scan is gone (bit-identical). Backpropagate binds the previous layer's error volume instead of calling the property getter twice per element. TNNetPad non-zero modes: interior columns map one for one onto source columns, so a padded row is one contiguous Move (forward) / one span Add (backward) plus the 2*Padding border columns, instead of one Move per column with a NeuralPadSourceIndex call each. TNNetBilinearUpsample: a far-corner weight is exactly zero when the sample lands on a source centre (every s-th pixel at odd s, the clamped border, all pixels at s = 1); those taps are skipped rather than blending a Depth-long run of zeros. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…folded inference affine NeuralPerChannelMeanInvStd read the volume twice and staged a centered column per position (Move + MulAdd + MulAdd). One pass now accumulates both moments along the depth run and the variance comes from E[x^2] - mean^2, clamped at zero before the inverse root. The AdaIN gradient checks are unchanged by it (content 0.00340 -> 0.00322, style 0.00294 -> 0.00245) and forward AVX-vs-scalar parity still reports 4.77e-7. TNNetAdaIN.Compute: only the backward pass consumes the normalized-content cache, so when the content layer carries no error volume the map folds to one per-channel affine applied in place over the content copy inherited Compute already made - two column passes instead of six. TNNetLocalProduct: the per-window error share is a reciprocal fixed at SetPrevLayer instead of a divide per output element, and ComputeCPU broadcasts the position product over the depth run with a fill plus a copy. TNNetGroupedConvolutionLinear.BackpropagateCPU: the activation kind is resolved into IsReLU/IsIdentity once, as TNNetConvolution.BackpropagateCPU already does, instead of re-reading FActivationFn per output element. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
… cached device index map TNNetUpsample walked depth outermost and stepped Y by two output rows, so every depth slot re-traversed the whole output volume by column. Compute and ComputePreviousLayerError now walk source positions outermost with the output depth run innermost, which makes each of the four corner writes contiguous. The forward pre-fill is gone with it: the four corners tile the doubled grid, so every output cell is written exactly once. TNNetPixelShuffle device paths rebuilt the output-to-source index map on every call; it depends only on the layer geometry, so it is built once and SetPrevLayer empties it when the shape changes. PrepareInputForGroupedConvolutionFast: the destination column is hoisted above the feature window, and the tap offset plus both per-group offsets are carried by addition instead of a multiply per copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…and bound volumes TNNetSpaceToDepth / TNNetDepthToSpace resolved FPrevLayer.FOutput (and the OutputError property getter) plus two GetRawPtr multiply-adds inside the innermost block-cell loop. Both volumes are bound once, the block walks rows outermost, and the source, destination and block-column offsets are carried by addition: the destination slots of one output column are continuous, and a block step is one row stride or one depth run. TNNetCoordConv.Compute made two passes over the volume and re-tested the degenerate-axis condition per pixel. One row-major pass writes the passthrough copy and both coordinate channels through a shared column base, and a zero slope with a zero origin expresses the degenerate axis without the test. TNNetSimpleGate: one column base addresses both input halves and both error halves, the previous-layer error getter is bound once, and both loops walk rows outermost. TNNetGatherTokens.Backpropagate carries the output-error column offset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
…f one per channel TNNetChannelShuffle and TNNetReverseChannels called CopyFromDepthToDepth / AddFromDepthToDepth once per channel, and each of those is a strided sweep over the whole volume - Depth full traversals for one permutation. Both now walk positions outermost and permute each depth column in a single pass, with one column base indexing input and output (the layers are shape preserving). TNNetReverseXY and TNNetFlipX recomputed both GetRawPos offsets per cell. A double flip reverses the raw position order outright, and a horizontal flip reverses it within a row, so each side is a carried walk: destination forwards, source backwards, one row stride between rows. TNNetGridAvgPool carries the pooling-window cell offset (a depth run across X, a row stride down Y) instead of a GetRawPtr multiply-add per cell. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThMi76PXQdj6iFu6zYmUFp
The Wqkv slab holds whole q|k|v thirds (LoadModernBertQKVWeights permutes rows inside each head but never interleaves heads), so each band is one contiguous split and every head sits at HeadCnt*HeadDim inside it. The per-head split/rotary forest becomes three band splits plus ONE head-tiled TNNetRotaryEmbedding (pRotaryHeadDim = HeadDim) per q/k band, which repeats the per-head frequency schedule across the whole band - the Falcon recipe. With nothing per-head left ahead of the attention math, the GLOBAL layers (which mask nothing) fuse into one TNNetFusedSDPA each via AddGQAAttentionFromSources with CausalMask=false; MaskBand with FCausal false and FWindow 0 yields the full bidirectional band, and the non-causal fused path is already gradient-checked through AddMultiHeadGroupedQueryAttention. The LOCAL layers carry the symmetric sliding window, which the fused layer has no argument for, so they keep per-head SDPA - they just share the hoisted band rotary now. The hidden-state parity test moves its HF oracle ahead of the structural counts and asserts the new shape. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
… loaders Four load-time cleanups, all mechanical: - Nemotron-H LoadChannelVector copied its channel vector one float at a time; it is a contiguous run, so it becomes one Move (with a dest-size check, since a Move has no bounds of its own). - Nemotron-H LoadDepthwiseConv copied each channel's kernel row with a scalar inner loop and a per-element multiply for the source index; one Move per row with the source offset carried by addition. - The Jamba MoE expert loop built each of its three tensor names TWICE (once to load, once to mark consumed) from four concatenations each; the expert prefix and the three names are now built once per expert. - Both ModernBERT slab loaders re-derived r*Hidden and the row byte count per row, re-tested B <> nil inside the loop, and indexed the neuron mirror twice; the offset is carried, the size and the bias-present verdict are hoisted, and the neuron is bound once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
…fills FlushWeightCache rebuilds a layer's whole bias output and concatenated / interleaved weight cache. Four import sites fill several DISJOINT neuron ranges of the SAME layer in a row and paid that rebuild once per range: - LoadQwen35DeltaNetWeights: in_proj_z / _b / _a into LinZBA (3 rebuilds per hybrid block, now 1); - the Qwen3.5-MoE, Llama-4 and granitemoe shared-expert SharedGateUp up/gate halves (2 rebuilds, now 1); - LoadGraniteMoEExperts' streamable branch: 2*NumExperts rebuilds of the fused gate|up bank, now one after the loop (the staged fallback already flushed once). Deferring is safe here: the cache is derived state rebuilt in full from the neuron weights and biases, the loads only write neuron volumes, and EnsureWritableImportWeights' dequantize does not read it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
…stream rows Qwen3.5's double-width q_proj is imported as 2*NumHeads slab slices (a query half and a gate half per head). LoadLlamaLinearWeights row-streams a slice only when the reader offers a row view; packed NF4, quantized GGUF and the synthetic view readers do not, so each slice fell back to LoadTensorFlat of the WHOLE slab - 2*NumHeads full decodes per block. LoadLlamaLinearWeights gains pSrcSlab: an already-materialized flat FP32 [SrcRows, InDim] copy of the tensor to slice from RAM instead of the reader, built by the new StageTensorSlabFlat (which expands packed NF4 the same way the internal fallback did). The Qwen3.5 q_proj caller stages the slab once when the reader cannot stream rows, and is otherwise unchanged - streamable readers keep per-slice streaming and never pay the slab-sized transient. A staged slab and the direct-to-int8 row path are mutually exclusive by construction (int8 requires row streaming), and the staged size is validated against SrcRows x InDim. TestQwen35QProjStagedSlabSliceParity loads every head half both ways from the fixture checkpoint and asserts bit-identical neuron weights and biases, pinning the rotate_half permutation and the Scale fold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
LoadVaeConv is the workhorse of the VAE and SD-UNet import (66 call sites) and was the un-optimized twin of LoadResNetConvFoldBN: it re-fetched FArrNeurons[o].Weights through the list accessor on every element and recomputed the whole source offset per weight. Mirror the ResNet loader -- bind the neuron and its weight volume once per output channel, hoist the kernel-slot base, carry the source index with its fixed K*K stride, and take the contiguous Move for K = 1. Also in the same importers: - ViT encoder blocks fill Q/K/V into disjoint neuron ranges of one fused QKV layer, so only the V call has to rebuild the concat cache (pDeferFlush). - MaxViT window attention: every window slices the same qkv slab, which a reader with no row view decodes in full per slice. Stage it once for those readers and slice from RAM, the idiom already used for the Qwen3.5 q_proj. - MaxViTBuildHeadBias carries the flat index instead of forming i*p2+j twice. - The SigLIP probe copy is a contiguous Move. - ReparamRepVGGBlock builds the two conv tensor names once instead of eight times each. RunMaskRCNN gains an explicit pFeatsUnchanged opt-in: the FPN backbone does not depend on the proposal box, so a second box over the same feature maps restarts the forward pass at the first RoIAlign (AddLayerAfter always appends, so the layer list is topological and every box consumer sits at or after that index). It also skips the FPN levels below the chosen one. Default is false, so existing single-box callers are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
…e check TestIPAdapterInferenceNet compared the inference-built net against the trainable one with an exact-equality assertion. It has failed since it was introduced in 2dacda4 ("fix(ipadapter): honour pTrainable net-wide and free the net on import error"), but only in a -dRelease -dAVX2 -dAVX64 build: max |diff| = 1.49e-8, about one float32 ulp. The net-level `NN.SetTrainable()` that commit added takes the default pLowMemory=True, so the inference net's pointwise convolutions reduce through ComputeLowMemoryCPU (per-neuron dot products) while the trainable net reduces through DotProductsTiled over the concatenated weight cache. The two accumulation orders agree bit-for-bit in a scalar build and diverge by a rounding step once the tiled kernel vectorizes, so the assertion was only ever true by accident of the default build flavour. Pinning the memory mode is not available here: with low memory armed AfterWeightUpdate never builds FConcatedWeights, so clearing the flag after the weights are loaded would leave ComputeTiledCPU reading an empty buffer. Assert the outputs agree to 1e-5 instead, and record why in the test comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
The old name claimed a whole-GPU fp16 mode; the flag only narrows the activation (B) operand of the int8 OpenCL matmuls, and the user wants the flag to stay free for future half-precision experiments on other backends. The lifecycle word now lives in the flag, so the duplicate "experimental" prose was dropped from --help and the startup notice. Renamed: the CLI literal, TChatOptions.GpuFP16 -> ExperimentalFP16, the ChatTerminal self-test, and the README row. Unchanged on purpose: TNNet.OpenCLFP16, TNNetConvolutionBase.ShouldOpenCLFP16/FP16Active and TDotProductSharedKernel.FFP16Activations name the effect, not the flag. No alias for --gpu-fp16. Tests: 2932/2932 pass; ChatTerminal --selftest OK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A4YUGhHrXod7QVZsEZdKSw
TVolume.ForceMaxAbs printed the pre-clamp maximum with a formatted WriteLn whenever it rescaled. A numeric library routine must not write to stdout: in a GUI or service host with no console FPC raises EInOutError 105 on that call, so a clamp that works in a console build faults elsewhere, and the write would dominate the cost if the routine were called per batch. The clamp itself is unchanged. TestVolumeForceMaxAbs pins the semantics: an out-of-range volume is scaled so max|x| equals the bound with the cell ratios kept, and an in-range volume is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
…ke nil PointwiseNorm and PointwiseMul both carry a row base across the Y loop and step it by RowStride. The storage-order rewrite (6d02234) dropped the RowStride assignment when it renamed colBase to rowBase, so both routines advanced the base by an uninitialized value and read past row 0 for any volume with SizeY > 1. Assign RowStride := FSizeX * FDepth again in both. PointwiseMul also tested Assigned(pNorms) at the ReSize but dereferenced pNorms per element, so a nil call crashed instead of doing what the guard promised. pNorms is optional in PointwiseNorm, so make it optional here too: no recorded norms means no scaling to reapply, and the routine exits before the loops rather than testing a pointer per element. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
TVolume.SoftMax wrapped its whole body in "if ShiftedMin <> 0", so an all-equal volume kept its raw values and returned 0 instead of the uniform 1/N with total sum N. The guard was only needed by the low-end rescale, which divides by ShiftedMin and already runs solely when ShiftedMin < -1000; the shift/exp/normalize path now always runs, with an empty volume the one case that exits early. An all-zero logits row - a masked row, or a layer that has not learned yet - therefore stayed at zero, and TNNetSoftMax.Backpropagate reads that output as y, so the Jacobian gradient was zero too. Sibling PointwiseSoftMax already normalized a constant span. The cai_softmax device kernel carried a matching all-equal early return; it is gone, so both paths agree again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
…nRange The scan seeded SmallestValue from FData[StartPos] and then started the loop at StartPos, comparing that element with itself. The comparison is strict (<), so the extra iteration never changed the result; it was only wasted work, and the "if FinishPos > StartPos" guard around the loop only skipped an iteration a Pascal for-loop already skips on its own. The loop now starts at StartPos + 1 and the inner guard is gone. The outer "FinishPos >= StartPos" guard stays: it is what keeps an empty range (Len <= 0) returning 0 instead of StartPos. No sibling routine has this shape; GetSmallestIdxInRange is the only *IdxInRange in TVolume. New TestVolumeSmallestIdxInRange covers a single-element range, a normal range, a tie (first position wins), the minimum sitting at StartPos, the clip to the volume size, an out-of-range start, and an empty range. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C8KE61qj4dXF79vKgzA3Zr
First step of int8-input convolution. TNNetVolume.DotProductInt8Int8 returns the exact int32 sum of code_a[i] * code_b[i]; the caller applies scale_a * scale_b once per output. Codes must stay in [-127, 127], the QuantizeInt8 range. AVX2 kernel AVXDotProductInt8Int8: a*b = |a| * (sign(a)*b) via vpabsb/vpsignb feeds vpmaddubsw, vpmaddwd against in-register int16 ones widens to int32 (no memory constant, so nothing PIC-sensitive), vpaddd accumulates; 32 elements per iteration, Pascal tail. Plain Pascal loop on every other build. Nothing existing computed an int8 x int8 sum: DotProductInt8 takes an FP32 B operand. Tests: exact int32 length sweep around the 32-element block with the +-127 worst-case pairs, and agreement with the DotProductInt8 FP32 path. Suite 2939/2939 on both plain and -dAVX2 builds; ChatTerminal lazbuild links. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
Step 2 of int8 x int8 convolution. TNNetConvolutionBase gains FInputCopyInt8 / FInputPreparedInt8 (TNNetVolumeQuant8) and one per-tensor activation scale FInputScaleInt8. EnableInt8Input sizes both buffers once after SetPrevLayer (nil by default, since most convolutions never run int8 input); QuantizeInputInt8 runs MaxAbsFinite + QuantizeInt8 over the whole FInputCopy with the QuantizeInt8RowTolerant scale (MaxAbs/127, zero input -> zero codes and scale 1); PrepareInputForConvolutionInt8 is the byte Move im2col in exactly the layout of PrepareInputForConvolutionFast. Pointwise aliases FInputPreparedInt8 to FInputCopyInt8 as the FP32 path does. Neither per-forward routine allocates. Padding is code 0 because CopyPadding zero-fills the border. Not covered: TNNetGroupedConvolutionLinear's grouped im2col layout. Tests: padded 3x3, stride-2, pointwise, and not-enabled cases comparing scale * code to the FP32 FInputPrepared within half a step. Suite 2943/2943 on plain and -dAVX2 builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
Step 3 of int8 x int8 convolution. TNNetVolume.DotProductsTiledInt8Int8 mirrors DotProductsTiledInt8's tiling and output layout with DotProductInt8Int8 as the inner kernel and one FP32 store per output (acc * Codes.Scale[row] * BScale). TNNetConvolution.ComputeInt8Int8CPU runs QuantizeInputInt8, PrepareInputForConvolutionInt8, the tiled int32 reduction, bias and activation. Compute takes it when ShouldComputeInt8Int8CPU holds (int8 weights, EnableInt8Input called, not OpenCL, shapes agree) and then skips the dead FP32 im2col. ChunkEligible excludes int8-input layers: ComputeRange still runs the int8 x FP32 kernel. Measured (AVX2, 64x64x64 input, 3x3, 64 neurons, pad 1, single run under the 3 GB ulimit): 23.90 ms/forward int8 weights x FP32 input -> 8.45 ms/forward int8 x int8 (2.8x). Tests: padded/strided/pointwise parity against the int8-weight path within the half-step input quantization bound, 5% max-abs check against FP32, and the not-enabled layer keeps the old path. Suite 2947/2947 on plain and -dAVX2 builds; ChatTerminal lazbuild links. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
…ages The four EnableInt8Input/QuantizeInputInt8/PrepareInputForConvolutionInt8 messages hard-coded TNNetConvolutionBase, so a TNNetConvolutionReLU named a class that is not in the net. ClassName names the real layer, as the other base-class messages in this unit do. TTestNeuralLayers 92/92 on plain and -dAVX2 builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
EnableInt8Input/DisableInt8Input are now virtual no-ops on TNNetLayer, declared beside EnableOpenCL (outside the OpenCL ifdef), so TNNet can walk every layer. TNNetLayerConcatedWeights - the class that owns the int8 weights - owns FInputCopyInt8, FInputScaleInt8, AllocInputCopyInt8, the virtual Int8InputSource (FPrevLayer.Output by default), the two overrides and QuantizeInputInt8. TNNetConvolutionAbstract returns FInputCopy as the source and sizes the copy with FPadding (source and sizing on the same class, so TNNetDepthwiseConv cannot mismatch). TNNetConvolutionBase keeps only FInputPreparedInt8 and the byte im2col. DisableInt8Input runs from DequantizeWeightsInt8 and the destructor. TNNet.EnableInt8Input arms every quantized TNNetLayerConcatedWeights and returns how many actually armed; TNNet.DisableInt8Input reverses it. It must run after BuildQuantInt8/QuantizeWeightsInt8: an FP32 layer is skipped. The eight existing int8-input tests pass unchanged. New tests: net-wide count on a mixed net, DequantizeWeightsInt8 drops the buffers, TNNetFullConnect input quantization. Suite 2950/2950 on plain and -dAVX2 builds; ChatTerminal lazbuild links. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
ChatTerminal/ChatServer option ExperimentalInt8Input: after the weights are int8 and after EnableOpenCL, the engine calls TNNet.EnableInt8Input and prints the armed layer count as the confirmation that the ordering held. --fp32 makes it a reported no-op, like --experimental-fp16. Today only TNNetConvolution has an int8 x int8 kernel; the fully connected blocks of an LLM arm the input copy but still run int8 x FP32, so on ChatTerminal's models the flag changes nothing yet. The help text and the README row say so. ChatTerminal and ChatServer lazbuild -B link; --selftest 0 FAIL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
The step-3 ChunkEligible exclusion took every int8-input layer off the intra-layer threading; on Qwen3.5-4B (--cpu --low-memory) that was 169 TNNetPointwiseConvLinear layers at 387 -> 1716 ms/forward and decode 5.5 -> 1.7 tok/s. PrepareChunkedForward now runs QuantizeInputInt8 + PrepareInputForConvolutionInt8 once, after PrepareForwardPrologue and before the scheduler publishes any slice (SchedEnqueueReady calls it single-threaded first), and skips the FP32 im2col the chunks no longer read. ComputeRange takes the ranged DotProductsTiledInt8Int8 on both chunk axes, with the shared bias and activation tail. The exclusion is gone. Tests: chunked vs serial int8 x int8 output bit-identical (tolerance 0) on padded, pointwise and neuron-axis shapes, FP32 im2col proven skipped on the chunk path, ChunkEligible True with the int8 input armed. Suite 2952/2952 on plain and -dAVX2 builds; ChatTerminal lazbuild links. Synthetic 16 x (1x1x2048 -> 2048) pointwise stack, threading on, AVX2, single run under the 3 GB ulimit: int8 x FP32 27.85 ms/forward, int8 x int8 14.45 ms/forward. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
TNNetDeconvolution overrides Compute with the transposed scatter but inherited TNNetConvolution.ChunkEligible and ComputeRange, so under intra-layer threading a deconvolution ran the forward-convolution ranged kernel. ChunkEligible now returns False. Test: on a threaded net the sibling TNNetConvolution is chunk-eligible and the deconvolution is not. TTestNeuralLayers 98/98 on plain and -dAVX2 builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
Step 1 of --int4: block-of-32 int4 weight storage and the kernels that consume it, no layer wiring yet. TNNetVolumeQuant4 keeps one row per (x,y) in the GGUF Q4_0 layout: per block of 32 codes, 16 packed bytes (low nibbles = elements 0..15, high nibbles = 16..31, code biased by 8) and one float scale, quantized with the Q4_0 rule (the largest-magnitude value maps to code -8, so the scale carries its sign). A Q4_0 checkpoint row can therefore be copied in as it is (ImportPackedRow). ReSize refuses a depth that is not a multiple of 32. TNNetVolume.DotProductInt4Int8 returns sum over blocks of BlockScale * sum(code_w * code_b); the caller applies the one input scale. The AVX2 kernel processes two blocks per iteration: one 32-byte packed load unpacked by mask + subtract 8, two vperm2i128 to line the input up with the nibble order, then the sign-magnitude vpmaddubsw chain of AVXDotProductInt8Int8 and a per-block FMA with the block scale. Constants are built in registers (PIC-safe). Float accumulation per lane means it is not bit-exact against the Pascal loop. DotProductsTiledInt4Int8 (full and ranged) mirrors DotProductsTiledInt8Int8 with the A rows addressed by packed bytes and block scales. Nothing in neuralvolume.pas held int4 codes before; neuralgguf.pas parses Q4_0 blocks but dequantizes them to FP32 at load. Single-core AVX2 GEMV 8192x4096 (noisy box, best of 4): int8 x int8 2.05 ms, int4 x int8 2.13 ms - parity per call at half the weight bytes. Tests: Q4_0 layout and geometry, quantize round trip (one-code bound on the saturating side), kernel vs dequantized reference over 0..33 blocks, tiled full + ranged vs DotProductsTiled. 2957/2957 on plain and -dAVX2 builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
Step 2 of --int4: the layer side of the Q4_0 weight path, no switch yet. TNNetLayerConcatedWeights gains FQuantInt4/FQuantTableInt4 and the virtual QuantizeWeightsInt4: Q4_0-quantizes each neuron row (from the FP32 weights, or by requantizing the int8 table - the direct-load route until a Q4_0 row importer exists), shrinks the FP32 rows as QuantizeWeightsInt8 does, and enables the int8 input copy, since the only int4 kernel is int4 x int8. It refuses, leaving the layer FP32, when the row size is not a multiple of 32, when the class has no int4 forward (SupportsInt4Weights, true only on TNNetConvolution), or when OpenCL is already enabled. EnableOpenCL returns early on an int4 layer, so WillOpenCL stays false. TNNetConvolution routes both quantized-weight x int8-input forwards through one predicate (ShouldComputeQuantInt8InputCPU) and one ranged helper (DotProductsQuantInt8Input) that picks the tiled kernel by FQuantInt4; ComputeInt8Int8CPU is renamed ComputeQuantInt8InputCPU. Compute, PrepareChunkedForward and both chunk axes of ComputeRange use them; an int4 layer without a usable int8 input path raises instead of reading the shrunk FP32 rows. Winograd and Backpropagate refuse int4 as they refuse int8. TNNet.QuantizeWeightsInt4 sweeps the layers that SupportsInt4Weights and returns how many hold int4 weights afterwards. Tests (7): padded/strided/pointwise parity against FP32 (exact bound 0.5*InputScale*sum|w| against Q4_0-rounded weights, relative L2 < 12% against plain FP32 - Q4_0's uniform rounding error is ~7-10% on Gaussian weights), the int8 -> int4 route, the 27-element refusal, chunked == serial bit-identical on both axes, and the sweep count. 2964/2964 on plain and -dAVX2 builds; ChatTerminal links. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
Step 3 of --int4. TChatOptions.Int8 becomes WeightMode: TChatWeightMode (cwmFP32, cwmInt8, cwmInt4); the last of --fp32/--int8/--int4 on the command line wins, int8 stays the default. --int4 loads exactly like --int8 (the checkpoint streams into int8 rows, no FP32 spike), then the chat engine calls TNNet.QuantizeWeightsInt4 BEFORE EnableOpenCL - a layer refuses int4 once OpenCL is enabled on it - and prints how many layers hold Q4_0 int4 weights. The int4 layers run on the CPU; with --gpu a notice says so and the remaining layers keep OpenCL. --experimental-fp16 is ignored under --int4 as under --fp32; --experimental-int8-input still arms the remaining int8 layers, and its notice says the count now includes the int4 layers. The int8 KV cache default follows --int4 as it follows --int8. README flag-table row and --selftest cases (default, --int4, --int4 --fp32, --fp32 --int4, KV default). ChatTerminal --selftest: 105 PASS / 0 FAIL; ChatTerminal and ChatServer link. No real-model run: no checkpoint on this box. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
…ts with block sums sum((u-8)*b) = sum(u*b) - 8*sum(b): the unsigned nibble goes straight into vpmaddubsw's u8 operand and the bias is undone by one short float dot of the row's block scales against 8*sum(b) per block. Per pair of blocks the loop loses vpsubb x2, vpsignb x2 and vpabsb x2 (6 of 22 ops); the correction costs 3 ops per 8 blocks. The block sums are input-only, so they are computed once per forward: TNNetVolumeQuant8 gains a float plane (EnableBlockSums allocates it once, ReSize keeps it, ComputeBlockSums fills 8*sum per block of every row). TNNetConvolution.QuantizeWeightsInt4 enables the plane on FInputPreparedInt8; the new PrepareQuantInt8Input (shared by Compute and PrepareChunkedForward) fills it after the byte im2col when the weights are int4. ShouldComputeInt4Int8CPU also requires the plane, so a missing plane routes to the existing raise instead of a nil read. DotProductsTiledInt4Int8 raises when VBs has no block sums. No allocation on the forward path. Timing on this box cannot resolve the change (int4/int8 ratio per binary old ~1.13, new ~1.07, both +-0.3 across interleaved runs); parity tests cover the correction on both builds. 2964/2964 plain and -dAVX2; ChatTerminal links. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
TNNet.EnableInt8Input visited only WeightsQuantizedInt8 layers, so under --int4 it skipped the int4 layers (which arm the copy themselves) and the --experimental-int8-input notice reported "0 layers" while its text claimed the count included them. It still arms the int8 layers only, and now counts every TNNetLayerConcatedWeights whose InputCopyInt8 is assigned afterwards. TestNetQuantizeWeightsInt4CountsConvertedLayers asserts the count is 2 on a net with no int8 layer. TTestNeuralLayers 105/105 on plain and -dAVX2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
Under --int4, a GGUF Q4_0 weight tensor whose target layer SupportsInt4Weights is copied into TNNetVolumeQuant4 rows as it is: same codes, the block's f16 d widened with the same DecodeF16 DequantizeQ4_0 uses, no FP32 and no int8 row in between. Every other tensor keeps the dequantize -> int8 -> requantize route and TNNet.QuantizeWeightsInt4 converts it after the load. Layer: QuantizeWeightsInt4's tail is now FinishInt4WeightConversion (shared; the TNNetConvolution override enables the block sums there, so both routes get them) and the import API is BeginInt4QuantImport (eligibility + sizing) / ImportInt4QuantRow / EndInt4QuantImport (refuses unless every neuron row was imported). Reader: TNNetGGUFReader.LoadTensorPackedRowsQ4_0 serves row ranges of packed blocks + widened scales, sharing ValidateRowStreamRange with LoadTensorRowsFlat; the q/k de-interleave moves whole rows, so those projections stream packed too. The GGUF writer gains gwQ4_0 so tests can build a Q4_0 fixture. Loader: LoadLlamaLinearWeights takes the direct route only when this call fills the whole layer (NeuronBase 0, every neuron) - a layer filled by several calls (fused gate/up halves, per-head q slices) would end up half int4 and half int8 - and folds a row Scale into the block scales, which is exact. The route is switched by the unit global NeuralImportInt4FromQ4_0 (630 call sites made a parameter too invasive) and counted in NeuralImportInt4LayerCount; the --int4 notice reports direct vs requantized layers. Tests (4): Q4_0 requantize-after-dequantize is byte-identical and ImportInt4QuantRow lands those bytes; ineligible layers refused; packed rows dequantize bit-identically to DequantizeQ4_0; a tiny Llama written as Q4_0 direct-loads 10 layers (q/k/v/o/down x 2 blocks) with logits at least as close to the FP32 forward as the old route, and the Q8_0 file direct-loads 0. 2968/2968 on plain and -dAVX2; ChatTerminal and ChatServer link; --selftest OK. Not measured: load time / RAM on a real Q4_0 checkpoint (none on this box). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
…ermps per scale pair The two vperm2i128 that lined the input up with the nibble order were the same permutation for every neuron row, so TNNetVolumeQuant8 now keeps the input row in that order once per forward: EnableInt4InputPlanes (renamed from EnableBlockSums) allocates a paired-codes plane beside the block sums, ComputeInt4InputPlanes fills both - per pair of blocks k,k+1 the 64 codes as [k 0..15][k+1 0..15][k 16..31][k+1 16..31], an odd last block in order. The kernel takes PairedCodesRowPtr, folds the input loads into vpmaddubsw's memory operand, and expands the two block scales with one vpermps against an index vector built at entry, replacing two broadcasts and a blend. Per pair of blocks the loop goes from 16 to 12 instructions. The Pascal fallback reads the same layout, so the parity tests cover it on both builds. NumElements must be the input row's depth or a prefix of whole block pairs (the paired order is a property of the row); the kernel test now builds its input row per length. No allocation on the forward path. Measured by the user on Qwen3.5-4B (8 workers): the previous kernel pulled ~22 GB/s where int8 x int8 pulled ~30 GB/s, so the headroom was in the instruction count. 2968/2968 on plain and -dAVX2; ChatTerminal links. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
Adds cai_dot_product_int4_splitk, the Q4_0 twin of the int8 split-K pass 1. The packed codes ride the device in the interleaved pair layout packed[a + p*FNumAs] (p = k/2, low nibble = code k, high = code k+1) with one float scale per (row, block of 32) in scales[a + blk*FNumAs], so adjacent work-items still read adjacent bytes at half the int8 traffic. The block scale is folded into the partial inside pass 1, which lets the existing cai_dot_product_int8_splitk_reduce finish the job unchanged with a per-row scale of 1.0 - no reduce twin. Slabs are whole blocks and KSplits may be 1, so there is no single-pass int4 kernel. TDotProductSharedKernel gains PrepareForComputeInt4 / ComputeInt4 / Int4Ready; ComputeInt8 and ComputeInt4 share ComputeResidentCodes and PrepareBiasOperand. TNNetLayerConcatedWeights.PrepareInt4DotCL repacks FQuantTableInt4 (the Q4_0 row keeps elements j and j+16 in one byte) and EnableOpenCL no longer skips int4 layers; TNNetConvolution's device route and its resident-source binding require Int4Ready, and ComputeOpenCLInt8 is renamed ComputeOpenCLQuantized. QuantizeWeightsInt4 still has to run before EnableOpenCL, the same ordering the int8 arming has. Activations stay FP32 on the device (no half twin yet). TestInt4OpenCLParity checks the device against an FP32 twin net whose rows went through the same Q4_0 quantizer: max |diff| 2e-6 on the decode GEMV (16 slabs), the KSplits = 1 shape, and two spatial shapes through the device im2col. ChatTerminal's --gpu notice and README row updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
On the L4 the --int4 run spends 21.2 ms per decode token while the net forward accounts for about 14.5 ms, so a third of decode is host work outside TNNet.Compute that no report shows. GenerateFromIds now accumulates the decode loop in six phases - net forward, logits copy + softmax, processor chain (repetition penalty), sampler, detokenize + emit, and the remainder - and --stats prints their per-step means on a second line after the tok/s line. The marks are Now() reads, six per token, on both the sampled and the greedy path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
…ioning the vocabulary The --stats phase line on the L4 put TNNetSamplerTopP.GetToken at 3.8-4.2 ms per token over a 248k vocabulary - 18% of an int4 decode step - all of it in GetTokenArray (a 2 MB window build) and the quickselect over that window, whose data-dependent branches mispredict on a softmaxed row. GetToken now tries LoadNucleusCandidates first: one GetMax pass and one linear gather (TNNetVolume.GetTokenArrayAbove) of the tokens scoring at least max/1024, written straight into the window. When the gathered mass reaches TopP the nucleus provably lies inside that set, because every token left out scores below every token kept; the few dozen entries are sorted whole and the cumulative scan and weighted draw run unchanged (SampleFromGatheredNucleus). Otherwise - a flat row, or TopP >= 1 - the full-window path runs exactly as before. SampleFromNucleus is split into NucleusWidth and DrawFromNucleus so both routes share the scan and the draw. GetTokenOnPixel takes the same route. Dev-box A/B at 248320 tokens, AVX2: 13.6-15.4 ms -> 0.6-0.8 ms per token. TestTopPLargeVocabGatherMatchesFullPath draws 200 paired seeds through both routes and requires the same token; TestGetTokenArrayAboveGathersAndSums covers the gather and its pixel variant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvNmDwHhRiv2PNAA1oDBQT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.