Rethinking Neural Thickets Without Weight Perturbations

Nicolas Dufour1, Elouan Gardès2,3, Amil Dravid4, Patrick Pérez5*, Loic Landrieu3

1Kyutai  ·  2Meta FAIR  ·  3LIGM, CNRS, Univ Gustave Eiffel, ENPC, Institut Polytechnique de Paris  ·  4UC Berkeley  ·  5Google DeepMind
*Work done while at Kyutai.

Neural Thickets perturbs every weight of a language model, keeps the copies that score best on a small labeled set, and has them vote. We show that a random bias on the output logits, with all other weights unchanged, matches or exceeds these gains on a diverse set of tasks. We also extend the method to non-generative models such as DINOv3.

Each pair of lines is one perturbation drawn twice with the same final logit shift. Top: the shift builds up over the 32 decoder blocks. Bottom: it is added at the output bias. Schematic.

The logits are enough

A random bias on the output logits matches full-weight perturbation: MATH-500 goes from 63.3 to 81.3 on OLMo-3-7B-Instruct, against 74.7.

Weight noise acts through the logits

To first order, a weight perturbation is a shift of the logits. At the noise scale used, this reproduces 89.7% of argmax changes.

It carries over to vision

Under domain shift, perturbing the logits of a DINOv3 probe adds 6.2 points of worst-region accuracy on FMoW; perturbing the weights adds 1.0.

1 · BackgroundNeural Thickets and perturb, select, vote

Neural Thickets improves a pretrained language model without training. Gan and Isola add random noise to its weights, keep the noisy copies that score best on a small labeled set, and let them vote.

We refer to this procedure as perturb, select, vote (PSV). It is defined by what is perturbed, a labeled selection set and a voting rule:

  1. PerturbDraw many copies of the model, called members. Neural Thickets adds Gaussian noise to every weight: each member is θ + σε with ε ~ 𝒩(0, I).
  2. SelectScore each member on 200 labeled examples and keep the best K, with K = 50.
  3. VoteThe kept members answer each test question, and the majority answer is returned.

With a few thousand members, they report these gains in accuracy points over greedy decoding, at a small fraction of the cost of fine-tuning:

+20.2Countdown
OLMo-3-7B-Instruct
+13.1MATH-500
OLMo-3-7B-Instruct
+52.8Countdown
Llama-3.1-8B-Instruct

They explain the gain geometrically: the pretrained weights would lie in a dense region of task-specific models, a thicket, that random perturbations can reach. We keep the procedure and test whether the gain depends on perturbing the weights. The figure below runs it on a toy question.

24 copies, 8 kept. The paper uses 1000 to 5000 copies and keeps 50.

PSV with noise on every weight (schematic). Left: each copy of the model traced through the 32 decoder blocks. Middle: the validation accuracy of each copy, then, for the kept copies, the shift of their logits toward the correct answer on one test question. Right: the vote of the kept copies and, for comparison, the vote of all 24. The question and the scores are illustrative.

2 · Localized perturbationsOne block is enough

Noise in a single decoder block gives about the same gain as noise in every weight.

If the gain required a coordinated change of all the weights, perturbing a small part of the network should lose most of it. On Countdown with OLMo-3-7B-Instruct (K = 50), the unperturbed model scores 66.6% and full-weight PSV 90.8%. Perturbing a single decoder block, with the other 31 fixed, improves on the base model for every block; the later blocks come close to 90.8%, and the last one reaches 91.8%.

The same holds for single matrices of the first block:

  • components that write into the residual stream recover the full gain;
  • the value and feed-forward input projections recover 96 to 99% of it;
  • the query and key path, which acts through the attention softmax, recovers 85 to 89%.

Qwen2.5 from 0.5B to 3B shows the same pattern.

One block at a time

Countdown accuracy of OLMo-3-7B-Instruct (K = 50) with a single decoder block perturbed.

One matrix of block 0

writes into the residual streamQK-normother matrices

Share of the full-weight gain recovered by perturbing a single component of the first decoder block (Countdown, OLMo-3-7B-Instruct, K = 50).

3 · Perturbing only the outputLogit PSV

A random bias on the output logits matches or exceeds full-weight PSV on most reasoning tasks.

Section 2 shows that the perturbation can be much smaller than the full network. We therefore look for the minimal perturbation that can still explain the gains: we keep the network fixed and perturb only its output. Each member draws one Gaussian vector over the vocabulary and adds it to the logits, at every decoding step and for every input:

ℓi = ℓ + σ εi,   εi ~ 𝒩(0, I)

This amounts to perturbing the output-layer bias, one value per vocabulary token, with all other weights unchanged. Selection and voting are unchanged. We call this logit PSV.

A toy example

Adding a bias to the logits multiplies the probability of each token by a fixed factor. The example below applies one such bias to eight validation questions. Most members do no better than the unperturbed model. A few fix several questions at once, for example by lowering a token that the model predicts too often, and selection keeps those.

Bias of this memberadded to the logits of every question

1.00

Members by validation accuracy

Questions correct out of 8. Dashed: the unperturbed model.

Toy example. Made-up logits for one decoding step, the first digit of the answer, on eight validation questions. Bars show the member's next-token probabilities; ticks show those of the unperturbed model. Each member draws one bias and adds it to the logits of all eight questions; the numbers under the bias are the factors it applies to each token's probability before renormalization. Selection keeps the members with the highest validation accuracy, whose accuracy on test questions can be lower.

Results

The figure below runs PSV with the noise in four places, from every weight to the output bias only; the last row is logit PSV.

24 members per run, 8 kept, a new test question each run.

Schematic. The same procedure in every row: members are placed at their validation accuracy, the best 8 of 24 are kept, and they answer a test question that the unperturbed model gets wrong (36 instead of 26). The paper uses 1000 to 5000 members and keeps 50.

Across four models, logit PSV is more accurate than full-weight PSV on 13 of 15 reasoning cells and within 0.6 points on the other two. It is usually slightly lower on MBPP and ROCStories. Most of these differences are within test-set noise.

Accuracy at K = 50

unperturbed (greedy)full-weight PSVlogit PSV

N = 1000 members and K = 50 for every method (N = 2000 for full-weight PSV on Countdown). The logit noise scale is fixed per model. USPTO is omitted: there, higher accuracy mostly comes from predicting the majority reaction class more often (see the paper).

Selection and voting

In terms of logits, PSV combines three known components:

  • Perturbation makes the members' outputs differ.
  • Selection on a validation set keeps the members whose shift helps across many examples. It is similar to verifier-guided selection, but done once before test time.
  • Voting is self-consistency over members.

To isolate selection, we compare with temperature-tuned majority voting (TT-MV), which samples the unperturbed model at T = 0.7 and votes without selection.

MATH-500 · OLMo-3-7B-InstructOn a strong model, sampling and voting account for much of the gain.

TT-MV
77.7%
Full-weight PSV, 1000 members
74.7%
Full-weight PSV, 5000 members
80.7%

Countdown · Qwen2.5-1.5B-InstructOn a weaker model, selection matters.

TT-MV, best temperature (T = 0.12)
34.3%
Full-weight PSV
37.4%

Accuracy against ensemble size K

logit PSVfull-weight PSVTT-MV (sampling at T = 0.7, no selection)unperturbed

K sweep from the appendix. Logit PSV tracks or exceeds full-weight PSV. TT-MV is mostly behind on the smaller models, where selection matters most.

4 · First-order analysisWeight noise acts through the logits

To first order, noise on every weight acts as a shift of the logits.

A perturbed model changes its answer only through its logits. For noise σε on every weight, as in Neural Thickets, the first-order change of the logits is

Δℓ ≈ σ J(x) ε.

The change made by each block enters the residual stream, these changes add up, and the unembedding maps their sum to the logits. Adding this shift as a bias on the logits, with all other weights unchanged, gives the same logits to first order. Logit PSV samples such a bias directly.

Noise on every weight vs. a bias on the logits

Schematic, for one input and one decoding step. Route 1 perturbs every weight; route 2 adds route 1's first-order logit shift as an output bias. Shifts are drawn per unit of σ. Agreement is measured on OLMo-3-7B-Instruct for Countdown: the fraction of argmax changes under weight noise that the first-order logits reproduce.

At σ = 10⁻³, the scale used in the experiments, the first-order logits reproduce 89.7% of the argmax changes of the model with every weight perturbed. Agreement decreases as σ grows.

The match holds for each input separately. J(x) depends on the input, so a weight perturbation shifts the logits differently on each input, whereas a logit bias applies the same shift to all of them. Logit PSV uses this simpler form, with a random bias, and still matches the gains.

Share of the exact model's argmax flips reproduced by the first-order logits, against the weight noise scale σ (OLMo-3-7B-Instruct, Countdown; bars are 95% confidence intervals over 32 perturbations). The red point is the scale used throughout the paper.

5 · VisionVision under domain shift

Under domain shift, perturbing the class logits of a DINOv3 probe helps; perturbing the backbone weights adds at most one point.

Vision backbones are usually adapted with a linear probe, and a classifier makes a single prediction per image. We define a domain expert as a perturbed backbone whose features give a better probe on a shifted domain. The two variants differ in what a member is:

  • Full-weight PSV perturbs a DINOv3 backbone and fits a probe for each member.
  • Logit PSV fits one probe and perturbs its class logits.

We use ImageNet as a control and two datasets with a strong shift between training and test, FMoW and Waterbirds.

ImageNet

balanced top-1, control
An English springer spaniel
English springer
A parachute over a beach
parachute
A garbage truck on a road
garbage truck
A golf ball on a tee
golf ball

1000 classes with no shift between training and test. It checks that the method does not help where there is nothing to correct.

FMoW

worst-region accuracy
Satellite image of a shipyard in Africa
Africa · 2017 · shipyard
Satellite image of a golf course in the Americas
Americas · 2016 · golf course
Satellite image of a stadium in Asia
Asia · 2016 · stadium
Satellite image of a place of worship in Europe
Europe · 2016 · place of worship

Satellite images in 62 land-use classes. Test images are taken in later years than training images, and regions are unevenly represented. The score is the accuracy on the worst geographic region.

Waterbirds

worst-group accuracy
A landbird on a bamboo forest background
landbird · land
A waterbird on a lake background
waterbird · water
A waterbird on a bamboo forest background
waterbird · land
A landbird on a lake background
landbird · water

Birds pasted on land or water backgrounds. In training, 95% of waterbirds are on water and 95% of landbirds on land; the test set is balanced. The score is the accuracy on the worst of the four groups, usually one of the rare ones (outlined).

Sample images. ImageNet (Deng et al., 2009; images from the Imagenette subset), FMoW-WILDS (Christie et al., 2018; Koh et al., 2021) and Waterbirds (Sagawa et al., 2020, built from CUB and Places).

Results

Weight perturbations improve accuracy by at most one point on the three datasets, even with the 6.7B-parameter backbone. Logit PSV has no effect on ImageNet and improves worst-group accuracy by 6.2 points on FMoW and 5.0 on Waterbirds.

Gain over the linear probe, DINOv3-7B

full-weight PSVpatch-embedding PSVlogit PSVclosed-form recalibration

Gains in accuracy points over the linear probe on the frozen DINOv3-7B backbone. PSV variants keep K = 10 members. Closed-form recalibration is not applicable on balanced ImageNet.

Closed-form recalibration

We compare with a fixed correction that needs no sampling, selection or voting. It adds to the logit of each class c

bc = log πtrue(c) − log πpred(c),

where πtrue(c) and πpred(c) are the true and predicted frequencies of class c on the validation group of interest. It recovers 3.7 of the 6.2 points on FMoW and matches logit PSV on Waterbirds (+5.0). The vision gains therefore look like recalibration under distribution shift rather than the discovery of domain experts.

6 · ConclusionConclusion

Neural Thickets showed that a pretrained language model can be improved on a task without training: perturb its weights at random, keep the copies that score best on 200 examples and let them vote. We kept this procedure and changed where the noise goes.

  • One block is enough. Noise in a single decoder block gives about the same gain as noise in every weight: on Countdown with OLMo-3-7B-Instruct, the last block alone reaches 91.8%, against 90.8% for every weight.
  • The output bias is enough. Noise on the output bias alone, with every other weight unchanged, is more accurate than full-weight PSV on 13 of 15 reasoning cells.
  • Weight noise acts through the logits. The first-order analysis shows why the location matters little: wherever the noise is applied, its leading effect is a shift of the logits.
  • Vision gains look like recalibration. A closed-form correction of the class frequencies gives 3.7 of the 6.2 points of logit PSV on FMoW, and the same +5.0 points on Waterbirds.

Under logit PSV, each member is a single vector over the vocabulary. It multiplies the probability of each token by a fixed factor, the same at every decoding step and on every input, and selection keeps the vectors that help on the task. Our results do not exclude useful structure near the pretrained weights; they show that it is not needed for the gains measured here.

Logit PSV is also simpler to run. It is training-free, and since members differ only in the output bias, they share the rest of the network and can be decoded in the same batch, each with its own bias.

CitationBibTeX

BibTeX
@article{dufour2026rethinking,
  title   = {Rethinking Neural Thickets Without Weight Perturbations},
  author  = {Dufour, Nicolas and Gard{\`e}s, Elouan and Dravid, Amil and P{\'e}rez, Patrick and Landrieu, Loic},
  journal = {arXiv preprint},
  year    = {2026}
}