Why Temperature 0 Is not Deterministic

In September 2025, Horace He and colleagues at Pondering Machines Lab ran a easy experiment. They despatched the immediate “Inform me about Richard Feynman” to Qwen3-235B 1,000 instances at temperature 0 and requested for 1,000 tokens every time. Temperature 0 means the mannequin at all times picks its most possible token, so you’d count on 1,000 an identical solutions. They acquired 80 distinctive completions.

What caught my eye is the place the solutions break up. All 1,000 completions have been an identical for the primary 102 tokens. At token 103, 992 of them continued with “Queens, New York” and eight with “New York Metropolis.” Eight runs in a thousand flipped at a single token, after 102 tokens the place none did.

Their submit explains why outputs differ in any respect. This text asks a special query: how usually ought to a token flip, and why does that danger arrive in sudden spikes? I will derive a one-line system, examine it with a simulation, and take a look at it on an actual mannequin. All the pieces runs on a laptop computer, and the code is within the article.

The place the noise comes from

At temperature 0 the mannequin picks the token with the best logit (its uncooked rating). That’s deterministic arithmetic, however your laptop computes logits in floating level, the place addition isn’t associative:

>>> 0.1 + 0.2 + 0.30.6000000000000001>>> 0.1 + (0.2 + 0.3)0.6

At scale it will get worse. I summed 100,000 random float32 numbers 3 ways. Ahead order gave −90.82497, reverse order gave −90.82674, and NumPy’s pairwise sum gave −90.82513 (a float64 reference provides −90.82508). Similar numbers, three solutions.

A neural community is billions of such sums, so the order the {hardware} makes use of issues. A preferred clarification blames GPU threads that end in random order. He et al. present that this is not the primary trigger: a typical LLM ahead go accommodates primarily no atomic provides, and the identical kernel on the identical enter returns the identical bits each time. The true perpetrator is that many kernels aren’t batch-invariant. Their discount order adjustments with the batch measurement, and on a shared server the batch measurement is dependent upon what number of different persons are sending requests at that second. Your reply is dependent upon strangers.

The authors be aware that this is not particular to GPUs, and that with batch-invariant kernels all 1,000 Feynman completions got here out an identical. The repair has a worth: of their serving take a look at (Qwen-3-8B on one GPU), the unoptimized deterministic model took 55 seconds in opposition to 26 for default vLLM, and 42 after an improved consideration kernel.

Solely near-ties matter

Right here is the half that pursuits me as a mathematician. Let z₁ and z₂ be the 2 largest logits, and name their distinction the hole, $M = z₁ − z₂ ≥ 0$

A token with a niche of 8 logits won’t ever flip, nonetheless noisy the kernel. Solely near-ties are in danger. Averaging over all positions, with f the density of gaps, the flip chance is

p = ∫ P(Δ < −m) f(m) dm

If the noise is small, f is roughly fixed, equal to f(0), over the slim vary the place $P(Δ < −m)$

p ≈ f(0) · E|Δ| / 2

The flip chance per token is how crowded the neighbourhood of a tie is, instances how giant the noise is. The primary issue belongs to the mannequin and the textual content. The second belongs to your {hardware}, precision and kernels. That is an elementary small-noise argument, so I make no declare of novelty. A associated formalization, the “background temperature” of Messina and Scotta (TMLR, 2026), treats such perturbations as an efficient temperature.

Does the system maintain?

Earlier than touching an actual mannequin, I checked the system on simulated gaps with a identified density and Gaussian noise of identified measurement $σ$. The core is 9 traces:

def flip_rate(sampler, sigma, n, rng, chunk=5_000_000):    flips, executed = 0, 0    whereas executed < n:        m = min(chunk, n - executed)        hole = sampler(rng, m)            # top-1 minus top-2 logit hole        err = rng.regular(0.0, sigma, m)  # numerical error on that hole        flips += np.count_nonzero(hole + err < 0)        executed += m    return flips / n

For an exponential and a half-normal hole, the simulation matches the system inside 4% for σ as much as 0.03, and the half-normal stays inside 2% even at σ = 0.3 (Determine 2). It breaks when the noise is as giant as the standard hole: at σ = 1 the exponential case flips 24% of the time in opposition to a predicted 40%, as a result of the density is not flat over the area that issues.

A 3rd case exhibits the system’s key assumption, $f(0) > 0$

An actual mannequin

I ran GPT-2 on 256 home windows of 128 tokens from WikiText-2 and in contrast a $bf16$ ahead go with an $fp32$ one. It is a stand-in for lifelike serving noise: my laptop computer cannot reproduce the batch-size impact of a loaded GPU server, however it will probably measure the logit error a lower-precision kernel produces. I run the transformer physique in $bf16$and the output projection in $fp32$, so the ultimate logits aren’t rounded to $bf16$‘s coarse grid.

import copy, numpy as np, torchfrom datasets import load_datasetfrom transformers import AutoModelForCausalLM, AutoTokenizerm32 = AutoModelForCausalLM.from_pretrained("gpt2").eval().float()m16 = copy.deepcopy(m32).to(torch.bfloat16)W = m32.get_output_embeddings().weight   # fp32 output matrix, each runstok = AutoTokenizer.from_pretrained("gpt2")information = load_dataset("wikitext", "wikitext-2-raw-v1", break up="take a look at")textual content = "nn".be a part of(t for t in information["text"] if t.strip())ids = torch.tensor(tok(textual content)["input_ids"])n = len(ids) // 128ids = ids[: n * 128].reshape(n, 128)decide = torch.randperm(n, generator=torch.Generator().manual_seed(0))[:256]ids = ids[pick]

For each place I preserve the $fp32$ top-20 logits, the hole, the error on that hole within the $bf16$ run, and whether or not the argmax flipped. (Plotting code omitted.)

gaps, deltas, flips, top20 = [], [], [], []with torch.no_grad():    for x in ids.break up(8):                 # x: [8, 128] token ids        h32 = m32.base_model(x).last_hidden_state[:, 4:]        h16 = m16.base_model(x).last_hidden_state[:, 4:].float()        ref = (h32 @ W.T).reshape(-1, W.form[0])        alt = (h16 @ W.T).reshape(-1, W.form[0])        v, i = ref.topk(20, dim=1)         # fp32 top-20 logits        a = alt.collect(1, i)               # identical tokens, bf16 run        hole = v[:, 0] - v[:, 1]        gaps.append(hole.numpy())        deltas.append(((a[:, 0] - a[:, 1]) - hole).numpy())        flips.append((alt.argmax(1) != i[:, 0]).numpy())        top20.append(v.numpy())hole, delta, flip, top20 = map(np.concatenate,                              (gaps, deltas, flips, top20))

# density of the hole at 0 (per logit), and the error close to tiesf0 = np.imply(hole < 0.25) / 0.25err = np.abs(delta[gap < 0.5]).imply()print(f"f(0) = {f0:.3f}, error close to ties = {err:.4f}")print(f"flip fee: measured {flip.imply():.4f}, "      f"predicted {f0 * err / 2:.4f}")# inject noise of identified measurement into the actual logitsfor s in np.logspace(-3, 0.5, 8):    noise = np.random.randn(*top20.form) * s    print(f"{s:.4f}", np.imply((top20 + noise).argmax(1) != 0))

The density of the hole at zero is $f(0) ≈ 0.83$

From tokens to completions

For one completion, survival is a product of per-token survival possibilities, $S(t) ≈ exp(−Σ pₛ)$

Take the Feynman numbers (that is my arithmetic on their printed counts). Over the primary 102 tokens, 1,000 runs produced zero disagreements. That’s about 102,000 token selections with no flip, so the common per-token flip fee over that stretch is at most about 3×10⁻⁵ (the “rule of three” 95% certain). At token 103 the speed was 8 in 1,000, or 0.8%. That’s greater than 250 instances larger, at a single token. Danger is not unfold evenly. It sits on just a few knife-edge tokens, with lengthy stretches of near-zero hazard in between.

Prompts additionally differ in what number of knife-edge tokens they comprise. To see what that does to survival, I simulated a imply per-token flip fee of 0.005 with fragility various throughout prompts (lognormal, log-sd 1):

p_bar, sigma_ln = 0.005, 1.0t = np.arange(0, 1001)# per-prompt flip chance: lognormal with imply p_barp_i = p_bar * np.exp(rng.regular(0, sigma_ln, 400_000) - 0.5 * sigma_ln**2)uniform = (1 - p_bar) ** t                  # identical p for each immediatecombination = np.array([np.mean((1 - p_i) ** k) for k in t])  # p varies

If each immediate had p = 0.005, half of the completions would break up by token 139 and eight% would nonetheless match at token 500. With the identical imply however various fragility, the median strikes to token 210 and 28% nonetheless match at token 500. The common flip fee alone would not inform you how lengthy a completion survives. The unfold of fragility issues too.

The place this breaks

• My noise is not your server’s noise. bf16 in opposition to fp32 on a laptop computer reproduces the scale of a precision error, not the batch-size mechanism of a loaded GPU cluster. These numbers describe the mannequin, not any hosted API.

• Two tokens solely. The system ignores the third-best token and assumes the error is symmetric and unbiased of the hole.

• A flip is not an error. Many flips swap near-equivalent phrasings. The system says how usually outputs differ, not how usually they worsen.

What to do with it

Predict your flake fee. In case your stack has a per-token flip fee p, a completion of L tokens differs between runs with chance about $1 − exp(−p·L)$

Do not assert actual equality on temperature-0 outputs. Evaluate with a tolerance, or grade semantically. Should you management inference, batch-invariant kernels take away the impact for a throughput worth. Should you do not, assume two an identical calls can differ.

Takeaways

1. Temperature 0 is deterministic within the arithmetic, not within the arithmetic.

2. Solely near-ties matter: the per-token flip chance is about $f(0) · E|Δ| / 2$ the crowding of ties instances the noise degree.

3. Danger is spiky. A number of knife-edge tokens carry most of it, which is why completions keep an identical for a stretch after which break up.

···

Sources and credit

1. He, Horace and Pondering Machines Lab, “Defeating Nondeterminism in LLM Inference”, Pondering Machines Lab: Connectionism, September 10, 2025. https://thinkingmachines.ai/weblog/defeating-nondeterminism-in-llm-inference/ (supply of the Feynman experiment and its counts, the batch-invariance clarification, and the serving timings).

2. Messina, Alberto and Scotta, Stefano, “Introducing Background Temperature to Characterise Hidden Randomness in Massive Language Fashions”, Transactions on Machine Studying Analysis, 2026. https://openreview.web/discussion board?id=bz0he4bARF

3. Radford et al., “Language Fashions are Unsupervised Multitask Learners” (GPT-2), 2019, and Merity et al., “Pointer Sentinel Combination Fashions” (WikiText-2), 2016, each used by Hugging Face Transformers and Datasets.

4. The flip-probability system, simulations, real-model experiment, figures and the arithmetic on the Feynman counts are the writer’s personal.