The FID Lottery Quantifying Hidden Randomness in Generative-Model Evaluation

Open any recent image-generation paper and the central claim usually rests on a single number — the Fréchet Inception Distance (FID). FID is the closest thing image generation has to an arbiter: a half-unit shift reorders the leaderboard; a decade of recipes have been justified by single-Inception-unit gains; and budgets in the low millions of GPU-hours hinge on which architecture lands a few decimals lower. But behind every reported FID number sits a chain of pseudo-random draws (parameter initialisation, minibatch order, per-step Gaussian noise injected by the training loss, hardware stochasticity, and the initial noise drawn at sampling time), any of which could have produced a potentially different score had the seed been different. Conventional wisdom considers this variance in FID to be negligible, especially for well-trained models. ☞︎In this paper we show that the FID reproducibility gap is real and is a serious concern.

Each time one trains a generative model and reports its FID, one is playing two lotteries. The training lottery runs once during training: one draws an initialisation, a data ordering, and the per-step noise that the loss injects at every gradient step, and what comes out is one trained network among many that could have been produced by different seeds. The generation lottery runs on the trained network: one draws an initial noise x_T ∼ 𝒩(0, I) to seed the sampler, generates a sample set, and scores it. Practitioners have learned to mitigate the second lottery — redrawing the initial noise across several seeds and reporting an error bar or an averaged FID score — but no amount of resampling on a single trained network says anything about where a re-trained network run would have landed. The training lottery stays hidden behind the one ticket we actually drew. Diffusion makes the problem worse: the flow-matching or score-matching loss redraws a fresh Gaussian ε ∼ 𝒩(0, I) at every gradient step, so the training noise never settles — it is a permanent random injection that an independent training run would resolve differently, not transient noise that longer training averages out. Scale offers no automatic remedy: neural scaling laws characterise how the mean loss falls with parameters and tokens, leaving the seed-induced spread around that mean unspecified. Zhang et al. recently showed that independently-trained diffusion networks converge to nearly the same noise-to-image mapping. But does this also hold for the FID metric computed over a set of generated images?

Each lottery defines an axis of FID variance: a training axis of N independent training runs and a generation axis of K sampling seeds per run. To measure both, we score the resulting N×K panel of FID evaluations. The panel below renders a converged SiT-B/2: every column is one trained network, every dot is one FID evaluation, and the column-to-column spread already overshoots the within-column spread at a glance. On this panel we first decompose the training lottery into its independent random sources, then sweep four practitioner-controlled axes (classifier-free guidance, compute, model size, and learning rate, transferred across widths by hyperparameter transfer) to test whether any of them tightens it. Across several hundred SiT networks from S through XL on ImageNet 256×256, ☞︎the training axis dominates the evaluation axis at every scale we probe and none of the four knobs closes the gap; a single-seed Inception FID therefore sits on a noise floor that recipe-level gains regularly fall below. A closer look at this floor turns up two practical handles nonetheless: per-cell classifier-free-guidance tuning (GS-FID) halves the relative noise floor but reshuffles which seeds win (Spearman ρ = 0.73), and a lucky training seed reaches the same FID with up to 2× less compute than an unlucky one.