Why AI models seem to get worse right before a new release

Summary. In the weeks preceding a major model launch, users consistently report a decline in model quality — weaker reasoning, increased error rates, less consistent output. The probable mechanism is load balancing: vendors distribute traffic across server pools running different quantization levels, and pools serving the outgoing model are frequently quantized more aggressively to free compute for training the next release.

Facts

• Documentation: user reports of quality degradation clustering ahead of new model announcements, documented in research published April 2026
• Probable mechanism: load balancers distributing requests across server pools with variable quantization, with older models frequently subject to more aggressive quantization
• Academic response: a dedicated statistical framework, published at ICLR 2026 by Amazon researchers, built specifically to distinguish genuine degradation from ordinary variance
• Independent findings: Stanford and UC Berkeley research documented significant performance variance for GPT-3.5 and GPT-4 on identical tasks over time
• Distribution of effect: a model can improve on published benchmarks while measurably degrading on specific, narrower use cases concurrently

Mechanism

Running a model at full precision across all serving infrastructure carries meaningful compute cost. As a new model release approaches, pressure increases to reallocate compute toward training and evaluation. One available lever is more aggressive quantization of the outgoing model across a portion of its serving infrastructure, trading output quality for freed capacity.

Because load balancers distribute requests across pools without guaranteed consistency, two identical requests submitted simultaneously can return from pools running different quantization levels. One returns output at close to full quality. The other returns output from the same model at measurably reduced quality.

Verification

Reports of quality decline are frequent enough on their own that isolating a genuine, systemic drop from ordinary variance and confirmation bias presents a legitimate methodological challenge.

The research consistently indicates the effect occurs with enough frequency to matter, and with enough variance across use cases that published benchmarks can fail to capture it. A model posting a higher aggregate benchmark score while degrading on a specific workload your team depends on is not a contradiction, both outcomes occur at once.

Detection strategy

A vendor status page is not a detection mechanism, since this pattern does not appear there. Detection requires measuring your own workflows against a stable baseline continuously, rather than assuming current output matches a prior baseline.

That requires tracking cost and behavior per run, per model, over time, and flagging drift before it surfaces as a production incident requiring backward reconstruction. It also requires the ability to route around a degraded model immediately upon detection, without reconstructing the dependent workflow.

See how Exemplar tracks model quality · Request a briefing from Exemplar