Published 11 Nov 20261 min read

Retries with jitter: a ten-line fix for a self-inflicted outage

Every client retried after exactly one second. So the service fell over again, exactly one second later.

Bundles of yellow fibre cables in a server rack

A dependency had a short blip. Every client retried after exactly one second, at the same moment, and knocked it over again. Then again.

The fix

Wait longer after each failure, and add randomness so clients spread out.

function delayFor(attempt) {
  const base = 200 * 2 ** attempt;          // 200, 400, 800 ms...
  const capped = Math.min(base, 10_000);
  return Math.random() * capped;           // "full jitter"
}

Three more rules

Cap the number of attempts. Three is usually plenty.
Only retry what is safe to repeat.
Give up quickly on errors that will not fix themselves, such as a bad request.