technical · sourced answer
Exponential Backoff Retries
Exponential backoff retries are an error handling strategy where the delay between consecutive retries of a failed operation increases exponentially. Instead of retrying at fixed intervals, the system waits longer after each failure to allow the receiving server time to recover from congestion or temporary outages.
Mechanical Operation
The process begins with an initial wait time, such as one second. If the first retry fails, the wait time is multiplied by a constant factor, typically two. The second retry occurs after two seconds, the third after four, the fourth after eight, and so on. This geometric progression continues until a maximum delay threshold or a maximum number of attempts is reached, at which point the message is marked as a permanent failure.
Importance for Senders
Using this method prevents a sender from inadvertently performing a Denial of Service attack on a receiving mail server. If thousands of messages fail simultaneously and all retry every ten seconds, the resulting traffic spike can keep the receiver offline. By spreading out the retry attempts, senders maintain a better reputation and increase the likelihood that a temporary 4xx SMTP error will resolve before the message is dropped.
Operational Considerations
A critical addition to this strategy is jitter, which adds a small amount of random noise to the delay. Without jitter, multiple failed requests that occurred at the same time will retry in synchronized waves, creating spikes of traffic. Implementing jitter ensures that retries are distributed evenly across the time window, further reducing pressure on the infrastructure.
Common Implementation Mistakes
Developers often forget to set a maximum retry limit or a ceiling on the delay. Without a cap, the wait time can grow to hours or days, causing unacceptable latency for transactional emails. Another mistake is treating 5xx permanent failures as retryable; exponential backoff should only be applied to 4xx transient errors, such as rate limiting or temporary greylisting.
Concrete Example
Consider a transactional email sent via an API. Attempt 1 fails due to a 421 server busy error. The system waits 2 seconds. Attempt 2 fails; the system waits 4 seconds. Attempt 3 fails; the system waits 8 seconds. By the time Attempt 4 occurs, the receiving server has likely cleared its queue, allowing the email to be accepted. SendHQ provides free tools at https://sendhq.cc/tools to help manage email infrastructure efficiency.
Questions teams ask
What is the difference between fixed and exponential backoff?
Fixed backoff retries every X seconds regardless of failure count. Exponential backoff increases the interval after every failure to reduce load on the target server.
When should you stop retrying?
Retries should stop when a 5xx permanent failure is returned, the maximum number of attempts is reached, or the maximum delay ceiling is hit.
Does jitter affect the exponential growth?
No, jitter adds a random offset to the calculated exponential delay to prevent synchronized retry spikes among multiple concurrent requests.
Primary sources
- RFC 5321: Simple Mail Transfer Protocol — RFC Editor
- M3AAWG Best Common Practices — M3AAWG