Rate limiting in practice

The best rate limit is the one you never have to explain, because the traffic never gets close to it. The second best is the one that produces a clear 429 with a Retry-After header, so the client knows exactly what happened and exactly when to try again. The worst is the one you discover when the load test finds it, or when a customer's integration starts failing with errors that look like bugs, because your service collapsed under a burst instead of saying no politely.

Rate limiting is one of those topics that every backend engineer has opinions about and very few have actually designed from scratch. The algorithms are simple enough to describe in a paragraph, and the engineering is entirely in the details: what counts as a request, who the limit applies to, what happens to the burst, and how you tell the client what you did. Get the details wrong and you have either an API that falls over or an API that refuses service to your best customers.

The algorithms, briefly

There are three algorithms that cover almost every real design, and they are worth knowing by name because the names come up constantly. The token bucket allows a steady rate with room for bursts: tokens are added at a fixed rate, a request consumes a token, and a burst is allowed as long as tokens remain. The leaky bucket smooths output instead: requests enter a queue and are processed at a fixed rate, so bursts are delayed rather than served. The sliding window counts requests over a moving time window, either by timestamp precision or by tracking fixed windows and weighting the overlap, and it is the closest to a clean "N requests per minute" without the burst of a fixed window reset.

The fixed window counter deserves a warning, because it is the default in a lot of code and it has a famous flaw. It counts requests per fixed window, and a request just before the window boundary and one just after count in different windows, so a client can send twice the limit in the gap. The sliding window is barely more work and does not have the hole. If you are designing a public API, start with the sliding window, and reach for the token bucket when the product needs the burst behavior.

The decisions that matter more than the algorithm

The algorithm is the easy part. The decisions around it are the product. The first decision is what the rate applies to: per API key, per user, per IP, per tenant. The answer determines your data model, because a per user limit is a per user counter, and a per tenant limit is a shared budget across a whole organization. The providers that get this right apply limits at the level that matches the business contract: the customer bought a rate, and the rate is per customer, not per key, and not per IP, because a customer behind a corporate proxy sharing an IP with ten other customers should not be punished for the shared address.

The second decision is the distinction between a fair use limit and a hard contract. A fair use limit is a backstop, set high enough that legitimate users never hit it, there to catch the runaway loop and the abusive scraper. A hard contract is the product: the customer bought 100 requests per minute, and the API enforces it, and the client is expected to handle the 429. The design of the headers and the documentation follows from which one you are building.

The third decision is what happens at the limit. Reject with 429, obviously, but also: does the burst get a response or a queue? For the token bucket, a burst is served by the tokens, and a full bucket is a queue or a rejection depending on the design. The honest answer is that most APIs reject, because queuing a request changes the semantics of the response, and a client that sees a 200 after a ten second delay is a client with a bug it cannot explain.

The headers are the interface

The HTTP status code is the start of the conversation, not the end. The client needs to know how much it has used, how much remains, and when it can try again. The standard headers exist for exactly this: RateLimit-Limit tells the limit, RateLimit-Remaining tells what is left, RateLimit-Reset tells when the window resets, and Retry-After tells when a rejected request can be retried.

The Retry-After header is the one that gets the most value per byte, because it turns a rejection into an instruction. A client that receives a 429 with Retry-After: 30 knows to wait thirty seconds, and a well behaved client will. A client that receives a bare 429 has to guess, and the guess is usually wrong, and the retry storm that follows is how rate limiting incidents become cascading outages.

The client side of the contract matters as much as the server side. The clients that handle 429 with exponential backoff and respect Retry-After are the ones that keep the API healthy. The clients that retry immediately and aggressively are the ones that turn a minor overload into an outage. A rate limit is only as good as the clients' behavior in response to it, which is why the headers and the documentation are part of the design, not an afterthought.

The distributed version

The naive implementation is a counter in memory, and it is correct for a single instance and wrong for a fleet. When the service runs on many instances behind a load balancer, an in memory counter sees a tenth of the traffic on each instance, and the effective limit is ten times what you intended. The distributed rate limiter needs a shared store: Redis is the classic answer, with its counter operations, its atomicity, and its TTLs, and the fixed window and sliding window implementations are a few commands.

The distributed rate limiter has its own failure mode, and it is the one that surprises people: the rate limit store is a single point of failure, and when it goes down, the API can either fail open or fail closed. Fail open means no limits, and the traffic that the limiter was protecting hits the backend. Fail closed means reject everything, and a healthy service suddenly refuses all traffic because its limiter is down. The teams that have been through this choose fail open with a safety valve, and they accept the risk because the alternative is worse. The limiter protects the backend, and the backend is still the last line of defense, so the limiter failing should degrade gracefully, not kill the service.

The operational truth

Rate limiting is not a config you set and forget. It is a system you observe. The useful signals are the distribution of limits, the number of 429s, and the number of clients that hit the limit repeatedly. A client that hits the limit every day at the same time is a client with a pattern, and a pattern is either a batch job that needs a higher limit or a bug that needs a fix. The 429 log is a product feedback channel, and teams that read it find the customers who need a plan upgrade and the scripts that need a fix.

The other operational truth is the load test. The rate limit is the thing that keeps the service alive during the load test, and the load test is the thing that proves the rate limit works. The team that has never tested the limiter at scale will find out during the test whether the limiter holds, and the honest teams run the test before the launch, not after the incident.

The polite wall

The metaphor I keep coming back to is a wall. The wall is not there to be mysterious. It is there to say no, politely, clearly, and at the right time, so the service behind it stays healthy and the legitimate traffic gets through. The 429 with a Retry-After header is the politest possible no: here is what happened, here is when to try again, here is the contract we both agreed to.

The teams that treat rate limiting as a product decision, not a config, are the teams whose APIs survive their own success. The ones that treat it as a checkbox are the ones whose APIs fall over at the worst possible moment, which is always the moment the traffic finally arrives. Build the wall before you need it, and make it polite.