Rate limiting was designed against a mental model of a human at a keyboard, or a script written by someone who read the docs and set a delay. The traffic it now has to survive is generated by systems that read a description, form a plan, and execute it as fast as the network allows.
The difference is not volume. A single agent does not necessarily send more requests than a busy user. The difference is shape: agent traffic arrives in tight bursts, retries when it is unsure rather than when it fails, and occasionally enters a loop that no human would sustain for more than a few seconds.
A limiter tuned for human traffic will either let those patterns through until something falls over, or block them so bluntly that legitimate agents cannot complete ordinary work.
How agent traffic actually differs
Four properties matter for limiter design.
Bursts are the normal case rather than an anomaly. A planner that decides it needs eight lookups will issue eight lookups immediately, in parallel, because there is no reason to wait. A limiter that treats any burst as abuse will reject work that is entirely reasonable. A limiter with no burst allowance at all forces agents into artificial serialisation, which multiplies latency and makes the API feel broken.
Retries happen on ambiguity, not just on failure. A human who gets an odd result investigates. An agent that gets a result it cannot interpret frequently tries again, sometimes with slightly different arguments, on the theory that it phrased something wrong. This means unclear responses generate traffic. An ambiguous empty result can produce three or four follow-up calls that a clear response would have prevented, which makes response clarity a load-shedding measure as much as a usability one.
Loops occur. Not often, but they occur, and when they do the request rate is bounded only by round-trip time. The usual causes are a plan whose termination condition is never satisfied, a pagination cursor that does not advance, or a retry with backoff that was implemented without a maximum. No human client produces this pattern.
Traffic is correlated across callers. Many agents run against the same handful of runtimes with the same default retry behaviour, so a transient error can produce a synchronised retry wave from callers that share nothing except a library. A limiter that survives one aggressive caller can still fall over when a thousand polite ones back off in lockstep.
Fixed windows and the double burst
The simplest limiter counts requests per caller in a fixed clock window: so many per minute, reset on the minute. It is easy to implement, easy to reason about, and it has an artefact that gets exercised constantly by automated callers.
Consider a limit of sixty requests per minute with windows aligned to the clock. A caller can send sixty requests in the final second of one window and sixty more in the first second of the next. Both windows are individually compliant. The observed traffic is a hundred and twenty requests inside a two second span, which is twice the rate the limit was written to permit.
Human traffic rarely finds this edge. Automated traffic finds it reliably, because a blocked agent typically waits for the reset time the API told it about and then fires everything at once. The limiter's own retry hint aligns callers onto the window boundary, which makes the double burst more likely rather than less. Under many callers this becomes a synchronised spike at the top of every minute.
Sliding window counters fix the arithmetic by considering a moving interval, either exactly by keeping timestamps or approximately by weighting the previous window. The approximate form is cheap and removes the worst of the artefact.
A token bucket is usually the better fit for agent traffic, because it separates two things a fixed window conflates: the sustained rate and the permitted burst. Tokens refill continuously at the sustained rate. The bucket has a maximum capacity, which is the largest burst allowed. A caller that has been idle accumulates up to that capacity and can spend it at once, which is exactly the behaviour a planner issuing eight parallel lookups needs. A caller that runs continuously is held to the refill rate, which is exactly the behaviour a loop needs.
Tuning is two decisions rather than one. Refill rate is the sustained throughput the backend can carry. Capacity is the burst tolerance, and setting it around the size of a realistic parallel fan-out is a good starting point. Capacity much larger than that turns the bucket into a fixed window with the same artefact.
Identifying the caller
A limiter needs a key. Getting the key wrong makes every other design decision irrelevant, because a limiter that keys on something the caller controls is not a limiter.
An authenticated identity is the correct key when one exists. It is stable, it survives network changes, it cannot be spoofed without stealing a credential, and it lets limits differ per consumer. Everything else is a fallback.
For keyless or public endpoints the fallback is usually the client address, and this is where the trouble is. A service behind a load balancer, a CDN, or a reverse proxy sees the proxy's address on every connection. Keying on it puts every caller in the world into one bucket. The standard remedy is to read the originating address from a forwarded header that the proxy sets.
The trust problem is specific and frequently missed. A forwarded header is a claim made by whatever spoke to the server. If the proxy sets it, it is trustworthy. If a client can reach the origin directly, bypassing the proxy, then the client sets it, and the client can set it to anything. An attacker who can reach the origin directly gets an unlimited supply of distinct rate limit keys by varying one header, which means no rate limit at all.
Directly reachable origins are common by accident. A service is deployed, put behind a CDN, and the origin remains publicly routable because nobody closed it off. Everything works, the header is honoured, and the limiter is decorative.
The defences are unglamorous: - Make the origin unreachable except from the proxy, by network policy, security group, or a shared secret the proxy attaches and the origin requires. - Trust the forwarded header only when the immediate peer address belongs to a known proxy range, and fall back to the peer address otherwise. - When parsing a forwarded chain, take the entry contributed by the trusted proxy rather than the leftmost entry, since the leftmost is client-supplied and the proxy appends to whatever it received. - Treat a forwarded header as absent if it fails to parse, rather than accepting a malformed value.
Address keys have a second weakness worth stating plainly. Callers behind shared network address translation share a key and interfere with each other, while callers with large address allocations can rotate through many keys. Address-based limiting is a blunt instrument that keeps honest traffic honest and does not stop a determined caller. Where the cost of a request is real, that argues for requiring identity rather than for tightening the address limit.
Bounding the limiter's memory
Every limiter keeps state per key. A hash map from key to counter or bucket, in process memory or in a shared store. The number of keys is bounded by the number of distinct callers, which is bounded by nothing.
A caller that varies its key, whether maliciously through a forgeable header or accidentally through address rotation, causes unbounded state growth. The limiter, which exists to protect the service from resource exhaustion, becomes the mechanism of resource exhaustion. This failure is quieter than a traffic flood because memory grows for a while before anything visibly breaks.
The state must be bounded structurally rather than by hoping the key space is small.
Expire entries aggressively. A bucket for a caller that has not been seen for several multiples of the window is worthless and should be dropped. In a shared store, set a time-to-live on write and let the store evict. In process memory, a periodic sweep works but only if it is guaranteed to run under load, which is exactly when it is most likely to be starved.
Cap the total number of tracked keys and evict under a defined policy, least-recently-used being the obvious one. Then decide deliberately what happens when the cap is reached. Failing open lets a flood of new keys bypass the limiter entirely. Failing closed rejects new callers and turns the flood into a denial of service against legitimate traffic. Neither is good, which is why the cap should be set high enough that reaching it is itself an alert rather than a routine condition.
For very large key spaces, approximate structures are worth considering. Counting with a fixed-size sketch trades exactness for a hard memory bound, which is often the right trade when the goal is stopping abuse rather than enforcing a contractual quota.
Whatever the design, the memory ceiling should be a number someone has calculated, not an emergent property. Keys times bytes per entry times replicas, computed once and written down.
Refusals a caller can act on
A rejected request is a message to an automated system, and what that system does next depends entirely on what it can extract from the message.
A bare status code with an empty body tells a caller it failed and nothing about when to try again. The common default behaviour is an immediate retry, then another, with whatever backoff the client library happens to implement. The refusal generates more traffic than the original request would have.
A useful refusal carries several things. A stable machine-readable error identifier, so the caller can branch on the reason without parsing prose. A clear statement that this is a rate limit rather than a permission failure or an outage, because those call for entirely different responses and a generic error invites the wrong one. A retry hint expressed as a delay in seconds, which is the single most valuable field because it converts guessing into waiting. And a human-readable message for the logs, which is the only part a person will read.
The retry hint should be a duration rather than a timestamp where possible, since a duration is immune to clock skew, and it should include jitter or the caller should be told to add some. Sending every blocked caller the same reset instant produces a synchronised retry wave, which is the failure mode the limiter was supposed to prevent.
The most important property is that the refusal must be unambiguous. An agent that cannot tell a rate limit from an empty result may interpret the refusal as a fact about the world, or as a phrasing problem it should retry with different arguments. Ambiguity converts one refusal into several requests. Clarity is load shedding.
Exposing limit state before the refusal
A caller that only learns about the limit by hitting it will hit it, repeatedly, because it has no way to plan. Publishing the state on every successful response lets a well-behaved caller stay under the line without ever being blocked.
Three values are enough: the limit in force for this key, the number of requests or tokens remaining, and when the allowance resets or refills. A caller with those can pace itself, decide whether to fan out or serialise, and choose which of several pending calls is worth spending remaining budget on.
This is one of the few places where an API can improve its own reliability by giving information away. A sophisticated caller will use it to avoid generating rejected traffic entirely. An unsophisticated one ignores it and is no worse off than before.
Two implementation notes. Keep the header names consistent across every endpoint, because a caller that has to special-case them will not bother. And make sure the values are accurate under concurrency: a remaining count computed before a concurrent request was counted will be optimistic, and a caller pacing itself against an optimistic number will overshoot.
Per-endpoint limits
A single global limit assumes every request costs the same. In practice the spread is enormous. A cached lookup returning a small object costs almost nothing. A spatial query over a large area, a request that triggers a paid third-party call, or an operation that runs a model costs orders of magnitude more, in latency, in compute, and sometimes in money.
A global limit tuned for the cheap endpoints leaves the expensive ones exposed, since a caller can spend its entire allowance on the most costly operation available. Tuned for the expensive ones, it throttles cheap calls that were never a problem and makes the API tedious to use for ordinary work.
The workable structure is two limits applied together. A global limit protects the front door and stops crude floods. A per-endpoint limit protects each expensive resource according to what it actually costs. A request must satisfy both.
For endpoints that cost real money, the per-endpoint limit should be treated as a spend control rather than a load control, which changes how it is set. Load limits are set from capacity and can be relaxed when capacity is added. Spend limits are set from a budget and should not be relaxed by an on-call engineer at three in the morning because a caller complained.
A refinement worth the complexity in mixed workloads is cost-weighted consumption: each endpoint deducts a number of tokens proportional to its real cost, from one shared bucket. This lets a caller choose its own mix, many cheap calls or a few expensive ones, without the operator having to guess the right ratio in advance. It also makes the limit explicable, since the weights can be published in the manifest and a planner can budget against them.
The limit that gets set too high
The predictable mistake is not setting limits too tight. It is setting them so generously that they never bind during testing, then discovering their real behaviour during an incident, when the limiter's own failure modes surface for the first time under the worst possible conditions.
Every limiter should be exercised deliberately. Drive a caller into the limit on purpose and confirm three things: that the refusal is structured and carries a retry hint, that a caller which honours the hint recovers cleanly without a synchronised stampede, and that the limiter's memory stays bounded while a flood of distinct keys arrives.
A limit that has never rejected anything in production is not evidence of good behaviour. It is an untested code path sitting on the critical route of every request.