Idempotency keys: the silent contract between your API and the world

There is a moment in every payment integration where a developer discovers, usually the hard way, that the network is not reliable and the API is not omniscient. A request times out, the client retries, and the server, which actually processed the first attempt, processes the payment again. Two charges for one purchase. The support ticket writes itself.

The fix for this specific class of disaster is so old and so boring that it is easy to underestimate. It is an idempotency key: a client generated identifier that accompanies a request, telling the server "if you have already processed this key, do not process it again, just tell me the result you already produced." Payments APIs have shipped this for years. Stripe made it famous. And yet a surprising number of internal APIs, order services, job runners, and webhook handlers still assume that a retry is somehow distinguishable from a new request. It is not. The bytes are identical.

The client is not your enemy, the network is

It is worth being precise about where the duplication comes from, because the fix follows from the cause. Requests fail in two ways, and only one of them tells you the request did not happen. A connection reset before the server sees the request means nothing happened, and retrying is correct. A timeout after the server processed the request and started writing the response means everything happened, and the client just never found out. From the client's point of view, both look like failure. The retry is not a bug in the client. It is the only move the client has.

The server, for its part, cannot tell a retry from a duplicate by looking at the payload. Two identical requests with the same body arrive, one from a user clicking twice, one from a client retrying a timeout. Same bytes. The only thing that separates them is the idempotency key, which is why the key has to be generated by the client and why it has to be stable across retries.

The contract

The contract has three parts, and each one is a decision. The client generates a unique key per logical operation, and reuses that same key on every retry of that operation. The server stores the result of the first request with that key, and returns the stored result for every subsequent request with the same key, without performing the operation again. The server expires keys after a window, because a key stored forever is a leaky abstraction and a privacy problem.

The interesting part of the contract is what counts as "the same logical operation". If a user submits a payment form and the network fails, the retry should carry the key from the first attempt. If the user changes the amount and resubmits, that is a different operation and must be a different key, because the server would otherwise return the stale result for the old amount. The key is not a random string. It is a label for the intent of the request, and intent changes require a new label.

The implementation is a database problem

There is a naive implementation that looks right and is a trap. Check the key in a cache, and if it is present, return the cached result. This fails on the exact day you need it to work: when the cache has an outage, or when the key has expired from the cache but the operation is still in flight, or when the check and the insert race between two concurrent requests. Idempotency is a consistency property, and consistency properties live in the database, not in the cache.

The shape that holds up is a unique constraint. Create a table for idempotency records, with the key as a unique index, and make the first insert of the key and the operation itself part of the same transaction. When a duplicate arrives, the insert violates the unique constraint, and the handler returns the recorded result instead of running the operation again. The database is the source of truth, and the database does not forget.

The concurrent case is where this earns its keep. Two requests with the same key arrive at the same time, from a double click or a retry racing the original. The unique constraint serializes them: one wins the insert, the other loses, and the loser waits, then reads the winner's result. No double charge, no double order, no double anything. The pattern is the same one used for signup tokens, invite codes, and every other "this thing must happen exactly once" problem, and it is the same reason it is the one that works.

The failure modes that still bite

Idempotency keys remove the retry duplication, and they are not a free pass. The remaining failures are instructive. If the key is generated after the network call, on the retry, the retry generates a new key and the duplication returns. The key must exist before the first attempt. If the server performs the operation and crashes before committing the idempotency record, the retry will run the operation again, and no key scheme fixes a crash at that exact point. That is the limit of the technique, and the honest answer is to make the operation itself idempotent as well, by design, so that even the duplicate is harmless.

The key should also be bound to the request it labels. If the server checks the key, finds a stored result, and returns it without verifying that the payload matches, a client error can return the wrong result. The safest implementations store a fingerprint of the request with the key and reject a mismatch, because a key that returns a stale result for different input is worse than no key at all.

The width of the idea

Payments are the classic use case, and the idea is much wider. Job runners get idempotency keys so a retried job does not send two emails. Order services get them so a retried checkout does not create two orders. Webhook processing gets them so a redelivered event does not apply twice. Signup flows get them so a retried registration does not create two accounts. Anywhere a retry is possible and a duplicate is expensive, an idempotency key is the answer, and the list of places where a retry is possible is "everywhere the network is involved".

The wider lesson is about contracts between systems. When two services talk over a network, neither can assume the other saw what it sent. The protocols that survive are the ones that make the retry safe by construction, and idempotency is the mechanism that makes the retry safe. It is not a payment feature. It is a property of any API that handles money, orders, notifications, or state changes, which is to say, any API that matters.

The boring discipline

Idempotency keys are one of those things that looks like a nice to have until it saves you from a specific, expensive incident, and the teams that skip them are not making a decision. They are accepting a risk they have not priced. The teams that adopt them are not doing anything clever. They are adding a column, a unique constraint, and a contract that says "this request may be retried, and retrying it is safe."

The silent part of the contract is the part worth remembering. The client and the server never discuss the retry. The client just resends the same key, the server just checks the same table, and the world never finds out the network misbehaved. That is the best kind of engineering: the incident that never happens, because someone made the retry boring instead of trusting the network to be good.