Designing Idempotent APIs in Distributed Systems

Designing Idempotent APIs in Distributed Systems#

Idempotency is the property that repeating the same request produces the same outcome. In distributed systems, retries are inevitable because of timeouts, network partitions, and client-side retry policies. Without idempotency, retries can double-charge a customer, create duplicate resources, or violate invariants.

Why Idempotency Breaks in Practice#

Common failure scenarios include:

Client times out, retries, and the server processes both requests.
Load balancer retries after a connection reset, but the server already committed.
Message brokers redeliver events due to consumer crashes.

Idempotency must be implemented explicitly for non-idempotent verbs like POST and side-effecting PATCH.

Idempotency Design Options#

1. Natural Idempotency Through Resource Modeling#

If the client can supply a stable resource identifier, PUT /orders/{id} is naturally idempotent. This is the simplest strategy, but it is not always possible for server-generated identifiers.

2. Idempotency Keys with Deduplication#

For POST requests, accept an Idempotency-Key header and store the outcome keyed by (client, key).

Key requirements:

Keys must be unique per logical operation.
Store the response body, status, and headers.
Expire keys carefully, aligned with client retry windows.

3. Conditional Writes and Optimistic Concurrency#

Use conditional updates to ensure only one write is applied. Examples include SQL INSERT ... ON CONFLICT DO NOTHING or optimistic version checks with WHERE version = ?.

4. Idempotent Side Effects#

Downstream calls must also be idempotent. This often means propagating the same idempotency key across services or using request correlation IDs with deduplication in each tier.

Storage Model for Deduplication#

A typical dedup store has:

idempotency_key
request_hash
response_status
response_body
created_at

Store the hash of the request payload to detect key reuse with a different request body and return a 409 Conflict.

Spring Boot Example with Redis-backed Idempotency#

@RestController
@RequestMapping("/payments")
public class PaymentController {
    private final PaymentService paymentService;

    public PaymentController(PaymentService paymentService) {
        this.paymentService = paymentService;
    }

    @PostMapping
    public ResponseEntity<PaymentResponse> createPayment(
            @RequestHeader("Idempotency-Key") String key,
            @RequestBody PaymentRequest request) {
        return paymentService.createPayment(key, request);
    }
}

@Service
public class PaymentService {
    private final IdempotencyStore idempotencyStore;
    private final PaymentRepository paymentRepository;

    public PaymentService(IdempotencyStore idempotencyStore,
                          PaymentRepository paymentRepository) {
        this.idempotencyStore = idempotencyStore;
        this.paymentRepository = paymentRepository;
    }

    @Transactional
    public ResponseEntity<PaymentResponse> createPayment(String key, PaymentRequest request) {
        return idempotencyStore.execute(key, request, () -> {
            Payment payment = paymentRepository.save(Payment.from(request));
            PaymentResponse response = PaymentResponse.from(payment);
            return ResponseEntity.status(HttpStatus.CREATED).body(response);
        });
    }
}

@Component
public class IdempotencyStore {
    private final RedisTemplate<String, IdempotentRecord> redisTemplate;

    public IdempotencyStore(RedisTemplate<String, IdempotentRecord> redisTemplate) {
        this.redisTemplate = redisTemplate;
    }

    public ResponseEntity<PaymentResponse> execute(
            String key,
            PaymentRequest request,
            Supplier<ResponseEntity<PaymentResponse>> action) {
        String redisKey = "idempotency:" + key;
        IdempotentRecord cached = redisTemplate.opsForValue().get(redisKey);
        if (cached != null) {
            if (!cached.requestHash().equals(request.hash())) {
                return ResponseEntity.status(HttpStatus.CONFLICT).build();
            }
            return ResponseEntity.status(cached.status()).body(cached.response());
        }

        ResponseEntity<PaymentResponse> response = action.get();
        IdempotentRecord record = IdempotentRecord.from(request, response);
        redisTemplate.opsForValue().set(redisKey, record, Duration.ofHours(24));
        return response;
    }
}

Operational Considerations#

Key scope: include tenant or client ID to avoid collisions.
TTL selection: align TTL with client retry policy and incident window.
Storage durability: use durable stores for payment or billing APIs.
Tracing: tag traces with idempotency key for auditability.

Common Pitfalls#

Accepting the same key with different request payloads.
Deleting keys too soon, causing duplicate operations on retries.
Ignoring downstream idempotency, which still creates duplicates.

Summary#

Idempotency is a cross-cutting concern across API design, persistence, and observability. Implementing idempotency keys with deterministic responses and request hashing provides a robust and scalable solution for distributed systems.