Designing Idempotent APIs in Distributed Systems

Designing Idempotent APIs in Distributed Systems

Idempotency is the property that repeating the same request produces the same outcome. In distributed systems, retries are inevitable because of timeouts, network partitions, and client-side retry policies. Without idempotency, retries can double-charge a customer, create duplicate resources, or violate invariants.

Why Idempotency Breaks in Practice

Common failure scenarios include:

• Client times out, retries, and the server processes both requests.
• Load balancer retries after a connection reset, but the server already committed.
• Message brokers redeliver events due to consumer crashes.

Idempotency must be implemented explicitly for non-idempotent verbs like POST and side-effecting PATCH.

Idempotency Design Options

1. Natural Idempotency Through Resource Modeling

If the client can supply a stable resource identifier, PUT /orders/{id} is naturally idempotent. This is the simplest strategy, but it is not always possible for server-generated identifiers.

2. Idempotency Keys with Deduplication

For POST requests, accept an Idempotency-Key header and store the outcome keyed by (client, key).

Key requirements:

• Keys must be unique per logical operation.
• Store the response body, status, and headers.
• Expire keys carefully, aligned with client retry windows.

3. Conditional Writes and Optimistic Concurrency

Use conditional updates to ensure only one write is applied. Examples include SQL INSERT ... ON CONFLICT DO NOTHING or optimistic version checks with WHERE version = ?.

4. Idempotent Side Effects

Downstream calls must also be idempotent. This often means propagating the same idempotency key across services or using request correlation IDs with deduplication in each tier.

Storage Model for Deduplication

A typical dedup store has:

• idempotency_key
• request_hash
• response_status
• response_body
• created_at

Store the hash of the request payload to detect key reuse with a different request body and return a 409 Conflict.

Spring Boot Example with Redis-backed Idempotency

@RestController
@RequestMapping("/payments")
public class PaymentController {
    private final PaymentService paymentService;

    public PaymentController(PaymentService paymentService) {
        this.paymentService = paymentService;
    }

    @PostMapping
    public ResponseEntity<PaymentResponse> createPayment(
            @RequestHeader("Idempotency-Key") String key,
            @RequestBody PaymentRequest request) {
        return paymentService.createPayment(key, request);
    }
}

@Service
public class PaymentService {
    private final IdempotencyStore idempotencyStore;
    private final PaymentRepository paymentRepository;

    public PaymentService(IdempotencyStore idempotencyStore,
                          PaymentRepository paymentRepository) {
        this.idempotencyStore = idempotencyStore;
        this.paymentRepository = paymentRepository;
    }

    @Transactional
    public ResponseEntity<PaymentResponse> createPayment(String key, PaymentRequest request) {
        return idempotencyStore.execute(key, request, () -> {
            Payment payment = paymentRepository.save(Payment.from(request));
            PaymentResponse response = PaymentResponse.from(payment);
            return ResponseEntity.status(HttpStatus.CREATED).body(response);
        });
    }
}

@Component
public class IdempotencyStore {
    private final RedisTemplate<String, IdempotentRecord> redisTemplate;

    public IdempotencyStore(RedisTemplate<String, IdempotentRecord> redisTemplate) {
        this.redisTemplate = redisTemplate;
    }

    public ResponseEntity<PaymentResponse> execute(
            String key,
            PaymentRequest request,
            Supplier<ResponseEntity<PaymentResponse>> action) {
        String redisKey = "idempotency:" + key;
        IdempotentRecord cached = redisTemplate.opsForValue().get(redisKey);
        if (cached != null) {
            if (!cached.requestHash().equals(request.hash())) {
                return ResponseEntity.status(HttpStatus.CONFLICT).build();
            }
            return ResponseEntity.status(cached.status()).body(cached.response());
        }

        ResponseEntity<PaymentResponse> response = action.get();
        IdempotentRecord record = IdempotentRecord.from(request, response);
        redisTemplate.opsForValue().set(redisKey, record, Duration.ofHours(24));
        return response;
    }
}

Operational Considerations

• Key scope: include tenant or client ID to avoid collisions.
• TTL selection: align TTL with client retry policy and incident window.
• Storage durability: use durable stores for payment or billing APIs.
• Tracing: tag traces with idempotency key for auditability.

Common Pitfalls

• Accepting the same key with different request payloads.
• Deleting keys too soon, causing duplicate operations on retries.
• Ignoring downstream idempotency, which still creates duplicates.

Summary

Idempotency is a cross-cutting concern across API design, persistence, and observability. Implementing idempotency keys with deterministic responses and request hashing provides a robust and scalable solution for distributed systems.