Event Schema Evolution

Event schema evolution without breaking consumers

Schema evolution is inevitable in long-lived event streams. The goal is to add fields and behaviors without breaking downstream services or losing the ability to replay historical data.

Prerequisites

• Schema Registry (Confluent or compatible)
• Node.js 20+
• @kafkajs/confluent-schema-registry and kafkajs

Compatibility strategies

• Backward compatible: new consumers can read old data.
• Forward compatible: old consumers can read new data.
• Full compatible: both directions, safest for replay.

Use backward or full compatibility for event streams that are replayed.

Versioning rules that scale

• Never remove or change the meaning of existing fields.
• Add new fields with defaults to preserve backward compatibility.
• Prefer additive changes and keep old fields until all consumers migrate.

Registering schemas from Node.js

import { Kafka } from "kafkajs";
import { SchemaRegistry, SchemaType } from "@kafkajs/confluent-schema-registry";

const registry = new SchemaRegistry({ host: "http://localhost:8081" });

const schemaV1 = {
  type: "record",
  name: "OrderCreated",
  namespace: "com.example.orders",
  fields: [
    { name: "orderId", type: "string" },
    { name: "total", type: "double" }
  ]
};

const { id: schemaId } = await registry.register({
  type: SchemaType.AVRO,
  schema: JSON.stringify(schemaV1)
});

Evolving a schema safely

Add optional fields with defaults so old consumers can continue to deserialize.

const schemaV2 = {
  type: "record",
  name: "OrderCreated",
  namespace: "com.example.orders",
  fields: [
    { name: "orderId", type: "string" },
    { name: "total", type: "double" },
    { name: "currency", type: "string", default: "USD" }
  ]
};

await registry.register({
  type: SchemaType.AVRO,
  schema: JSON.stringify(schemaV2)
});

Publishing with schema-aware serialization

const kafka = new Kafka({ clientId: "orders-api", brokers: ["localhost:9092"] });
const producer = kafka.producer();
await producer.connect();

const payload = { orderId: "o-1001", total: 420.55, currency: "EUR" };
const encoded = await registry.encode(schemaId, payload);

await producer.send({
  topic: "orders",
  messages: [{ value: encoded }]
});

Operational guardrails

• Enforce compatibility in the registry, not in code reviews.
• Run consumer contract tests against recorded payloads.
• Track schema IDs in logs to help with incident triage.

Things to remember

• Use additive, backward-compatible changes as the default.
• Keep old fields until all consumers have migrated and historical replays are no longer needed.
• Store schema IDs alongside payloads to ease debugging.