How Kafka Handles Schema Evolution with Avro Schemas: A Complete Guide

Kafka delegates schema management to the Confluent Schema Registry, which stores versioned Avro schemas and issues numeric IDs that producers embed in message headers, enabling consumers to fetch the correct schema version and safely handle backward- and forward-compatible changes.

The DataTalksClub/data-engineering-zoomcamp repository demonstrates production-grade patterns for Kafka schema evolution with Avro in its streaming modules. Unlike JSON or CSV, Avro requires strict schema contracts, making the Schema Registry essential for maintaining compatibility between producers and consumers as data structures change over time.

How the Confluent Schema Registry Manages Avro Schemas

Schema Separation from Message Storage

Kafka brokers store only binary message payloads, not the associated schemas. The Confluent Schema Registry operates as a separate service that stores Avro schema definitions, assigns each version a unique numeric ID, and enforces compatibility rules. When a producer serializes a record using AvroSerializer, it registers the schema with the registry and receives a schema ID that is written into the Kafka message metadata.

Schema Retrieval During Consumption

Consumers use AvroDeserializer to read the schema ID from the incoming message, fetch the corresponding schema version from the registry, and deserialize the payload accordingly. This decoupling allows producers and consumers to evolve independently as long as schema compatibility rules are maintained.

Implementing Schema-Aware Producers and Consumers

The repository provides concrete Python implementations in 07-streaming/extras/python/avro_example/ showing how to wire Schema Registry clients with Kafka clients.

Producer Configuration with AvroSerializer

In 07-streaming/extras/python/avro_example/producer.py (lines 27-34), the producer instantiates a SchemaRegistryClient, loads the Avro schema files, and creates serializers that automatically register schemas and embed their IDs:

# 07-streaming/extras/python/avro_example/producer.py

# lines 27-34

schema_registry_props = {'url': props['schema_registry.url']}
schema_registry_client = SchemaRegistryClient(schema_registry_props)
self.key_serializer = AvroSerializer(schema_registry_client, key_schema_str, ride_record_key_to_dict)
self.value_serializer = AvroSerializer(schema_registry_client, value_schema_str, ride_record_to_dict)

Consumer Configuration with AvroDeserializer

The consumer implementation in 07-streaming/extras/python/avro_example/consumer.py (lines 22-28) demonstrates fetching schema versions dynamically:

# 07-streaming/extras/python/avro_example/consumer.py

# lines 22-28

self.avro_key_deserializer = AvroDeserializer(
      schema_registry_client=schema_registry_client,
      schema_str=key_schema_str,
      from_dict=dict_to_ride_record_key)
self.avro_value_deserializer = AvroDeserializer(
      schema_registry_client=schema_registry_client,
      schema_str=value_schema_str,
      from_dict=dict_to_ride_record)

Schema Evolution Compatibility Strategies

The Schema Registry enforces three primary compatibility modes when evolving Kafka Avro schemas:

• BACKWARD compatible: New consumers can read data written with the old schema. This is the default mode.
• FORWARD compatible: Old consumers can read data written with the new schema, which requires providing default values for all newly added fields.
• FULL compatible: Changes satisfy both backward and forward compatibility, allowing complete interoperability between old and new clients.

Step-by-Step Schema Evolution Workflow

When evolving a schema—for example, adding a payment_type field to a taxi ride record—follow this pattern demonstrated in the DataTalksClub repository:

1. Define the new schema version with default values for added fields:

{
  "type": "record",
  "name": "Ride",
  "fields": [
    {"name": "vendor_id", "type": "int"},
    {"name": "pickup_datetime", "type": "string"},
    {"name": "dropoff_datetime", "type": "string"},
    {"name": "passenger_count", "type": ["null", "int"], "default": null},
    {"name": "payment_type", "type": ["null", "string"], "default": null}
  ]
}

2. Configure the producer to use the new schema file path (e.g., resources/schemas/taxi_ride_value_v2.avsc):

config = {
    'bootstrap.servers': BOOTSTRAP_SERVERS,
    'schema_registry.url': SCHEMA_REGISTRY_URL,
    'schema.key': RIDE_KEY_SCHEMA_PATH,
    'schema.value': 'resources/schemas/taxi_ride_value_v2.avsc'
}
producer = RideAvroProducer(props=config)

3. Verify registry compatibility settings ensure that old consumers can still process messages. When old consumers encounter messages with the new schema ID, they receive null for the missing payment_type field due to the default value specification.

Summary

• Kafka does not store schemas; the Confluent Schema Registry manages Avro schema versions and assigns numeric IDs.
• Producers embed schema IDs into message metadata via AvroSerializer, while consumers fetch schemas using AvroDeserializer and the stored ID.
• Compatibility modes (BACKWARD, FORWARD, FULL) govern how schemas can evolve without breaking existing consumers.
• Default values are required for forward-compatible changes, allowing old consumers to read new data.
• Repository reference: The 07-streaming/extras/python/avro_example/ directory in DataTalksClub/data-engineering-zoomcamp contains working implementations of producer.py, consumer.py, and sample .avsc schema files.

Frequently Asked Questions

Does Kafka store Avro schemas internally?

No. According to the DataTalksClub/data-engineering-zoomcamp implementation, Kafka stores only the binary message payload and a schema ID. The actual Avro schema definitions reside in the Confluent Schema Registry, which operates as a separate service that producers and consumers query to serialize and deserialize data.

What happens when a consumer receives a message with an unknown schema version?

The consumer reads the schema ID from the message header and fetches the corresponding schema from the Schema Registry. If the schema evolution follows backward compatibility rules, the consumer deserializes the data successfully, applying default values for any fields present in the new schema but missing from the old message format.

How do you make a schema change forward compatible?

To achieve forward compatibility, you must provide default values for all new fields added to the schema. As shown in the repository's example schema, fields use JSON syntax like "default": null alongside union types such as ["null", "string"]. This allows old consumers to read new messages by substituting the default value for missing fields.

Can you evolve schemas without using the Schema Registry?

While technically possible using manual schema management in application code, the DataTalksClub repository demonstrates that production systems rely on the Schema Registry to prevent schema poisoning and enforce compatibility rules. Without the registry, you risk deserialization failures when producers and consumers run different schema versions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}