Without Schema Registry, Kafka topics are blind byte streams. A producer changes the JSON format — consumer crashes with NullPointerException. According to statistics, 70% of production incidents are related to schema incompatibility. We solve this problem with Confluent Schema Registry: the message schema is versioned, evolution is controlled, incompatible changes are blocked before publication. We set up Schema Registry for Apache Avro, Protobuf, or JSON Schema turnkey — we'll assess your project in 1 day.
Schema Registry is a separate HTTP service that stores schemas in the _schemas Kafka topic. On first send, the producer registers the schema and receives a schema_id (integer). Instead of the full schema, only the schema_id (4 bytes) is embedded in each message — this is Confluent's wire format. An Avro message takes 2-3 times less space than an equivalent JSON schema.
Producer → [magic byte 0x00][schema_id 4 bytes][serialized payload] → Kafka Consumer → reads schema_id → requests schema from Registry → deserializes Why Schema Registry is mandatory for production?
Without Schema Registry, schema evolution becomes a nightmare. You add a field to JSON — old consumers that don't expect it may crash. Schema Registry with BACKWARD mode guarantees that the new schema is compatible with previous versions. This prevents downtime. Our experience: more than 50 projects with Kafka, and none went without Registry. Adopting Schema Registry reduces debugging time by 80%, saving 200,000 to 500,000 rubles per year.
Installation and basic setup
Typical setup via Docker Compose:
version: '3.8' services: schema-registry: image: confluentinc/cp-schema-registry:7.6.0 ports: - "8081:8081" environment: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "kafka-1:9092,kafka-2:9092,kafka-3:9092" SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8081" SCHEMA_REGISTRY_KAFKASTORE_TOPIC: "_schemas" SCHEMA_REGISTRY_KAFKASTORE_TOPIC_REPLICATION_FACTOR: 3 SCHEMA_REGISTRY_SCHEMA_COMPATIBILITY_LEVEL: "BACKWARD" SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL: PLAINTEXT restart: unless-stopped For production — at least 2 instances behind a load balancer, one is master. We guarantee fault tolerance.
How to choose compatibility mode?
| Mode | Description | When to use |
|---|---|---|
| BACKWARD | New schema can read data written by old schema | Standard choice for production |
| FORWARD | Old schema can read data written by new schema | When consumers update slower than producers |
| FULL | Both directions | Only when strictly necessary |
| NONE | No checks | Only for development, not for production |
We recommend BACKWARD for most scenarios. It allows adding fields with default and removing fields without default.
What if schemas are incompatible?
If CI/CD fails with an incompatibility error, there are two options: either roll back the schema change and refine it, or create a new topic with the new schema version and migrate producers/consumers. Schema Registry makes it easy to roll back by re-registering the previous version.
Registering schemas via REST API
After defining the schema, register it in Schema Registry using the REST API:
curl -X POST http://schema-registry:8081/subjects/order-events-value/versions \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d '{ "schema": "{\"type\":\"record\",\"name\":\"OrderEvent\",\"namespace\":\"com.example.orders\",\"fields\":[{\"name\":\"event_id\",\"type\":\"string\"},{\"name\":\"order_id\",\"type\":\"long\"},{\"name\":\"status\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"created_at\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}}]} }' Comparison of Avro, Protobuf, and JSON Schema formats
| Format | Advantages | Disadvantages |
|---|---|---|
| Avro | Compact binary, native Confluent integration | Complexity without code generation |
| Protobuf | Faster than Avro, supports many languages | Requires compiling .proto to classes |
| JSON Schema | Human-readable, no code generation required | Larger size, fewer tools |
Choice depends on ecosystem: Avro is standard for Kafka, Protobuf for microservices with gRPC, JSON Schema for simple integrations. More details in the official Confluent Schema Registry documentation.
Java producer with Schema Registry integration
For Java, use KafkaAvroSerializer. Add dependencies to pom.xml:
<dependency> <groupId>io.confluent</groupId> <artifactId>kafka-avro-serializer</artifactId> <version>7.6.0</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>1.11.3</version> </dependency> Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092"); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class); props.put("schema.registry.url", "http://schema-registry:8081"); props.put("auto.register.schemas", false); // In production — disable auto-registration props.put("use.latest.version", true); KafkaProducer<String, OrderEvent> producer = new KafkaProducer<>(props); OrderEvent event = OrderEvent.newBuilder() .setEventId(UUID.randomUUID().toString()) .setOrderId(12345L) .setUserId(67890L) .setStatus(OrderStatus.CREATED) .setCreatedAt(Instant.now().toEpochMilli()) .build(); producer.send(new ProducerRecord<>("order-events", event.getOrderId().toString(), event)); Setting auto.register.schemas=false and use.latest.version=true is mandatory for production to avoid accidental registration of unverified schemas.
How to integrate compatibility checks into CI/CD?
Before deploying a new service version, run a script to check compatibility:
#!/bin/bash SCHEMA_FILE="src/main/avro/OrderEvent.avsc" SUBJECT="order-events-value" REGISTRY_URL="http://schema-registry:8081" SCHEMA_JSON=$(jq -c . "$SCHEMA_FILE") RESPONSE=$(curl -s -X POST \ "${REGISTRY_URL}/compatibility/subjects/${SUBJECT}/versions/latest" \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d "{\"schema\": $(echo $SCHEMA_JSON | jq -R .)}") COMPATIBLE=$(echo $RESPONSE | jq -r '.is_compatible') if [ "$COMPATIBLE" != "true" ]; then echo "FAIL: Schema is not compatible: $RESPONSE" exit 1 fi echo "OK: Schema is backward compatible" If compatibility is violated — the pipeline fails, and the incompatible schema does not reach production.
What's included in turnkey Schema Registry setup
- Deploy Schema Registry in production (minimum 2 nodes)
- Define and register Avro schemas for all topics
- Configure compatibility modes (BACKWARD / FORWARD / FULL)
- Integrate producers (Java/Python) with serializers
- Add compatibility checks to CI/CD pipeline
- Document schema evolution process for the team
- Train developers (1 day)
Timeline: from 3 to 5 days depending on the number of topics. Cost is calculated individually.
Monitoring and support
Schema Registry exports Prometheus metrics. We configure alerts for incompatible changes and master failures. We guarantee: after setup, no incompatible change reaches production. Experience: over 5 years of working with Kafka, 50+ projects. Order a consultation on your Kafka architecture — we will help implement Schema Registry in 3 days.