Excellent question. This is one of the most frequently asked Architect interview questions from DDIA.
The statement:
"Unknown fields should be preserved, not destroyed."
is true for Protocol Buffers (modern implementations) but not generally true for Avro. This is a key difference that interviewers often test.
First understand the problem
Suppose we have three services:
Order Service (V2)
|
| orderId, amount, couponCode
|
V
Kafka
|
V
Inventory Service (V1)
V2 sends
{
"orderId":101,
"amount":1000,
"couponCode":"NEW100"
}
Inventory Service V1 knows only
orderId
amount
It doesn't know couponCode.
What happens in Avro?
Assume the schemas are:
Producer Schema (V2)
{
"type":"record",
"name":"Order",
"fields":[
{"name":"orderId","type":"int"},
{"name":"amount","type":"double"},
{"name":"couponCode","type":["null","string"],"default":null}
]
}
Consumer Schema (V1)
{
"type":"record",
"name":"Order",
"fields":[
{"name":"orderId","type":"int"},
{"name":"amount","type":"double"}
]
}
During deserialization,
Avro matches fields by name.
It reads
orderId ✔
amount ✔
couponCode ❌ Unknown
Unknown field is simply ignored.
The consumer object becomes
Order
orderId = 101
amount = 1000
couponCode never exists inside the Java object.
If Inventory republishes the message
Suppose
Producer
↓
Kafka
↓
Inventory
↓
Kafka Again
Inventory reads
orderId
amount
and writes again.
The new message becomes
{
"orderId":101,
"amount":1000
}
The field
couponCode
is lost.
This is because Avro does not automatically preserve unknown fields during normal deserialize → serialize cycles.
Why?
Avro is schema resolution based.
It creates an object using only fields present in the reader schema.
Unknown fields are discarded during deserialization.
So they cannot be written back later unless the application explicitly carries them along.
Protocol Buffers behave differently
Protocol Buffers internally stores unknown fields.
Imagine
Producer
↓
Protobuf Binary
↓
Consumer
Consumer understands
orderId
amount
Unknown
couponCode
is kept inside an internal UnknownFieldSet.
The object looks conceptually like:
Order
orderId
amount
UnknownFieldSet
|
couponCode
If the consumer forwards the message,
serialize()
includes
couponCode
again.
Nothing is lost.
Visual Comparison
Avro
Producer V2
↓
orderId
amount
couponCode
↓
Consumer V1
↓
Reads
orderId
amount
↓
couponCode discarded ❌
↓
Writes again
↓
orderId
amount
Protocol Buffers
Producer V2
↓
orderId
amount
couponCode
↓
Consumer V1
↓
Reads
orderId
amount
↓
UnknownFieldSet
↓
couponCode preserved
↓
Serialize again
↓
orderId
amount
couponCode
How does Avro achieve compatibility then?
Avro's strength is schema evolution, not unknown field preservation.
It achieves compatibility using writer schema + reader schema resolution:
The writer's schema (or its ID via a Schema Registry) is available.
The reader uses its own schema.
Avro resolves differences:
Matching fields are read.
New writer fields missing from the reader are ignored.
Reader fields missing from the writer use default values (if defined).
The application works without errors, but unknown fields are not retained after deserialization.
Real-world Kafka Example
Order Service V2
↓
Kafka
↓
Payment Service V1
↓
Notification Service V2
If Payment Service only validates payment and republishes the event:
With Avro: If it deserializes into a V1 object and reserializes,
couponCodeis lost unless the application preserves it explicitly.With Protocol Buffers: The unknown field is retained automatically and forwarded.
Architect Interview Answer ⭐
Question: How does Avro preserve unknown fields?
Answer:
Avro generally does not preserve unknown fields during a deserialize–serialize cycle. It resolves schema differences using the writer and reader schemas, ignoring fields that the reader doesn't know. This enables schema compatibility, but unknown fields are discarded unless the application explicitly carries them forward. In contrast, Protocol Buffers stores unknown fields internally (using an
UnknownFieldSetin many implementations) so they can be reserialized without loss.
This distinction between schema compatibility (Avro) and unknown field preservation (Protocol Buffers) is an important concept for distributed systems and frequently comes up in Senior Java Architect interviews.
