Yes. This is an important Kafka + Dead Letter Topic (DLT) interview topic, and there is one subtle point in your notes that interviewers often test: producer retry failure is not the same thing as consumer processing failure.
1. First understand the complete flow
Suppose an Order Service consumes an event and then publishes another Kafka event.
Kafka Input Topic
│
▼
Consumer Service
│
│ process
▼
Kafka Producer
│
│ send()
▼
Output TopicIf the producer temporarily cannot get an acknowledgement:
Producer
│
│ send()
▼
Kafka Broker
│
│ ACK lost / timeout
X
ProducerKafka producer retries the send if retries are enabled.
Producer
│
├──── Attempt 1 ────► Broker
│ X
│
├──── wait 500 ms
│
├──── Attempt 2 ────► Broker
│ X
│
├──── wait 500 ms
│
└──── Attempt 3 ────► Broker
│
▼
ACKSo:
Retries are the first line of defence against transient Kafka failures. DLT is generally for messages that ultimately cannot be processed successfully.
2. Why retries=0 can be dangerous
Suppose:
Producer
│
│ send Order-101
▼
Kafka Broker
│
│ Write succeeds ✓
│
│ ACK lost
X
Producer thinks → FAILUREThe producer may not know that Kafka actually stored the message.
If:
retries = 0the application can immediately treat the send as failed.
This can result in:
Kafka actually has:
Order-101 ✓
Application thinks:
Order-101 ❌If application-level error handling then sends the original message to a DLT, you could potentially have:
Main Topic
│
└── Order-101 ✓
DLT
│
└── Order-101 ✓Now you have a duplicate representation of the event.
Interview statement
I generally keep producer retries enabled for transient failures. Setting
retries=0can unnecessarily convert recoverable failures into application-level failures and can be especially tricky when the broker accepted the record but the acknowledgement was lost.
3. Understand the four timeout/retry settings
Your configuration:
request.timeout.ms = 10000
delivery.timeout.ms = 60000
retry.backoff.ms = 500
max.block.ms = 10000Think of them as different clocks.
request.timeout.ms = 10000
This is the timeout for one request.
Producer
│
│ Request
▼
Broker
│
│
│ maximum wait ≈ 10 sec
▼
Response / TimeoutIt does not mean:
"The whole send operation gets only 10 seconds."
It is per request attempt.
4. delivery.timeout.ms = 60000
This is the overall delivery deadline.
Think:
delivery.timeout.ms
60 sec
┌──────────────────────────┐
│ │
▼ ▼
Attempt 1 Retry Attempt 2 Retry ...
│ │
└──────────► Success / FailureKafka producer must successfully deliver the record within this overall time.
If it cannot:
60 seconds exceeded
│
▼
Send failsInterview answer
request.timeout.mscontrols how long an individual request waits for a broker response, whiledelivery.timeout.mscontrols the total time allowed for the record to be successfully delivered, including retries.
5. retry.backoff.ms = 500
This controls the delay before retrying.
Attempt 1
│
X
│
│ 500 ms
▼
Attempt 2
│
X
│
│ 500 ms
▼
Attempt 3Why have a delay?
Because immediately hammering a temporarily unhealthy broker isn't useful.
Failure
│
▼
wait
│
▼
retryThis gives the broker/network a small amount of time to recover.
6. max.block.ms = 10000
This one is different.
It doesn't primarily control broker response time.
It controls how long producer operations can block while waiting for things such as:
metadata
buffer space
Example:
Producer
│
▼
KafkaProducer.send()
│
├── metadata available?
│
└── buffer space available?
│
▼
wait if necessary
│
max 10 secAfter the configured blocking period, the producer operation can fail.
Easy way to remember
request.timeout.ms
↓
"How long do I wait for THIS broker request?"
delivery.timeout.ms
↓
"How long do I give THIS RECORD to get delivered?"
retry.backoff.ms
↓
"How long do I wait BEFORE retrying?"
max.block.ms
↓
"How long can the producer operation BLOCK waiting for metadata/buffer?"7. The most important issue: Consumer + Producer
This is where your notes become senior-level Kafka knowledge.
Imagine:
Input Topic
│
▼
Consumer
│
▼
Process
│
▼
Produce Output TopicSuppose processing the message takes too long because the producer keeps retrying.
Consumer poll
│
▼
Process message
│
▼
Kafka Producer
│
├── retry
├── retry
├── retry
└── retry
│
▼
60 secondsMeanwhile, the consumer isn't polling Kafka.
If this takes longer than the consumer's configured polling interval:
Consumer
│
│ processing
│
│ no poll for too long
▼
Kafka detects consumer timeout
│
▼
RebalanceAnother consumer can receive the same partition/message.
8. Example of duplicate processing
Suppose:
Consumer Group
C1 → Partition 0
C2 → Partition 1C1 receives:
Order-101Then C1 starts processing:
C1
│
├── process Order-101
│
├── producer retry
│
├── producer retry
│
└── still processing...C1 doesn't poll within the allowed interval.
Kafka causes a rebalance:
C1
│
X
│
▼
Rebalance
C2
│
▼
Partition 0C2 may now receive:
Order-101So:
C1 → Order-101
│
│ processing
▼
Rebalance
│
▼
C2 → Order-101 againThis is why idempotency is critical.
9. Consumer max.poll.interval.ms
This is the configuration you should connect with the concept above.
For example:
max.poll.interval.ms = 300000means the consumer must call poll() within that interval.
Conceptually:
poll()
│
▼
process records
│
▼
poll()If processing takes longer than the allowed interval:
poll()
│
▼
very long processing
│
│ > max.poll.interval.ms
▼
Consumer considered failed
│
▼
RebalanceSenior interview statement
When a consumer triggers downstream Kafka publishing, I make sure the maximum processing/retry duration is compatible with
max.poll.interval.ms. Otherwise long processing can trigger a rebalance and result in duplicate processing.
10. Where does DLT fit?
Now let's put everything together.
Main Topic
│
▼
Consumer
│
▼
Process
│
┌──────────┴──────────┐
│ │
Success Failure
│ │
▼ ▼
Commit Retry
│
┌───────────┴──────────┐
│ │
Success Failure
│ │
▼ ▼
Commit Retry
│
Max retries
│
▼
DLTFor example:
orders
│
▼
Order Consumer
│
├── attempt 1 → failure
│
├── attempt 2 → failure
│
├── attempt 3 → failure
│
└── max retries reached
│
▼
orders.DLT11. But not every error should be retried
This is another excellent interview question.
Transient error
Example:
Database temporarily unavailable
Network timeout
HTTP 503
Kafka broker temporarily unavailableRetry makes sense:
Failure
│
▼
Retry
│
▼
Success ✓Permanent error
Example:
Invalid JSON
Invalid customer ID
Missing mandatory field
Business validation failureRetrying 10 times won't fix it.
Invalid Order
│
▼
Retry
│
▼
Same Invalid Order
│
▼
Retry
│
▼
Same Invalid OrderWasteful.
Instead:
Invalid message
│
▼
DLT12. Real-world architecture
A robust Kafka architecture might look like:
┌──────────────────┐
│ Order Service │
└────────┬─────────┘
│
▼
Kafka orders
│
▼
┌──────────────────┐
│ Order Consumer │
└────────┬─────────┘
│
┌──────────┴──────────┐
│ │
Success Failure
│ │
▼ ▼
Commit Offset Retry
│
┌───────┴───────┐
│ │
Success Failure
│ │
▼ ▼
Commit DLT
│
▼
Investigation /
Replay13. What should happen to a DLT message?
DLT shouldn't simply become a graveyard.
A good production architecture is:
DLT
│
┌─────────┴─────────┐
│ │
▼ ▼
Monitoring Investigation
│
▼
Fix problem
│
▼
Replay
│
▼
Main TopicExample:
orders.DLT
│
│ investigate why Order-101 failed
▼
Fix consumer bug
│
▼
Replay Order-101
│
▼
orders14. Very important: DLT doesn't automatically solve duplicates
Suppose:
Consumer
│
▼
Process payment
│
▼
Payment DB ✓
│
X
Offset commit failsKafka may redeliver:
Same Order
│
▼
Consumer again
│
▼
Payment DBWithout idempotency:
₹1000 payment
₹1000 payment
↓
₹2000 ❌Therefore:
Kafka retries
+
Idempotent consumer
+
DLTis much safer.
15. Best interview answer
If the interviewer asks:
"Explain Kafka retries, timeouts and DLT."
You can answer:
"I normally keep Kafka producer retries enabled because transient broker or network failures should be retried automatically.
request.timeout.mscontrols the timeout for an individual broker request,retry.backoff.mscontrols the delay between retry attempts, anddelivery.timeout.msdefines the overall delivery deadline for a record.max.block.mscontrols how long producer operations can block waiting for metadata or buffer availability.If Kafka publishing is performed as part of consumer processing, I also make sure the retry and delivery duration is compatible with the consumer's
max.poll.interval.ms; otherwise long processing can trigger a rebalance and cause another consumer to process the same record.For consumer processing failures, transient errors should be retried, while permanent or repeatedly failing messages can eventually be sent to a Dead Letter Topic. The consumer should also be idempotent because at-least-once processing can result in duplicate delivery."
That is a strong Senior Java/Spring Boot Architect answer.
One final diagram to remember
Kafka Input Topic
│
▼
Consumer
│
▼
Processing
│
┌───────────┴───────────┐
│ │
Success Failure
│ │
▼ ▼
Produce/Commit Retry
│
┌────────┴────────┐
│ │
Success Failure
│ │
▼ ▼
Commit DLT
│
▼
Replay
Producer retry side:
send()
│
▼
Broker
│
X timeout
│
▼
wait 500ms
│
▼
retry
│
▼
success / delivery timeout
Consumer safety:
poll()
│
▼
process + producer retry
│
│ must fit within
▼
max.poll.interval.ms
│
├── within limit → poll again ✓
│
└── exceeds limit → rebalance → possible duplicateThe key distinction to memorize: Producer retries handle Kafka delivery problems; consumer retries/DLT handle application processing problems. They are related, but they are not the same retry mechanism.
No comments:
Post a Comment