Wednesday, 26 August 2026

Kafka - Retries - Dead Letter Queue

Yes. This is an important Kafka + Dead Letter Topic (DLT) interview topic, and there is one subtle point in your notes that interviewers often test: producer retry failure is not the same thing as consumer processing failure.

1. First understand the complete flow

Suppose an Order Service consumes an event and then publishes another Kafka event.

              Kafka Input Topic
                    │
                    ▼
             Consumer Service
                    │
                    │ process
                    ▼
             Kafka Producer
                    │
                    │ send()
                    ▼
             Output Topic

If the producer temporarily cannot get an acknowledgement:

Producer
   │
   │ send()
   ▼
Kafka Broker
   │
   │ ACK lost / timeout
   X
Producer

Kafka producer retries the send if retries are enabled.

Producer
   │
   ├──── Attempt 1 ────► Broker
   │                         X
   │
   ├──── wait 500 ms
   │
   ├──── Attempt 2 ────► Broker
   │                         X
   │
   ├──── wait 500 ms
   │
   └──── Attempt 3 ────► Broker
                             │
                             ▼
                            ACK

So:

Retries are the first line of defence against transient Kafka failures. DLT is generally for messages that ultimately cannot be processed successfully.

2. Why `retries=0` can be dangerous

Suppose:

Producer
   │
   │ send Order-101
   ▼
Kafka Broker
   │
   │ Write succeeds ✓
   │
   │ ACK lost
   X
Producer thinks → FAILURE

The producer may not know that Kafka actually stored the message.

If:

retries = 0

the application can immediately treat the send as failed.

This can result in:

Kafka actually has:
Order-101 ✓

Application thinks:
Order-101 ❌

If application-level error handling then sends the original message to a DLT, you could potentially have:

Main Topic
   │
   └── Order-101 ✓
   
DLT
   │
   └── Order-101 ✓

Now you have a duplicate representation of the event.

Interview statement

I generally keep producer retries enabled for transient failures. Setting retries=0 can unnecessarily convert recoverable failures into application-level failures and can be especially tricky when the broker accepted the record but the acknowledgement was lost.

3. Understand the four timeout/retry settings

Your configuration:

request.timeout.ms    = 10000
delivery.timeout.ms   = 60000
retry.backoff.ms      = 500
max.block.ms          = 10000

Think of them as different clocks.

`request.timeout.ms = 10000`

This is the timeout for one request.

Producer
   │
   │ Request
   ▼
Broker
   │
   │
   │ maximum wait ≈ 10 sec
   ▼
Response / Timeout

It does not mean:

"The whole send operation gets only 10 seconds."

It is per request attempt.

4. `delivery.timeout.ms = 60000`

This is the overall delivery deadline.

Think:

              delivery.timeout.ms
                    60 sec
       ┌──────────────────────────┐
       │                          │
       ▼                          ▼
   Attempt 1   Retry   Attempt 2   Retry ... 
       │                          │
       └──────────► Success / Failure

Kafka producer must successfully deliver the record within this overall time.

If it cannot:

60 seconds exceeded
       │
       ▼
Send fails

Interview answer

request.timeout.ms controls how long an individual request waits for a broker response, while delivery.timeout.ms controls the total time allowed for the record to be successfully delivered, including retries.

5. `retry.backoff.ms = 500`

This controls the delay before retrying.

Attempt 1
   │
   X
   │
   │ 500 ms
   ▼
Attempt 2
   │
   X
   │
   │ 500 ms
   ▼
Attempt 3

Why have a delay?

Because immediately hammering a temporarily unhealthy broker isn't useful.

Failure
   │
   ▼
wait
   │
   ▼
retry

This gives the broker/network a small amount of time to recover.

6. `max.block.ms = 10000`

This one is different.

It doesn't primarily control broker response time.

It controls how long producer operations can block while waiting for things such as:

metadata
buffer space

Example:

Producer
   │
   ▼
KafkaProducer.send()
   │
   ├── metadata available?
   │
   └── buffer space available?
            │
            ▼
       wait if necessary
            │
         max 10 sec

After the configured blocking period, the producer operation can fail.

Easy way to remember

request.timeout.ms
        ↓
"How long do I wait for THIS broker request?"

delivery.timeout.ms
        ↓
"How long do I give THIS RECORD to get delivered?"

retry.backoff.ms
        ↓
"How long do I wait BEFORE retrying?"

max.block.ms
        ↓
"How long can the producer operation BLOCK waiting for metadata/buffer?"

7. The most important issue: Consumer + Producer

This is where your notes become senior-level Kafka knowledge.

Imagine:

Input Topic
     │
     ▼
Consumer
     │
     ▼
Process
     │
     ▼
Produce Output Topic

Suppose processing the message takes too long because the producer keeps retrying.

Consumer poll
     │
     ▼
Process message
     │
     ▼
Kafka Producer
     │
     ├── retry
     ├── retry
     ├── retry
     └── retry
           │
           ▼
       60 seconds

Meanwhile, the consumer isn't polling Kafka.

If this takes longer than the consumer's configured polling interval:

Consumer
   │
   │ processing
   │
   │ no poll for too long
   ▼
Kafka detects consumer timeout
   │
   ▼
Rebalance

Another consumer can receive the same partition/message.

8. Example of duplicate processing

Suppose:

Consumer Group

C1 → Partition 0
C2 → Partition 1

C1 receives:

Order-101

Then C1 starts processing:

C1
 │
 ├── process Order-101
 │
 ├── producer retry
 │
 ├── producer retry
 │
 └── still processing...

C1 doesn't poll within the allowed interval.

Kafka causes a rebalance:

C1
 │
 X
 │
 ▼
Rebalance

C2
 │
 ▼
Partition 0

C2 may now receive:

Order-101

So:

C1 → Order-101
          │
          │ processing
          ▼
       Rebalance
          │
          ▼
C2 → Order-101 again

This is why idempotency is critical.

9. Consumer `max.poll.interval.ms`

This is the configuration you should connect with the concept above.

For example:

max.poll.interval.ms = 300000

means the consumer must call poll() within that interval.

Conceptually:

poll()
  │
  ▼
process records
  │
  ▼
poll()

If processing takes longer than the allowed interval:

poll()
  │
  ▼
very long processing
  │
  │ > max.poll.interval.ms
  ▼
Consumer considered failed
  │
  ▼
Rebalance

Senior interview statement

When a consumer triggers downstream Kafka publishing, I make sure the maximum processing/retry duration is compatible with max.poll.interval.ms. Otherwise long processing can trigger a rebalance and result in duplicate processing.

10. Where does DLT fit?

Now let's put everything together.

                 Main Topic
                     │
                     ▼
                Consumer
                     │
                     ▼
                 Process
                     │
          ┌──────────┴──────────┐
          │                     │
       Success                Failure
          │                     │
          ▼                     ▼
       Commit              Retry
                                │
                    ┌───────────┴──────────┐
                    │                      │
                 Success                 Failure
                    │                      │
                    ▼                      ▼
                 Commit                   Retry
                                           │
                                      Max retries
                                           │
                                           ▼
                                          DLT

For example:

orders
   │
   ▼
Order Consumer
   │
   ├── attempt 1 → failure
   │
   ├── attempt 2 → failure
   │
   ├── attempt 3 → failure
   │
   └── max retries reached
              │
              ▼
        orders.DLT

11. But not every error should be retried

This is another excellent interview question.

Transient error

Example:

Database temporarily unavailable
Network timeout
HTTP 503
Kafka broker temporarily unavailable

Retry makes sense:

Failure
  │
  ▼
Retry
  │
  ▼
Success ✓

Permanent error

Example:

Invalid JSON
Invalid customer ID
Missing mandatory field
Business validation failure

Retrying 10 times won't fix it.

Invalid Order
     │
     ▼
Retry
     │
     ▼
Same Invalid Order
     │
     ▼
Retry
     │
     ▼
Same Invalid Order

Wasteful.

Instead:

Invalid message
      │
      ▼
     DLT

12. Real-world architecture

A robust Kafka architecture might look like:

                       ┌──────────────────┐
                       │   Order Service  │
                       └────────┬─────────┘
                                │
                                ▼
                         Kafka orders
                                │
                                ▼
                       ┌──────────────────┐
                       │ Order Consumer   │
                       └────────┬─────────┘
                                │
                     ┌──────────┴──────────┐
                     │                     │
                  Success               Failure
                     │                     │
                     ▼                     ▼
                Commit Offset           Retry
                                           │
                                   ┌───────┴───────┐
                                   │               │
                                Success          Failure
                                   │               │
                                   ▼               ▼
                                Commit            DLT
                                                   │
                                                   ▼
                                            Investigation /
                                               Replay

13. What should happen to a DLT message?

DLT shouldn't simply become a graveyard.

A good production architecture is:

                 DLT
                  │
        ┌─────────┴─────────┐
        │                   │
        ▼                   ▼
   Monitoring            Investigation
                            │
                            ▼
                       Fix problem
                            │
                            ▼
                         Replay
                            │
                            ▼
                       Main Topic

Example:

orders.DLT
    │
    │ investigate why Order-101 failed
    ▼
Fix consumer bug
    │
    ▼
Replay Order-101
    │
    ▼
orders

14. Very important: DLT doesn't automatically solve duplicates

Suppose:

Consumer
   │
   ▼
Process payment
   │
   ▼
Payment DB ✓
   │
   X
Offset commit fails

Kafka may redeliver:

Same Order
    │
    ▼
Consumer again
    │
    ▼
Payment DB

Without idempotency:

₹1000 payment
₹1000 payment
     ↓
₹2000 ❌

Therefore:

Kafka retries
      +
Idempotent consumer
      +
DLT

is much safer.

15. Best interview answer

If the interviewer asks:

"Explain Kafka retries, timeouts and DLT."

You can answer:

"I normally keep Kafka producer retries enabled because transient broker or network failures should be retried automatically. request.timeout.ms controls the timeout for an individual broker request, retry.backoff.ms controls the delay between retry attempts, and delivery.timeout.ms defines the overall delivery deadline for a record. max.block.ms controls how long producer operations can block waiting for metadata or buffer availability.
If Kafka publishing is performed as part of consumer processing, I also make sure the retry and delivery duration is compatible with the consumer's max.poll.interval.ms; otherwise long processing can trigger a rebalance and cause another consumer to process the same record.
For consumer processing failures, transient errors should be retried, while permanent or repeatedly failing messages can eventually be sent to a Dead Letter Topic. The consumer should also be idempotent because at-least-once processing can result in duplicate delivery."

That is a strong Senior Java/Spring Boot Architect answer.

One final diagram to remember

                 Kafka Input Topic
                         │
                         ▼
                    Consumer
                         │
                         ▼
                    Processing
                         │
             ┌───────────┴───────────┐
             │                       │
          Success                  Failure
             │                       │
             ▼                       ▼
       Produce/Commit              Retry
                                     │
                            ┌────────┴────────┐
                            │                 │
                         Success           Failure
                            │                 │
                            ▼                 ▼
                         Commit              DLT
                                              │
                                              ▼
                                           Replay


Producer retry side:

send()
  │
  ▼
Broker
  │
  X timeout
  │
  ▼
wait 500ms
  │
  ▼
retry
  │
  ▼
success / delivery timeout


Consumer safety:

poll()
  │
  ▼
process + producer retry
  │
  │ must fit within
  ▼
max.poll.interval.ms
  │
  ├── within limit → poll again ✓
  │
  └── exceeds limit → rebalance → possible duplicate

The key distinction to memorize: Producer retries handle Kafka delivery problems; consumer retries/DLT handle application processing problems. They are related, but they are not the same retry mechanism.

Kafka - Interview Questions - important shard router -> kafka producer find the logic moview to orders

Absolutely. For a Java/Spring Boot senior architect interview, Kafka questions are often asked from basic concepts → architecture → delivery semantics → partitions → consumer groups → failures → performance → real-world design.

Below is an interview-focused set with examples and text-flow diagrams.

Kafka Interview Questions & Answers

1. What is Apache Kafka?

Answer:

Kafka is a distributed event-streaming platform used to publish, store, and consume streams of events/messages at high scale.

Typical architecture:

                    ┌───────────────┐
                    │   Producer    │
                    │ Spring Boot   │
                    └───────┬───────┘
                            │
                            │ Event
                            ▼
                    ┌───────────────┐
                    │     Kafka     │
                    │    Topic      │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │   Consumer    │
                    │ Spring Boot   │
                    └───────────────┘

Example:

Order Service
     │
     │ OrderCreated
     ▼
Kafka: order-events
     │
     ├──────────────► Payment Service
     │
     ├──────────────► Inventory Service
     │
     └──────────────► Notification Service

The important advantage is that producer and consumers are decoupled.

2. Kafka vs traditional message queue?

A common interview question.

Traditional queue

Producer
   │
   ▼
 Queue
   │
   ▼
Consumer

Usually, once the consumer successfully processes the message, the message is removed/acknowledged.

Kafka

Producer
   │
   ▼
Kafka Topic
   │
   ├── Consumer Group A
   ├── Consumer Group B
   └── Consumer Group C

Kafka stores events for a configured retention period.

For example:

Topic: orders

Offset
  0     Order A
  1     Order B
  2     Order C
  3     Order D

Consumer A can read:

0 → 1 → 2 → 3

Later, another consumer can independently read the same events.

Interview point:

Kafka is not simply a queue. It is a distributed, durable, partitioned event log.

3. What is a Kafka Topic?

A topic is a logical category/name to which producers publish messages.

Example:

order-events
payment-events
customer-events
notification-events

Flow:

Order Service
     │
     │ publish()
     ▼
┌──────────────────────┐
│ Topic: order-events  │
└──────────────────────┘
     │
     ├── Payment Consumer
     ├── Inventory Consumer
     └── Notification Consumer

A topic is divided into partitions.

order-events

Partition 0
[0] [1] [2] [3]

Partition 1
[0] [1] [2] [3]

Partition 2
[0] [1] [2] [3]

4. What is a Kafka Partition?

A partition is the fundamental unit of parallelism and storage in Kafka.

Example:

Topic: orders

Partition 0
   │
   ├── Order-101
   ├── Order-104
   └── Order-108

Partition 1
   │
   ├── Order-102
   ├── Order-105
   └── Order-109

Partition 2
   │
   ├── Order-103
   ├── Order-106
   └── Order-110

Each partition maintains message ordering.

Important interview statement

Kafka guarantees ordering within a partition, not across the entire topic.

5. Why does Kafka use partitions?

Mainly for:

1. Parallelism

                Topic
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
      P0         P1         P2
       │          │          │
       ▼          ▼          ▼
 Consumer-1   Consumer-2   Consumer-3

Three consumers can process three partitions simultaneously.

2. Scalability

Partitions can be distributed across multiple Kafka brokers.

Broker 1
  └── P0

Broker 2
  └── P1

Broker 3
  └── P2

6. What is a Kafka Broker?

A broker is a Kafka server.

A Kafka cluster can contain multiple brokers.

                 Kafka Cluster
        ┌────────────┼────────────┐
        ▼            ▼            ▼
    Broker-1     Broker-2     Broker-3
       │             │             │
      P0             P1            P2

Multiple brokers provide:

scalability
fault tolerance
replication
load distribution

7. What is Replication Factor?

Replication factor tells how many copies of a partition Kafka maintains.

Suppose:

Replication Factor = 3

Then:

Partition P0

Broker 1 → Leader
Broker 2 → Replica
Broker 3 → Replica

If Broker 1 fails:

Broker 1 ❌

Broker 2 → becomes Leader
Broker 3 → Replica

This provides fault tolerance.

8. What is a Kafka Leader and Follower?

For each partition, one replica is the leader.

Other replicas are followers.

Partition P0

Broker 1
  └── Leader

Broker 2
  └── Follower

Broker 3
  └── Follower

Producer normally sends writes to the leader.

Producer
   │
   ▼
Broker 1
Leader
   │
   ├────────► Broker 2
   │           Follower
   │
   └────────► Broker 3
               Follower

If the leader fails, Kafka can elect another replica.

9. What is an Offset?

An offset uniquely identifies the position of a record within a partition.

Example:

Partition 0

Offset     Message
  0        Order-101
  1        Order-102
  2        Order-103
  3        Order-104

Offset is maintained per partition.

Important:

P0 → offset 10
P1 → offset 25
P2 → offset 17

There is no single global offset for the entire topic.

10. What is a Consumer Group?

A consumer group is a group of consumers that collectively consume a topic.

Example:

Topic
 ├── P0
 ├── P1
 └── P2

Consumer Group: payment-service

 ├── Consumer-1 → P0
 ├── Consumer-2 → P1
 └── Consumer-3 → P2

Each partition is assigned to only one consumer within a consumer group at a time.

11. Can two consumers in the same group consume the same partition?

Normally No.

Example:

Topic
 P0
 P1
 P2

Consumer Group A

 C1 → P0
 C2 → P1
 C3 → P2

But different consumer groups can independently consume the same partition:

                Topic
                  │
                  ▼
                 P0
               /    \
              /      \
             ▼        ▼
      Consumer Group A   Consumer Group B
             │                  │
      Payment Service     Analytics Service

This is a very important interview concept.

12. What happens if consumers > partitions?

Suppose:

3 partitions
5 consumers

Only 3 consumers can actively consume.

P0 → Consumer-1
P1 → Consumer-2
P2 → Consumer-3

Consumer-4 → idle
Consumer-5 → idle

Therefore:

Maximum active consumers in a consumer group = number of partitions.

13. What happens if consumers < partitions?

Suppose:

3 partitions
2 consumers

Kafka distributes multiple partitions to consumers.

Consumer-1 → P0 + P1
Consumer-2 → P2

Therefore partition count is important when designing consumer scalability.

14. How does Kafka guarantee ordering?

Kafka guarantees ordering inside a partition.

Example:

P0

Offset 0 → Order Created
Offset 1 → Payment Started
Offset 2 → Payment Completed

A consumer reads:

0 → 1 → 2

in order.

But:

P0:
Order A
Order B

P1:
Order C
Order D

Kafka does not guarantee:

A → B → C → D

across partitions.

15. How do you guarantee ordering for a particular customer/order?

Use a key.

Example:

Producer
   │
   │ key = customerId
   ▼
Kafka

Kafka hashes the key to determine the partition.

customer-101 → P2
customer-101 → P2
customer-101 → P2

Therefore:

Customer-101 events

Event 1 → P2
Event 2 → P2
Event 3 → P2

Their relative order is preserved.

Interview answer

If ordering is required for an entity, use a stable key such as orderId or customerId so all events for that entity go to the same partition.

16. How does Kafka Producer decide which partition to use?

Common cases:

Key specified

key = customerId

       │
       ▼
hash(customerId)
       │
       ▼
partition

Example:

customer-100 → P0
customer-101 → P2
customer-102 → P1

No key

Kafka can distribute records across partitions according to the producer's partitioning behavior/configuration.

17. What is Producer Acknowledgement (`acks`)?

One of the most common Kafka interview questions.

`acks=0`

Producer doesn't wait for broker acknowledgement.

Producer ─────────► Kafka
       returns immediately

Fast but riskier.

`acks=1`

Leader acknowledges after writing the record.

Producer
   │
   ▼
Leader
   │
   │ ACK
   ▼
Producer

Good balance of performance and durability.

`acks=all`

Leader waits for the required in-sync replicas to acknowledge.

Producer
   │
   ▼
Leader
   │
   ├────► Replica 1
   │
   └────► Replica 2
          │
          ▼
        ACK

Highest durability among these settings, generally with more latency.

18. What is ISR?

ISR = In-Sync Replicas.

Suppose:

P0

Broker 1 → Leader
Broker 2 → ISR
Broker 3 → ISR

All are sufficiently caught up with the leader.

If Broker 3 becomes too far behind:

Broker 1 → Leader
Broker 2 → ISR
Broker 3 → Out of Sync

ISR is important for:

durability
leader election
replication health

19. What is Consumer Offset Commit?

Consumers need to remember which messages they have processed.

Example:

Consumer processed:

0
1
2

It commits its position.

Conceptually:

Committed Offset = 3

If consumer crashes:

Consumer
    ↓
   ❌

New consumer can resume from the committed position.

20. What is Consumer Rebalancing?

Suppose:

P0 → C1
P1 → C2
P2 → C3

C2 crashes.

Kafka detects the consumer group membership change and redistributes partitions.

Before:

C1 → P0
C2 → P1
C3 → P2

C2 ❌

After:

C1 → P0 + P1
C3 → P2

This is called rebalance.

21. What is at-most-once delivery?

Message can be processed zero or one time, so messages may be lost but are not normally redelivered.

Typical approach:

Read message
   │
   ▼
Commit offset
   │
   ▼
Process message

If application crashes after commit but before processing:

Commit ✓
Process ❌

The message may effectively be lost.

22. What is at-least-once delivery?

Message is processed one or more times.

Typical flow:

Read
 │
 ▼
Process
 │
 ▼
Commit offset

If processing succeeds but the application crashes before committing:

Process ✓
Commit ❌
Crash

After restart:

Same message
     ↓
Processed again

So duplicates are possible.

Interview answer

At-least-once delivery provides stronger durability but requires consumers to be idempotent.

23. What is exactly-once processing?

Exactly-once semantics aim to ensure that an event's effect is applied exactly once within the supported Kafka processing/transaction model.

A simplified flow:

Consumer
   │
   ▼
Process
   │
   ├── Produce result
   │
   └── Commit offset
          │
          ▼
       Transaction

Kafka supports transactional mechanisms for atomic processing of Kafka records and produced records.

Senior-level interview point:

Don't simply say:

"Kafka guarantees exactly once."

Better:

Kafka provides exactly-once semantics for supported transactional Kafka workflows, but end-to-end exactly-once behavior with an external database or external API requires additional design, such as idempotency or transactional/outbox patterns.

24. What is Idempotency?

An operation is idempotent if executing it multiple times produces the same final result.

Example:

Kafka sends:

OrderId = 101
Payment = ₹1000

Consumer receives it twice.

Without idempotency:

₹1000 charged
₹1000 charged

Total = ₹2000 ❌

With idempotency:

Payment event 101
      │
      ▼
Check processed_event table
      │
      ├── Already processed → Ignore
      │
      └── New → Process

Result:

₹1000 charged ✓

25. Kafka Retry Pattern

Suppose payment processing fails.

Kafka
  │
  ▼
Payment Consumer
  │
  ▼
Payment Service
  │
  X
Failure

Possible retry architecture:

                 ┌──────────────┐
                 │ order-events │
                 └──────┬───────┘
                        │
                        ▼
                Payment Consumer
                        │
                     Failure
                        │
                        ▼
                  retry-topic
                        │
                        ▼
                Payment Consumer
                        │
                     Failure
                        │
                        ▼
                     DLQ

DLQ = Dead Letter Queue/Topic.

26. What is a Dead Letter Topic?

Messages that repeatedly fail processing can be sent to a DLQ/DLT.

Main Topic
    │
    ▼
Consumer
    │
    X
Repeated Failure
    │
    ▼
DLT

Example:

order-events
      │
      ▼
Order Consumer
      │
      X
Invalid message
      │
      ▼
order-events.DLT

Operations team can inspect and replay the failed events later.

27. How would you design Kafka for high throughput?

I would consider:

                    Kafka Cluster
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Broker-1     Broker-2     Broker-3
          │            │            │
         P0/P3        P1/P4        P2/P5

Key techniques:

Increase partitions appropriately.
Run multiple consumers in a consumer group.
Use batching.
Use compression.
Tune producer batch size/linger carefully.
Avoid unnecessary synchronous operations.
Scale brokers horizontally.
Monitor consumer lag.
Ensure adequate disk/network capacity.

28. What is Consumer Lag?

Consumer lag tells us how far behind a consumer is from the latest available records.

Example:

Latest offset = 1000
Consumer offset = 850

Lag = 150

Flow:

Producer
   │
   ▼
Kafka
   │
   │ 1000
   ▼
Consumer
   │
   │ processed till 850
   ▼

Lag = 150

High lag could indicate:

slow consumer
insufficient consumer instances
downstream dependency latency
too few partitions
processing bottleneck
broker/network issues

29. How would you troubleshoot increasing Kafka consumer lag?

My interview answer would be:

Consumer Lag increasing
        │
        ▼
Check consumer processing time
        │
        ├── Slow DB/API?
        │
        ├── CPU?
        │
        ├── GC?
        │
        ├── Network?
        │
        ├── Partition imbalance?
        │
        └── Insufficient consumers?

Then:

Check consumer lag per partition.
Check consumer CPU/memory.
Check processing latency.
Check downstream DB/API latency.
Check rebalance frequency.
Check partition distribution.
Increase consumers only if there are enough partitions.
Increase partitions if the architecture requires more parallelism.

30. Can we increase Kafka partitions later?

Yes.

For example:

Before:
Topic → 3 partitions

After:
Topic → 6 partitions

But there is an important caveat:

Increasing partitions can affect key-based partition distribution and therefore ordering behavior for future records.

So partition count should be planned carefully.

31. What happens when a Kafka broker goes down?

Suppose:

P0

Broker-1 → Leader
Broker-2 → Replica
Broker-3 → Replica

Broker-1 fails:

Broker-1 ❌

Broker-2 → New Leader
Broker-3 → Replica

Producers and consumers reconnect to the new leader.

This is why replication is important.

32. Kafka Architecture — Complete Flow

This is a good diagram to remember for interviews:

                 ┌─────────────────┐
                 │ Producer        │
                 │ Order Service   │
                 └────────┬────────┘
                          │
                          │ OrderCreated
                          ▼
                ┌────────────────────┐
                │    Kafka Cluster   │
                │                    │
                │ Topic: orders      │
                │                    │
                │ P0   P1   P2       │
                └──┬────┬────┬───────┘
                   │    │    │
                   ▼    ▼    ▼
                 C1     C2    C3
                  \      |    /
                   \     |   /
                 Consumer Group
                   Payment
                       │
                       ▼
                ┌──────────────┐
                │ Payment DB   │
                └──────────────┘

Another consumer group can independently consume the same topic:

                     Kafka Topic
                         │
            ┌────────────┴────────────┐
            │                         │
            ▼                         ▼
    Consumer Group A          Consumer Group B
    Payment Service           Analytics Service
            │                         │
            ▼                         ▼
       Payment DB               Data Lake

33. Kafka vs Redis Streams vs RabbitMQ

A senior architect may be asked this.

Feature	Kafka	RabbitMQ	Redis Streams
Primary model	Distributed log	Message broker	Stream
High throughput	Excellent	Good	Excellent
Long retention	Excellent	Less typical	Depends on configuration
Replay	Excellent	Not primary strength	Yes
Partitioning	Core feature	Different model	Different model
Consumer groups	Yes	Consumer patterns	Yes
Event streaming	Excellent	Good	Good
Best fit	Large event streams	Messaging/work queues	Low-latency streams

A good interview answer:

I would choose Kafka when durable event streaming, replay, high throughput and multiple independent consumers are important. RabbitMQ is often a better fit for traditional task/message routing patterns. Redis Streams can be attractive when very low latency and an existing Redis-centric architecture are priorities.

34. Real-world Spring Boot Kafka Flow

For your Java/Spring Boot background, explain it like this:

Spring Boot Order Service
          │
          │ KafkaTemplate
          ▼
     Kafka Producer
          │
          ▼
     orders topic
          │
          ▼
Payment Spring Boot Service
          │
          │ @KafkaListener
          ▼
    Payment Processor
          │
          ▼
      Payment DB

Example conceptually:

kafkaTemplate.send("orders", orderId, orderEvent);

Consumer:

@KafkaListener(topics = "orders", groupId = "payment-service")
public void consume(OrderEvent event) {
    paymentService.process(event);
}

The important thing isn't memorizing annotations. In a senior interview, explain:

Producer
   ↓
Partition selection
   ↓
Leader
   ↓
Replication
   ↓
Consumer Group
   ↓
Partition assignment
   ↓
Processing
   ↓
Offset commit

35. Senior Architect Question: Kafka + Database — How do you avoid data inconsistency?

This is very important.

Suppose:

Order Service
   │
   ├── Save order to DB
   │
   └── Publish Kafka event

What happens?

DB commit ✓
Kafka publish ❌

Now DB has the order but Kafka doesn't.

Solution: Transactional Outbox Pattern

             Order Service
                  │
                  ▼
          ┌─────────────────┐
          │   DB Transaction│
          │                 │
          │ Orders          │
          │ Outbox Events   │
          └────────┬────────┘
                   │
                   ▼
              Outbox Table
                   │
                   ▼
             CDC / Outbox
              Publisher
                   │
                   ▼
                 Kafka

The order and outbox event are committed in the same database transaction.

Then a CDC/outbox publisher sends the event to Kafka.

This is an excellent senior-level Kafka interview answer.

36. Most Important Kafka Questions to Prepare

If you have limited interview time, focus on these 15:

What is Kafka?
Kafka vs RabbitMQ?
Topic vs Partition?
What is a broker?
What is replication factor?
Leader vs follower?
What is ISR?
What is offset?
What is consumer group?
How does partition assignment work?
How is ordering guaranteed?
acks=0/1/all?
At-most-once vs at-least-once vs exactly-once?
What is consumer lag and how do you troubleshoot it?
Kafka + DB consistency / Transactional Outbox?

One master flow to memorize

             PRODUCER
                 │
                 │ Event + Key
                 ▼
        ┌─────────────────┐
        │ Kafka Cluster   │
        │                 │
        │ Topic           │
        │  ├─ P0          │
        │  ├─ P1          │
        │  └─ P2          │
        └───────┬─────────┘
                │
        Partition Assignment
                │
        ┌───────┼────────┐
        ▼       ▼        ▼
       C1      C2       C3
        └───────┼────────┘
                │
          Consumer Group
                │
                ▼
            Processing
                │
                ▼
          Offset Commit
                │
        ┌───────┴────────┐
        │                │
       Success          Failure
        │                │
        ▼                ▼
     Continue       Retry / DLT

For a Senior Java Architect interview, the next level is not just Kafka definitions—it is scenario questions. The highest-value ones are: “Kafka consumer is slow,” “duplicate payment happened,” “broker went down,” “message ordering broke,” “DB commit succeeded but Kafka publish failed,” “10K TPS system design,” and “how to implement retry/DLT in Spring Kafka.”

Dynamic Sharding & Consistent Hashing

Dynamic Sharding & Consistent Hashing — Class Notes

1. Why do we need Dynamic Sharding?

In a distributed database, data is split across multiple database servers called shards.

A common approach is hash-based sharding:

shard = hash(recordKey) % N

Where:

recordKey → ID/key of the record
hash() → hash function
N → number of database shards

Example

Suppose we have 4 shards:

hash(userId) % 4

User ID	Hash % 4	Shard
User A	0	Shard 0
User B	1	Shard 1
User C	2	Shard 2
User D	3	Shard 3

This works well as long as the number of shards doesn't change.

2. Problem with Traditional Hash-Based Sharding

The biggest problem occurs when we add or remove database servers.

Initially

Number of shards = 4

shard = hash(key) % 4

Suppose:

hash(User123) = 10

10 % 4 = 2

So User123 is stored in:

Shard 2

What happens when we add a server?

Now we have 5 shards:

shard = hash(key) % 5

For the same user:

10 % 5 = 0

The system now expects User123 in:

Shard 0

But the record is actually still in:

Shard 2

Result

A huge number of records need to be redistributed.

Before:

        Shard 0
        Shard 1
        Shard 2
        Shard 3

             ↓ Add Shard 4

        Shard 0
        Shard 1
        Shard 2
        Shard 3
        Shard 4

             ↓

Many keys get different shard assignments
             ↓
Large-scale data migration

The same problem occurs when removing a server.

3. Problems with Traditional Hashing

There are two major problems.

Problem 1 — Adding/removing nodes

Changing N changes:

hash(key) % N

Therefore, many keys get assigned to different shards.

This causes:

Large data movement
High network traffic
Disk I/O
Increased load
Potential downtime/performance degradation

Problem 2 — Different server capacities

Suppose we have:

Server A → 16 GB RAM
Server B → 64 GB RAM
Server C → 128 GB RAM

Traditional hashing treats them equally.

Server A → 33%
Server B → 33%
Server C → 33%

But this isn't necessarily ideal.

The more powerful server should potentially handle more data/traffic.

4. Consistent Hashing

Consistent hashing solves the major redistribution problem by hashing:

Database records/keys
Database nodes

into the same hash space.

Instead of:

hash(key) % N

we create a circular hash space called a hash ring.

5. Consistent Hashing Ring

Imagine the hash space as a circle:

                     25
                +----------+
             10              40
           /                    \
          /                      \
        90                        55
          \                      /
           \                    /
             75              65
                +----------+

The hash values wrap around.

For example:

0 → 1 → 2 → ... → 99 → back to 0

So:

99 → 0

The hash space is continuous.

6. Hash Both Nodes and Keys

Suppose we have:

Node 1
Node 2
Node 3

We hash their identifiers:

hash(Node 1) → 31
hash(Node 2) → 99
hash(Node 3) → 58

We also hash data keys:

hash(Key A) → 40
hash(Key B) → 57
hash(Key C) → 75
hash(Key D) → 2

Everything exists on the same ring.

7. How is a Key Assigned to a Node?

A key belongs to the next node encountered while moving clockwise around the ring.

This is the most important rule to remember.

Example

Node 1 → position 31
Node 5 → position 58
Node 2 → position 99

Consider:

Key A → position 40

Move clockwise:

40 → 58

Therefore:

Key A → Node 5

Another key:

Key B → position 75

Move clockwise:

75 → 99

Therefore:

Key B → Node 2

Another key:

Key C → position 10

Move clockwise:

10 → 31

Therefore:

Key C → Node 1

8. Why is Consistent Hashing Better?

The biggest advantage is:

Adding or removing a node affects only a small portion of the keys.

We don't need to redistribute the entire database.

9. Removing a Node

Suppose:

Node 1 → 31
Node 5 → 58
Node 2 → 99

And we remove:

Node 2 → 99

The keys belonging to Node 2 simply move to the next node clockwise.

Because the ring is circular, that might be Node 1.

Before:

             Node 2
               99
                |
        Keys belonging
        to Node 2
                |
                ↓
              Node 1
                31

After removing Node 2:

Keys of Node 2
      ↓
Node 1

Important

Keys belonging to other nodes don't need to move.

Node 5 → unchanged
Node 1 → mostly unchanged
Node 2 → removed

This dramatically reduces data migration.

10. Adding a Node

Now suppose we add:

New Node → position 31

Previously:

Node 1 → position 40

Some of Node 1's keys now fall into the new node's range.

Only those keys need to move.

Before:

       Node 1
       40

       ↓ Add New Node

New Node
   31

Node 1
   40

Only the affected range moves.

Old Node 1
     |
     | affected keys
     ↓
New Node

The rest of the cluster remains untouched.

11. Key Advantage of Consistent Hashing

Traditional hashing:

hash(key) % N

Changing N can cause:

Many keys
   ↓
Different shard
   ↓
Large data migration

Consistent hashing:

Add/remove node
       ↓
Only affected ring range
       ↓
Small data migration

Interview statement

Consistent hashing minimizes key redistribution when nodes are added or removed.

12. Handling Servers with Different Capacities

Another advantage is that consistent hashing can handle heterogeneous servers.

Suppose:

Node 0 → Weak
Node 1 → 2× more powerful
Node 2 → 3× more powerful

We don't necessarily want:

Node 0 → 33%
Node 1 → 33%
Node 2 → 33%

Instead, we can assign more positions on the ring to powerful nodes.

13. Virtual Nodes

This is achieved using virtual nodes, also called vnodes.

Instead of representing one physical server with one position:

Physical Node 0 → 1 virtual node
Physical Node 1 → 2 virtual nodes
Physical Node 2 → 3 virtual nodes

We might have:

Node 0:
    V0

Node 1:
    V1a
    V1b

Node 2:
    V2a
    V2b
    V2c

On the hash ring:

             V2a
              |
       V1a         V2b
          \       /
           \     /
            V0
           /  \
        V2c   V1b

Now Node 2 owns multiple portions of the ring.

Therefore, statistically:

Node 2 → more keys
Node 1 → medium number of keys
Node 0 → fewer keys

This allows us to use servers with different hardware capabilities.

14. Why Virtual Nodes Are Important

Virtual nodes provide two benefits:

1. Capacity weighting

Powerful server:

More virtual nodes
        ↓
Larger portion of hash ring
        ↓
More data

Weak server:

Fewer virtual nodes
        ↓
Smaller portion of hash ring
        ↓
Less data

2. Better load distribution

Multiple positions spread a physical node's responsibility across the ring.

This reduces the chance that one server receives an unusually large continuous range.

15. Uneven Load Distribution Problem

Even with consistent hashing, there is another problem.

Suppose we have only three nodes:

Node A
Node B
Node C

After hashing their identifiers:

Node A → 10
Node B → 15
Node C → 90

The ring could look like:

0 ---- A -- B -------------------------- C ---- 100

Node C may own a very large or small portion depending on the positions.

Therefore:

Node A → 5% keys
Node B → 10% keys
Node C → 85% keys

This creates a hotspot.

                Too much traffic
                      ↓
                   Node C
                      ↓
                 Bottleneck

While:

Node A → underutilized
Node B → underutilized

16. Solution — Multiple Hash Positions

One solution is to map each physical node to multiple positions on the ring.

For example, instead of:

Node 0 → 1 position
Node 1 → 1 position
Node 2 → 1 position

we can have:

Node 0 → 2 positions
Node 1 → 2 positions
Node 2 → 2 positions

Conceptually:

Hash Function 1:

Node 0 → 99
Node 1 → 16
Node 2 → 65


Hash Function 2:

Node 0 → another position
Node 1 → another position
Node 2 → another position

Now every physical node has multiple points on the ring.

17. Better Distribution

With a single position:

Node 0 ───────────── Node 1 ─── Node 2

The ranges can be very uneven.

With multiple positions:

Node 0 ─ Node 2 ─ Node 1 ─ Node 0 ─ Node 2 ─ Node 1

Each physical node owns multiple smaller ranges.

Statistically, this produces a much more balanced distribution.

18. Consistent Hashing Architecture

A typical distributed database can look like:

                  Client
                    |
                    v
             Application Layer
                    |
                    v
          +---------------------+
          | Hashing / Router    |
          +---------------------+
                    |
              Hash(recordKey)
                    |
                    v
             Consistent Hash Ring
                    |
       +------------+------------+
       |            |            |
       v            v            v
    Node A        Node B       Node C
       |            |            |
       v            v            v
    Database      Database     Database

The router determines which node owns the key.

19. Complete Flow

Write

Client
  |
  | Write(key, value)
  v
Application
  |
  | hash(key)
  v
Consistent Hash Ring
  |
  | Find next clockwise node
  v
Database Node
  |
  v
Store record

Read

Client
  |
  | Read(key)
  v
Application
  |
  | hash(key)
  v
Consistent Hash Ring
  |
  | Find owner
  v
Database Node
  |
  v
Return record

The important point is that the application/router can determine where the key should live without querying every database server.

20. Node Addition Flow

Existing Cluster

Node A
Node B
Node C

      ↓

Add Node D

      ↓

Hash Node D

      ↓

Node D gets a position/range
on the consistent hash ring

      ↓

Only keys in that affected
range are migrated

      ↓

Cluster continues operating

21. Node Removal Flow

Node B fails/removes
       ↓
Identify Node B's ring ranges
       ↓
Transfer those ranges
to the next responsible node
       ↓
Other ranges remain unchanged

This makes scaling much easier.

22. Traditional Hashing vs Consistent Hashing

Feature	Traditional Hashing	Consistent Hashing
Formula	`hash(key) % N`	Hash key onto ring
Hash nodes?	Usually no	Yes
Hash keys?	Yes	Yes
Data structure	Fixed partitions	Circular hash ring
Add node	Large redistribution	Small redistribution
Remove node	Large redistribution	Small redistribution
Heterogeneous hardware	Difficult	Easy with virtual nodes
Load balancing	Can be uneven	Better with multiple positions/vnodes
Scalability	Limited	Excellent
Common use	Simple sharding	Distributed systems

23. Important Terminology

Hash Space

The complete range of possible hash values.

0 → 99

Hash Ring

The hash space represented as a circle:

0 → 99 → 0

Physical Node

An actual database server.

DB Server 1
DB Server 2
DB Server 3

Virtual Node

A logical position on the hash ring representing a physical node.

Physical Node A
    ↓
Virtual A1
Virtual A2
Virtual A3

Key Redistribution

Moving records from one database node to another when ownership changes.

Consistent hashing minimizes this redistribution.

24. Key Takeaways

Remember these 5 points for interviews:

1. Traditional hashing has a scaling problem

hash(key) % N

Changing N changes the mapping of many keys.

2. Consistent hashing uses a hash ring

Both:

Database keys
+
Database nodes

are mapped into the same hash space.

3. Adding/removing nodes causes minimal movement

Only the affected range of keys needs to move.

4. Virtual nodes handle different server capacities

Powerful server
      ↓
More virtual nodes
      ↓
More hash-ring ranges
      ↓
More data

5. Multiple positions improve load balancing

Instead of giving each physical node one position, give it multiple positions.

Multiple positions
        ↓
Smaller ranges
        ↓
Better statistical distribution
        ↓
Fewer hotspots

Interview-ready definition

Consistent hashing is a distributed hashing technique that maps both data keys and nodes onto the same circular hash space. When nodes are added or removed, only a small portion of the keys need to be redistributed, making it highly suitable for dynamically scalable distributed systems. Virtual nodes can be used to support servers with different capacities and improve load distribution.

System Desing - DropBox - chunks,client agent

Dropbox System Design — Class Notes

Dropbox is a cloud file-storage and synchronization system. Users should be able to upload files, download files, synchronize files across multiple devices, share files, and recover files reliably.

The key challenge is that files can be very large, while the system needs to support millions of users and huge amounts of data.

1. Requirements

Functional requirements

The system should support:

Upload a file
Download a file
Synchronize files across devices
Create folders
Rename/move/delete files
Share files/folders
Maintain file versions
Recover deleted files
Detect changes made from different devices

Non-functional requirements

We want:

High availability
High durability
Low download latency
Efficient synchronization
Horizontal scalability
Fault tolerance
Strong protection against data loss

2. High-Level Architecture

A simplified Dropbox architecture:

                         ┌──────────────┐
                         │    Client    │
                         │ Laptop/Mobile│
                         └──────┬───────┘
                                │
                                ▼
                       ┌─────────────────┐
                       │ Load Balancer   │
                       └────────┬────────┘
                                │
                ┌───────────────┼───────────────┐
                │               │               │
                ▼               ▼               ▼
          Metadata Service   Sync Service   Sharing Service
                │               │               │
                └───────────────┼───────────────┘
                                │
                                ▼
                         Metadata Database
                                │
                                │
                         File Metadata
                                │
                                ▼
                         Object Storage
                         /      |       \
                        /       |        \
                   Replica    Replica   Replica
                                │
                                ▼
                              CDN
                                │
                                ▼
                              Users

The most important design decision is:

Store file metadata in a database, but store the actual file contents in object storage.

3. Metadata vs File Content

We should not put large files directly inside a relational database.

For example:

File:
Vacation.jpg
Size: 25 MB

Instead, maintain metadata:

FileMetadata

file_id
user_id
file_name
folder_id
file_size
file_hash
version
created_time
modified_time
storage_key

The actual file goes to object storage:

Object Storage

bucket
   │
   ├── user123/file789/chunk001
   ├── user123/file789/chunk002
   ├── user123/file789/chunk003
   └── ...

So:

Metadata DB
     │
     └── "Where is the file?"
              │
              ▼
        Object Storage
              │
              └── Actual file

4. Why Object Storage?

Dropbox can contain enormous amounts of data.

Imagine:

1 billion users
×
100 GB average storage
=
100 exabytes

A traditional database is not the right place for this amount of file content.

Object storage is designed for:

Huge files
Massive capacity
High durability
Replication
Distributed storage
Large-scale reads/writes

Examples of object-storage concepts include:

Bucket
   │
   ├── Object A
   ├── Object B
   ├── Object C
   └── Object D

5. File Upload

Let's look at a normal upload.

User wants to upload:

presentation.pdf
100 MB

A naive design would be:

Client
   │
   │ 100 MB
   ▼
Application Server
   │
   ▼
Storage

This creates several problems.

The application server has to:

Receive 100 MB
Keep the connection open
Transfer 100 MB
Potentially consume memory/resources
Forward 100 MB to storage

With millions of users, this becomes expensive.

6. Direct Upload to Object Storage

A better architecture is:

Client
   │
   │ Request upload
   ▼
Application Server
   │
   │ Generate upload permission
   ▼
Client
   │
   │ Direct upload
   ▼
Object Storage

The application server does not need to carry the entire file.

It primarily handles:

Authentication
Authorization
Metadata
Upload session

while the file goes directly to storage.

This is a very important system-design pattern:

Use the application server for control-plane operations and object storage for the data plane.

7. Chunking

Large files should be divided into smaller pieces.

For example:

100 MB file

       File
        │
 ┌──────┼──────┐
 ▼      ▼      ▼
Chunk1 Chunk2 Chunk3
 10MB   10MB    10MB

Actually, a production system can use many more chunks depending on its design.

Why chunk files?

If the upload fails at 90%:

Without chunking:

100 MB
   ↓
Upload fails
   ↓
Start again ❌

With chunking:

Chunk 1 ✅
Chunk 2 ✅
Chunk 3 ✅
...
Chunk 9 ❌

Retry only Chunk 9

This provides resumable uploads.

8. Chunk Hashing

Each chunk can have a hash.

Chunk 1 → Hash A
Chunk 2 → Hash B
Chunk 3 → Hash C

The complete file can be represented by metadata containing its chunks:

File ID: F123

Chunk 1 → Hash A
Chunk 2 → Hash B
Chunk 3 → Hash C
...

This gives us an important capability:

We can identify whether a particular chunk already exists.

9. Deduplication

Suppose User A uploads:

movie.mp4

and User B uploads the exact same file.

We don't necessarily want to store two physical copies.

User A
   │
   ▼
Hash X ─────────┐
                │
                ▼
             Object
                ▲
                │
Hash X ─────────┘
   ▲
   │
User B

Both metadata records can point to the same underlying object.

This is called deduplication.

Benefits

Saves storage
Reduces network bandwidth
Reduces upload cost

But we must carefully manage reference counts and deletion.

If User A deletes the file:

User A ──X──► Object
                  ▲
                  │
               User B

We cannot delete the underlying object because User B still references it.

10. Synchronization

Synchronization is the heart of Dropbox.

Suppose we have:

Laptop
   │
   │
   ▼
Dropbox Server
   ▲
   │
   │
Mobile

User changes:

document.txt

on the laptop.

Dropbox needs to detect:

What changed?

and then synchronize that change to other devices.

11. Client-Side Sync Agent

A Dropbox-like system can have a background process running on the user's device.

                 Laptop
        ┌──────────────────────┐
        │                      │
        │   Local Files        │
        │       │              │
        │       ▼              │
        │   Sync Agent         │
        │       │              │
        └───────┼──────────────┘
                │
                ▼
          Dropbox Server

The sync agent monitors changes.

For example:

document.txt
     │
     ▼
Modified
     │
     ▼
Sync Agent detects change
     │
     ▼
Calculate hash/chunks
     │
     ▼
Upload changed chunks

12. Incremental Synchronization

This is one of the most important Dropbox concepts.

Suppose:

File = 100 MB

User changes only:

1 MB

We don't want:

100 MB → upload again

Instead:

100 MB
 │
 ├── Chunk 1
 ├── Chunk 2
 ├── Chunk 3 ← modified
 ├── Chunk 4
 └── ...

Upload only the changed chunk.

Client
  │
  └── Changed Chunk 3
            │
            ▼
        Object Storage

This dramatically reduces bandwidth.

13. File Versioning

Suppose:

document.txt
Version 1

User modifies it:

Version 2

Later:

Version 3

We can maintain:

File
 │
 ├── Version 1
 ├── Version 2
 └── Version 3

Metadata could contain:

file_id
version_id
created_time
modified_time
storage_location

This allows:

Undo
File recovery
Version history
Protection against accidental changes

14. Delete Operation

Deleting a file should not necessarily mean immediately deleting the physical object.

Instead:

User
 │
 ▼
Delete File
 │
 ▼
Metadata
 │
 └── marked_deleted = true

The actual object can remain temporarily.

Metadata
   │
   └── Deleted

Object Storage
   │
   └── Still exists

Later, a background cleanup process can permanently remove objects that are no longer needed.

This is useful for:

Recovery
Version history
Trash
Deduplication

15. Conflict Resolution

A major synchronization problem occurs when two devices modify the same file.

Example:

Laptop
document.txt → Version A
     │
     │
     ├──────────────┐
     │              │
     ▼              ▼
Server           Mobile
                   │
              document.txt
              Version B

Both devices modify the same file before synchronization.

Now the server receives:

Laptop → Version A
Mobile → Version B

What should happen?

This is a conflict.

A simple strategy is:

Latest version wins

But that can cause data loss.

A safer approach is to create a conflict copy:

document.txt

document (Laptop's conflicted copy).txt

More sophisticated systems can use:

Version numbers
Timestamps
Vector clocks
Operation logs
Application-specific merge logic

16. Metadata Database

The metadata database might contain:

Users
-----
user_id
name
email


Files
-----
file_id
user_id
folder_id
file_name
size
version
hash
created_at
updated_at
deleted


Chunks
------
chunk_id
file_id
chunk_hash
size
storage_key


Folders
-------
folder_id
user_id
parent_folder_id
folder_name

The database stores metadata, not the huge file contents.

17. Folder Hierarchy

Dropbox has a hierarchical folder structure.

Example:

Root
 │
 ├── Documents
 │    ├── Resume.pdf
 │    └── Design.docx
 │
 ├── Photos
 │    ├── India.jpg
 │    └── USA.jpg
 │
 └── Videos
      └── Trip.mp4

We can represent this using:

folder_id
parent_folder_id

Example:

Root
folder_id = 1

Documents
folder_id = 2
parent_folder_id = 1

Resume.pdf
folder_id = 2

18. Sharing

Suppose Ramesh wants to share:

Documents/Design.pdf

with another user.

We should not simply expose the storage object directly.

Instead:

User A
  │
  ▼
Sharing Service
  │
  ▼
Permission DB
  │
  ├── User A → Owner
  └── User B → Read

Then:

User B
  │
  ▼
Authorization
  │
  ▼
Can access?
  │
  ├── YES → File
  └── NO  → 403

19. Download Flow

A download can work similarly to upload.

Client
   │
   │ Request file
   ▼
API Server
   │
   │ Check authentication
   │ Check authorization
   ▼
Metadata DB
   │
   │ Find storage location
   ▼
Object Storage / CDN
   │
   ▼
Client

Again, the application server doesn't necessarily need to stream the entire file.

It can provide a secure temporary URL or equivalent controlled access mechanism.

20. CDN

For frequently downloaded files:

Client
   │
   ▼
CDN
   │
   ├── HIT ──► File
   │
   └── MISS
         │
         ▼
    Object Storage

CDN provides:

Lower latency
Reduced load on storage
Better global performance

21. Reliability and Redundancy

Files must not disappear because one machine fails.

We can replicate storage:

                 File
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
   Storage A   Storage B   Storage C
      ✅           ✅          ✅

If one fails:

Storage A ❌

Storage B ✅
Storage C ✅

The file remains available.

For even stronger durability, object storage can replicate data across:

Multiple machines
Multiple racks
Multiple availability zones
Potentially multiple regions

22. Metadata Database Replication

The metadata database also needs redundancy.

                 Application
                      │
                 ┌────▼────┐
                 │ Primary │
                 │   DB    │
                 └────┬────┘
                      │
                 Replication
                      │
             ┌────────┴────────┐
             ▼                 ▼
          Replica 1          Replica 2

If the primary fails:

Primary DB ❌
     │
     ▼
Replica promoted
     │
     ▼
System continues

23. Sharding Metadata

Eventually, one metadata database may become too large.

We can shard based on UserID.

Shard = UserID % N

Example:

UserID = 12345

12345 % 10 = 5

User's metadata → Shard 5

Architecture:

                  Metadata Service
                         │
                   Shard Router
                         │
       ┌─────────┬───────┼───────┬─────────┐
       ▼         ▼       ▼       ▼         ▼
    Shard 0   Shard 1  Shard 2 ...      Shard 9

This allows horizontal scaling.

24. Notification / Sync Service

We don't want every client to continuously ask:

"Did anything change?"
"Did anything change?"
"Did anything change?"

Instead, the server can notify connected clients.

For example:

Laptop ───────────────┐
                      │
Mobile ───────────────┼──► Sync Service
                      │
Tablet ───────────────┘

When a file changes:

File Change
    │
    ▼
Sync Service
    │
    ├──► Laptop
    ├──► Mobile
    └──► Tablet

The clients then retrieve the required changed metadata/chunks.

25. Complete Dropbox Flow

Upload

                 UPLOAD
                   │
                   ▼
              Client App
                   │
                   ▼
            Metadata Service
                   │
            Create upload session
                   │
                   ▼
             Chunk File
                   │
                   ▼
          Calculate Chunk Hash
                   │
                   ▼
          Object Storage Upload
                   │
                   ▼
            Update Metadata DB
                   │
                   ▼
             Notify Devices

Download

                 DOWNLOAD
                     │
                     ▼
                   Client
                     │
                     ▼
              Metadata Service
                     │
              Authorization
                     │
                     ▼
                CDN / Storage
                     │
                     ▼
                   Client

Synchronization

Local File Change
       │
       ▼
   Sync Agent
       │
       ▼
Calculate Hash
       │
       ▼
Find Changed Chunks
       │
       ▼
Upload Chunks
       │
       ▼
Update Metadata
       │
       ▼
Notify Other Devices
       │
       ▼
Other Device Downloads
Changed Chunks

26. Most Important Design Decisions

Problem	Solution
Huge file storage	Object storage
Large uploads	Chunking
Failed uploads	Resumable upload
Only small part changed	Incremental/chunk-level sync
Duplicate files	Deduplication
File recovery	Versioning + trash
Concurrent edits	Conflict detection/resolution
Global downloads	CDN
Metadata scalability	Database sharding
Database failure	Replication/failover
Storage failure	Replicated object storage
Too many polling requests	Notification/sync service
Secure sharing	Authorization + controlled file access
Application server overload	Direct client ↔ object-storage transfer

⭐ 27. Interview Mental Model

Remember Dropbox using this simple flow:

              ┌──────────────────────┐
              │       CLIENT         │
              │   Sync Agent         │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │    API / Sync        │
              │      Service         │
              └──────────┬───────────┘
                         │
             ┌───────────┴───────────┐
             │                       │
             ▼                       ▼
       Metadata DB             Object Storage
       ───────────             ──────────────
       Files                   Actual data
       Folders                 Chunks
       Versions                Replicas
       Permissions
             │                       │
             └───────────┬───────────┘
                         │
                         ▼
                       CDN
                         │
                         ▼
                       Users

The five concepts to remember

1. Metadata DB → tells us what the file is and where it is.

2. Object Storage → stores the actual file/chunks.

3. Chunking → enables resumable and incremental uploads.

4. Sync Service → keeps multiple devices consistent.

5. Replication + CDN + Sharding → provides reliability, performance, and scalability.

Core Dropbox principle: Don't treat a large file as one giant database record. Break the problem into metadata + chunks + object storage + synchronization, and then scale each component independently.

System Design - Instagram Design - Object Storage, Redis, Kafka, Microservices Architecture

Absolutely. For Instagram system design, the best way is not to jump directly into Kafka/Redis. Think like an architect:

Requirements → APIs → Data model → Database choice → Core flows → Scaling → Bottlenecks → Reliability

I'll explain in your Ramesh style: simple first, then deep system-design concepts.

Instagram — System Design

1. First: What are we building?

Instagram is basically a social media platform with:

User management
Follow / unfollow
Upload photo/video
Like
Comment
Feed
Stories
Notifications
Search
Direct messaging

For an interview, don't try to design everything.

Core Instagram MVP

We'll concentrate on:

User → Follow → Upload Post → Generate Feed → Like/Comment

This gives us almost all important distributed-system concepts.

2. High-Level Architecture

Think:

                    Mobile App
                        |
                        v
                 API Gateway / LB
                        |
          +-------------+-------------+
          |             |             |
       User Service  Post Service  Feed Service
          |             |             |
          v             v             v
       User DB       Post DB       Feed Cache
                        |
                        v
                  Object Storage
                 (Images / Videos)
                        |
                        v
                       CDN

Other services:

Follow Service
Like Service
Comment Service
Notification Service
Search Service
Media Service

3. Most Important Design Decision

The biggest question:

Where do we store the image/video?

❌ Don't put 10 MB image inside MySQL.

Instead:

Image
  |
  v
Object Storage
(S3 / GCS)
  |
  v
CDN
  |
  v
User

Database stores only metadata:

Post
-----
post_id
user_id
image_url
caption
created_at

Why?

Suppose:

100 million images × 5 MB

That's:

500 TB

Database is not the right place for this.

Object storage is designed for large blobs.

4. Database Selection — Very Important

This is where interviewers expect architectural reasoning.

Data	Database	Why
User profile	MySQL/PostgreSQL	Strong consistency + relational
Follow relationships	Cassandra/DynamoDB	Huge scale + simple access
Posts metadata	Cassandra/DynamoDB	Massive write/read scale
Likes	Cassandra/DynamoDB	Very high writes
Comments	Cassandra/DynamoDB	High volume
Feed	Redis + Cassandra	Extremely fast reads
Images/videos	S3/Object Storage	Large binary data
Search	Elasticsearch/OpenSearch	Text search
Cache	Redis	Low latency
Analytics	Kafka + Data Lake	Huge event stream

Important interview statement

Don't say:

"Instagram uses Cassandra."

Say:

"For this access pattern, I would choose Cassandra/DynamoDB because the workload is high-volume, horizontally scalable, and mostly key-based."

Database choice should follow access pattern.

5. User Service

User table:

User
----------------
user_id
username
email
profile_image
bio
created_at

Example:

user_id = 1001
username = ramesh

PostgreSQL/MySQL works well initially.

Why relational?

Because user/profile operations aren't necessarily the highest-volume part.

6. Follow System

This is actually one of the most important parts.

Suppose:

Ramesh follows:
A
B
C
D

We need queries like:

Who does Ramesh follow?

and:

Who follows Ramesh?

So maintain two access patterns.

Following
-----------
user_id
following_id
created_at

and potentially:

Followers
-----------
user_id
follower_id
created_at

Example:

Ramesh → Virat
Ramesh → Sachin
Ramesh → Kohli

7. Upload Post

User uploads:

Photo
 |
 v
API Gateway
 |
 v
Post Service
 |
 +----> Object Storage
 |
 +----> Post DB
 |
 +----> Kafka Event

The database stores:

post_id
user_id
media_url
caption
timestamp

Kafka event:

POST_CREATED

{
  post_id: 123,
  user_id: 1001
}

Now other services can consume it.

8. Why Kafka?

This is a classic decoupling problem.

Without Kafka:

Post Service
    |
    +--> Feed Service
    +--> Notification Service
    +--> Search Service
    +--> Analytics

Post service becomes dependent on everything.

Instead:

              Post Service
                   |
                   v
                 Kafka
          /        |       \
         v         v        v
       Feed   Notification Search
       Service    Service   Service

Benefit

If Notification Service is down:

Post creation should still work.

That's the power of asynchronous architecture.

9. The BIG Question — Feed

This is probably the most important Instagram system-design question.

User opens Instagram:

GET /feed

We need to return:

Post A
Post B
Post C
Post D
...

But imagine:

Ramesh follows 2,000 people.

We cannot query all 2,000 users' posts every time.

So we need a better design.

10. Two Approaches to Feed

Approach 1 — Fan-out on Read

When Ramesh opens Instagram:

Get people Ramesh follows
       |
       v
Get their latest posts
       |
       v
Merge
       |
       v
Sort by timestamp/ranking
       |
       v
Return feed

Problem

Every read does huge work.

If:

10 million users

open Instagram simultaneously:

💥 Database gets hammered.

11. Approach 2 — Fan-out on Write

When someone creates a post:

Virat creates Post A
       |
       v
Find Virat's followers
       |
       v
Push Post A into their feeds

So:

Ramesh Feed
----------------
Post A
Post B
Post C

Then when Ramesh opens Instagram:

GET Redis Feed

Very fast.

This is called:

Fan-out on Write

12. But There Is a BIG Problem

Imagine:

Virat has 300 million followers.

He creates one post.

Fan-out means:

1 post
   |
   +--> 300 million feed updates

💥 Impossible/expensive.

This is the celebrity problem.

13. Real Solution — Hybrid Feed

Use:

Normal users

Fan-out on write

Post created
    ↓
Push into followers' feeds

Celebrities

Fan-out on read

Celebrity post
      ↓
Don't push to 300M feeds
      ↓
Merge celebrity posts when user opens feed

Therefore:

                Feed
                 |
       +---------+---------+
       |                   |
       v                   v
Precomputed Feed      Celebrity Posts
    Redis                   DB
       |                   |
       +---------+---------+
                 |
                 v
              Merge
                 |
                 v
              Ranking
                 |
                 v
               User

🔥 This hybrid approach is a very important interview concept.

14. Redis Feed

For every user:

feed:{userId}

Example:

feed:1001

contains:

post123
post456
post789

Use Redis Sorted Set:

ZADD feed:1001 timestamp post123

Then:

ZREVRANGE feed:1001 0 49

gives latest 50 posts.

Why Redis?

Because feed is:

read-heavy + latency-sensitive.

We want:

10–50 ms

rather than hitting the database every time.

15. But Redis Cannot Store Everything

Suppose:

500 million users
×
100 feed entries

That's huge.

So Redis can hold the hot/recent feed.

Older feed data can remain in persistent storage.

Think:

              Feed
               |
       +-------+-------+
       |               |
     Redis          Cassandra
   recent feed      older data

16. Like System

User clicks:

❤️ Like

Request:

POST /posts/123/like

Like table:

Like
----------------
post_id
user_id
created_at

Primary/access key:

post_id + user_id

This prevents duplicate likes.

17. Like Counter Problem

Suppose a popular post receives:

1 million likes

Don't constantly update:

UPDATE posts
SET likes = likes + 1

because one row becomes extremely hot.

Instead:

Like events
     ↓
Kafka
     ↓
Like processors
     ↓
Aggregated count
     ↓
Cache / DB

This reduces contention.

18. Comment System

Comment:

comment_id
post_id
user_id
text
created_at

Query:

Get comments for post 123

Partition by:

post_id

But another problem:

A viral post could have:

50 million comments

Don't load all.

Use:

pagination

Example:

GET /posts/123/comments?cursor=abc&limit=20

Prefer cursor pagination over:

OFFSET 1000000

because deep offsets become expensive.

19. Image Delivery — CDN

User uploads:

photo.jpg

Store:

S3

Then:

CloudFront / CDN

serves it.

Flow:

User
 |
 v
CDN
 |
 +---- cache hit ---> Image
 |
 +---- cache miss
          |
          v
         S3

Why CDN?

Without CDN:

India user → US server → image

With CDN:

India user → India CDN edge → image

Much faster.

20. Image Processing

Don't make upload request wait for:

Resize
Compress
Thumbnail
Multiple resolutions
Moderation

Instead:

Upload
  |
  v
Object Storage
  |
  v
Kafka
  |
  v
Media Processing Workers
  |
  +--> Thumbnail
  +--> 720p
  +--> 1080p
  +--> Compression

This is another important async processing pattern.

21. Complete Architecture

Put everything together:

                         Mobile App
                             |
                             v
                       Load Balancer
                             |
                             v
                        API Gateway
                             |
       +----------+----------+----------+----------+
       |          |          |          |          |
       v          v          v          v          v
     User       Post       Follow      Feed      Like
   Service    Service     Service     Service   Service
       |          |          |          |          |
       v          v          v          v          v
    MySQL     Cassandra   Cassandra    Redis     Cassandra
                  |
                  v
               Kafka
                  |
        +---------+---------+
        |         |         |
        v         v         v
      Feed    Notification Search
    Workers    Workers     Workers
        |
        v
      Redis
        |
        v
   Feed Generation

Post Service
     |
     v
 Object Storage
     |
     v
    CDN
     |
     v
    User

22. Scaling Strategy

Now comes the "how do we scale?" part.

Level 1 — Vertical scaling

Initially:

1 application server
1 database

Simple.

But eventually:

CPU ↑
Memory ↑
Traffic ↑

Vertical scaling has limits.

23. Horizontal Scaling

Add more instances:

             Load Balancer
              /    |    \
             /     |     \
          Server Server Server

Now:

10K requests/sec

can become:

100K requests/sec

by adding instances.

Services should ideally be stateless.

24. Database Scaling

Eventually one database isn't enough.

Use:

Read replicas

             Primary
            /       \
           v         v
       Replica     Replica

Writes:

Primary

Reads:

Replicas

25. Sharding

If one database becomes too large:

Users 1–10M     → Shard 1
Users 10–20M    → Shard 2
Users 20–30M    → Shard 3

Possible shard key:

user_id

But choose carefully.

Bad shard key:

country

because India could become a huge hot shard.

26. Hot Key Problem

Suppose:

post_id = 123

is a viral post.

Millions of users request:

GET post/123

One database partition can become overloaded.

Solutions:

CDN
Redis
replicated cache
request coalescing
read replicas

This is called:

Hot key / hot partition problem

27. Feed Ranking

Real Instagram doesn't simply show:

latest timestamp

We can introduce a ranking service.

Example score:

Score =
  recency
+ relationship
+ engagement
+ user interest

Then:

Candidate Posts
       |
       v
Ranking Service
       |
       v
Top N posts

For interview:

Don't build ML initially.

Say:

"Initially I'll use a simple chronological ranking strategy and later introduce a ranking service/ML model."

That's a strong architectural answer.

28. Reliability

What happens if:

Feed Service DOWN

Instagram should still allow:

Upload
Like
Comment
Profile

because services are loosely coupled.

Kafka helps.

Also:

Timeout
Retry
Circuit Breaker
Dead Letter Queue

29. CAP Thinking

For social media, we generally prefer:

Availability + Partition tolerance

over strict consistency for many operations.

Example:

If I like a post:

My UI → ❤️ immediately

The global like count can become consistent slightly later.

That's acceptable.

But for something like:

Username uniqueness

we need stronger consistency.

30. Interviewer's Favorite Question

"What happens when a celebrity posts?"

Answer:

"I wouldn't fan out the post to all followers. For normal users, I'd use fan-out-on-write and maintain precomputed feeds in Redis. For celebrity accounts with very large follower counts, I'd use fan-out-on-read and merge their posts during feed generation. This hybrid model avoids the celebrity write amplification problem."

🔥 That's a high-value system-design answer.

31. Core Concepts You Should Remember

For your class notes, remember Instagram through these 10 concepts:

#	Concept	Instagram Example
1	Load Balancer	Distribute API traffic
2	Stateless Services	User/Post services
3	Object Storage	Images/videos
4	CDN	Image delivery
5	Kafka	Async events
6	Redis	Feed/cache
7	NoSQL	Posts/likes/follows
8	Sharding	Huge datasets
9	Fan-out	Feed generation
10	Hybrid architecture	Celebrity problem

The most important mental model

             Instagram
                 |
       +---------+---------+
       |                   |
    Metadata              Media
       |                   |
       v                   v
  DB + Redis         Object Storage
       |                   |
       v                   v
     Kafka                CDN
       |
       v
 Async Processing
       |
       +---- Feed
       +---- Notification
       +---- Search
       +---- Analytics

If you understand this architecture, you already understand a large portion of modern distributed-system design.

Wednesday, 26 August 2026

Kafka - Retries - Dead Letter Queue

1. First understand the complete flow

2. Why retries=0 can be dangerous

Interview statement

3. Understand the four timeout/retry settings

request.timeout.ms = 10000

4. delivery.timeout.ms = 60000

Interview answer

5. retry.backoff.ms = 500

6. max.block.ms = 10000

Easy way to remember

7. The most important issue: Consumer + Producer

8. Example of duplicate processing

9. Consumer max.poll.interval.ms

Senior interview statement

10. Where does DLT fit?

11. But not every error should be retried

Transient error

Permanent error

12. Real-world architecture

13. What should happen to a DLT message?

14. Very important: DLT doesn't automatically solve duplicates

15. Best interview answer

"Explain Kafka retries, timeouts and DLT."

One final diagram to remember

Kafka - Interview Questions - important shard router -> kafka producer find the logic moview to orders

Kafka Interview Questions & Answers

1. What is Apache Kafka?

2. Kafka vs traditional message queue?

Traditional queue

Kafka

3. What is a Kafka Topic?

4. What is a Kafka Partition?

Important interview statement

5. Why does Kafka use partitions?

1. Parallelism

2. Scalability

6. What is a Kafka Broker?

7. What is Replication Factor?

8. What is a Kafka Leader and Follower?

9. What is an Offset?

10. What is a Consumer Group?

11. Can two consumers in the same group consume the same partition?

12. What happens if consumers > partitions?

13. What happens if consumers < partitions?

14. How does Kafka guarantee ordering?

15. How do you guarantee ordering for a particular customer/order?

Interview answer

16. How does Kafka Producer decide which partition to use?

Key specified

No key

17. What is Producer Acknowledgement (acks)?

acks=0

acks=1

acks=all

18. What is ISR?

19. What is Consumer Offset Commit?

20. What is Consumer Rebalancing?

21. What is at-most-once delivery?

22. What is at-least-once delivery?

Interview answer

23. What is exactly-once processing?

24. What is Idempotency?

25. Kafka Retry Pattern

26. What is a Dead Letter Topic?

27. How would you design Kafka for high throughput?

28. What is Consumer Lag?

29. How would you troubleshoot increasing Kafka consumer lag?

30. Can we increase Kafka partitions later?

31. What happens when a Kafka broker goes down?

32. Kafka Architecture — Complete Flow

33. Kafka vs Redis Streams vs RabbitMQ

34. Real-world Spring Boot Kafka Flow

35. Senior Architect Question: Kafka + Database — How do you avoid data inconsistency?

Solution: Transactional Outbox Pattern

36. Most Important Kafka Questions to Prepare

One master flow to memorize

Dynamic Sharding & Consistent Hashing

Dynamic Sharding & Consistent Hashing — Class Notes

2. Why `retries=0` can be dangerous

`request.timeout.ms = 10000`

4. `delivery.timeout.ms = 60000`

5. `retry.backoff.ms = 500`

6. `max.block.ms = 10000`

9. Consumer `max.poll.interval.ms`

17. What is Producer Acknowledgement (`acks`)?

`acks=0`

`acks=1`

`acks=all`