System Design - Instagram Design - Object Storage, Redis, Kafka, Microservices Architecture

 Absolutely. For Instagram system design, the best way is not to jump directly into Kafka/Redis. Think like an architect:

Requirements → APIs → Data model → Database choice → Core flows → Scaling → Bottlenecks → Reliability

I'll explain in your Ramesh style: simple first, then deep system-design concepts.

Instagram — System Design

1. First: What are we building?

Instagram is basically a social media platform with:

1. User management

2. Follow / unfollow

3. Upload photo/video

4. Like

5. Comment

6. Feed

7. Stories

8. Notifications

9. Search

10. Direct messaging

For an interview, don't try to design everything.

Core Instagram MVP

We'll concentrate on:

User → Follow → Upload Post → Generate Feed → Like/Comment

This gives us almost all important distributed-system concepts.

---

2. High-Level Architecture

Think:

                    Mobile App
                        |
                        v
                 API Gateway / LB
                        |
          +-------------+-------------+
          |             |             |
       User Service  Post Service  Feed Service
          |             |             |
          v             v             v
       User DB       Post DB       Feed Cache
                        |
                        v
                  Object Storage
                 (Images / Videos)
                        |
                        v
                       CDN

Other services:

Follow Service
Like Service
Comment Service
Notification Service
Search Service
Media Service

---

3. Most Important Design Decision

The biggest question:

Where do we store the image/video?

❌ Don't put 10 MB image inside MySQL.

Instead:

Image
  |
  v
Object Storage
(S3 / GCS)
  |
  v
CDN
  |
  v
User

Database stores only metadata:

Post
-----
post_id
user_id
image_url
caption
created_at

Why?

Suppose:

100 million images × 5 MB

That's:

500 TB

Database is not the right place for this.

Object storage is designed for large blobs.

---

4. Database Selection — Very Important

This is where interviewers expect architectural reasoning.

Data	Database	Why
User profile	MySQL/PostgreSQL	Strong consistency + relational
Follow relationships	Cassandra/DynamoDB	Huge scale + simple access
Posts metadata	Cassandra/DynamoDB	Massive write/read scale
Likes	Cassandra/DynamoDB	Very high writes
Comments	Cassandra/DynamoDB	High volume
Feed	Redis + Cassandra	Extremely fast reads
Images/videos	S3/Object Storage	Large binary data
Search	Elasticsearch/OpenSearch	Text search
Cache	Redis	Low latency
Analytics	Kafka + Data Lake	Huge event stream

Important interview statement

Don't say:

"Instagram uses Cassandra."

Say:

"For this access pattern, I would choose Cassandra/DynamoDB because the workload is high-volume, horizontally scalable, and mostly key-based."

Database choice should follow access pattern.

---

5. User Service

User table:

User
----------------
user_id
username
email
profile_image
bio
created_at

Example:

user_id = 1001
username = ramesh

PostgreSQL/MySQL works well initially.

Why relational?

Because user/profile operations aren't necessarily the highest-volume part.

---

6. Follow System

This is actually one of the most important parts.

Suppose:

Ramesh follows:
A
B
C
D

We need queries like:

Who does Ramesh follow?

and:

Who follows Ramesh?

So maintain two access patterns.

Following
-----------
user_id
following_id
created_at

and potentially:

Followers
-----------
user_id
follower_id
created_at

Example:

Ramesh → Virat
Ramesh → Sachin
Ramesh → Kohli

---

7. Upload Post

User uploads:

Photo
 |
 v
API Gateway
 |
 v
Post Service
 |
 +----> Object Storage
 |
 +----> Post DB
 |
 +----> Kafka Event

The database stores:

post_id
user_id
media_url
caption
timestamp

Kafka event:

POST_CREATED

{
  post_id: 123,
  user_id: 1001
}

Now other services can consume it.

---

8. Why Kafka?

This is a classic decoupling problem.

Without Kafka:

Post Service
    |
    +--> Feed Service
    +--> Notification Service
    +--> Search Service
    +--> Analytics

Post service becomes dependent on everything.

Instead:

              Post Service
                   |
                   v
                 Kafka
          /        |       \
         v         v        v
       Feed   Notification Search
       Service    Service   Service

Benefit

If Notification Service is down:

Post creation should still work.

That's the power of asynchronous architecture.

---

9. The BIG Question — Feed

This is probably the most important Instagram system-design question.

User opens Instagram:

GET /feed

We need to return:

Post A
Post B
Post C
Post D
...

But imagine:

Ramesh follows 2,000 people.

We cannot query all 2,000 users' posts every time.

So we need a better design.

---

10. Two Approaches to Feed

Approach 1 — Fan-out on Read

When Ramesh opens Instagram:

Get people Ramesh follows
       |
       v
Get their latest posts
       |
       v
Merge
       |
       v
Sort by timestamp/ranking
       |
       v
Return feed

Problem

Every read does huge work.

If:

10 million users

open Instagram simultaneously:

💥 Database gets hammered.

---

11. Approach 2 — Fan-out on Write

When someone creates a post:

Virat creates Post A
       |
       v
Find Virat's followers
       |
       v
Push Post A into their feeds

So:

Ramesh Feed
----------------
Post A
Post B
Post C

Then when Ramesh opens Instagram:

GET Redis Feed

Very fast.

This is called:

Fan-out on Write

---

12. But There Is a BIG Problem

Imagine:

Virat has 300 million followers.

He creates one post.

Fan-out means:

1 post
   |
   +--> 300 million feed updates

💥 Impossible/expensive.

This is the celebrity problem.

---

13. Real Solution — Hybrid Feed

Use:

Normal users

Fan-out on write

Post created
    ↓
Push into followers' feeds

Celebrities

Fan-out on read

Celebrity post
      ↓
Don't push to 300M feeds
      ↓
Merge celebrity posts when user opens feed

Therefore:

                Feed
                 |
       +---------+---------+
       |                   |
       v                   v
Precomputed Feed      Celebrity Posts
    Redis                   DB
       |                   |
       +---------+---------+
                 |
                 v
              Merge
                 |
                 v
              Ranking
                 |
                 v
               User

🔥 This hybrid approach is a very important interview concept.

---

14. Redis Feed

For every user:

feed:{userId}

Example:

feed:1001

contains:

post123
post456
post789

Use Redis Sorted Set:

ZADD feed:1001 timestamp post123

Then:

ZREVRANGE feed:1001 0 49

gives latest 50 posts.

Why Redis?

Because feed is:

read-heavy + latency-sensitive.

We want:

10–50 ms

rather than hitting the database every time.

---

15. But Redis Cannot Store Everything

Suppose:

500 million users
×
100 feed entries

That's huge.

So Redis can hold the hot/recent feed.

Older feed data can remain in persistent storage.

Think:

              Feed
               |
       +-------+-------+
       |               |
     Redis          Cassandra
   recent feed      older data

---

16. Like System

User clicks:

❤️ Like

Request:

POST /posts/123/like

Like table:

Like
----------------
post_id
user_id
created_at

Primary/access key:

post_id + user_id

This prevents duplicate likes.

---

17. Like Counter Problem

Suppose a popular post receives:

1 million likes

Don't constantly update:

UPDATE posts
SET likes = likes + 1

because one row becomes extremely hot.

Instead:

Like events
     ↓
Kafka
     ↓
Like processors
     ↓
Aggregated count
     ↓
Cache / DB

This reduces contention.

---

18. Comment System

Comment:

comment_id
post_id
user_id
text
created_at

Query:

Get comments for post 123

Partition by:

post_id

But another problem:

A viral post could have:

50 million comments

Don't load all.

Use:

pagination

Example:

GET /posts/123/comments?cursor=abc&limit=20

Prefer cursor pagination over:

OFFSET 1000000

because deep offsets become expensive.

---

19. Image Delivery — CDN

User uploads:

photo.jpg

Store:

S3

Then:

CloudFront / CDN

serves it.

Flow:

User
 |
 v
CDN
 |
 +---- cache hit ---> Image
 |
 +---- cache miss
          |
          v
         S3

Why CDN?

Without CDN:

India user → US server → image

With CDN:

India user → India CDN edge → image

Much faster.

---

20. Image Processing

Don't make upload request wait for:

Resize
Compress
Thumbnail
Multiple resolutions
Moderation

Instead:

Upload
  |
  v
Object Storage
  |
  v
Kafka
  |
  v
Media Processing Workers
  |
  +--> Thumbnail
  +--> 720p
  +--> 1080p
  +--> Compression

This is another important async processing pattern.

---

21. Complete Architecture

Put everything together:

                         Mobile App
                             |
                             v
                       Load Balancer
                             |
                             v
                        API Gateway
                             |
       +----------+----------+----------+----------+
       |          |          |          |          |
       v          v          v          v          v
     User       Post       Follow      Feed      Like
   Service    Service     Service     Service   Service
       |          |          |          |          |
       v          v          v          v          v
    MySQL     Cassandra   Cassandra    Redis     Cassandra
                  |
                  v
               Kafka
                  |
        +---------+---------+
        |         |         |
        v         v         v
      Feed    Notification Search
    Workers    Workers     Workers
        |
        v
      Redis
        |
        v
   Feed Generation

Post Service
     |
     v
 Object Storage
     |
     v
    CDN
     |
     v
    User

---

22. Scaling Strategy

Now comes the "how do we scale?" part.

Level 1 — Vertical scaling

Initially:

1 application server
1 database

Simple.

But eventually:

CPU ↑
Memory ↑
Traffic ↑

Vertical scaling has limits.

---

23. Horizontal Scaling

Add more instances:

             Load Balancer
              /    |    \
             /     |     \
          Server Server Server

Now:

10K requests/sec

can become:

100K requests/sec

by adding instances.

Services should ideally be stateless.

---

24. Database Scaling

Eventually one database isn't enough.

Use:

Read replicas

             Primary
            /       \
           v         v
       Replica     Replica

Writes:

Primary

Reads:

Replicas

---

25. Sharding

If one database becomes too large:

Users 1–10M     → Shard 1
Users 10–20M    → Shard 2
Users 20–30M    → Shard 3

Possible shard key:

user_id

But choose carefully.

Bad shard key:

country

because India could become a huge hot shard.

---

26. Hot Key Problem

Suppose:

post_id = 123

is a viral post.

Millions of users request:

GET post/123

One database partition can become overloaded.

Solutions:

CDN
Redis
replicated cache
request coalescing
read replicas

This is called:

Hot key / hot partition problem

---

27. Feed Ranking

Real Instagram doesn't simply show:

latest timestamp

We can introduce a ranking service.

Example score:

Score =
  recency
+ relationship
+ engagement
+ user interest

Then:

Candidate Posts
       |
       v
Ranking Service
       |
       v
Top N posts

For interview:

Don't build ML initially.

Say:

"Initially I'll use a simple chronological ranking strategy and later introduce a ranking service/ML model."

That's a strong architectural answer.

---

28. Reliability

What happens if:

Feed Service DOWN

Instagram should still allow:

Upload
Like
Comment
Profile

because services are loosely coupled.

Kafka helps.

Also:

Timeout
Retry
Circuit Breaker
Dead Letter Queue

---

29. CAP Thinking

For social media, we generally prefer:

Availability + Partition tolerance

over strict consistency for many operations.

Example:

If I like a post:

My UI → ❤️ immediately

The global like count can become consistent slightly later.

That's acceptable.

But for something like:

Username uniqueness

we need stronger consistency.

---

30. Interviewer's Favorite Question

"What happens when a celebrity posts?"

Answer:

"I wouldn't fan out the post to all followers. For normal users, I'd use fan-out-on-write and maintain precomputed feeds in Redis. For celebrity accounts with very large follower counts, I'd use fan-out-on-read and merge their posts during feed generation. This hybrid model avoids the celebrity write amplification problem."

🔥 That's a high-value system-design answer.

---

31. Core Concepts You Should Remember

For your class notes, remember Instagram through these 10 concepts:

#	Concept	Instagram Example
1	Load Balancer	Distribute API traffic
2	Stateless Services	User/Post services
3	Object Storage	Images/videos
4	CDN	Image delivery
5	Kafka	Async events
6	Redis	Feed/cache
7	NoSQL	Posts/likes/follows
8	Sharding	Huge datasets
9	Fan-out	Feed generation
10	Hybrid architecture	Celebrity problem

The most important mental model

             Instagram
                 |
       +---------+---------+
       |                   |
    Metadata              Media
       |                   |
       v                   v
  DB + Redis         Object Storage
       |                   |
       v                   v
     Kafka                CDN
       |
       v
 Async Processing
       |
       +---- Feed
       +---- Notification
       +---- Search
       +---- Analytics

If you understand this architecture, you already understand a large portion of modern distributed-system design.