Saturday, 29 August 2026

SPOTIFY - System Desing - Elastic Search DB, Redis, Object Storage

 Absolutely. Here are interview-ready Spotify System Design class notes, structured from your transcript and expanded with the key architecture reasoning you should explain in an interview.

๐ŸŽต Spotify System Design — Class Notes

1. Problem Statement

Design a large-scale audio streaming platform like Spotify supporting:

  • Music

  • Podcasts

  • Audiobooks

  • Creator audio uploads

  • Search

  • Recommendations

  • User libraries/playlists

  • Smooth audio streaming

The key challenge is not simply storing audio files—it is serving massive amounts of audio concurrently with low latency and high availability.


2. First 3 Core Assumptions

These are extremely important in a system-design interview.

① Read-heavy or Write-heavy?

Read-heavy.

Users continuously stream existing content, while uploads happen much less frequently.

Typical pattern:

Creators
   │
   │ Upload
   ▼
Spotify
   │
   │ Huge number of reads
   ▼
Users → Stream Audio

Compared with Netflix, Spotify may have a larger creator-upload component, but the system is still overwhelmingly read-heavy.

② Distributed or Single Server?

Distributed system.

A single server cannot handle:

  • Millions of users

  • Huge audio storage

  • High concurrent streaming

  • Search

  • Recommendations

  • User libraries

  • Global traffic

Therefore:

Users
  ↓
Load Balancer
  ↓
API Servers
  ↓
Multiple distributed services
  ↓
Distributed databases + Object Storage + CDN

③ Consistency or Availability?

Availability is more important.

Spotify should remain available even if some components fail.

For example:

A slightly stale recommendation is acceptable; not being able to play music is not.

Therefore:

Availability > Strong consistency

However, some operations still require stronger consistency—for example:

  • Payment/subscription state

  • Playlist ownership

  • Certain account operations

So the real answer is:

AP-oriented overall, with selective strong consistency where business correctness requires it.


3. Functional Requirements

Separate requirements into core and supporting functionality.

Core Requirements

1. Upload Audio

Creators can upload:

  • Songs

  • Podcasts

  • Audiobooks

Creator
   ↓
Upload API
   ↓
Object Storage

2. Stream Audio

Users can:

  • Search/select content

  • Start playback

  • Pause/resume

  • Seek

  • Skip

The system should stream audio progressively rather than downloading the entire file.


3. User Library

Users can:

  • Like songs

  • Save albums

  • Follow artists

  • Save podcasts

  • Maintain playlists


Supporting Requirements

4. Search

Search by:

  • Song title

  • Artist

  • Album

  • Podcast

  • Genre

  • Audiobook

  • Description

Example:

"Imagine Dragons"
       ↓
Search Service
       ↓
Elasticsearch
       ↓
Songs / Albums / Artists

5. Recommendations

Recommend content based on:

  • Listening history

  • Likes

  • Skips

  • Search history

  • Genres

  • Similar users

  • Trending content

Example:

User
 ↓
Listening History
 ↓
Recommendation Engine
 ↓
Personalized Songs

6. Thumbnails / Metadata

Store metadata such as:

Song
 ├── title
 ├── artist
 ├── album
 ├── genre
 ├── duration
 ├── thumbnail
 └── audio location

4. Non-Functional Requirements

1. Availability ⭐

Spotify should be available 24×7.

Why?

If streaming is unavailable:

No playback
   ↓
Bad UX
   ↓
User dissatisfaction
   ↓
Potential revenue loss

2. Reliability

The system should tolerate failures.

Examples:

  • Server failure

  • Database failure

  • Network failure

  • CDN failure

  • Storage failure

Use:

  • Replication

  • Failover

  • Retries

  • Health checks

  • Circuit breakers

  • Multi-region deployment


3. Scalability ⭐

System should scale horizontally.

10K users
   ↓
100K users
   ↓
1M users
   ↓
100M users

Avoid depending on a single machine.


4. Smooth Streaming ⭐⭐⭐

This is probably the most important Spotify-specific requirement.

Users should not experience:

Play
 ↓
Buffer
 ↓
Play
 ↓
Buffer

Instead:

Play → continuous audio → continuous audio → continuous audio

Important techniques:

  • CDN

  • Chunked streaming

  • Multiple bitrate versions

  • Prefetching/buffering

  • Adaptive bitrate streaming

  • Geographic distribution


5. Scope Questions

In a real interview, don't immediately jump into architecture.

First ask questions.

User Scale

How many total users?

How many Daily Active Users (DAU)?

Example assumption:

Total users = 500M
DAU = 100M

Listening Behavior

How many minutes does an average user listen per day?

Example:

Average = 60 minutes/day

Traffic Spikes

Ask:

Are there events causing traffic spikes?

Examples:

  • New album release

  • Major artist release

  • Viral podcast

  • Festival

  • Promotional campaign


Audio Size

Ask:

What's the average size of one minute of audio?

This is critical for estimating storage and bandwidth.


Chunk Size

Ask:

How much audio is sent in one chunk?

For example:

Chunk = 5–10 seconds

The exact number is an assumption—you should state it explicitly.


Read/Write Ratio

Ask:

What is the read/write ratio?

Expected:

Very read-heavy

Replication

Ask:

What replication factor do we require?

For example:

Replication factor = 3

6. High-Level Architecture

This is the architecture you should draw first.

                         ┌───────────────┐
                         │    Creator    │
                         └───────┬───────┘
                                 │
                              Upload
                                 │
                                 ▼
                         ┌────────────────┐
                         │ Upload Service │
                         └───────┬────────┘
                                 │
                                 ▼
                         ┌────────────────┐
                         │ Object Storage │
                         │   S3-like      │
                         └───────┬────────┘
                                 │
                         Audio Processing
                                 │
                                 ▼
                         ┌────────────────┐
                         │ Audio Versions │
                         │ 64/128/320kbps │
                         └───────┬────────┘
                                 │
                                 ▼
                              CDN
                                 │
                         ┌───────┴───────┐
                         │               │
                         ▼               ▼
                       User 1          User 2

For user requests:

                    Users
                      │
                      ▼
               ┌─────────────┐
               │     CDN     │
               └──────┬──────┘
                      │
              Cache Miss?
                      │
                      ▼
             ┌────────────────┐
             │ Streaming API  │
             └───────┬────────┘
                     │
                     ▼
              Object Storage

7. Why Object Storage?

Never store huge audio files directly inside a relational database.

Bad design:

MySQL
 └── BLOB
      └── 5 MB audio

Better:

Metadata DB
    │
    └── audio_id
         ↓
Object Storage
         ↓
audio file

Database stores metadata:

{
  "songId": "S123",
  "title": "Song A",
  "artistId": "A456",
  "duration": 240,
  "storagePath": "audio/S123/..."
}

Object storage handles large media efficiently.


8. Audio Processing Pipeline

When creator uploads an audio file:

Creator
   ↓
Upload API
   ↓
Object Storage
   ↓
Kafka / Message Queue
   ↓
Audio Processing Workers
   ↓
Transcoding
   ↓
Multiple Bitrates
   ↓
Object Storage
   ↓
CDN

For example:

Original
   │
   ├── 64 kbps
   ├── 128 kbps
   └── 320 kbps

Why?

Different users have different network conditions.


9. CDN — Most Important Component

Without CDN:

User
  ↓
Spotify Server
  ↓
Object Storage

Millions of users repeatedly downloading audio creates enormous load.

With CDN:

                   Origin
                     │
                Object Storage
                     │
              ┌──────┴──────┐
              │     CDN      │
              └──────┬──────┘
                     │
       ┌─────────────┼─────────────┐
       ▼             ▼             ▼
    User A         User B        User C

Popular songs can remain cached close to users.

Result

  • Lower latency

  • Lower origin load

  • Better scalability

  • Better streaming experience

  • Lower bandwidth cost


10. Streaming Flow

Suppose the user clicks:

"Blinding Lights"

Flow:

User
 ↓
API Gateway
 ↓
Playback Service
 ↓
Metadata Service
 ↓
Get audio manifest
 ↓
CDN
 ↓
Audio chunks
 ↓
Player

The player receives audio progressively.

Song
│
├── Chunk 1
├── Chunk 2
├── Chunk 3
├── Chunk 4
└── Chunk 5

The player can buffer upcoming chunks while playing the current chunk.


11. Why Chunking?

Imagine a 5-minute song.

Instead of:

Download entire 5-minute file
        ↓
Start playback

use:

Chunk 1 → Play
Chunk 2 → Buffer
Chunk 3 → Buffer
Chunk 4 → Buffer
...

Advantages:

  • Faster startup

  • Less wasted bandwidth

  • Better seeking

  • Better failure recovery

  • Easier CDN caching


12. Search Architecture

Use a dedicated search engine such as Elasticsearch/OpenSearch.

User
 ↓
Search API
 ↓
Search Service
 ↓
Elasticsearch
 ↓
Search Results

Don't run every search directly against the primary database.

Why?

Search requires:

  • Full-text search

  • Fuzzy matching

  • Ranking

  • Autocomplete

  • Filtering

Example:

"cold pl"
      ↓
Coldplay
Coldplay - Yellow
Coldplay - Fix You

13. Database Design

We can divide data based on access patterns.

User DB

User
 ├── user_id
 ├── name
 ├── subscription
 └── preferences

Content DB

Song
 ├── song_id
 ├── title
 ├── artist_id
 ├── album_id
 ├── duration
 └── storage_location

Playlist DB

Playlist
 ├── playlist_id
 ├── user_id
 └── name

PlaylistSong
 ├── playlist_id
 ├── song_id
 └── position

14. Caching

Caching is extremely important.

Potential cache:

Redis

Cache:

  • Song metadata

  • Artist information

  • Popular playlists

  • User recommendations

  • Subscription status

  • Frequently searched content

Example:

User
 ↓
Redis
 ↓ cache hit
Song metadata

Instead of:

User
 ↓
Database

for every request.


15. Recommendation System

This can be a separate subsystem.

                  User Events
                      │
       ┌──────────────┼──────────────┐
       │              │              │
     Play           Skip           Like
       │              │              │
       └──────────────┼──────────────┘
                      ▼
                Event Stream
                   Kafka
                      │
                      ▼
             Recommendation
                 Pipeline
                      │
              ┌───────┴────────┐
              ▼                ▼
        Batch Processing    ML Models
              │                │
              └───────┬────────┘
                      ▼
              Recommendation DB
                      │
                      ▼
                    User

This is an excellent place to mention event-driven architecture.


16. Kafka/Event Streaming

Spotify generates enormous numbers of events:

PLAY
PAUSE
SKIP
LIKE
SEARCH
FOLLOW
PLAYLIST_ADD

Don't synchronously process every event through the main API.

Instead:

User
 ↓
Playback Service
 ↓
Kafka
 ↓
Consumers
 ├── Recommendation
 ├── Analytics
 ├── Trending
 └── Personalization

This decouples the systems.


17. Availability & Fault Tolerance

Suppose one streaming server fails.

Don't make users lose playback.

Use:

             Load Balancer
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
     Server1    Server2    Server3
       ✓          ✗          ✓

Health checks remove Server2.

For databases:

Primary
   │
   ├── Replica 1
   └── Replica 2

For global Spotify:

US Region
EU Region
Asia Region

Multi-region deployment improves resilience and latency.


18. CAP Theorem Interview Point ⭐

If interviewer asks:

Why availability over consistency?

Answer:

Spotify is primarily a read-heavy streaming system where users value continuous playback. Temporary stale metadata or recommendations are generally acceptable, while an unavailable service directly affects user experience. Therefore, the architecture favors availability and eventual consistency for many read paths, while maintaining stronger consistency for critical business operations.

Excellent interview answer.


19. Reliability vs Availability

Don't confuse them.

Availability

Is the system accessible?

Reliability

Does it continue behaving correctly over time?

Example:

Spotify is reachable
BUT
plays the wrong audio file

Availability = Yes

Reliability = No


20. Handling Traffic Spikes

Imagine Taylor Swift releases a new album.

Millions of users request the same songs.

Without CDN:

Millions
   ↓
Origin
   ↓
๐Ÿ’ฅ

With CDN:

Millions
   ↓
CDN
 ┌─┼─┬─┐
 ▼ ▼ ▼ ▼
Cache hits

This is why CDN + caching is critical.


21. Hot Content Problem

A newly released song can become extremely popular.

This creates a hot key/content problem.

Solutions:

  • CDN caching

  • Multiple CDN PoPs

  • Replicate popular content

  • Cache metadata

  • Avoid single origin bottleneck


22. Security

Important interview points:

Authentication

User → OAuth/JWT → API

Authorization

Verify:

Can this user access this content?

Content protection

Audio URLs should not simply expose permanent storage paths.

Use:

  • Signed URLs

  • Short-lived tokens

  • DRM/content protection where applicable


23. Observability

For a production system:

Services
   │
   ├── Metrics
   ├── Logs
   └── Traces

Monitor:

  • Playback start latency

  • Buffering rate

  • CDN cache hit ratio

  • Streaming errors

  • API latency

  • Kafka lag

  • Database latency

  • Error rate

Important Spotify-specific metric

Buffering ratio / rebuffering rate

This directly measures streaming quality.


24. Back-of-the-Envelope Estimation

This is a very important interview skill.

Assume:

DAU = 100M
Average listening = 60 min/day
Average audio bitrate = 128 kbps

Daily listening

100M × 60 minutes

= 6 billion minutes/day

Convert to seconds:

6B × 60
= 360B seconds/day

At 128 kbps:

128,000 bits/sec

Daily data:

360B × 128,000 bits

Approximately:

45.9 petabits/day

Divide by 8:

≈ 5.76 PB/day

So you can say:

At this scale, Spotify could potentially deliver several petabytes of audio traffic per day, which strongly justifies CDN-based distribution.

The exact number depends heavily on bitrate, DAU, and listening duration.


25. Storage Estimation

Suppose:

1 minute audio ≈ 1 MB

And Spotify has:

100M minutes of unique audio

Then:

100M × 1 MB
≈ 100 TB

But real storage is higher because of:

  • Multiple bitrates

  • Replication

  • Backups

  • Transcoded versions

  • Podcasts/audiobooks

Object storage is therefore appropriate.


26. Complete Architecture

The final architecture can look like this:

                         ┌─────────────┐
                         │   Creator   │
                         └──────┬──────┘
                                │
                              Upload
                                │
                                ▼
                        ┌────────────────┐
                        │ Upload Service │
                        └───────┬────────┘
                                │
                                ▼
                        ┌────────────────┐
                        │ Object Storage │
                        └───────┬────────┘
                                │
                              Kafka
                                │
                                ▼
                       ┌──────────────────┐
                       │ Audio Processing │
                       │ / Transcoding    │
                       └────────┬─────────┘
                                │
                                ▼
                         ┌────────────┐
                         │   Storage  │
                         └─────┬──────┘
                               │
                               ▼
                              CDN
                               │
                  ┌────────────┴────────────┐
                  │                         │
                User                      User
                  │                         │
                  └────────────┬────────────┘
                               │
                         API Gateway
                               │
             ┌─────────────────┼─────────────────┐
             │                 │                 │
             ▼                 ▼                 ▼
       User Service      Search Service    Playback Service
             │                 │                 │
             ▼                 ▼                 ▼
        User DB          Elasticsearch       Metadata DB
             │
             ▼
           Redis

                 User Events
                     │
                     ▼
                   Kafka
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       Analytics   Trending   Recommendation
                                  │
                                  ▼
                              ML Models

27. Interview Flow — How You Should Present It

When interviewer says "Design Spotify", follow this sequence:

Step 1 — Clarify scope

"I'll first clarify users, DAU, listening duration, audio size, traffic spikes and streaming chunk size."

Step 2 — Requirements

Core:
• Upload
• Search
• Stream
• Library

Support:
• Recommendations
• Playlists
• Likes
• Following

Step 3 — NFR

Availability
Scalability
Reliability
Low latency
Smooth streaming

Step 4 — High-level architecture

Start with:

Client
 ↓
API Gateway
 ↓
Services
 ↓
DB / Cache

Then add:

Object Storage
 ↓
CDN
 ↓
Audio Streaming

Step 5 — Deep dive

Focus on:

Audio upload → transcoding → storage → CDN → streaming

This is the heart of Spotify.

Step 6 — Scaling

Discuss:

  • CDN

  • Caching

  • Horizontal scaling

  • Database replication

  • Partitioning

  • Multi-region

Step 7 — Reliability

Discuss:

  • Replication

  • Failover

  • Retry

  • Circuit breaker

  • Health checks

  • Multi-region

Step 8 — Events & recommendations

Discuss:

Playback events
      ↓
Kafka
      ↓
Recommendation / Analytics

⭐ 28. Top 10 Interview Takeaways

#ConceptRemember
1TrafficVery read-heavy
2ArchitectureDistributed
3CAPAvailability > consistency for many paths
4AudioObject storage, not DB BLOBs
5StreamingChunk-based
6ScalabilityCDN is critical
7SearchElasticsearch/OpenSearch
8EventsKafka
9RecommendationsEvent-driven + ML pipeline
10ReliabilityReplication + failover + multi-region

๐Ÿง  One-line interview answer

"I would design Spotify as a highly available, distributed, read-heavy audio platform where audio is stored in object storage, transcoded into multiple bitrates, distributed through a CDN using chunked streaming, while metadata is served through scalable databases and Redis, search through Elasticsearch, and user listening events flow through Kafka into recommendation and analytics pipelines."

That sentence is worth memorizing—it gives the interviewer the entire architecture in ~20 seconds.

No comments:

Post a Comment