Wednesday, 26 August 2026

System Desing - DropBox - chunks,client agent

 

Dropbox System Design — Class Notes

Dropbox is a cloud file-storage and synchronization system. Users should be able to upload files, download files, synchronize files across multiple devices, share files, and recover files reliably.

The key challenge is that files can be very large, while the system needs to support millions of users and huge amounts of data.

1. Requirements

Functional requirements

The system should support:

  1. Upload a file

  2. Download a file

  3. Synchronize files across devices

  4. Create folders

  5. Rename/move/delete files

  6. Share files/folders

  7. Maintain file versions

  8. Recover deleted files

  9. Detect changes made from different devices

Non-functional requirements

We want:

  • High availability

  • High durability

  • Low download latency

  • Efficient synchronization

  • Horizontal scalability

  • Fault tolerance

  • Strong protection against data loss

2. High-Level Architecture

A simplified Dropbox architecture:

                         ┌──────────────┐
                         │    Client    │
                         │ Laptop/Mobile│
                         └──────┬───────┘
                                │
                                ▼
                       ┌─────────────────┐
                       │ Load Balancer   │
                       └────────┬────────┘
                                │
                ┌───────────────┼───────────────┐
                │               │               │
                ▼               ▼               ▼
          Metadata Service   Sync Service   Sharing Service
                │               │               │
                └───────────────┼───────────────┘
                                │
                                ▼
                         Metadata Database
                                │
                                │
                         File Metadata
                                │
                                ▼
                         Object Storage
                         /      |       \
                        /       |        \
                   Replica    Replica   Replica
                                │
                                ▼
                              CDN
                                │
                                ▼
                              Users

The most important design decision is:

Store file metadata in a database, but store the actual file contents in object storage.

3. Metadata vs File Content

We should not put large files directly inside a relational database.

For example:

File:
Vacation.jpg
Size: 25 MB

Instead, maintain metadata:

FileMetadata

file_id
user_id
file_name
folder_id
file_size
file_hash
version
created_time
modified_time
storage_key

The actual file goes to object storage:

Object Storage

bucket
   │
   ├── user123/file789/chunk001
   ├── user123/file789/chunk002
   ├── user123/file789/chunk003
   └── ...

So:

Metadata DB
     │
     └── "Where is the file?"
              │
              ▼
        Object Storage
              │
              └── Actual file

4. Why Object Storage?

Dropbox can contain enormous amounts of data.

Imagine:

1 billion users
×
100 GB average storage
=
100 exabytes

A traditional database is not the right place for this amount of file content.

Object storage is designed for:

  • Huge files

  • Massive capacity

  • High durability

  • Replication

  • Distributed storage

  • Large-scale reads/writes

Examples of object-storage concepts include:

Bucket
   │
   ├── Object A
   ├── Object B
   ├── Object C
   └── Object D

5. File Upload

Let's look at a normal upload.

User wants to upload:

presentation.pdf
100 MB

A naive design would be:

Client
   │
   │ 100 MB
   ▼
Application Server
   │
   ▼
Storage

This creates several problems.

The application server has to:

  • Receive 100 MB

  • Keep the connection open

  • Transfer 100 MB

  • Potentially consume memory/resources

  • Forward 100 MB to storage

With millions of users, this becomes expensive.

6. Direct Upload to Object Storage

A better architecture is:

Client
   │
   │ Request upload
   ▼
Application Server
   │
   │ Generate upload permission
   ▼
Client
   │
   │ Direct upload
   ▼
Object Storage

The application server does not need to carry the entire file.

It primarily handles:

Authentication
Authorization
Metadata
Upload session

while the file goes directly to storage.

This is a very important system-design pattern:

Use the application server for control-plane operations and object storage for the data plane.

7. Chunking

Large files should be divided into smaller pieces.

For example:

100 MB file

       File
        │
 ┌──────┼──────┐
 ▼      ▼      ▼
Chunk1 Chunk2 Chunk3
 10MB   10MB    10MB

Actually, a production system can use many more chunks depending on its design.

Why chunk files?

If the upload fails at 90%:

Without chunking:

100 MB
   ↓
Upload fails
   ↓
Start again ❌

With chunking:

Chunk 1 ✅
Chunk 2 ✅
Chunk 3 ✅
...
Chunk 9 ❌

Retry only Chunk 9

This provides resumable uploads.

8. Chunk Hashing

Each chunk can have a hash.

Chunk 1 → Hash A
Chunk 2 → Hash B
Chunk 3 → Hash C

The complete file can be represented by metadata containing its chunks:

File ID: F123

Chunk 1 → Hash A
Chunk 2 → Hash B
Chunk 3 → Hash C
...

This gives us an important capability:

We can identify whether a particular chunk already exists.

9. Deduplication

Suppose User A uploads:

movie.mp4

and User B uploads the exact same file.

We don't necessarily want to store two physical copies.

User A
   │
   ▼
Hash X ─────────┐
                │
                ▼
             Object
                ▲
                │
Hash X ─────────┘
   ▲
   │
User B

Both metadata records can point to the same underlying object.

This is called deduplication.

Benefits

  • Saves storage

  • Reduces network bandwidth

  • Reduces upload cost

But we must carefully manage reference counts and deletion.

If User A deletes the file:

User A ──X──► Object
                  ▲
                  │
               User B

We cannot delete the underlying object because User B still references it.

10. Synchronization

Synchronization is the heart of Dropbox.

Suppose we have:

Laptop
   │
   │
   ▼
Dropbox Server
   ▲
   │
   │
Mobile

User changes:

document.txt

on the laptop.

Dropbox needs to detect:

What changed?

and then synchronize that change to other devices.

11. Client-Side Sync Agent

A Dropbox-like system can have a background process running on the user's device.

                 Laptop
        ┌──────────────────────┐
        │                      │
        │   Local Files        │
        │       │              │
        │       ▼              │
        │   Sync Agent         │
        │       │              │
        └───────┼──────────────┘
                │
                ▼
          Dropbox Server

The sync agent monitors changes.

For example:

document.txt
     │
     ▼
Modified
     │
     ▼
Sync Agent detects change
     │
     ▼
Calculate hash/chunks
     │
     ▼
Upload changed chunks

12. Incremental Synchronization

This is one of the most important Dropbox concepts.

Suppose:

File = 100 MB

User changes only:

1 MB

We don't want:

100 MB → upload again

Instead:

100 MB
 │
 ├── Chunk 1
 ├── Chunk 2
 ├── Chunk 3 ← modified
 ├── Chunk 4
 └── ...

Upload only the changed chunk.

Client
  │
  └── Changed Chunk 3
            │
            ▼
        Object Storage

This dramatically reduces bandwidth.

13. File Versioning

Suppose:

document.txt
Version 1

User modifies it:

Version 2

Later:

Version 3

We can maintain:

File
 │
 ├── Version 1
 ├── Version 2
 └── Version 3

Metadata could contain:

file_id
version_id
created_time
modified_time
storage_location

This allows:

  • Undo

  • File recovery

  • Version history

  • Protection against accidental changes

14. Delete Operation

Deleting a file should not necessarily mean immediately deleting the physical object.

Instead:

User
 │
 ▼
Delete File
 │
 ▼
Metadata
 │
 └── marked_deleted = true

The actual object can remain temporarily.

Metadata
   │
   └── Deleted

Object Storage
   │
   └── Still exists

Later, a background cleanup process can permanently remove objects that are no longer needed.

This is useful for:

  • Recovery

  • Version history

  • Trash

  • Deduplication

15. Conflict Resolution

A major synchronization problem occurs when two devices modify the same file.

Example:

Laptop
document.txt → Version A
     │
     │
     ├──────────────┐
     │              │
     ▼              ▼
Server           Mobile
                   │
              document.txt
              Version B

Both devices modify the same file before synchronization.

Now the server receives:

Laptop → Version A
Mobile → Version B

What should happen?

This is a conflict.

A simple strategy is:

Latest version wins

But that can cause data loss.

A safer approach is to create a conflict copy:

document.txt

document (Laptop's conflicted copy).txt

More sophisticated systems can use:

  • Version numbers

  • Timestamps

  • Vector clocks

  • Operation logs

  • Application-specific merge logic

16. Metadata Database

The metadata database might contain:

Users
-----
user_id
name
email


Files
-----
file_id
user_id
folder_id
file_name
size
version
hash
created_at
updated_at
deleted


Chunks
------
chunk_id
file_id
chunk_hash
size
storage_key


Folders
-------
folder_id
user_id
parent_folder_id
folder_name

The database stores metadata, not the huge file contents.

17. Folder Hierarchy

Dropbox has a hierarchical folder structure.

Example:

Root
 │
 ├── Documents
 │    ├── Resume.pdf
 │    └── Design.docx
 │
 ├── Photos
 │    ├── India.jpg
 │    └── USA.jpg
 │
 └── Videos
      └── Trip.mp4

We can represent this using:

folder_id
parent_folder_id

Example:

Root
folder_id = 1

Documents
folder_id = 2
parent_folder_id = 1

Resume.pdf
folder_id = 2

18. Sharing

Suppose Ramesh wants to share:

Documents/Design.pdf

with another user.

We should not simply expose the storage object directly.

Instead:

User A
  │
  ▼
Sharing Service
  │
  ▼
Permission DB
  │
  ├── User A → Owner
  └── User B → Read

Then:

User B
  │
  ▼
Authorization
  │
  ▼
Can access?
  │
  ├── YES → File
  └── NO  → 403

19. Download Flow

A download can work similarly to upload.

Client
   │
   │ Request file
   ▼
API Server
   │
   │ Check authentication
   │ Check authorization
   ▼
Metadata DB
   │
   │ Find storage location
   ▼
Object Storage / CDN
   │
   ▼
Client

Again, the application server doesn't necessarily need to stream the entire file.

It can provide a secure temporary URL or equivalent controlled access mechanism.

20. CDN

For frequently downloaded files:

Client
   │
   ▼
CDN
   │
   ├── HIT ──► File
   │
   └── MISS
         │
         ▼
    Object Storage

CDN provides:

  • Lower latency

  • Reduced load on storage

  • Better global performance

21. Reliability and Redundancy

Files must not disappear because one machine fails.

We can replicate storage:

                 File
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
   Storage A   Storage B   Storage C
      ✅           ✅          ✅

If one fails:

Storage A ❌

Storage B ✅
Storage C ✅

The file remains available.

For even stronger durability, object storage can replicate data across:

  • Multiple machines

  • Multiple racks

  • Multiple availability zones

  • Potentially multiple regions

22. Metadata Database Replication

The metadata database also needs redundancy.

                 Application
                      │
                 ┌────▼────┐
                 │ Primary │
                 │   DB    │
                 └────┬────┘
                      │
                 Replication
                      │
             ┌────────┴────────┐
             ▼                 ▼
          Replica 1          Replica 2

If the primary fails:

Primary DB ❌
     │
     ▼
Replica promoted
     │
     ▼
System continues

23. Sharding Metadata

Eventually, one metadata database may become too large.

We can shard based on UserID.

Shard = UserID % N

Example:

UserID = 12345

12345 % 10 = 5

User's metadata → Shard 5

Architecture:

                  Metadata Service
                         │
                   Shard Router
                         │
       ┌─────────┬───────┼───────┬─────────┐
       ▼         ▼       ▼       ▼         ▼
    Shard 0   Shard 1  Shard 2 ...      Shard 9

This allows horizontal scaling.

24. Notification / Sync Service

We don't want every client to continuously ask:

"Did anything change?"
"Did anything change?"
"Did anything change?"

Instead, the server can notify connected clients.

For example:

Laptop ───────────────┐
                      │
Mobile ───────────────┼──► Sync Service
                      │
Tablet ───────────────┘

When a file changes:

File Change
    │
    ▼
Sync Service
    │
    ├──► Laptop
    ├──► Mobile
    └──► Tablet

The clients then retrieve the required changed metadata/chunks.

25. Complete Dropbox Flow

Upload

                 UPLOAD
                   │
                   ▼
              Client App
                   │
                   ▼
            Metadata Service
                   │
            Create upload session
                   │
                   ▼
             Chunk File
                   │
                   ▼
          Calculate Chunk Hash
                   │
                   ▼
          Object Storage Upload
                   │
                   ▼
            Update Metadata DB
                   │
                   ▼
             Notify Devices

Download

                 DOWNLOAD
                     │
                     ▼
                   Client
                     │
                     ▼
              Metadata Service
                     │
              Authorization
                     │
                     ▼
                CDN / Storage
                     │
                     ▼
                   Client

Synchronization

Local File Change
       │
       ▼
   Sync Agent
       │
       ▼
Calculate Hash
       │
       ▼
Find Changed Chunks
       │
       ▼
Upload Chunks
       │
       ▼
Update Metadata
       │
       ▼
Notify Other Devices
       │
       ▼
Other Device Downloads
Changed Chunks

26. Most Important Design Decisions

ProblemSolution
Huge file storageObject storage
Large uploadsChunking
Failed uploadsResumable upload
Only small part changedIncremental/chunk-level sync
Duplicate filesDeduplication
File recoveryVersioning + trash
Concurrent editsConflict detection/resolution
Global downloadsCDN
Metadata scalabilityDatabase sharding
Database failureReplication/failover
Storage failureReplicated object storage
Too many polling requestsNotification/sync service
Secure sharingAuthorization + controlled file access
Application server overloadDirect client ↔ object-storage transfer

⭐ 27. Interview Mental Model

Remember Dropbox using this simple flow:

              ┌──────────────────────┐
              │       CLIENT         │
              │   Sync Agent         │
              └──────────┬───────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │    API / Sync        │
              │      Service         │
              └──────────┬───────────┘
                         │
             ┌───────────┴───────────┐
             │                       │
             ▼                       ▼
       Metadata DB             Object Storage
       ───────────             ──────────────
       Files                   Actual data
       Folders                 Chunks
       Versions                Replicas
       Permissions
             │                       │
             └───────────┬───────────┘
                         │
                         ▼
                       CDN
                         │
                         ▼
                       Users

The five concepts to remember

1. Metadata DB → tells us what the file is and where it is.

2. Object Storage → stores the actual file/chunks.

3. Chunking → enables resumable and incremental uploads.

4. Sync Service → keeps multiple devices consistent.

5. Replication + CDN + Sharding → provides reliability, performance, and scalability.

Core Dropbox principle: Don't treat a large file as one giant database record. Break the problem into metadata + chunks + object storage + synchronization, and then scale each component independently.

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)