ఇది File Synchronization / Dropbox-style System Design interviewకి చాలా ముఖ్యమైన topic. మీ transcriptని unnecessary details లేకుండా class notes formatలో organize చేశాను.
📚 File Synchronization – Class Notes
1. File Synchronization ఎందుకు challenging?
File-sharing systemsలో పెద్ద challenge:
Multiple clients మధ్య filesని efficiently synchronize చేయడం.
ఒక large fileలో చిన్న change వచ్చిన ప్రతిసారీ మొత్తం fileని upload/download చేస్తే bandwidth waste అవుతుంది.
Example
1 GB video file ఉందనుకుందాం.
Editor 3 చిన్న changes చేశాడు.
Naive approach:
Change 1 → Upload 1 GB
Change 2 → Upload 1 GB
Change 3 → Upload 1 GB
Total Upload = 3 GBమరో client కూడా download చేస్తే:
Total Upload = 3 GB
Total Download = 3 GB
Total Network = 6 GBఅందుకే large filesకి whole-file synchronization inefficient.
2. ❌ Naive File Synchronization
Client
|
| Entire 1 GB file
↓
Server
|
| Entire 1 GB file
↓
Other ClientProblem:
High bandwidth consumption
High latency
Slow synchronization
Network outage వచ్చినప్పుడు download restart కావచ్చు
More users/files → bandwidth requirement increases significantly
3. ✅ Better Solution — Block-Level Synchronization
మొత్తం fileని పంపకుండా:
Fileని small fixed-size blocksగా split చేసి, changed blocks మాత్రమే synchronize చేయాలి.
Example:
1 GB File
↓
┌────┬────┬────┬────┬────┐
│ B1 │ B2 │ B3 │ B4 │ B5 │
└────┴────┴────┴────┴────┘ఒక block మాత్రమే change అయితే:
Before:
B1 B2 B3 B4 B5
After:
B1 B2 B3' B4 B5
Only B3' needs to be transferred.అంటే:
Whole file → ❌
Changed blocks → ✅4. ⭐ Rsync Algorithm
ఈ problem కోసం famous algorithm:
rsync
Main idea:
Sender మరియు receiver దగ్గర ఉన్న file versionsలో ఏ blocks same, ఏ blocks different అనేది identify చేసి, అవసరమైన data మాత్రమే networkలో పంపడం.
5. Rsync Basic Example
Suppose:
Client → Version A1
Server → Version A0A1 = Latest
A0 = OldGoal:
A0 → A1Instead of sending entire A1:
1 GB → Serveronly changed/missing pieces పంపాలి.
6. Rsync Flow
Step 1 — File split
Fileని fixed-size, non-overlapping blocksగా divide చేస్తాం.
A1:
┌────┬────┬────┬────┬────┐
│ B1 │ B2 │ B3 │ B4 │ B5 │
└────┴────┴────┴────┴────┘Step 2 — Calculate checksum/hash
Server దగ్గర ఉన్న old version A0ని కూడా blocksగా split చేస్తాం.
ప్రతి blockకి checksum calculate చేస్తాం.
B1 → Hash1
B2 → Hash2
B3 → Hash3
B4 → Hash4
B5 → Hash5Step 3 — Send hashes
Server entire blocks పంపదు.
Instead:
Server
↓
Block hashes
↓
ClientHashes చిన్నవిగా ఉంటాయి కాబట్టి network bandwidth చాలా తక్కువ.
Step 4 — Compare hashes
Client తన latest file A1 blocksకి hashes calculate చేసి server hashesతో compare చేస్తుంది.
Client Block Server Block
B1 ────────────→ Same
B2 ────────────→ Same
B3 ────────────→ Different
B4 ────────────→ Same
B5 ────────────→ Sameఅందువల్ల:
Only B3 changedఅని తెలుసుకోవచ్చు.
Step 5 — Send instructions + missing blocks
Client serverకి:
Keep B1
Keep B2
Replace B3
Keep B4
Keep B5అనే instructions పంపుతుంది.
అవసరమైన changed blocks మాత్రమే పంపుతుంది.
Step 6 — Server reconstructs latest version
Server:
B1 + B2 + B3(new) + B4 + B5చేసి latest file versionని పొందుతుంది.
7. 🎯 Important Optimization
Transcriptలో simplified explanation ఇచ్చారు.
Actual rsync algorithmలో checksum mechanism మరింత sophisticatedగా ఉంటుంది, including two checksums.
Interviewలో basic concept చెప్పేటప్పుడు:
Block → checksum → compare → transfer only differences
అంటే సరిపోతుంది.
8. 🏗️ Scalable File Synchronization Architecture
ఇప్పుడు algorithmని actual system architectureలో ఎలా integrate చేస్తామో చూద్దాం.
Client Side
Client
|
┌────────┴────────┐
↓ ↓
Watch Service Local Update Service
|
↓
Server Update Service
|
↓
Database9. Watch Service
Client machineలో file changesని monitor చేస్తుంది.
Example:
User modifies video.mp4
↓
Watch Service
↓
Change detectedWatch Service యొక్క job:
Change detect చేసి synchronization processని trigger చేయడం.
ఇది actual synchronization logicని handle చేయదు.
10. Server Update Service
Watch Service change detect చేసిన తర్వాత:
Watch Service
↓
Server Update ServiceServer Update Service:
Changed fileని blocksగా split చేస్తుంది
Block hashes calculate చేస్తుంది
Metadata persist చేస్తుంది
Serverకి hashes పంపుతుంది
Server ఏ blocks కావాలో చెబుతుంది
Required blocks మాత్రమే upload చేస్తుంది
11. Client-side Database
ప్రతి blockకి metadata maintain చేయవచ్చు:
File ID
Block ID
Block Hash
Block Size
Version
TimestampExample:
File: movie.mp4
Block 1 → Hash ABC
Block 2 → Hash DEF
Block 3 → Hash XYZదీంతో ప్రతి synchronization సమయంలో మళ్లీ unnecessary computation తగ్గుతుంది.
12. Server Side Architecture
Server
|
┌─────────────┴─────────────┐
↓ ↓
Sync Service Notification Service
|
↓
File Service
|
↓
Block Storage / Metadata DB13. Sync Service
Client నుంచి hashes వచ్చినప్పుడు:
Client
|
| Block hashes
↓
Sync ServiceSync Service:
Incoming hashes receive చేస్తుంది
Server stored hashesతో compare చేస్తుంది
Missing/different blocks identify చేస్తుంది
Clientకి required blocks list పంపుతుంది
Blocks receive చేసి synchronization complete చేస్తుంది
14. File Service
File Service responsibility:
Read blocks
Write blocks
Store block metadata
Retrieve block information
Manage file/block versions
Important:
Server తప్పనిసరిగా ప్రతి synchronizationలో complete fileని reconstruct చేయాల్సిన అవసరం లేదు.
Block-level storage maintain చేయవచ్చు.
15. 🔔 Server → Clients Synchronization
ఇది second major flow.
ఇప్పటివరకు:
Client → Serverచూశాం.
ఇప్పుడు:
Server → Other ClientsStep 1
Serverలో file update అవుతుంది.
New Block arrives
↓
Database updatedStep 2 — Server Watch Service
Serverలో కూడా watcher ఉండవచ్చు.
Database / Block Storage
↓
Watch Service
↓
Change detectedStep 3 — Notification Service
Watch Service:
Watch Service
↓
Notification ServiceNotification Service affected clientsకి notification పంపుతుంది.
Server
|
Notification Service
/ | \
↓ ↓ ↓
Client A Client B Client C16. Client receives synchronization notification
Clientలో:
Notification
↓
Local Update ServiceLocal Update Service:
Server hashes receive చేస్తుంది
Local hashesతో compare చేస్తుంది
Missing blocks identify చేస్తుంది
Serverని required blocks కోసం request చేస్తుంది
Blocks download చేస్తుంది
Local file update చేస్తుంది
17. Complete Architecture
Interviewలో ఈ diagram verbally explain చేయవచ్చు:
CLIENT
┌──────────────────────────┐
│ │
Files →│ Watch Service │
│ ↓ │
│ Server Update Service │
│ ↓ │
│ Local Metadata/Blocks │
└───────────┬──────────────┘
│
Hashes / Blocks
│
↓
┌──────────────────────────┐
│ SERVER │
│ │
│ Sync Service │
│ ↓ │
│ File Service │
│ ↓ │
│ Block Storage / DB │
│ ↓ │
│ Notification Service │
└───────────┬──────────────┘
│
Notifications
/ | \
↓ ↓ ↓
Client A Client B Client C18. 🔄 Two Important Flows
Flow 1: Client → Server
User changes file
↓
Watch Service
↓
Server Update Service
↓
Split into blocks
↓
Calculate hashes
↓
Send hashes
↓
Server compares hashes
↓
Returns missing blocks
↓
Client sends changed blocks
↓
Server stores blocksFlow 2: Server → Clients
Server receives changed blocks
↓
Watch Service
↓
Notification Service
↓
Notify all relevant clients
↓
Clients compare hashes
↓
Request missing blocks
↓
Download blocks
↓
Local Update Service
↓
Update local file19. 🧠 Most Important Interview Concepts
Q1. Why not upload the entire file?
Because large files + frequent small changes cause:
High bandwidth
High latency
Poor scalability
Q2. How do you reduce bandwidth?
Divide files into blocks and synchronize only changed blocks.
Q3. How do you identify changed blocks?
Calculate checksums/hashes for blocks and compare them between client and server.
Q4. Why send hashes instead of blocks?
Hash is much smaller than the actual block.
1 MB block
vs
small checksumSo comparison requires much less network bandwidth.
Q5. What does Watch Service do?
Detects local file changes and triggers synchronization.
Q6. What does Sync Service do?
Compares block hashes and determines which blocks need to be synchronized.
Q7. What does Notification Service do?
Notifies relevant clients that a file has changed.
Q8. What does Local Update Service do?
Downloads missing blocks and updates the local file.
20. ⭐ Interview Golden Statement
ఈ sentence గుర్తుపెట్టుకోండి:
“For efficient file synchronization, we should avoid transferring the entire file. We divide the file into fixed-size blocks, calculate checksums, compare block metadata between client and server, and transfer only the changed or missing blocks. A watch service detects local changes, a sync service handles block-level synchronization, and a notification service informs other clients about updates.”
🔥 One-line memory trick
CHANGE
↓
WATCH
↓
HASH
↓
COMPARE
↓
DIFF
↓
TRANSFER ONLY CHANGED BLOCKS
↓
NOTIFY OTHER CLIENTS
↓
DOWNLOAD MISSING BLOCKS
↓
SYNCInterview focus: Rsync → Block-level sync → Hash/checksum → Watch Service → Sync Service → Notification Service → Local Update Service — ఈ sequence clearగా explain చేయగలిగితే Dropbox/File Synchronization system-design questionలో core synchronization part strongగా cover అవుతుంది.
No comments:
Post a Comment