Final Memory Map
STORAGE
|
┌────────────┼────────────┐
↓ ↓ ↓
FILE OBJECT BLOCK
| | |
Folders Flat Chunks
| | |
Simple Scalable Fast
| + Metadata |
| | Partial
| | updates
↓ ↓ ↓
Human use Big data File Sync
⭐📚 File Storage – Class Notes
ఈ lectureలో Dropbox / File Sharing Systemలో server-side filesని ఎలా store చేయాలి? అనే problemని solve చేస్తున్నాం.
మనం ఇప్పటికే:
File synchronization
Communication protocols
Message Broker
చూశాం.
ఇప్పుడు missing component:
Serverలో files/blocksని efficiently persist చేయడం ఎలా?
Lectureలో 3 storage models compare చేశారు:
File Storage → Object Storage → Block Storage
మరియు File Sharing use caseకి Block Storage ఎందుకు strong fit అవుతుందో explain చేశారు.
1. 🗂️ File Storage
File storageలో dataని traditional files + folders structureలో store చేస్తాం.
Root
├── Users
│ ├── UserA
│ │ ├── file1
│ │ └── file2
│ └── UserB
│ └── file3ఇది hierarchical structure.
ప్రతి fileకి metadata ఉంటుంది:
File path
Creation date
Modification date
File size
ఇతర file metadata
2. ✅ File Storage Advantages
Simple
Traditional file systems అందరికీ familiar.
Existing application support
Operating systems మరియు applicationsతో easyగా integrate చేయవచ్చు.
Relatively inexpensive
సాధారణ storage needsకి cost-effective.
Good for manual access
Users folders/files tree structure ద్వారా easily browse చేయగలరు.
3. ❌ File Storage Limitations
Files మరియు folders సంఖ్య పెరిగేకొద్దీ filesystemకి:
ఎక్కువ resource management
indexing/tracking overhead
performance challenges
వస్తాయి.
ముఖ్యమైన point:
Storage capacity పెంచడం మాత్రమే filesystem structural/performance problemsని solve చేయదు.
4. File Storage ఎప్పుడు ఉపయోగించాలి?
Suitable for:
Simple file management
Local archives
Data protection
User-driven file browsing
అంటే:
Human-friendly hierarchical file management అవసరమైనప్పుడు File Storage మంచి choice.
5. ☁️ Cloud File Storage
Lecture examples:
Azure Files
AWS Elastic File System
Google Cloud file-storage offering
Conceptually:
Application
↓
Cloud File Storage
↓
Files + Folders6. 🎯 Why File Storage is not ideal for our File Sharing System?
మన requirement:
Programmatic reads/writes + large scale + efficient synchronization
కాబట్టి traditional hierarchical filesystem కంటే better scalable storage model కావాలి.
అందుకే next:
Object Storage
7. 🪣 Object Storage
Object Storageలో hierarchical folders కాకుండా flat structure ఉంటుంది.
Object Storage
Object A
Object B
Object C
Object D
Object Eప్రతి objectలో:
Object
├── File Data
├── Metadata
└── Unique IDఉంటాయి.
8. ⭐ Object Storage Advantages
1. Flat structure
Traditional nested directories అవసరం లేదు.
2. Metadata support
Objectకి metadata attach చేయవచ్చు.
ఉదాహరణ:
File: video.mp4
Metadata:
User = Ramesh
Type = video
Created = ...
Version = 53. Unique identifier
ప్రతి objectకి unique ID ఉండటం వల్ల object locate/retrieve చేయడం easy.
4. Excellent scalability
Storage clusterకి additional nodes add చేయడం ద్వారా scale out చేయవచ్చు.
9. ❌ Object Storage Major Limitation
ఇది మన File Sync use caseకి చాలా important.
Objectని partialగా update చేయడం సాధారణంగా possible కాదు; object మొత్తం update/rewrite చేయాల్సి ఉంటుంది.
Example:
1 GB Object
Only 5 MB changedఅయినా conceptually:
Read entire object
↓
Modify
↓
Rewrite entire objectఅవుతుంది.
ఇది frequent small updatesకి inefficient.
10. Object Storage – Best Use Case
Object Storage excellent for:
Large amounts of unstructured data
Very large-scale storage
High scalability
Rich metadata
Files that don't need frequent partial updates
11. 🧱 Block Storage
ఇప్పుడు మన use caseకి చాలా interesting option:
Block Storage
Fileని ఒక single objectగా కాకుండా fixed-size blocksగా split చేస్తాం.
File
↓
┌────┬────┬────┬────┬────┐
│ B1 │ B2 │ B3 │ B4 │ B5 │
└────┴────┴────┴────┴────┘ప్రతి blockకి:
Unique address / identifier
ఉంటుంది.
12. 🔥 Why Block Storage is Perfect for Our Use Case
మనం previous File Synchronization lectureలో filesని already blocksగా split చేశాం.
File
↓
Blocks
↓
Hash blocks
↓
Compare
↓
Transfer changed blocksఅదే blocksని server-sideలో కూడా store చేయవచ్చు.
అంటే:
Synchronization strategy మరియు storage strategy naturally align అవుతాయి.
13. Example
Suppose 1 GB file:
1 GB
↓
1000 blocksOnly Block 347 changed.
Object Storage
Update B347
↓
Potentially rewrite whole 1 GB objectBlock Storage
Update B347
↓
Rewrite only B347అందువల్ల frequent partial updatesలో block storage efficient.
14. ✅ Block Storage Advantages
1. Efficient random access
Unique block IDs వల్ల required blockని directly access చేయవచ్చు.
2. Low data-transfer overhead
Entire file కాకుండా required blocks మాత్రమే transfer చేయవచ్చు.
3. Frequent writes
ఒక block మాత్రమే మార్చితే:
Old B347
↓
New B347చేస్తే సరిపోతుంది.
4. Low latency
High-performance accessకి suitable.
5. Large filesకి useful
Frequent updates ఉన్న large filesకి particularly useful.
15. ❌ Block Storage Limitations
1. Metadata support చాలా limited
Blockకి essentially:
Block ID / Addressలాంటిది మాత్రమే ఉంటుంది.
File name, owner, creation date, version వంటి rich metadata application/database layerలో manage చేయాలి.
అంటే:
Block Storage
+
Application Metadata DBఅవసరం కావచ్చు.
2. Cost
Block storage సాధారణంగా:
More expensive than basic file/object storage
అని lectureలో position చేశారు.
3. More complexity
Applicationకి metadata management responsibility పెరుగుతుంది.
16. Block Storage ఎప్పుడు ఉపయోగించాలి?
Best suited for:
High-performance applications
Mission-critical workloads
Low latency
Consistent I/O performance
Large files with frequent updates
17. 🆚 File vs Object vs Block Storage
| Feature | File Storage | Object Storage | Block Storage |
|---|---|---|---|
| Structure | Hierarchical | Flat | Blocks |
| Data unit | File | Object | Block |
| Metadata | Good | Excellent | Limited |
| Scalability | Moderate | ⭐ Excellent | Good |
| Partial update | Possible via file system | ❌ Whole object | ✅ Individual block |
| Performance | Good | Good | ⭐ High |
| Cost | Lower | Usually economical at scale | Higher |
| Large unstructured data | Good | ⭐ Excellent | Good |
| Frequent small updates | Moderate | ❌ Poor fit | ⭐ Excellent |
| File Sharing use case | ❌ | ⚠️ | ⭐ |
18. 🧠 Most Important Comparison
File Storage
Simple
+
Cheap
+
Human-friendlyBut:
Large scale / programmatic access
↓
Less idealObject Storage
Highly scalable
+
Metadata
+
Large unstructured dataBut:
Small portion changed
↓
Whole object update
↓
InefficientBlock Storage
File
↓
Blocks
↓
Update only changed blockThis matches our synchronization model.
19. 📄 Document Store ≠ File Storage
Lecture చివర్లో important clarification ఉంది.
Document Store అనే పేరు చూసి:
"ఇది documents/files store చేయడానికి ఉపయోగిస్తారేమో"
అనుకోకూడదు.
Document store అనేది:
Document-oriented database
ఉదాహరణ data:
{
"user": "Ramesh",
"file": "movie.mp4",
"version": 5
}లాంటివి.
ఇది large unstructured binary files store చేయడానికి traditional file storage alternative కాదు.
20. 🏗️ Dropbox-style Architectureలో Storage
మన previous lecturesని కలిపితే:
CLIENT
|
File changed
↓
Watch Service
↓
Server Update Service
↓
Split into Blocks
↓
Hash Blocks
↓
SERVER
|
┌─────┴─────┐
↓ ↓
Message Broker Sync Service
↓
Block Storage
+
Metadata DBఇక్కడ:
Block Storage
Actual file blocks store చేస్తుంది.
Metadata DB
Store చేయవచ్చు:
File ID
User ID
File Name
Version
Block ID
Block Hash
Block Order
Timestamp21. 🔄 Complete File Update Flow
User edits 1 GB file
↓
File split into blocks
↓
Identify changed block
↓
Hash comparison
↓
Only changed block
↓
Message Broker / Sync flow
↓
Block Storage
↓
Update block
↓
Notification Service
↓
Other Clients
↓
Download changed blockKey idea:
Storage should complement synchronization.
Synchronization says:
Which blocks changed?
Block storage says:
Where do we store those blocks efficiently?
🎯 Interview Questions & Answers
Q1. Why not use object storage?
Object storage is highly scalable and supports rich metadata, but partial updates are inefficient because the whole object generally needs to be rewritten. Our file-sharing system frequently changes small portions of large files, so block storage is a better fit.
Q2. Why block storage?
Because files are already divided into blocks for synchronization. We can store and update only the changed blocks, reducing data transfer and improving write performance.
Q3. What is the disadvantage?
Block storage has limited metadata capabilities, so file-level metadata must be managed separately at the application/database layer. It is also more expensive.
Q4. Why not file storage?
File storage is simple and cost-effective, but it is less optimized for highly scalable programmatic access to large amounts of frequently changing data.
🧠 Final Memory Map
STORAGE
|
┌────────────┼────────────┐
↓ ↓ ↓
FILE OBJECT BLOCK
| | |
Folders Flat Chunks
| | |
Simple Scalable Fast
| + Metadata |
| | Partial
| | updates
↓ ↓ ↓
Human use Big data File Sync
⭐⭐ Interview Golden Line
“For a Dropbox-like file-sharing system, block storage is a strong fit because our synchronization algorithm already divides large files into fixed-size blocks. When a small portion changes, we can update only the affected block instead of rewriting the entire file or object. The trade-off is that block storage provides limited metadata, so file-level metadata needs to be managed separately.”
🔥 మొత్తం Dropbox System Design chain
File Change → Watch Service → Block Hashing → Message Broker → Sync Service → Block Storage → Notification Service → SSE → Other Clients → Changed Block Download
ఈ chainని interviewలో end-to-end explain చేయగలిగితే Dropbox/File Sharing System Designలో core architecture చాలా strongగా ఉంటుంది.
No comments:
Post a Comment