Absolutely. This chapter is essentially “How do we take Redis from one machine to a production-scale distributed system?” I’ll explain it in your Ramesh style: concept → why → architecture → example → interview point → real-world trade-off.
The chapter covers Persistence, Replication, Partitioning, Hash Partitioning, Presharding, Consistent Hashing, Tagging, and Twemproxy.
1. First understand the BIG picture
Imagine you have:
Application
|
v
+-------------+
| Redis |
| 1 Server |
+-------------+
Initially this is very fast.
But production grows:
100 GB → 500 GB → 2 TB data
10K requests/sec → 100K → 1M
One server becomes a bottleneck
Server failure becomes a major problem
So Redis scaling has 3 fundamental dimensions:
Redis Scaling
|
+---------------+---------------+
| | |
v v v
Persistence Replication Partitioning
| | |
Don't lose Don't depend Spread data
data on one node across nodes
Ramesh formula:
Persistence = Don't lose data
Replication = Don't lose availability
Partitioning = Don't overload one machine
2. Persistence — “Redis memory is temporary”
Redis primarily stores data in memory.
Suppose:
Redis RAM
user:101 → Ramesh
user:102 → Sudha
user:103 → Yagna
Suddenly:
Redis crashes 💥
RAM disappears.
Therefore:
RAM = Fast
Disk = Durable
Redis provides two major persistence mechanisms:
Persistence
|
+--------+--------+
| |
v v
RDB AOF
Snapshot Write Log
The file explicitly describes RDB and AOF as the two Redis persistence mechanisms, which can also be enabled together.
3. RDB — “Take a photograph 📸”
Think of RDB as taking a snapshot of Redis.
At 10:00:
Redis:
A = 100
B = 200
C = 300
Redis creates:
dump.rdb
That file represents the dataset at that point in time.
Example
10:00 → Snapshot
10:05 → Snapshot
10:10 → Snapshot
If Redis crashes at:
10:12
You may restore from:
10:10 snapshot
But:
10:10 → 10:12
changes may be lost.
Ramesh rule
RDB = Fast recovery + possible recent-data loss
4. SAVE vs BGSAVE
This is an important interview question.
SAVE
Redis Main Process
|
| SAVE
v
Disk
Redis blocks while creating the snapshot.
Therefore:
❌ Don't use
SAVEcasually in production.
BGSAVE
Redis
|
fork()
/ \
/ \
Main Process Child
| |
Serve users RDB
Disk
BGSAVE creates the snapshot in a child process so the main process can continue serving requests.
But there is a hidden cost:
Copy-on-Write
Suppose:
Parent
100 GB
Child is creating RDB.
Meanwhile application modifies memory.
Redis may need additional memory for changed pages.
So:
Memory usage
↑
|
100GB| + changed pages
| /
|______/
Interview point:
BGSAVE is non-blocking from the application's perspective, but fork + copy-on-write can increase memory pressure.
5. Redis Snapshot Configuration
The file gives examples such as:
save 900 1
save 300 10
save 60 10000
Meaning:
| Condition | Snapshot |
|---|---|
| 1 change within 15 min | RDB |
| 10 changes within 5 min | RDB |
| 10,000 changes within 1 min | RDB |
Ramesh interpretation
This is not:
“Save every 60 seconds.”
It is:
“If enough writes happen within the specified interval, take a snapshot.”
6. AOF — “Write down every transaction 📝”
RDB:
Take photograph
AOF:
Record the commands
Suppose application executes:
SET user:1 Ramesh
SET user:2 Sudha
INCR pageview
INCR pageview
AOF records the write commands.
After restart:
Redis
|
v
Read AOF
|
v
Replay commands
|
v
Rebuild dataset
The file describes AOF as an append-only command log that can rebuild Redis state by replaying commands in order.
7. AOF fsync — VERY important
AOF has three important policies:
appendfsync
|
+-----+------+
| |
no everysec always
no
Redis → OS → Disk
OS decides when to flush.
✅ Fast
❌ More potential data loss
everysec
Redis
|
+---- flush every second
✅ Good balance
⭐ Common default in the source material
always
Every write
|
v
fsync
|
v
Disk
✅ Strongest durability
❌ Slowest
The source explicitly describes no as fastest, always as safest but slowest, and everysec as a performance/durability balance.
Interview answer
RDB optimizes snapshot-based recovery; AOF prioritizes durability by recording writes.
8. RDB vs AOF — Ramesh Table
| Feature | RDB | AOF |
|---|---|---|
| Concept | Snapshot | Command log |
| File | Binary | Append-only log |
| Recovery | Faster | Usually slower |
| Data loss | Possible between snapshots | Depends on fsync policy |
| File size | Smaller | Can become larger |
| Main use | Backup / DR | Durability |
| Performance | Generally better | More overhead |
| Can coexist? | Yes | Yes |
An important point from the source: when both exist, AOF takes precedence during startup because of its durability characteristics.
9. Replication — “One Redis is not enough”
Now imagine:
Application
|
v
MASTER
Redis-1
Redis-1 crashes.
Game over.
So create replicas:
MASTER
Redis-1
/ \
/ \
v v
Replica-1 Replica-2
Writes:
Application
|
v
MASTER
|
+--------> Replica 1
|
+--------> Replica 2
Redis replication allows one master to have multiple replicas.
10. Why Replication?
Three major reasons:
① Read scaling
MASTER
writes only
|
+-------+-------+
| |
v v
Replica Replica
reads reads
Instead of:
1 Redis → 100K reads
you can distribute:
Master → writes
Replica1 → reads
Replica2 → reads
Replica3 → reads
The source specifically identifies replicas as a way to handle read operations separately from writes.
② High availability
If master dies:
MASTER 💥
Replica
|
v
Promote → MASTER
③ Data redundancy
Multiple copies exist.
11. But replication has a BIG problem
Replication alone does not automatically mean failover in the single-instance setup described by the chapter.
Example:
Master A
|
+---- Replica B
+---- Replica C
Master A dies.
You manually need:
B → Master
C → replicate B
Clients → connect B
The source notes that automatic failover is the role of Redis Sentinel, covered in the following chapter.
Ramesh rule
Replication gives copies. Sentinel gives automatic failover.
And later:
Redis Cluster gives distributed data + cluster management.
12. Partitioning — THE BIG SCALING CONCEPT
Suppose:
Redis Server = 128 GB RAM
Your dataset becomes:
500 GB
Replication won't solve the capacity problem.
Why?
Because:
Master = 128 GB
Replica = copy of 128 GB
Replica = copy of 128 GB
You still need a machine capable of holding the entire dataset.
So we need:
SHARDING
The source defines partitioning as breaking data up and distributing it across hosts; in Redis, horizontal partitioning means distributing keys across instances.
13. Horizontal Partitioning
Imagine:
Redis Cluster
Server 1
user:1
user:2
user:3
Server 2
user:4
user:5
user:6
Server 3
user:7
user:8
user:9
Data is split by keys.
This is:
Horizontal partitioning = Sharding
14. Vertical Partitioning
Different idea:
User data
Profile
Orders
Payments
Activity
Could distribute different parts across servers.
Conceptually:
Redis-1 → Profile
Redis-2 → Orders
Redis-3 → Payments
The source distinguishes horizontal partitioning by keys from vertical partitioning by key values.
15. Range Partitioning
Very simple.
Suppose:
user:1
user:2
...
user:5000
Divide:
Redis-1 → 1–1000
Redis-2 → 1001–2000
Redis-3 → 2001–3000
Redis-4 → 3001–5000
The source uses exactly this type of incremental-ID example.
Problem #1 — Hotspot / uneven distribution
Suppose:
Redis-1 → 1 million keys
Redis-2 → 10,000 keys
Redis-3 → 10,000 keys
Bad distribution.
Problem #2 — Adding a server
Originally:
1 → Redis A
2 → Redis B
3 → Redis C
Add Redis D.
Ranges may need significant restructuring.
Therefore:
Range partitioning = simple, but difficult to rebalance.
16. Hash Partitioning
Now we become smarter.
Instead of:
key range → server
we do:
hash(key) % numberOfServers
Example:
hash("user:101") = 15
15 % 3 = 0
→ Redis-0
The source shows this exact basic formula.
Architecture:
user:101
|
v
hash()
|
v
15
|
% 3
|
+--------+--------+
| | |
0 1 2
| | |
R1 R2 R3
Usually distribution becomes much more balanced than naive ranges.
17. BUT Hash Partitioning has a killer problem
Suppose:
3 Redis servers
Then:
hash(key) % 3
Now add:
Redis-4
It becomes:
hash(key) % 4
For many keys:
old server ≠ new server
So keys move.
For a cache:
Old:
user:100 → Redis-2
New:
user:100 → Redis-4
Redis-4 doesn't have it.
Result:
CACHE MISS 💥
The source reports that changing the number of instances can invalidate a large portion of data; its example saw 75% invalidated after adding two servers.
18. Presharding — clever workaround
Idea:
Don't wait until you need more nodes. Create many logical partitions upfront.
For example:
Server 1
├── Redis 6379
├── Redis 6380
├── Redis 6381
├── Redis 6382
└── Redis 6383
Server 2
├── Redis 6379
├── Redis 6380
...
Instead of:
3 partitions
create:
15 partitions
Then later:
Small Server → Big Server
instead of changing:
15 partitions → 20 partitions
The source calls this presharding and explains that multiple Redis instances can be run per physical server.
Advantage
Hash mapping stays stable.
Disadvantage
More instances
↓
More monitoring
↓
More operational complexity
And it isn't truly elastic.
19. ⭐ Consistent Hashing — THE MOST IMPORTANT CONCEPT
This is the concept you were asking about earlier.
Normal hashing:
hash(key) % N
Problem:
N changes
↓
mapping changes massively
Consistent hashing says:
When nodes change, move only a small amount of data.
The source explains the idealized remapping as roughly K/n keys, where K is the number of keys and n is the number of servers.
20. Hash Ring — Think Like a Clock 🕐
Instead of:
0
1
2
3
imagine a circle:
Server B
●
.-------------.
. .
. .
Server A ● ● Server C
. .
. .
'-------------'
This is the:
HASH RING
Both:
Servers
Keys
are hashed onto the ring.
21. Consistent Hashing Example
Suppose:
Server-1 → hash 3
Server-2 → hash 7
Server-3 → hash 11
Keys:
key1 → 3
key2 → 4
key3 → 8
key4 → 12
Now clockwise:
0 ---- 3 ---- 7 ---- 11 ---- 15
S1 S2 S3
key1 = 3
Exactly at S1:
key1 → S1
key2 = 4
Next server clockwise:
4 → 7 → S2
key3 = 8
Next server:
8 → 11 → S3
key4 = 12
No server after 12.
So wrap around:
12 → 3 → S1
This is the same routing logic described in the source.
22. Why Adding a Server Is Better
Current:
S1 S2 S3
-------●--------●--------●------
Add:
S4
●
Only keys in the region immediately affected by S4 need to move.
Not everything.
That's the BIG WIN
Normal Hashing
Node added
↓
Many mappings change
↓
Many cache misses
Consistent Hashing
Node added
↓
Small portion moves
↓
Most cache mappings remain
23. Virtual Nodes — Very Important
Suppose only one point per server:
S1 ●
S2 ●
S3 ●
Distribution may be poor.
So create multiple points:
S1 → S1-1 S1-2 S1-3 S1-4 ...
S2 → S2-1 S2-2 S2-3 S2-4 ...
S3 → S3-1 S3-2 S3-3 S3-4 ...
These are:
Virtual Nodes / VNodes
The source's implementation defaults to 256 virtual nodes per client in its example.
Why?
To make distribution more uniform.
Physical Server
|
+-- vnode1
+-- vnode2
+-- vnode3
...
This is extremely important in distributed systems.
24. Consistent Hashing — Java Architect View
Think:
server = ring.getNextNode(hash(key));
Not:
server = servers.get(hash(key) % servers.size());
The difference:
Modulo hashing
↓
Node count is critical
Consistent hashing
↓
Ring position is critical
25. Tagging — Redis Multi-Key Problem
This is another very important Redis Cluster concept.
Suppose:
user:1
user:2
Hashing may send them to different nodes.
But what if you execute:
SINTER user:1 user:2
Redis needs both keys on the same Redis instance for this kind of operation.
Solution:
Hash Tags
Use:
user:1{users}
user:2{users}
user:3{users}
Redis hashes:
{users}
instead of the entire key.
Therefore:
user:1{users} ──┐
user:2{users} ──┼──> same Redis node
user:3{users} ──┘
The source specifically describes curly-brace tags as a way to force related keys onto the same instance.
Ramesh rule
If multiple keys must participate in one Redis operation, give them the same hash tag.
26. Cache vs Data Store — VERY IMPORTANT DESIGN DECISION
This is probably the most architect-level section.
Redis as CACHE
Database
↑
|
Redis Cache
If cache data disappears:
Cache miss
↓
Database
↓
Reload cache
Therefore:
Cache can tolerate remapping.
Recommended:
Redis Cache
↓
Consistent Hashing
The source explicitly recommends consistent hashing for Redis cache workloads to minimize cache misses.
27. Redis as PRIMARY DATA STORE
Now imagine Redis contains:
Customer account balance
Payment information
Order state
You cannot casually move:
user:101
from Redis-1 to Redis-2.
You need:
Data ownership
Replication
Failover
Routing
Consistency
Recovery
That's where:
Redis Cluster
becomes much more appropriate.
The source recommends Redis Cluster or an equivalent replicated routing solution when Redis is used as a data store.
28. Client vs Proxy vs Query Router
There are three places where sharding logic can live:
Sharding
|
+---------+---------+
| | |
v v v
Client Proxy Query Router
① Client-side
Application decides:
key → Redis node
Application
|
+--> Redis-1
+--> Redis-2
+--> Redis-3
② Proxy
Application thinks there is one Redis:
Application
|
v
Proxy
/ | \
R1 R2 R3
Proxy decides where the key goes.
③ Query Router
Redis cluster itself handles routing.
Application
|
v
Redis Cluster
|
+---+---+---+
R1 R2 R3
The source describes these three layers and notes that Redis Cluster acts as the query-routing layer.
29. Twemproxy
Twemproxy is essentially:
A proxy sitting between application and Redis servers to perform sharding.
Architecture:
Application
|
v
Twemproxy
/ | \
/ | \
R1 R2 R3
Application doesn't need to know:
Which key → which Redis
Twemproxy handles it.
The source describes twemproxy as a lightweight Redis/Memcached proxy supporting multiple hashing modes including consistent hashing.
30. BUT Twemproxy has a SPOF
Imagine:
Application
|
v
Twemproxy 💥
|
X X X
Redis Redis Redis
Redis servers are healthy.
But application can't reach them.
Therefore:
Proxy itself must be highly available.
Better:
Load Balancer
/ \
v v
Twemproxy-1 Twemproxy-2
| | | | | |
R1 R2 R3 R1 R2 R3
The source explicitly identifies a single twemproxy process as a single point of failure and proposes a load balancer in front of multiple proxy instances.
🧠 Ramesh Master Architecture
Now put everything together.
CLIENTS
|
v
Load Balancer
|
+-------------+-------------+
| |
v v
Redis Proxy 1 Redis Proxy 2
| |
+-------------+-------------+
|
Redis Cluster
|
+-----------------+-----------------+
| | |
v v v
Shard-1 Shard-2 Shard-3
| | |
Master Master Master
/ \ / \ / \
R1 R2 R1 R2 R1 R2
And persistence:
Redis Master/Replica
|
+---+---+
| |
RDB AOF
| |
Backup Durable log
🔥 Ramesh Interview Cheat Sheet
| Concept | One-line meaning |
|---|---|
| RDB | Snapshot of Redis |
| AOF | Record Redis write commands |
| Replication | Copy data to other Redis nodes |
| Replica | Read scaling + redundancy |
| Failover | Promote replica to master |
| Partitioning | Split data across machines |
| Sharding | Distribute keys across nodes |
| Range partitioning | Key range → server |
| Hash partitioning | hash(key) % N |
| Presharding | Create many partitions upfront |
| Consistent hashing | Add/remove nodes with minimal remapping |
| Hash ring | Circular hash space |
| Virtual node | Multiple logical positions for one server |
| Tagging | Force related keys onto same shard |
| Proxy | Routing layer outside Redis |
| Twemproxy | Redis/Memcached sharding proxy |
| Sentinel | Automatic failover/monitoring |
| Redis Cluster | Distributed Redis with sharding + cluster management |
⭐ The Most Important Mental Model
Remember this sequence:
REDIS SCALING
|
+----------+----------+
| | |
v v v
PERSISTENCE REPLICA SHARDING
| | |
v v v
RDB/AOF Read/HA Split data
|
+----------+----------+
| |
v v
Hashing Consistent
Hash % N Hashing
|
v
Hash Ring
|
v
VNodes
|
v
Tags
And the Ramesh Golden Rule:
RDB/AOF solves DATA LOSS.
Replication solves READ SCALE + REDUNDANCY.
Sentinel solves FAILOVER.
Sharding solves DATA SIZE + WRITE SCALE.
Consistent Hashing solves NODE CHANGE + CACHE MISS.
Hash Tags solve MULTI-KEY OPERATIONS.
Redis Cluster brings distributed Redis together.
This chapter concludes with exactly these major themes—persistence, replication, partitioning, presharding, consistent hashing, and twemproxy—before moving into Redis Sentinel and Redis Cluster.
No comments:
Post a Comment