Distributed Synchronization — Class Notes
1. Why Synchronization Is Difficult
A distributed system has three important characteristics:
Concurrent operation — multiple nodes perform computations at the same time.
Independent failures — one node can fail without telling us anything about when another node will fail.
No shared/global clock — there is no single clock that all nodes can rely on as the exact same source of time.
The third point creates a major challenge.
In a distributed system, we cannot guarantee that:
“All nodes agree that it is exactly the same moment.”
We can synchronize clocks approximately, but perfect synchronization is practically impossible.
2. Why Does “Now” Matter?
Consider a distributed data store with replicated data.
Suppose we have three replicas:
| Replica | Value |
|---|---|
| Node 1 | Americano |
| Node 2 | Americano |
| Node 3 | Bread |
Now the replicas disagree.
The important question is:
Which value was written last?
Without knowing the order of writes, we cannot confidently decide which value should win.
For example:
Write 1 → Americano
Write 2 → Breador perhaps:
Write 1 → Bread
Write 2 → AmericanoLooking only at the current replica values doesn't tell us which happened last.
3. Solution #1 — Timestamps
One approach is to attach a timestamp to every write.
Example:
Americano → 10:01:05
Bread → 10:01:08The system can then use:
Last Write Wins (LWW)
So the value with the latest timestamp becomes the winner.
Advantage
Very simple and works well in many real-world distributed systems.
Problem
It depends on having reasonably accurate clocks across nodes.
If Node A's clock says:
10:01:10while Node B's clock says:
10:01:05the timestamps may produce the wrong ordering.
Therefore, the question becomes:
How accurate does our clock synchronization need to be?
4. How Do We Synchronize Clocks?
Option 1 — GPS
We could install GPS receivers on servers and use GPS time as a common reference.
Conceptually:
GPS Time
↓
┌───────┼───────┐
↓ ↓ ↓
Node A Node B Node CThis can provide extremely accurate time.
However, for most distributed systems, it is:
expensive
complicated
unnecessary
difficult to deploy at large scale
So it is usually impractical.
5. NTP — Network Time Protocol
The common solution is Network Time Protocol (NTP).
NTP allows computers to synchronize their clocks over a network using reliable time sources.
Conceptually:
Time Server
↓
┌──────┼──────┐
↓ ↓ ↓
Node A Node B Node CNTP doesn't make every machine's clock perfectly identical.
Instead, it gets the clocks close enough for many practical applications.
Key idea
NTP gives us an estimate of physical time.
This is often sufficient for systems that use timestamps and Last Write Wins.
6. What If “Good Enough” Isn't Good Enough?
This is where things become interesting.
Suppose our application requires extremely precise ordering.
Even NTP cannot guarantee:
Node A: 10:00:00.000000
Node B: 10:00:00.000000
Node C: 10:00:00.000000at exactly the same instant.
So instead of asking:
“What time did this event happen?”
we can ask:
“What event happened before what other event?”
This leads to logical clocks.
7. Vector Clocks
A vector clock doesn't attempt to tell us the actual physical time.
Instead, it tells us about the ordering of events.
For example:
Event A → Event B → Event CThe important information is the sequence:
A happened before B
B happened before Crather than:
A happened at 10:01:03
B happened at 10:01:05
C happened at 10:01:08Why is this useful?
Because for distributed systems, we often don't actually care about the exact time.
We care about:
Which operation happened before another operation?
8. Physical Time vs Logical Time
| Physical Time | Logical Time |
|---|---|
| Uses real clocks | Uses event relationships |
| NTP can synchronize clocks | Vector clocks track ordering |
| Gives approximate timestamps | Gives ordering information |
| Can have clock skew | Doesn't depend on synchronized clocks |
| Useful for Last Write Wins | Useful for determining causality/order |
| Easier to implement | More complex |
9. Important Concept — Time vs Ordering
This is the key takeaway from this section.
Physical clocks
Tell us approximately:
WHEN did something happen?Logical clocks / Vector clocks
Tell us:
WHAT happened before WHAT?For distributed systems, ordering is often more important than exact time.
10. Trade-off
There is no perfect solution.
NTP
Pros
Simple
Widely deployed
Practical
Good enough for many applications
Cons
Not perfectly accurate
Clock skew remains
Cannot guarantee exact simultaneous time
Vector Clocks
Pros
Don't depend on synchronized physical clocks
Can establish a formally valid ordering
Useful for distributed consistency and conflict detection
Cons
More complex
Additional metadata
More application-level complexity
⭐ Interview Takeaways
Remember these points:
1. Why is synchronization difficult?
Because distributed nodes operate concurrently and there is no perfectly shared global clock.
2. Why do we need synchronization?
To determine the ordering of operations, especially when replicated data contains conflicting values.
3. Common solution for physical time?
NTP (Network Time Protocol).
4. Why not GPS everywhere?
It can provide highly accurate time but is impractical and expensive for most distributed systems.
5. What if physical time isn't accurate enough?
Use logical clocks, such as vector clocks.
6. What does a vector clock tell us?
Not the exact time, but the ordering/causal relationship between events.
🧠 One-line memory trick
NTP tells us approximately “WHEN”; Vector Clocks tell us “WHICH CAME BEFORE WHICH.”
-----
Vector Clock:
Absolutely. Vector Clock is much easier if you first understand the problem it solves.
1. The basic problem
Imagine 3 servers:
Distributed System
Node A Node B Node C
| | |
| | |
Event Event EventAll nodes work independently.
There is no perfectly synchronized clock.
So if:
Node A says: 10:05:01
Node B says: 10:05:00we cannot always trust the timestamps to determine which event actually happened first.
Instead, we want to know:
Did Event X happen before Event Y?
That's what a vector clock helps us determine.
2. Think of a vector clock as a scoreboard
Suppose we have 3 nodes:
A B CEach node maintains a vector containing a counter for every node.
Initially:
A = [0,0,0]
B = [0,0,0]
C = [0,0,0]The positions always mean:
A B C
↓ ↓ ↓
Vector [0, 0, 0]So:
position 1 → events known from A
position 2 → events known from B
position 3 → events known from C
3. Node A performs an event
Suppose A performs:
A: Write "Americano"A increments its own counter:
A = [1,0,0]Meaning:
A has performed 1 event, and A knows nothing about events from B or C.
4. A sends information to B
Now A sends its data to B.
A = [1,0,0]
↓
BB receives the message.
B merges the vectors and increments its own counter.
Before:
B = [0,0,0]After receiving A's information:
B = [1,1,0]Interpretation:
B knows about 1 event from A and has performed 1 event itself.
5. B performs another event
Suppose B writes:
B: Write "Bread"B increments its own counter:
B = [1,2,0]Now B knows:
A → 1 event
B → 2 events
C → 0 events6. B sends information to C
B sends its vector:
B = [1,2,0]
↓
CC originally has:
C = [0,0,0]C receives B's vector and merges:
C = [1,2,0]Then if C performs an event:
C = [1,2,1]Now C knows about:
A → 1
B → 2
C → 17. The REALLY important part — comparing vectors
This is where vector clocks become powerful.
Suppose we have two events:
Event X → [2,1,0]
Event Y → [2,3,0]Compare each position:
A B C
X = [2, 1, 0]
Y = [2, 3, 0]
↑ ↑ ↑For every position:
X <= Yand at least one position is strictly smaller.
Therefore:
X happened before Y.
We write:
X → YThis is called causal ordering.
8. What if vectors are incomparable?
This is the most important distributed-systems concept.
Suppose:
Event X = [2,1,0]
Event Y = [1,0,3]Compare:
A B C
X = [2, 1, 0]
Y = [1, 0, 3]X is greater than Y for A and B:
2 > 1
1 > 0But X is less than Y for C:
0 < 3So neither vector is completely greater than the other.
Therefore:
Neither event happened before the other.
They are concurrent events.
X
↗
?
↘
YMore accurately, they happened independently.
9. Why is this useful for replicated data?
Go back to our original example:
Replica A → Americano
Replica B → Americano
Replica C → BreadSuppose:
Americano → [2,1,0]
Bread → [1,0,1]The vectors are incomparable.
That tells us:
These two writes happened independently/concurrently.
This is very different from simply saying:
"Bread has the bigger timestamp, so Bread wins."
Vector clocks allow the system to detect the conflict instead of incorrectly assuming an ordering.
The application can then decide what to do:
Americano
+
Bread
↓
Conflict resolution10. Simple real-world analogy
Think of three people maintaining notebooks:
Ramesh Suresh MaheshEach person records:
"How many actions have I seen from each person?"
Ramesh's notebook:
[Ramesh=3, Suresh=2, Mahesh=1]This means Ramesh knows:
3 Ramesh events
2 Suresh events
1 Mahesh event
The vector isn't saying:
"It is 10:32 AM."
Instead, it's saying:
"Here is what I know about the history of events."
That's the key idea.
11. Vector Clock Rules
Remember these 3 rules for interviews.
Rule 1 — Local event
When a node performs an event:
increment your own counterExample:
[1,2,0] → [1,3,0]for Node B.
Rule 2 — Send message
Attach your vector to the message.
A = [2,1,0]
message
↓
B receives [2,1,0]Rule 3 — Receive message
Receiver takes the maximum of each position, then increments its own counter.
Example:
B current = [1,3,0]
message = [2,1,4]Take maximum:
[2,3,4]Then B increments its own position:
[2,4,4]12. The Most Important Comparison Rule
Given:
X = [x1,x2,x3]
Y = [y1,y2,y3]X happened before Y if:
x1 <= y1
x2 <= y2
x3 <= y3and at least one is strictly smaller.
Then:
X → YConcurrent if:
Some values are greater and some are smaller.
Example:
X = [2,4,1]
Y = [3,2,5]Because:
2 < 3
4 > 2
1 < 5Neither dominates the other.
Therefore:
X || YThey are concurrent.
🧠 One-line memory trick
Vector Clock = "Who knows how many events from whom?"
And remember:
NTP asks: "What time is it?"
Vector Clock asks: "What happened before what?"
That's why vector clocks are useful when exact physical time is unreliable but event ordering/causality matters.
------------------
NTP ***
NTP — Class Notes
1. What is NTP?
NTP = Network Time Protocol
NTP is the most widely used mechanism for synchronizing clocks across computers in a distributed system.
The basic problem is:
Every server has its own clock, but we want all servers to have a reasonably consistent idea of the current time.
NTP doesn't provide perfectly identical time. Instead, it tries to get clocks close enough by measuring and compensating for network delay.
2. Why Can't We Simply Ask a Time Server?
Suppose we have a highly accurate clock somewhere on the Internet:
Very Accurate Clock
|
↓
Time Server
|
Internet
|
↓
ClientThe clock itself might be extremely accurate.
But there is a problem:
Network latency
Suppose the client asks:
"What time is it?"
The request takes 40 ms to reach the server.
The response takes 60 ms to come back.
The client receives the answer 100 ms after sending the request.
If the client simply uses the returned timestamp, it could be approximately 60 ms behind the actual time.
So:
Accurate clock + inaccurate network = inaccurate result
NTP solves this by measuring the network delay and compensating for it.
3. NTP Uses Stratum
NTP organizes time servers into layers called strata.
Stratum 0
┌──────────────────────┐
│ Atomic Clock / GPS │
└──────────┬───────────┘
↓
Stratum 1
┌───────────────┐
│ Time Server │
└───────┬───────┘
↓
Stratum 2
┌───────────────┐
│ Time Server │
└───────┬───────┘
↓
Stratum 3
┌───────────────┐
│ Time Server │
└───────┬───────┘
↓
...Stratum 0
These are the actual highly accurate time sources:
Atomic clocks
GPS receivers
Other precision reference clocks
Technically, Stratum 0 is the reference clock itself, not a normal network server.
4. Stratum 1
A Stratum 1 server is directly connected to a Stratum 0 reference clock.
Atomic Clock
↓
Stratum 1 ServerThe server can obtain extremely accurate time from the reference clock.
There may still be tiny amounts of error due to:
Hardware
Device drivers
Interrupts
I/O
Operating-system scheduling
But the error is very small compared with Internet network latency.
5. Stratum 2 and Beyond
A Stratum 2 server synchronizes with a Stratum 1 server:
Stratum 0
↓
Stratum 1
↓
Stratum 2
↓
Stratum 3
↓
Stratum 4And so on.
The further you move away from the reference clock, the more uncertainty can accumulate.
The important practical point from the lecture is:
NTP normally aims for roughly ±10 ms accuracy over ordinary network paths.
NTP supports a maximum stratum depth of 15 for normal synchronization; Stratum 16 represents an unsynchronized/unusable state.
6. Why Do We Need Multiple Strata?
Imagine millions of computers all connecting directly to one Stratum 1 server:
Stratum 1
|
┌───────────┼───────────┐
↓ ↓ ↓
Client Client Client
↓ ↓ ↓
Client Client ClientThe server would have to handle enormous traffic.
Instead, we create a hierarchy:
Stratum 1
/ \
↓ ↓
Stratum 2 Stratum 2
/ \ / \
↓ ↓ ↓ ↓
S3 S3 S3 S3This creates fan-out and distributes the load.
In practice, many systems use relatively low stratum numbers such as 2, 3, 4, etc.; reaching the maximum depth is uncommon.
7. NTP Protocol Details
Some useful facts:
| Property | NTP |
|---|---|
| Protocol | UDP |
| Port | 123 |
| Timestamp | 64 bits |
| Seconds portion | 32 bits |
| Fraction portion | 32 bits |
So an NTP timestamp contains:
┌───────────────────────┬───────────────────────┐
│ 32 bits │ 32 bits │
│ Seconds │ Fractional seconds │
└───────────────────────┴───────────────────────┘This gives very high timestamp resolution.
However, the 32-bit seconds field creates a 2036 rollover issue for the traditional NTP era representation.
8. The Four Important Timestamps
This is the most important technical part.
NTP uses four timestamps:
T0 = Client sends request
T1 = Server receives request
T2 = Server sends response
T3 = Client receives responseThink of the communication like this:
CLIENT SERVER
T0
|
| -------- Request -------->
| T1
| |
| |
| T2
| <-------- Response --------
|
T3Let's understand each one.
T0 — Client transmit time
The client records:
"I am sending the request now."
T1 — Server receive time
The server records:
"I received the request now."
T2 — Server transmit time
The server records:
"I am sending the response now."
T3 — Client receive time
The client records:
"I received the response now."
9. Round-Trip Time
The total time between sending the request and receiving the response is:
T3 − T0
But this includes:
Client → Server network delay
+
Server processing time
+
Server → Client network delayThe server processing time is:
T2 − T1
Therefore, estimated network round-trip delay is:
(T3 − T0) − (T2 − T1)
This is a very important NTP calculation.
10. Clock Offset
NTP also estimates the difference between the client's clock and the server's clock.
The standard offset calculation is:
Offset = [(T1 − T0) + (T2 − T3)] / 2
You don't necessarily need to memorize the derivation immediately.
The intuition is more important:
NTP uses the four timestamps to estimate both network delay and clock difference.
Then the client can adjust its clock.
11. Simple Example
Suppose:
T0 = 10:00:00.000
T1 = 10:00:00.050
T2 = 10:00:00.052
T3 = 10:00:00.102Step 1 — Total round trip
T3 - T0
= 102 msStep 2 — Server processing time
T2 - T1
= 2 msStep 3 — Estimated network time
102 ms - 2 ms
= 100 msSo approximately 100 ms was spent traveling through the network.
Assuming roughly symmetric network delay, NTP can estimate the clock offset from the four timestamps.
12. Why Does NTP Need Repeated Measurements?
One request isn't enough.
Network latency constantly changes:
Request 1 → 80 ms
Request 2 → 120 ms
Request 3 → 95 ms
Request 4 → 200 ms
Request 5 → 90 msTherefore NTP continually takes measurements.
It then uses filtering/statistical techniques to determine:
"What is the most reasonable estimate of the current network delay and clock offset?"
This makes NTP adaptive.
13. The Key Idea
The most important thing to understand is:
Accurate Clock
↓
Time Server
↓
Network introduces delay
↓
NTP measures it
↓
NTP compensates for delay
↓
Client gets estimated timeSo NTP isn't magically making the network instantaneous.
It is essentially saying:
"I know the answer arrived late. Let me estimate how late it was and compensate for that."
14. NTP vs Vector Clock
This connects directly to your previous class.
| NTP | Vector Clock |
|---|---|
| Physical time | Logical time |
| Answers approximately "When?" | Answers "What happened before what?" |
| Synchronizes clocks | Tracks causal ordering |
| Uses network time servers | Uses counters/vectors |
| Subject to clock/network uncertainty | Doesn't require synchronized clocks |
| Useful for timestamps | Useful for distributed causality/conflicts |
🧠 Easy memory trick
NTP = approximate real-world time
Vector Clock = logical ordering of events
And the big distributed-systems lesson is:
There is no perfect "NOW" in a distributed system. NTP gives us a good approximation; vector clocks avoid needing exact physical time when all we really need is event ordering.
VECTOR CLOCKS:
Vector Clocks — Class Notes
1. First: What Is a Vector Clock?
The most important point from the lecture:
A vector clock does NOT tell us the actual time.
Despite the word “clock”, it is not a physical clock.
NTP gives us an approximate idea of when something happened. Vector clocks are used for a different purpose:
To prove the sequence/ordering of operations in a distributed system.
Think:
NTP → "When did it happen?"
Vector Clock → "Which operation happened before which?"2. The Problem Vector Clock Solves
Imagine a distributed database containing one value:
Coffee Meeting Day = ?Several users can modify this value concurrently.
The four actors are:
Alice
Bob
Kathy
DaveEach actor has:
A unique ID
A local counter
Every time that actor modifies the value, it increments its own counter.
For example:
Alice → A
Bob → B
Kathy → C
Dave → DThe important thing is that each actor maintains its own counter locally. It is not a distributed counter.
3. Alice Makes the First Change
Suppose Alice says:
WednesdayHer first write is:
Wednesday [A1]The A1 means:
A = Alice
1 = Alice's first modificationSo the vector is essentially:
[A:1]The vector travels along with the value. You cannot just send:
WednesdayYou send:
Wednesday + vector informationThis lets the system understand the history of the value.
4. Network Partition Happens
Now imagine the network gets partitioned.
Network Partition
Alice ─── Kathy Bob ─── DaveAlice and Kathy cannot communicate with Bob and Dave.
This is a very realistic distributed-systems situation.
5. Bob Changes the Value
Bob doesn't agree with Wednesday.
He says:
TuesdayBob knows about Alice's previous write:
A1So Bob creates:
Tuesday [A1 B1]Meaning:
A1 → Alice's first write
B1 → Bob's first writeBob's own counter is B1.
Importantly, Bob does not need to maintain Alice's counter himself. Each actor is responsible for its own counter.
6. Dave Agrees With Bob
Dave receives Bob's version:
Tuesday [A1 B1]Dave says:
Yes, Tuesday.
Dave performs his own write.
So:
Tuesday [A1 B1 D1]Now the history says:
Alice → Bob → Davebecause each subsequent operation contains the previous operation's vector information.
7. Kathy Doesn't Know About Bob or Dave
Meanwhile, Kathy is still on Alice's side of the partition.
She only knows:
Alice → Wednesday [A1]But Kathy decides:
ThursdayHer write becomes:
Thursday [A1 C1]Notice something very important:
Kathy doesn't know about Bob or Dave.
So her vector contains:
A1
C1but not:
B1
D18. Partition Heals — Conflict!
Now Dave receives Kathy's version:
Thursday [A1 C1]But Dave currently has:
Tuesday [A1 B1 D1]So we have:
Version 1:
Tuesday [A1 B1 D1]
Version 2:
Thursday [A1 C1]Which one is newer?
This is where vector clocks become powerful.
9. How Do We Detect a Conflict?
To determine whether one version is a descendant of another, we need to check whether the vector contains all the information from the earlier version.
Compare:
Tuesday → [A1 B1 D1]
Thursday → [A1 C1]Look at Tuesday:
A1
B1
D1Look at Thursday:
A1
C1Thursday does not contain:
B1
D1Therefore:
Thursday cannot be a descendant of Tuesday.
Likewise, Tuesday doesn't contain Kathy's C1.
Therefore:
Neither version is a descendant of the other.
They are concurrent/conflicting versions.
10. This Is the Heart of Vector Clocks
Remember this rule:
If Vector B contains everything represented by Vector A, plus something newer:
A → BB is a descendant of A.
But if:
A has something B doesn't have
AND
B has something A doesn't havethen:
A || BThey are concurrent.
Example
A = [1,2,0]
B = [1,3,0]B contains everything from A and has a newer B component.
Therefore:
A → BBut:
A = [1,2,0]
B = [1,0,3]Neither contains the other's complete history.
Therefore:
A || BConflict.
11. What Happens When There Is a Conflict?
This is a very important point from the class.
The database doesn't magically know which business decision is correct.
It tells the application:
"You have two different histories. Resolve the conflict."
In the example:
Tuesday [A1 B1 D1]
Thursday [A1 C1]Dave decides:
Thursdayand creates another version:
Thursday [A1 B1 C1 D2]Dave's counter is now D2 because this is his second write.
12. Why Does the Application Need to Resolve It?
This is one of the biggest disadvantages of vector clocks.
Suppose the value wasn't:
Tuesday / Thursdaybut a customer record:
Customer
---------
Name
Phone
Address
EmailYou might have:
Version A:
Phone changed
Version B:
Address changedInstead of throwing one version away, the application might be able to merge:
Phone ← Version A
Address ← Version BBut the database cannot necessarily know the business semantics.
Therefore:
Conflict resolution is pushed toward the application/client.
13. An Interesting Case: No Conflict
Later, Alice sees two versions:
Tuesday [A1 B1 D1]
Thursday [A1 B1 C1 D2]Now look carefully.
The Thursday vector contains the history from Tuesday:
A1
B1
D1plus:
C1
D2So Thursday is a descendant of Tuesday.
Therefore the database can automatically determine:
Tuesday
↓
ThursdayThere is no need to ask the application to resolve a conflict.
The newer history wins because it contains the complete history of the older version.
14. Visualizing the Whole Example
Alice
|
Wednesday [A1]
|
─── Network Partition ───
/ \
Kathy Bob
| |
Thursday [A1 C1] Tuesday [A1 B1]
|
Dave
|
Tuesday [A1 B1 D1]
Partition heals
|
↓
┌──────────────────────────────┐
│ │
Tuesday [A1 B1 D1] Thursday [A1 C1]
│ │
└────── CONFLICT ──────────────┘
|
↓
Application resolves
|
↓
Thursday [A1 B1 C1 D2]15. Vector Clock vs Last Write Wins
This is the important trade-off discussed in the lecture.
Last Write Wins (LWW)
LWW uses timestamps:
Value A → 10:05:01
Value B → 10:05:03
Winner → BSimple.
But clock differences can cause the system to get the ordering wrong.
Vector Clock
Vector clocks don't depend on physical timestamps to establish causal ordering.
They can determine:
A happened before Bor:
A and B happened concurrentlySo vector clocks cannot get causal sequence wrong, assuming the implementation is correct.
16. But Vector Clocks Have a Cost
The downside is:
Complexity moves into the client/application.
The database can tell you:
CONFLICT!But your application has to decide:
Which value should win?
OR
Can the two values be merged?And application-level conflict-resolution code can itself contain bugs.
So:
Vector Clock
↓
Correct causal ordering
↓
Detect conflicts
↓
Application must resolve conflict17. When Should You Care About Vector Clocks?
The lecture gives a useful practical guideline:
Ask yourself:
"Is time a first-class object in my domain?"
For example, if you're building a system where exact ordering is critical, vector clocks may be valuable.
If users are simply clicking buttons and two operations happen very close together, a tiny timestamp-ordering error may not matter much.
So don't automatically choose vector clocks just because they are theoretically more correct.
⭐ Interview Summary
What is a vector clock?
A mechanism for tracking causal ordering of events in a distributed system, not a mechanism for measuring physical time.
Why do we need it?
Because multiple nodes can modify the same data concurrently and there is no perfectly synchronized global clock.
What does each node maintain?
A local counter identified by that node's unique ID.
What does the vector represent?
The known history/sequence of modifications.
How do we detect a conflict?
If neither vector contains the complete history represented by the other, the versions are concurrent.
Who resolves the conflict?
Usually the application/client, based on business semantics.
Biggest advantage?
Causal sequence cannot be incorrectly determined when the vector-clock implementation is correct.
Biggest disadvantage?
Conflict-resolution complexity is pushed into the application.
🧠 The 3 things to remember
VECTOR CLOCK
1. NOT actual time
↓
2. Tracks causality/history
↓
3. Detects concurrent conflictsAnd the simplest mental model is:
"A vector clock is a distributed history tracker."
It answers:
“What changes does this version know about, and is this version a descendant of another version?”
No comments:
Post a Comment