Ramesh Style | Distributed Systems | Interview Preparation
The main idea in this lesson is to understand what a database transaction means first, then see why the same idea becomes difficult when the work is spread across multiple distributed actors/services.
1. What is a Transaction?
A transaction is a bundle of database changes/statements that should be treated as one unit.
Example:
BEGIN TRANSACTION
1. Update Customer A's drink
2. Update Customer B's drink
3. Update Customer C's drink
COMMITThe important question is:
Should all these changes happen together, or should none happen?
This leads to ACID.
2. ACID Properties
TRANSACTION
|
┌──────────┼──────────┐
↓ ↓ ↓
ACID ACID ACID
↓ ↓ ↓
A C I D
Atomicity Consistency Isolation DurabilityRemember:
A transaction should complete as one reliable unit.
3. A — Atomicity
Meaning
Everything succeeds, or nothing succeeds.
Suppose we have:
Transaction
1. Update Ramesh's drink
2. Update Sudha's drink
3. Update Yagna's drinkIf step 3 fails:
1 → SUCCESS
2 → SUCCESS
3 → FAILUREAtomicity says:
ROLLBACK
1 → undone
2 → undone
3 → undoneFinal result:
NONE of the changes are appliedMemory trick
Atomic = All or Nothing
ALL ✅
OR
NOTHING ❌4. C — Consistency
⚠️ Very important interview point
The word Consistency here is different from Consistency in CAP theorem.
ACID Consistency
Means:
After the transaction completes, the database remains in a valid state according to its rules/constraints.
Example:
Before transaction
Account A = ₹1000
Account B = ₹500Transfer ₹200:
A → -₹200
B → +₹200After transaction:
A = ₹800
B = ₹700The database remains valid.
Simple memory
ACID C = Don't leave the database in a broken/invalid state.
5. ACID Consistency vs CAP Consistency
🔥 Interview trap.
ACID C
Transaction
↓
Database rules/constraints
↓
Valid database stateCAP C
Distributed nodes
↓
Read
↓
Return latest valueSo:
| ACID Consistency | CAP Consistency |
|---|---|
| Transaction leaves DB valid | Read sees latest value |
| Database integrity | Distributed-system consistency |
| Transaction concept | Distributed-system concept |
Interview answer
"ACID consistency and CAP consistency use the same word but mean different things. ACID consistency is about maintaining database validity and constraints, whereas CAP consistency is about whether reads observe the latest write across distributed nodes."
6. I — Isolation
Simple meaning
One transaction should not improperly see another transaction's intermediate changes.
Example:
Transaction 1
|
| Update coffee
↓
"Espresso"At the same time:
Transaction 2
|
↓
Read coffeeThe basic idea of isolation is that Transaction 2 shouldn't simply observe Transaction 1's unfinished/intermediate state.
The source gives this as the simple first approximation of isolation.
7. But Isolation Is NOT That Simple
🔥 This is an important point from the lesson.
A common simplified explanation is:
"One transaction cannot see another transaction until it completes."
That's useful as a first approximation, but real relational databases normally provide multiple isolation levels.
Conceptually:
Isolation
|
├── Different isolation levels
|
├── Different guarantees
|
└── Different performance costsHigher isolation:
More protection
↓
More coordination
↓
Potentially worse performanceTherefore:
Strict isolation is often too expensive, so real systems choose an appropriate isolation level.
8. D — Durability
Meaning
Once a transaction commits, the database remembers it.
Example:
BEGIN
↓
UPDATE
↓
COMMIT
↓
Database remembers changeLater:
READ
↓
Committed value is still thereEven if the application restarts, the committed transaction should not simply disappear.
Memory trick
Durable = Committed means remembered.
9. ACID Summary
| Letter | Meaning | Easy Memory |
|---|---|---|
| A | Atomicity | All or nothing |
| C | Consistency | DB remains valid |
| I | Isolation | Transactions don't improperly interfere |
| D | Durability | Committed data is remembered |
A → ALL OR NOTHING
C → VALID DATABASE STATE
I → ISOLATED TRANSACTIONS
D → DATA REMEMBERED10. Now the Real Problem — Distributed Transactions
A normal transaction might happen inside one database:
Application
|
↓
Single Database
|
└── TransactionThe database can control the whole operation.
But imagine:
Application
|
┌────────┼────────┐
↓ ↓ ↓
Service A Service B Service C
↓ ↓ ↓
DB-A DB-B DB-CNow one business operation may involve:
DB-A
DB-B
DB-C🔥 Problem:
How do we make all three databases behave as one transaction?
11. Coffee Shop Example ☕
This lesson uses a coffee shop to explain distributed transactions.
A customer orders:
CappuccinoThe process can be viewed as multiple steps.
12. Step 1 — Place the Order
Customer
|
↓
Coffee Shop
|
↓
Order receivedExample:
Order ID = 101
Drink = Cappuccino13. Step 2 — Process Payment
Customer
|
↓
Payment
|
↓
Payment SystemPayment could be:
Cash
Credit Card
Mobile AppPayment processing itself might involve another system.
For example:
Coffee Shop
|
↓
Payment Service
|
↓
Bank/Card NetworkThat means we have another potential distributed operation.
14. Step 3 — Queue the Order
Once payment succeeds:
Payment SUCCESS
|
↓
Order placed in queueThe lesson describes the coffee cup itself as a simple queue mechanism:
Counter
|
↓
[CAPPUCCINO]
[ESPRESSO]
[LATTE]The order is marked on the cup and waits for the barista.
15. Step 4 — Barista Processes the Order
The barista takes the order from the queue:
Queue
|
↓
[CAPPUCCINO]
|
↓
Barista
|
↓
Make Coffee16. Step 5 — Deliver the Coffee
After preparation:
Barista
|
↓
Reads identifier/name
|
↓
"Ramesh, your Cappuccino!"
|
↓
CustomerThe name/order identifier acts as a correlation identifier connecting the order to the customer.
17. Complete Coffee-Shop Flow
This is the most important diagram to remember.
CUSTOMER
|
↓
1. Place Order
|
↓
2. Process Payment
|
↓
Payment Service
|
↓
3. Queue Order
|
↓
ORDER QUEUE
|
↓
4. Barista Takes
|
↓
Make Coffee
|
↓
5. Deliver Coffee
|
↓
CUSTOMERThe lesson describes this as involving multiple steps and potentially multiple actors, with asynchronous processing between order-taking and coffee preparation/delivery.
18. Why Is This a Distributed Transaction?
Imagine each step is handled by a different system:
Order Service
|
↓
Payment Service
|
↓
Queue
|
↓
Coffee Service
|
↓
NotificationNow ask:
What if payment succeeds but coffee preparation fails?
For example:
Order → SUCCESS
Payment → SUCCESS
Queue → SUCCESS
Coffee → FAILUREWe cannot simply say:
ROLLBACK EVERYTHINGbecause these are potentially different systems, possibly with different databases and independent processes.
That's the core difficulty of distributed transactions.
19. Failure Scenario
Imagine:
Customer
|
↓
Order Service ✅
|
↓
Payment Service ✅
|
↓
Queue ✅
|
↓
Coffee Service ❌Now what should happen?
Should we:
Refund payment?Should we:
Cancel order?Should we:
Retry coffee preparation?Should we:
Compensate the previous operations?This is why distributed transactions become much harder than ordinary database transactions.
20. Synchronous vs Asynchronous
The lesson also emphasizes that the coffee-shop process involves asynchronous actors.
Synchronous
A → B
↓
Wait
↓
ResponseAsynchronous
A
|
↓
Message / Order
|
↓
Queue
|
↓
B processes laterCoffee-shop example:
Cashier
|
↓
writes order on cup
|
↓
Queue
|
↓
Barista processes laterThe cashier doesn't have to stand there waiting while the coffee is prepared.
That's asynchronous processing.
21. Correlation Identifier
The lesson mentions the name written on the coffee cup as a correlation identifier.
Example:
Order ID = 12345
Customer = RameshThe identifier travels with the work:
Order Service
|
| Order ID 12345
↓
Queue
|
| Order ID 12345
↓
Barista Service
|
↓
NotificationThe system can determine:
Which result belongs to which request?
Interview definition
A correlation ID is an identifier used to associate messages/events belonging to the same business operation across asynchronous services.
22. ACID Transaction vs Distributed Transaction
Traditional transaction
Application
|
↓
Database
|
↓
BEGIN
|
├── Update A
├── Update B
└── Update C
|
COMMITDatabase controls the whole transaction.
Distributed transaction
Business Operation
|
┌────────────┼────────────┐
↓ ↓ ↓
Service A Service B Service C
↓ ↓ ↓
DB-A DB-B DB-CNow:
Who controls COMMIT?That's the difficult question.
23. Why Distributed Transactions Are Difficult
🔥 Remember these five problems:
1. Network failure
2. Partial failure
3. Different databases
4. Asynchronous processing
5. Rollback across independent systemsExample:
DB-A → SUCCESS
DB-B → SUCCESS
DB-C → FAILUREYou now have a partial success.
In a single database:
ROLLBACKmay solve it.
Across distributed services:
ROLLBACK DB-A
ROLLBACK DB-B
ROLLBACK DB-Cis much harder because the systems may be independent and communication itself can fail.
24. Connection to CAP Theorem
This is a very important connection with your previous class.
Distributed Transaction
|
↓
Multiple nodes/services
|
↓
Network can fail
|
↓
CAP trade-offs
|
↓
Consistency vs AvailabilitySo distributed transactions aren't just a database problem.
They are fundamentally connected to:
Network failures
Partial failures
Coordination
Consistency
Availability
Messaging
🎯 Ramesh Interview Questions
Q1. What is a transaction?
A transaction is a group of database operations treated as one logical unit.
Q2. Explain ACID.
A → All or nothing
C → Valid database state
I → Transaction isolation
D → Committed data is durableQ3. What is atomicity?
Either the complete transaction succeeds or none of its changes are applied.
Q4. What is ACID consistency?
The transaction must leave the database in a valid state according to its rules and constraints.
Q5. Is ACID consistency the same as CAP consistency?
No. ACID consistency concerns database validity after a transaction, while CAP consistency concerns visibility of the latest value across distributed nodes.
Q6. Why are distributed transactions difficult?
Because one business operation may span multiple independent services/databases, and failures can occur after some operations succeed but before others complete.
Q7. What is asynchronous processing?
A producer submits work and does not need to wait for the consumer to finish immediately.
Q8. What is a correlation ID?
An identifier used to track and correlate messages belonging to the same business operation across distributed services.
🧠 Final Ramesh Memory Map
TRANSACTION
|
ACID
|
┌─────────────┼─────────────┐
↓ ↓ ↓
Atomicity Consistency Isolation
All/Nothing Valid DB No improper
interference
|
↓
Durability
Committed = Saved
DISTRIBUTED TRANSACTION
|
↓
Multiple Services
|
┌─────────────┼─────────────┐
↓ ↓ ↓
Order Payment Coffee
Service Service Service
| | |
DB-A DB-B DB-C
\ | /
\ | /
Network
|
↓
Partial failures
|
↓
Hard to achieve one
atomic transaction
|
↓
Distributed Systems⭐ One sentence to remember
A normal ACID transaction coordinates changes inside one transactional boundary; a distributed transaction tries to coordinate one business operation across multiple independent systems, where network failures and partial successes make atomicity much harder.
-----------------------
📚 Distributed Transactions — Why Split the Work?
Ramesh Style Class Notes | Interview Preparation
The key idea in this section is:
We split a business process into multiple workers/services to improve throughput and specialization, but once we split the work, failures and coordination become much harder.
1. Why Do We Split the Work?
Consider a coffee shop with only one worker.
Customer
↓
Take Order
↓
Process Payment
↓
Make Coffee
↓
Deliver Coffee
↓
Next CustomerOne person has to do everything sequentially.
That creates a bottleneck.
Split the work
COFFEE SHOP
|
┌─────────┴─────────┐
↓ ↓
Cashier Barista
| |
Take Order Make Coffee
PaymentNow two people can work simultaneously.
2. Benefit #1 — Parallelism
Instead of:
Worker
|
├── Order
├── Payment
├── Coffee
└── Deliverywe have:
Cashier Barista
| |
Take Order Make Coffee
| |
Payment DeliverThey can work at the same time.
Result
Parallel work
↓
More work completed
↓
Higher throughputInterview phrase
Splitting work allows independent tasks to execute concurrently, increasing system throughput.
3. Benefit #2 — Specialization
The cashier specializes in:
Order
Payment
Customer interactionThe barista specializes in:
Coffee preparationInstead of requiring every worker to know everything:
Worker
↓
Order + Payment + Coffeewe have:
Cashier → Order/Payment
Barista → CoffeeThis is similar to microservices:
Order Service
|
↓
Payment Service
|
↓
Coffee/Fulfillment ServiceEach component can specialize in its responsibility.
4. Benefit #3 — Uneven Workloads
🔥 This is an important distributed-systems reason for splitting work.
Different tasks don't necessarily take the same amount of time.
Example:
Taking order + payment
↓
20 seconds
Making complicated coffee
↓
2 minutesIf one person does both:
Customer 1
↓
Order
↓
Coffee
↓
Customer 2 waitsThe slow operation blocks everything.
Instead:
Cashier
↓
Orders continuously
↓
Queue
↓
Barista
↓
Makes coffeeThe queue absorbs the difference in workload.
5. Queue = Buffer Between Workers
This is a very important distributed-systems concept.
Cashier
|
| produces orders
↓
┌─────────────────┐
│ QUEUE │
│ Order 1 │
│ Order 2 │
│ Order 3 │
└─────────────────┘
|
| consumes orders
↓
BaristaThe cashier doesn't have to wait for the barista.
This gives us:
Producer
↓
Queue
↓
ConsumerThis is the basic model behind many messaging systems.
6. Why Not Keep Everything Together?
Because workloads may be different.
Suppose:
Payment workload = 1000 requests/minute
Coffee preparation = 200 requests/minuteWe can scale independently:
Payment
↓
10 workers
Coffee
↓
3 workersInstead of:
One giant workerThis is the distributed-systems principle:
Split components according to workload and responsibility, then scale each independently.
7. But Splitting Creates a New Problem
🔥 This is the central lesson.
Before splitting:
ONE PROCESS
|
└── EverythingIf something fails:
Transaction
↓
ROLLBACKAfter splitting:
Service A
↓
Service B
↓
Service CNow:
A → SUCCESS
B → SUCCESS
C → FAILUREWe have partial failure.
The system cannot simply pretend that nothing happened.
8. What Can Go Wrong?
The source identifies several failure scenarios.
8.1 Payment Failure
Customer
↓
Order
↓
Payment ❌Example:
Credit card rejected
Fraud detected
Card expired
Payment service unavailableThe transaction cannot continue normally.
9. Insufficient Resources
The service may not have the resources required to complete the work.
Example:
Customer orders:
Americano
↓
Coffee beans unavailable ❌Or:
Cappuccino
↓
Milk unavailable ❌Distributed equivalent:
Service
↓
Required resource
↓
Unavailable10. Equipment Failure
Example:
Coffee machine
↓
FAILURE ❌The order may already have:
Payment → SUCCESS
Order → SUCCESS
Queue → SUCCESSBut fulfillment cannot happen.
This creates partial completion.
11. Worker Failure
Suppose:
Order
↓
Queue
↓
BaristaThen:
Barista leaves / crashes
↓
No worker
↓
Order cannot be processedDistributed equivalent:
Consumer service
↓
CRASH
↓
Messages remain unprocessedThis is why distributed systems need:
retries
failover
monitoring
queues
recovery mechanisms
12. Consumer Failure
Even after everything succeeds:
Payment ✅
Order ✅
Coffee ✅the customer might disappear.
Example:
Customer ordered coffee
↓
Paid
↓
Coffee prepared
↓
Customer leavesNow the coffee has already been created.
You cannot magically undo the consumed resources.
This is a classic example of why real-world distributed processes don't behave like one database transaction.
13. If We Had One Big ACID Transaction...
Imagine:
BEGIN TRANSACTION
Order
Payment
Queue
Make Coffee
Delivery
COMMITIf coffee preparation fails:
ROLLBACKEverything disappears as though it never happened.
Conceptually:
Payment → undone
Order → undone
Queue → undoneThis would be convenient.
But real-world distributed systems generally don't work this way.
14. Three Responses to Failure
🔥 Very important class-note topic.
When we can't simply rollback everything, we have three broad strategies:
FAILURE
|
┌─────────┼─────────┐
↓ ↓ ↓
WRITE-OFF RETRY COMPENSATE15. Strategy 1 — Write-Off
Meaning
Accept the loss and discard the work.
Example:
Customer paid
↓
Coffee made
↓
Customer disappeared
↓
Coffee cannot be reused
↓
WRITE-OFFYou accept that the work/resources have been consumed.
Distributed example
Message processed
↓
Side effect occurred
↓
Consumer disappeared
↓
Cannot practically undo
↓
Write-off16. Strategy 2 — Retry
Meaning
Try the failed operation again.
Example:
Payment
↓
FAIL
↓
Retry
↓
SUCCESSUseful when the failure is temporary.
Examples:
Network timeout
Temporary payment gateway failure
Temporary equipment problem
Temporary service unavailableFlow
Operation
↓
Failure
↓
Retry
↓
Success?
/ \
YES NO
↓ ↓
Done Retry again /
compensate⚠️ In distributed systems, retries must be designed carefully because repeating an operation can accidentally create duplicate effects.
17. Strategy 3 — Compensating Action
This is one of the most important concepts.
Suppose:
Payment → SUCCESSThen:
Coffee preparation → FAILUREWe can't simply rollback the payment as if it never happened.
Instead:
Payment SUCCESS
↓
Coffee FAILURE
↓
Refund PaymentThe refund is a new transaction/action that compensates for the previous successful operation.
18. Rollback vs Compensation
🔥 Very important interview distinction.
Rollback
Transaction
↓
Failure
↓
ROLLBACK
↓
Pretend changes never happenedCompensation
Operation A → SUCCESS
Operation B → FAILURE
↓
New action
↓
Compensate AExample:
Charge ₹500
↓
Payment SUCCESS
↓
Booking FAILURE
↓
Refund ₹500The payment was not rolled back.
A new refund transaction compensated for it.
19. Three Strategies — Memory Table
| Strategy | Meaning | Example |
|---|---|---|
| Write-off | Accept the loss | Coffee discarded |
| Retry | Try operation again | Retry payment |
| Compensation | Perform another action to offset previous success | Refund payment |
Memory:
WRITE-OFF → Accept loss
RETRY → Try again
COMPENSATE → Correct using another action20. Why Starbucks Doesn't Use One Big Distributed Transaction
The key architectural insight:
One giant transaction
↓
Strong coordination
↓
Workers must wait
↓
Lower throughput
↓
More latencyInstead:
Order
↓
Payment
↓
Queue
↓
Barista
↓
DeliveryEach stage can operate independently.
Result:
More concurrency
↓
Higher throughput
↓
Better customer volume21. The Extreme Atomicity Example
Imagine a coffee shop with:
One barista per customer.
The barista:
1. Take order
2. Authorize card
3. Hold funds
4. Make coffee
5. Return to customer
6. Complete paymentFlow:
Customer
↓
Dedicated Barista
↓
Authorize payment
↓
Make coffee
↓
Return
↓
Charge cardThis approximates a distributed transaction.
But look at the problem:
1 customer
↓
1 dedicated worker
↓
Worker busy making coffee
↓
Cannot efficiently serve next customerThroughput falls dramatically.
22. Why Throughput Matters More
Suppose:
Design A — Strong coordination
1 worker/customer
↓
High coordination
↓
Low throughputDesign B — Distributed workflow
Cashiers ──→ Queue ──→ Baristas
↓ ↓
Parallel work Parallel workResult:
Higher throughput
Lower waiting time
More customers served🔥 The central insight is:
Distributed systems often sacrifice some transaction-style atomicity/coordination to achieve higher throughput and scalability.
23. Distributed Transaction vs Throughput
Think of it like this:
More Coordination
↑
|
|
|
↓
More Atomicity
↕
Less Coordination
↓
|
|
↓
Higher ThroughputNot every application wants maximum transactional guarantees.
Sometimes:
Business throughput is more valuable than global atomicity.
24. Two-Phase Commit Connection
The lesson connects this idea to Two-Phase Commit (2PC).
Conceptually:
Coordinator
|
┌───┼────┐
↓ ↓ ↓
DB1 DB2 DB3Phase 1 — Prepare
Coordinator
|
├──→ DB1: PREPARE?
├──→ DB2: PREPARE?
└──→ DB3: PREPARE?If everyone says:
YESthen:
Phase 2 — Commit
Coordinator
|
├──→ DB1: COMMIT
├──→ DB2: COMMIT
└──→ DB3: COMMITThis provides stronger transactional coordination.
But:
2PC
↓
More coordination
↓
More waiting
↓
Potential blocking
↓
Lower throughputThat's why systems designed for massive throughput often avoid wrapping an entire business workflow in one global transaction.
25. Important Nuance — Distributed Transactions Are Not Impossible
Don't say:
❌ "Distributed transactions cannot be implemented."
Better answer:
They can be implemented, but the coordination cost can be significant, so many distributed systems avoid global transactions and instead use retries, queues, idempotency, and compensating actions.
The source also notes that technologies such as ZooKeeper can be used to achieve stronger coordination, and Cassandra has Lightweight Transactions, but these are limited mechanisms rather than a universal "BEGIN → everything → COMMIT" transaction across an entire distributed system.
26. Cassandra Lightweight Transactions
Important distinction:
Cassandra Lightweight Transaction
≠
Global Distributed TransactionLWT can provide stronger transactional behavior for specific limited operations, rather than turning an entire distributed workflow into one large ACID transaction.
Think:
Limited atomic operation
↓
Usefulrather than:
Order
↓
Payment
↓
Inventory
↓
Shipping
↓
Notification
↓
One global COMMIT27. Complete Distributed Workflow
This is the diagram I recommend remembering for interviews:
CUSTOMER
|
↓
ORDER SERVICE
|
↓
PAYMENT SERVICE
|
↓
QUEUE
|
↓
FULFILLMENT
|
↓
NOTIFICATION
|
↓
CUSTOMERFailures can happen anywhere:
Order → ❌
Payment → ❌
Queue → ❌
Fulfillment → ❌
Consumer → ❌Therefore:
FAILURE
|
┌───────┼────────┐
↓ ↓ ↓
Write-off Retry Compensation🎯 Interview Questions
Q1. Why split work in distributed systems?
To achieve parallelism, specialization, independent scaling, and better throughput.
Q2. What problem does splitting introduce?
Partial failures and the need to coordinate multiple independent components.
Q3. What is a compensating transaction?
A new action that semantically reverses or compensates for a previously completed operation when a global rollback isn't possible.
Q4. Rollback vs compensation?
Rollback undoes changes within a transaction boundary; compensation performs a new operation to offset a previously completed distributed operation.
Q5. What are three common failure responses?
Write-off
Retry
Compensating actionQ6. Why avoid distributed transactions?
Global coordination can significantly reduce throughput and increase latency and operational complexity.
Q7. Are distributed transactions impossible?
No. They are possible, but their coordination cost can be high. Limited mechanisms such as Cassandra Lightweight Transactions can provide stronger guarantees for specific operations.
🧠 Ramesh Final Memory Map
WHY SPLIT WORK?
|
┌──────────────┼──────────────┐
↓ ↓ ↓
Parallelism Specialization Uneven Load
| | |
└──────────────┼──────────────┘
↓
HIGH THROUGHPUT
|
↓
BUT → PARTIAL FAILURE
|
┌───────────────┼───────────────┐
↓ ↓ ↓
Payment fails Worker fails Consumer fails
| | |
└───────────────┼───────────────┘
↓
NO GLOBAL
ROLLBACK
|
┌─────────┼─────────┐
↓ ↓ ↓
WRITE-OFF RETRY COMPENSATE
|
↓
New action
e.g. REFUND⭐ One-line interview answer
"We split distributed work to improve parallelism, specialization, independent scaling, and throughput. The trade-off is that once the workflow is split across independent services, a failure can occur after some steps have succeeded, so instead of relying on one global ACID rollback, we often use retries, write-offs, and compensating actions."
