Relaxed Reads in OpenRaft: Linearizable Reads Without Waiting for Apply to Catch Up
Article link: https://blog.openacid.com/distributed/raft-relaxed-read/

OpenRaft 0.10.1 will add relaxed reads. A relaxed read is a linearizable read on the leader that waits only until the current leader’s no-op log entry is applied, rather than until the state machine has applied everything up to the CommitIndex. The no-op is the first log entry a leader writes after it is elected, so once the leader has been running for a while, its no-op has long been applied. A relaxed read therefore almost never waits for apply, and its result is still linearizable.
When the state machine’s apply is delayed, relaxed reads clearly lower read latency. In OpenRaft’s benchmark, when each apply batch takes an extra 500 µs, the p50 latency of reads on the leader drops from 1.2–2.0 ms to 0.92–0.96 ms. When each apply batch takes an extra 2 ms, apply falls behind commit: standard reads reach a p50 latency of 2.3 s, while relaxed reads stay at 1.2 ms.
This article derives relaxed reads in 3 steps. Step 1 reads only on the leader and waits only for the no-op; as long as the leader does not change, such a read is linearizable. Step 2 looks at a leader that steps down in the middle of a read: a read that waits only for the no-op can then return an older value. Step 3 derives the fix from the counterexample in step 2: a follower applies only the log entries that the leader has already applied. The article then covers the TLA+ check, and how relaxed reads are used in OpenRaft, how they perform and what they cost.

Step 1: When reads stay on the leader, waiting for the no-op is enough
If reads run only on the leader, and the leader does not change while a read runs, then a read only needs to wait until the state machine has applied the current leader’s no-op, not until the state machine reaches the CommitIndex.
A standard ReadIndex read waits for the CommitIndex. When the leader receives a read, it first records the current CommitIndex as the ReadIndex. Next, it confirms through a heartbeat or a lease that it is still the leader. Finally, it waits until the state machine has applied up to the ReadIndex, and only then queries the state machine. OpenRaft uses max(noop, CommitIndex) as the ReadIndex; Openraft’s optimization of ReadIndex explains why. If the state machine’s apply is delayed, apply falls behind commit, and every read has to wait for apply to catch up with the CommitIndex.
But linearizability does not require a read to wait for the CommitIndex. Linearizability requires a read to see every client write that was acknowledged before the read started, but it does not require the read to see client writes that have not been acknowledged yet; see Linearizable Transactions in Distributed Systems. Below the CommitIndex there can be client writes that are committed but not yet acknowledged. Those client writes overlap the read in time, so the read is linearizable whether or not it sees them.
Therefore a relaxed read runs on the leader and uses the current leader’s no-op as its ReadIndex, instead of the CommitIndex. The leader still confirms its leadership through a heartbeat or a lease. As soon as the state machine has applied the no-op, the read queries the state machine.
As long as the leader does not change, a read that waits only for the no-op is linearizable:
- The log entries that earlier leaders committed all come before the current leader’s no-op. So once the state machine has applied the no-op, it has applied those log entries too, and the read is sure to see them.
- A client write that the current leader has acknowledged was already readable from the leader’s state machine before it was acknowledged, so the read is sure to see it as well.
- Earlier reads also queried the same leader’s state machine, and a state machine only moves forward, so a later read never sees an older state.
Point 2 rests on a premise: before a client write is acknowledged, its result must already be readable from the leader’s state machine. The application’s state machine guarantees this premise. For example, if a state machine makes the results of a batch of log entries visible only after the whole batch is applied, it must hold back the batch’s client-write responses until those results are visible.
But the argument above assumes that the leader does not change. After a read confirms leadership and before it queries the state machine, the leader may already have stepped down.
Step 2: The leader steps down mid-read, and a later read sees an older value
Waiting only for the no-op is not enough. If the leader steps down after a read has confirmed leadership, then a read that starts after the leader change can see an older value than an earlier read saw.
In the counterexample below, x starts at 0. Node A is the leader of term T, and A’s no-op is at log index 10. Time runs from top to bottom in the figure, and 10, 21 and 22 are log indexes:

- t1: A confirms leadership for the read
R_A. The ReadIndex ofR_Ais no-op 10, which is already applied, butR_Apauses before it queries the state machine. - t2: A steps down, and B becomes the leader of term
T+1. B commits and applies its own no-op 21. - t3: B commits log entry 22, which sets
xto 1. But B’s apply stops at 21, soxis still 0 in B’s state machine, and the client write of log entry 22 has not been acknowledged yet. - t4: B sends log entry 22 and CommitIndex 22 to A. Standard Raft does not require a follower to wait for the leader to apply first, so A, now a follower, applies log entry 22, and
xbecomes 1 in A’s state machine. - t5:
R_Aresumes on A, readsx = 1, and returns 1 to the client. - t6: Only after the client receives the result of
R_Adoes it start the readR_Bon B. B confirms leadership, and the ReadIndex ofR_Bis no-op 21, which is already applied. - t7: B has not applied log entry 22 yet, so
R_Breadsx = 0.
R_A returned its result before R_B started, so linearizability orders R_B after R_A: R_B must see at least the state that R_A saw. R_A saw x = 1, so R_B must not return 0. But R_B returned 0, and the later read saw an older value.
The root cause is that A, which became a follower when it stepped down, applied log entry 22 before the current leader B did, and R_A happened to answer from A’s state machine. Both R_A and R_B went to the leader at the time, so reading only on the leader does not keep the value from going back.
Standard ReadIndex does not let the value go back. With standard ReadIndex, the ReadIndex of R_B is CommitIndex 22, so R_B waits until B has applied log entry 22 before it queries the state machine, and R_B reads x = 1.
Step 3: A follower applies only the log entries the leader has applied
The counterexample in step 2 comes from a follower that applies a log entry before the current leader does. So the fix is to let a follower apply only the log entries that the leader has already applied.
The fix changes the apply limit that the leader sends to followers. In standard Raft, the leader sends its CommitIndex to followers in append entries and heartbeats, and a follower may apply up to that CommitIndex. With the fix, the leader instead sends the smaller of its own CommitIndex and AppliedIndex, min(CommitIndex, AppliedIndex). The leader also resends the limit each time it applies new log entries, so followers do not wait for the next heartbeat. Every node must follow this rule whenever it is the leader.
Back to the counterexample in step 2: at t4, B allows A to apply only up to 21, the index that B has applied, so A cannot apply log entry 22 before B does, and R_A can only read x = 0. B allows A to apply log entry 22 only after B itself has applied it. So if a read on A sees x = 1, B must already have applied log entry 22, and a later R_B on B is sure to see x = 1 too.
The fix works for any leader change. With the fix, every log entry that any node applies has already been applied by the leader of that time. Therefore, wherever an earlier read ran, each log entry it saw has either been applied by the current leader already, or belongs to an earlier term and comes before the current leader’s no-op. So a read on the current leader that waits only for the no-op is sure to see the state that earlier reads saw, and the conclusion of step 1 still holds after a leader change.
The fix does not affect Raft’s safety: commit is still decided by a quorum, and followers only apply a little later.
The original consensus-essence article also gives another fix: only the leader serves reads, and if the last log entry that a read sees has a newer term than the term in which the read confirmed leadership, the read discards its result and reads again.
Relaxed reads take effect on the leader that issues them
A relaxed read takes effect on the leader that issues the read. If an application lets the leader confirm leadership and then hands the read to a follower, the follower uses the standard ReadIndex even when the application requested a relaxed read: the follower waits until its own state machine has applied up to max(noop, CommitIndex). The reason is that a follower’s apply lags behind the leader’s, so the follower may not have applied a client write that the leader has already acknowledged. If the follower waited only for the no-op, a read on the follower could miss that acknowledged client write. The OpenRaft documentation describes the API for reads on followers.
Checking with TLA+: all 3 steps hold within the checked bounds
The conclusions of all 3 steps are checked with the TLA+ model RelaxedReadApplied.tla. The checked bounds are 3 nodes, up to 3 terms, up to 4 log entries and 2 reads. Within these bounds, TLC explored every state and found no execution that violates linearizability.
Control experiments show that each rule below is necessary: when any one of them is removed, TLC finds an execution that violates linearizability.
- The fix in step 3: if the leader sends its CommitIndex to followers, TLC finds exactly the counterexample of step 2.
- Every node follows the fix of step 3: if only some nodes follow the fix when they lead, TLC also finds a violation.
- The premise of step 1: if a client write is acknowledged after it commits but before its result is readable from the state machine, TLC also finds a violation.
TLC’s results cover the bounds above: this is bounded model checking, not a proof for clusters of any size. The TLA+ verification report lists every case and its bounds.
Using relaxed reads in OpenRaft
To enable relaxed reads, first change the config of every node, and then change the application’s reads.
First, set Config::enable_relaxed_read to Some(true) on every node and restart the nodes. The fix in step 3 requires every node to limit follower apply whenever it leads, so the application starts requesting relaxed reads once every node has this setting enabled.
Then the application requests a relaxed read with LinearizerOption::with_relaxed(true):
// Confirm leadership with a lease, and request a relaxed read
let option = LinearizerOption::from(ReadPolicy::LeaseRead).with_relaxed(true);
// On return, the local state machine has applied at least the current leader's no-op
raft.ensure_linearizable(option).await?;
// state_machine is the application's own state machine
let value = state_machine.read().await.get("key").cloned();
On a node without enable_relaxed_read, a relaxed read request automatically uses the standard ReadIndex max(noop, CommitIndex), and the read is still linearizable. In addition, the application’s state machine must meet the premise of step 1: before a client write is acknowledged, its result is already readable from the state machine.
Performance: clear gains when apply is delayed
A relaxed read skips the wait for apply to catch up with the CommitIndex, so when apply is delayed, relaxed reads clearly lower read latency.
OpenRaft’s benchmark runs 3 voters in 1 process, with the log and the state machine in memory. In total, 64 reader clients send 90,000 reads per second, and 32 writer clients send 10,000 client writes per second. All reads run on the leader and confirm leadership with a lease. To simulate a delayed apply, the benchmark makes each apply batch spend extra time before it reads its log entries; this article calls that extra time the apply delay. Each number in the table below is the median of 3 runs:
| Apply delay | Standard read p50 / p99 | Relaxed read p50 / p99 |
|---|---|---|
| 0 | 1.25 ms / 2.51 ms | 1.25 ms / 2.44 ms |
| 500 µs | 2.01 ms / 6.04 ms | 0.92 ms / 2.06 ms |
| 2 ms | 2.32 s / 3.16 s | 1.21 ms / 3.32 ms |
- With a 500 µs apply delay, the p50 of relaxed reads drops below half that of standard reads. Relaxed reads are also more stable: in 2 other test groups, standard reads had a p50 of 1.21 ms and 1.24 ms, while relaxed reads stayed at 0.92–0.96 ms in all 3 groups.
- With a 2 ms apply delay, apply cannot keep up with commit, and the number of unapplied log entries keeps growing for the whole run. A standard read waits until those log entries are applied, so its p50 reaches 2.32 s; a relaxed read waits only for the no-op, so its p50 stays at 1.21 ms.
- The apply delay can also happen while the state machine holds its write lock, and then a read also waits for that lock. In that case relaxed reads still lower the p50: from 2.11 ms to 1.39 ms with a 500 µs apply delay, and from 2.55 s to 2.08 s with a 2 ms apply delay.
- With no apply delay, both kinds of reads have the same latency. Read latency is measured from the time each read is scheduled to start and includes the time the read waits in the client, so the p50 is about 1.2 ms even with no apply delay.
relaxed-reads-results.md has all the data and the test method.
Costs and limitations
The main cost of relaxed reads comes from the fix in step 3: a follower waits for the leader to apply first, so followers apply a little later.
- Reads on followers get slower. A follower now applies an entry about one leader apply plus one message delay after the leader does, so standard reads on a follower also wait that much longer. But the slowdown in the benchmark is far larger than one leader apply plus one message delay: with a 500 µs apply delay, the p50 of standard reads on a follower rises from 1.8 ms to 95 ms, and when the apply delay happens while the state machine holds its write lock, the p50 rises from 2.0 ms to 336 ms. The cause of the slowdown is still under investigation. Until the cause is found, we recommend that applications that read on followers hold off on enabling
enable_relaxed_read. - Standard reads on the leader also get slightly slower. With
enable_relaxed_readon, even reads that do not request a relaxed read had up to 15% higher p50 on the leader, in the test with a 500 µs apply delay that happens while the state machine holds its write lock. Client writes have no measurable extra cost.
Summary
Relaxed reads remove a wait that linearizability does not need: a read on the leader waits only for the current leader’s no-op, not for apply to catch up with the CommitIndex. As long as the leader does not change, a read that waits only for the no-op is linearizable. When the leader steps down mid-read, such a read can see an older value; but as long as followers apply only the log entries that the leader has already applied, the value never goes back. When apply is delayed, relaxed reads clearly lower the latency of reads on the leader, and the cost of the fix is that followers apply a little later. So relaxed reads fit best in applications whose apply is delayed and whose reads go mostly to the leader.
Acknowledgments and references
The relaxed read design was proposed by neog24.
- OpenRaft issue #1267: the proposal and its discussion.
- OpenRaft PR #2317: the implementation of relaxed reads.
- ReadIndex: Less Latency through Relaxed Ordering: the description and proof of the design.
- RelaxedReadApplied.tla and the TLA+ verification report: the TLA+ model and the check results.
- Opt-in relaxed leader reads: relaxed reads in the OpenRaft documentation.
- relaxed-reads-results.md: all benchmark results.
Reference:
-
bench-results : https://github.com/databendlabs/openraft/blob/a728203cdc6dd15da28a053e5ae72a5d444bc8d1/benchmarks/minimal/relaxed-reads-results.md
-
ce-relaxed-read : https://github.com/drmingdrmer/consensus-essence/blob/main/src/list/raft-read-index-relaxed-order/raft-read-index-relaxed-order.md
-
ce-tla-report : https://github.com/drmingdrmer/consensus-essence/blob/main/src/list/raft-read-index-relaxed-order/tla/report.md
-
code-enable-relaxed-read : https://github.com/databendlabs/openraft/blob/a728203cdc6dd15da28a053e5ae72a5d444bc8d1/openraft/src/config/config.rs#L476-L531
-
code-with-relaxed : https://github.com/databendlabs/openraft/blob/a728203cdc6dd15da28a053e5ae72a5d444bc8d1/openraft/src/raft/linearizable_read/linearizer_option.rs#L120-L135
-
docs-openraft-relaxed-read : https://github.com/databendlabs/openraft/blob/a728203cdc6dd15da28a053e5ae72a5d444bc8d1/openraft/src/docs/protocol/read.md#opt-in-relaxed-leader-reads
-
github-neog24 : https://github.com/neog24
-
issue-1267 : https://github.com/databendlabs/openraft/issues/1267
-
Linearizable Transactions in Distributed Systems : https://blog.openacid.com/algo/linearizable/
-
Openraft’s optimization of ReadIndex : https://blog.openacid.com/algo/openraft-read/
-
pr-2317 : https://github.com/databendlabs/openraft/pull/2317
-
raft-noop : https://raft.github.io/raft.pdf#page=13
-
raft-read-index : https://web.stanford.edu/~ouster/cgi-bin/papers/OngaroPhD.pdf#page=90
-
openraft : https://github.com/databendlabs/openraft
Article link: https://blog.openacid.com/distributed/raft-relaxed-read/

Leave a comment