OpenACID Blog

The Pitfalls of Raft Membership Change

2026-08-27T00:00:00+00:00

The Problem

A while back, over coffee, a friend described a problem their team had hit with Raft in production. It looks like a small detail. It cost them a whole cluster.

Their implementation uses single-server change: to change the replica set, you add or remove one node at a time. Moving from abc to bcd takes two steps. First add d, which gives abcd. Then remove a, which gives bcd.

The trouble lives in the middle step. While the cluster has four nodes, a network split of the shape ad | bc leaves it unable to elect a leader. That shape of split is easy to get when the nodes live in different datacenters. Say a, b and c each sit in their own datacenter:

 a      b      c
----   ----   ----
DC-1   DC-2   DC-3

        | add `d` in DC-1
        v

 a      b      c     partitioned     a   |  b      c
 d                   no leader!      d   |
----   ----   ----   ------------>  ---- | ----   ----
DC-1   DC-2   DC-3                  DC-1 | DC-2   DC-3

        | remove `a`,
        | healthy again
        v

        b      c
 d
----   ----   ----
DC-1   DC-2   DC-3

In the steady state, the cluster has three nodes. If any one datacenter loses contact with the outside, the other two still hold a majority. They elect a leader and keep serving.
In the middle state, DC-1 holds two nodes, a and d. A majority of four nodes needs three nodes. DC-1 has only two, so it cannot elect a leader on its own. DC-2 and DC-3 together also have only two, so they cannot either. One datacenter going quiet now stops the whole cluster.

Look at what happened to DC-1. In the four-node middle state, every majority has to include a node from DC-1. DC-1 became a single point of failure, and only because a membership change was in progress.

The root cause is that single-server change is rigid about one thing: a quorum is always a majority, and never anything else. Once we let go of that rule, the problem dissolves. So we will look at membership change through quorum sets, and that view leads us straight to joint consensus.

If quorum sets are new to you, my earlier article A Minority Implementation of Majority Read/Write introduces the way of thinking we use below.

Looking at the Problem Through Quorum Sets

Instead of describing a cluster by its nodes, describe it by its quorums. A quorum is a group of nodes that is allowed to commit something. The list of all such groups is the quorum set.

One rule keeps a quorum set safe: any two groups in it must share at least one node. That shared node is what stops two different values from both being committed, and it is the only thing Paxos and Raft really need from a quorum.

Here are the quorum sets in our story:

The starting state abc uses every majority of abc: M(abc) = {ab, ac, bc}. The full group abc is a quorum too, but it already contains ab, and any group that contains a quorum is a quorum. So listing the bigger groups adds nothing, and we list only the smallest ones.
The final state bcd uses M(bcd) = {bc, cd, bd}.
The middle state abcd of a single-server change is again a majority set: M(abcd) = {abc, abd, acd, bcd}.

So a single-server change is a walk through three quorum sets:

M(abc) → M(abcd) → M(bcd)

Now the availability problem has a one-line explanation. Every quorum in the middle set needs three nodes. When the network splits into ad | bc, neither side has three nodes. Neither side can elect a leader, and the cluster stops.

A First Patch: Let

bc Commit

The middle set is where it hurts, so let us change the middle set. Suppose we also allow bc to commit. The middle quorum set becomes:

Q(abcd) = M(abcd) ∪ {bc}

In plain words: an entry is committed once it reaches bc, or any three of the four nodes.

This is safe, and the check is quick. bc shares a node with every group in M(abcd), because bc has two nodes, each three-node majority leaves out only one node, and it cannot leave out both b and c. So the sharing rule still holds across the whole set. Paxos and Raft run on this middle state exactly as before, and consistency is untouched.

The change now reads: M(abc) → M(abcd) ∪ {bc} → M(bcd).

It is also still a legal membership change. Raft’s safety argument for a single-server change asks for one thing: every quorum of the old node set must share a node with every quorum of the new one. Our added group bc meets ab at b, meets ac at c, and meets bc at both. So if Raft is safe for M(abc) → M(abcd), it is equally safe for M(abc) → M(abcd) ∪ {bc}. The same check passes for the second step, M(abcd) ∪ {bc} → M(bcd).

That treats the symptom. The cluster now survives the ad | bc split while the change is running.

Why Majority Runs Out of Room

The four-node state has this weakness for a reason worth naming: M(abcd) is not the largest safe quorum set for four nodes.

With an odd number of nodes, majority is the best you can do. Take three nodes abc: you cannot add any smaller group to {ab, ac, bc}, because a single node such as a fails to meet bc. Majority is already maximal, so nothing is lost.

With an even number of nodes, majority leaves quorums on the table. A four-node system has four three-node majorities. On top of those, it can safely hold three more two-node groups:

Q’(abcd) = M(abcd) ∪ {ab, bc, ac}

Every pair in Q’(abcd) shares a node. ab and bc share b. ab and ac share a. bc and ac share c. Each two-node group also meets each three-node group, because two plus three is more than four. Paxos and Raft run on Q’(abcd) with no changes at all, and it tolerates strictly more failures than M(abcd).

Majority is the first weak spot in Raft’s design. By writing majority into the algorithm, Raft gives away availability that an even-sized cluster could have had.

How to Expand a Majority

Here is the general recipe. Let the node set be C, for example C = {a,b,c}.

For an odd node count, n = 2k+1, keep the majorities. They are already maximal:
For an even node count, n = 2k, notice that any n/2 nodes must share a node with any n/2+1 nodes: together they count n+1 nodes in a cluster of only n. So we may add groups of size n/2 to M(C). The only extra thing to check is that the added groups share nodes with each other.

In our four-node example:
- Q’ = M(abcd) ∪ {ab, bc, ca} works: the three added groups pairwise share a node.
- Q’ = M(abcd) ∪ {bc, cd, bd} works for the same reason.
- Q’ = M(abcd) ∪ {ab, bc, cd} does not work: ab and cd share nothing, so two leaders could be elected at the same time.
There is an easy way to produce a good one. Treat the even cluster as an odd cluster C plus one extra node x:

Then the quorum set for the even cluster can be an expansion of M(D):

In words: keep every majority of the four nodes, and also accept every majority of the three nodes left when you ignore x. Picking x = d produces the first example above, and picking x = a produces the second. Both hold more quorums than M(abcd), so both are more available, and both survive the datacenter split we started with.

What the Middle State Really Needs

Those examples make one thing clear. The middle state of a membership change does not have to be a majority set. It only has to be safe, and for our datacenter problem it has to contain bc.

Several middle states qualify:

M(abcd) ∪ {ab, bc, ac},
{abc, abd, acd, bcd, bc},
and even {abd, acd, bcd, bc}, with abc dropped.

Joint consensus qualifies too. It looks complicated on paper, and it turns out to be the simplest of all.

The Correctness Conditions

Before comparing algorithms, let us write down what a membership change has to guarantee. Describe each state by its quorum set, the way we have been doing, and let the change go from Q₁ to Q₂. It has to meet three conditions:

A committed change stays visible. If one change is already committed, every uncommitted change must be recognizable as uncommitted. Otherwise a new leader cannot tell which of them to keep.
Concurrent changes exclude each other. Only one of several concurrent changes may succeed, so every process proposing a change must commit it against the same quorum set. The only thing all processes already agree on is Q₁. So a change must be committed to Q₁, or to an expansion of Q₁ that every process derives in the same way.
The change reaches the new configuration. It must also be committed to a quorum of Q₂. Otherwise a leader elected under Q₂ may never see it.

Raft’s original single-server change misses the first condition. The author fixed it later, and we will come back to that.

Joint Consensus Gives Us Exactly That

Joint consensus meets all three conditions. It also handles our datacenter problem, without anyone designing it for that.

In a change from abc to bcd, the joint middle state is the product of the two majority sets:

Q = M(abc) x M(bcd)

A joint quorum is any group that contains one quorum of M(abc) and one quorum of M(bcd) at the same time. With M(abc) = {ab, bc, ca} and M(bcd) = {bc, cd, bd}, the product is:

M(abc) x M(bcd) = {
    ab ∪ bc,
    ab ∪ cd,
    ab ∪ bd,
    bc ∪ bc,
    bc ∪ cd,
    bc ∪ bd,
    ac ∪ bc,
    ac ∪ cd,
    ac ∪ bd,
} = {
    abc,
    abcd,
    abd,
    acd,
    bc,
    bcd,
}

That is exactly M(abcd) ∪ {bc} — the very quorum set we built by hand a few sections ago.

So joint consensus hands us everything we were after:

It tolerates one node failure.
It always contains bc, so it survives the ad | bc split that started this article.
The whole change finishes with two committed log entries, whether or not the leader changes along the way.

The Bug in Single-Server Change

Single-server change has a second problem, and this one is heavier than availability. As first published, it was simply incorrect.

The bug appears when a leader change and a membership change run at the same time. The author announced it in 2015:

Unfortunately, I need to announce a bug in the dissertation version of membership changes (the single-server changes, not joint consensus). The bug is potentially severe, but the fix I’m proposing is easy to implement.

Here is how it goes wrong. The cluster starts with the four nodes abcd. One process wants to add u, another wants to add v. A leader change in the middle loses a committed entry:

C₀ = {a, b, c, d}
Cᵤ = C₀ ∪ {u}
Cᵥ = C₀ ∪ {v}

Lᵢ: Leader in term `i`
Fᵢ: Follower in term `i`
☒ : crash

    |
 u  |         Cᵤ                  F₂  Cᵤ
--- | ----------------------------------
 a  | C₀  L₀  Cᵤ  ☒               L₂  Cᵤ
 b  | C₀  F₀          F₁          F₂  Cᵤ
 c  | C₀  F₀          F₁  Cᵥ          Cᵤ
 d  | C₀              L₁  Cᵥ  ☒       Cᵤ
--- | ----------------------------------
 v  |                     Cᵥ                  time
    +-------------------------------------------->
          t₁  t₂  t₃  t₄  t₅  t₆  t₇  t₈

t₁: the four nodes abcd elect a as leader in term 0, with followers b and c.
t₂: a appends a change entry Cᵤ and switches to the new config Cᵤ right away. The entry reaches only a and u, so it is not committed.
t₃: a crashes.
t₄: d is elected leader in term 1, with followers b and c.
t₅: d appends another change entry Cᵥ and switches to Cᵥ. The entry reaches c, d and v, which is a majority of the five nodes in Cᵥ, so it is committed.
t₆: d crashes.
t₇: a comes back and is elected leader in term 2. It runs under Cᵤ, the config it sees in its own log, and collects votes from u and b.
t₈: a replicates its own log to everyone, and the committed Cᵥ is gone.

Read t₅ and t₈ together, because that is where the damage is. Cᵥ was committed under every rule Raft gives us, and then it was overwritten. The reason is that a was allowed to run an election under a configuration nobody else had ever committed.

The author’s fix is short, and it echoes a rule Raft already has for ordinary entries:

The solution I’m proposing is exactly like the dissertation describes except that a leader may not append a new configuration entry until it has committed an entry from its current term.

In our timeline, d must commit a no-op entry in term 1 before it may append Cᵥ. Once b and c hold that no-op, a can no longer win the term-2 election: b sees that a’s log is behind and refuses to vote for it. So a never becomes L₂, and the committed Cᵥ survives.

Look Closely at That Fix

The fix quietly turns single-server change into joint consensus.

Both end up doing the same job. A change has to pass through a quorum of the old configuration first, so that only one change out of several concurrent ones can be considered committed. Single-server change reaches that point with an extra entry: an ordinary application entry if one is handy, or a no-op if not. Joint consensus reaches it directly, because its middle state already is the old configuration and the new configuration at the same time.

A correct single-server change costs two log commits, every time.

Single-server change was proposed to make things simpler, and it does not. Changing abc to bcd costs 2 to 4 log entries with single-server change. With joint consensus it costs 2.

There is a fair objection here: single-server change often needs only 2 entries, since the leader usually has a committed entry of its own term already and no no-op is required. That is true, and it does not help. Code is not a bet on probability. Every branch that can run has to be written, tested and maintained, including the one that fires once in ten thousand changes. So a correct single-server change carries almost the same logic as joint consensus, implements a two-step change anyway, and wins nothing at runtime.

Closing Thoughts

Raft is a beautiful bridge from theory to working code, and that beauty is exactly why one design mistake in it travelled so far.

If you are building or maintaining a Raft implementation, the advice is short: use joint consensus. It closes the availability hole in the middle state, it removes the bug, and it is less code than a correct single-server change.

Reference:

多数派读写的少数派实现 : https://blog.openacid.com/algo/quorum/
Raft : https://raft.github.io/
Single-server membership change : https://gist.github.com/ongardie/a11f32b70581e20d6bcd

Build a Distributed Timestamp Oracle with EzRaft

2026-08-11T00:00:00+00:00

A timestamp oracle issues one globally increasing sequence. Transaction systems built on MVCC consume two of those values on every commit, so the oracle sits on the critical path of every write. The naive way to keep the sequence safe is one Raft commit per request. That design is correct. It is also too slow.

This article builds the other design: reserve a range with one Raft write, then allocate from memory. The service is timestamp_oracle, about 200 lines on EzRaft. It runs as a three-node cluster, stays available after any one node fails, and answers next_timestamp without waiting for a Raft commit.

EzRaft, from the previous article, is a small Rust API over OpenRaft. It hides the Raft, networking, and storage machinery, so you can start with the service itself.

Three calls carry the design:

EzRaft::write() commits the upper bound of a reserved range and applies it to the state machine.
EzRaft::wait_metrics() lets a background task sleep until its node becomes the leader.
EzRaft::linearizable(ReadPolicy::LeaseRead) checks the leader lease and returns the current term, so each cached range is bound to one term.

The idea underneath those calls is simple. Raft owns a ceiling. The leader spends from below that ceiling in memory. A lease and a term tag decide whether it is still allowed to spend.

What a Timestamp Oracle Must Guarantee

The interface is one call: a client sends next_timestamp, and the service returns a timestamp.

Those values usually order distributed transactions. Models such as Percolator give each transaction a start timestamp and a commit timestamp. Those two numbers decide which transaction came first. TiDB’s TSO (Timestamp Oracle) is the same component.

The service has three hard requirements:

Monotonicity. Timestamps increase in the order requests reach the oracle, and they never go backward after a failover. Every timestamp from the new leader must be greater than every timestamp the old leader already returned.
Fault tolerance. The service stays available while a majority of the cluster is alive. OpenRaft already provides that, so a three-node cluster can lose any one node and keep issuing timestamps.
Low latency. next_timestamp sits on the critical path of every operation that needs an ordering point, so each request has to be cheap.

There is a softer requirement as well. The values should stay close to wall-clock time. Correctness only requires the sequence to increase. Closeness to wall-clock time is a quality goal, not a safety property.

Safe Baseline: One Raft Write per Timestamp

The cluster only has to agree on one durable value: reserved_end, the exclusive end of all timestamp space reserved so far. Every value below reserved_end is permanently consumed, whether or not a client ever received it.

The obvious safe design writes one Raft log entry for every next_timestamp request. Once that entry is replicated and applied, the state machine returns the timestamp it claimed. Every leader walks the same log, so a failover cannot reuse an earlier value.

The state machine is correspondingly small:

struct Reserve {
    reserve_upto_us: u64,
}

struct Interval {
    start: u64,
    end: u64,
}

struct TimeState {
    reserved_end: u64,
}

impl EzApp for TimeState {
    type Request = Reserve;
    type Response = Interval;

    async fn apply(&mut self, req: Reserve) -> Interval {
        let start = self.reserved_end;
        let end = start.max(req.reserve_upto_us);
        self.reserved_end = end;
        Interval { start, end }
    }
}

Reserve asks the state machine to move reserved_end to at least reserve_upto_us. The response is the newly claimed half-open interval [start, end). Those types and apply() are all that [EzApp][docs-ezraft-ezapp] needs. See timestamp_oracle.rs:66-112.

The max in apply() makes the transition idempotent. Applying the same upper bound again leaves reserved_end unchanged, so a timed-out reservation can be retried safely.

This design is safe. It is also slow: every request pays for Raft replication and state-machine application. The rest of the article takes both of those costs off the request path.

Fast Path: Reserve Timestamps in Batches

One Raft entry can reserve a wide range instead of a single value. By default the service reserves one second of timestamp space, then the leader hands out individual values from that range in memory.

reserved_end is durable. The leader’s position inside the reserved range is not: it lives only in memory. If the leader fails, its successor skips the unused suffix and reserves a new range. The sequence may contain gaps. It never goes backward.

A background reservation task keeps the leader’s cache filled. Each successful reservation is stored in a Reserved value with three fields: the term that owns the range, the next available timestamp, and the exclusive upper bound.

struct Reserved {
    term: u64,
    next: u64,
    end: u64,
}

run_reserver() uses wait_metrics() to wait until the node becomes the leader. Once it is, the task calls refill() and repeats every half-reservation interval. See timestamp_oracle.rs:170-186:

struct TimeService {
    // ...
    reserved: Mutex<Reserved>,
}

impl TimeService {
    async fn run_reserver(&self) {
        let refresh_interval = Duration::from_micros(self.reservation_width.get() / 2);
        loop {
            let leads = self.raft.wait_metrics(None, |m| m.state == ServerState::Leader, "reserve timestamps");
            if let Err(error) = leads.await {
                return;
            }
            if let Err(error) = self.refill().await {
                warn!("{error}");
            }
            tokio::time::sleep(refresh_interval).await;
        }
    }
}

refill() first obtains a valid leader term. It then computes now + width, commits that upper bound through Raft, and stores the returned interval in TimeService.reserved. See timestamp_oracle.rs:188-205:

async fn refill(&self) -> io::Result<()> {
    // ensure leader lease and get leader term.
    let term = self.leader_term().await?;
    let now = unix_timestamp_micros();
    let width = self.reservation_width.get();
    let upto = now.saturating_add(width);

    let interval = self.raft.write(Reserve { reserve_upto_us: upto }).await?;

    self.reserved.lock().await.install(term, interval);
    Ok(())
}

The Raft write rate is now independent of the request rate. For a reservation width of width, the task writes once every width / 2. A single distributed commit can serve 100,000 next_timestamp requests.

Preserve Monotonicity Across Leader Failover

Batch allocation stays safe because the committed upper bound, not the in-memory cursor, is the handoff point. When a new leader is elected, it already has the log. After it applies those entries, reserved_end is at least as large as anything the old leader reserved. The new leader never starts from the old leader’s in-memory cursor, because that cursor was never replicated.

That committed bound protects the new leader. It does not stop the old one. A network partition can isolate the old leader long enough for the other nodes to elect a replacement, while the old leader still believes it owns the cluster.

Once the new leader issues a timestamp from a later range, any later allocation from the old leader’s range would move time backward. The old leader must refuse requests as soon as it can no longer prove that it still holds leadership.

Reject a Stale Leader with a Lease

A lease is a time-bounded promise: as long as the leader has heard from a quorum recently, no other node can win an election. Checking that promise is a local read of heartbeat state, so it adds no network round trip to a timestamp request. timestamp_oracle performs that check with a linearizable read using ReadPolicy::LeaseRead.

OpenRaft refreshes the lease from heartbeat acknowledgments. A recent acknowledgment from a quorum opens a window during which another leader cannot be elected. The current leader can allocate timestamps for the length of that window.

Once the latest quorum acknowledgment is older than the lease timeout, the node can no longer prove it is the only leader. linearizable() returns an error, and next_timestamp() stops before it touches the cache.

[EzRaft::linearizable(ReadPolicy::LeaseRead)][code-ezraft-linearizable] reads the most recent heartbeat state in memory. It does not contact the quorum again. Internally it calls OpenRaft’s [Raft::ensure_linearizable()][docs-openraft-ensure-linearizable] and returns (term, index). The term identifies the current leadership; the allocator uses it to validate the cached range:

async fn leader_term(&self) -> io::Result<u64> {
    let (term, _index) = self.raft.linearizable(ReadPolicy::LeaseRead).await?;
    Ok(term)
}

Serve

next_timestamp from Memory

With a reserved range in place, the request path has three steps:

Call leader_term() to validate the lease and obtain the current term.
Allocate one timestamp from the cache and advance the cursor.
Return the timestamp to the client.

The HTTP handler, the request path, and the cache allocation fit in a few lines. See timestamp_oracle.rs:124-133 and timestamp_oracle.rs:162-168:

async fn get_time(service: web::Data<TimeService>) -> Result<web::Json<String>, actix_web::Error> {
    let micros = service.next_timestamp().await.map_err(actix_web::error::ErrorServiceUnavailable)?;
    Ok(web::Json(format_timestamp(micros)))
}

async fn next_timestamp(&self) -> io::Result<u64> {
    let mut reserved = self.reserved.lock().await;
    let term = self.leader_term().await?;
    let timestamp = reserved.take(term, unix_timestamp_micros());
    timestamp.ok_or_else(|| io::Error::other("no reserved timestamp is currently available"))
}

fn take(&mut self, leader_term: u64, now: u64) -> Option<u64> {
    let timestamp = now.max(self.next);
    if self.term != leader_term || timestamp >= self.end {
        return None;
    }
    self.next = timestamp + 1;
    Some(timestamp)
}

now.max(self.next) keeps timestamps close to wall-clock time while preserving local monotonicity. If the clock advances, the allocator follows it. If several requests arrive in the same microsecond—or the wall clock steps backward—self.next keeps every result unique and increasing.

The term comparison in take() is not part of the lease check. The next section is why that field exists. leader_term() keeps the term and discards the log index; see timestamp_oracle.rs:207-210. The OpenRaft documentation describes the other ReadPolicy options. The underlying read optimization is covered in How OpenRaft Optimizes ReadIndex.

Bind Cached Ranges to a Leader Term

A valid lease proves that this node is the leader now. It does not prove that the cached range belongs to the current term.

Suppose node A reserved a range in term 1, lost leadership, and won it back in term 3. The leftover cache is still in the same process, so a restart is not there to wipe it. In between, node B may have been the leader in term 2 and may already have issued later timestamps. Node A’s leftover cache now sits behind time. Using it would go backward.

That is why Reserved stores the term next to next and end. A range reserved in term 1 is valid only in term 1. The term-3 leader must commit a new reservation before it can serve requests again.

take() enforces both cache invariants in one guard. A term mismatch rejects a range from an earlier leadership. timestamp >= self.end rejects an exhausted range. Later requests succeed only after the background task installs a valid range.

if self.term != leader_term || timestamp >= self.end {
    return None;
}

With the lease check and the term check in place, the in-memory path is safe across leader changes.

Run the Service

Start a Three-Node Cluster

Open three terminals and start one node in each. The first command creates the cluster; the other two use --seed to find the first node and join it:

cargo run --example timestamp_oracle -- --addr 127.0.0.1:8090
cargo run --example timestamp_oracle -- --addr 127.0.0.1:8091 --seed 127.0.0.1:8090
cargo run --example timestamp_oracle -- --addr 127.0.0.1:8092 --seed 127.0.0.1:8090

The cluster assigns node IDs automatically. When a node joins, EzRaft writes a blank log entry and uses its index as the new node’s ID. Other log entries may be committed between two joins, so IDs such as 0, 8, and 17 are expected. /api/metrics now shows three voters with node 0 as the leader:

{"id":0,"current_term":1,"state":"Leader","current_leader":0,
 "membership_config":{"membership":{"configs":[[0,8,17]], ...}}}

Request Timestamps

curl -X POST 127.0.0.1:8090/time

Run the command three times. Each request returns an RFC 3339 timestamp as a JSON string:

"2026-08-11T14:43:16.966734Z"
"2026-08-11T14:43:16.973216Z"
"2026-08-11T14:43:16.979556Z"

The six-to-seven-millisecond gaps come from starting a new curl process for each request. The timestamp allocation itself stays entirely in the leader’s memory.

A follower rejects /time and names the current leader in its response:

$ curl -s -X POST 127.0.0.1:8091/time
has to forward request to: Some(0), Some(BasicNode { addr: "127.0.0.1:8090" })   [HTTP 503]

Stop the Leader

Press Ctrl-C in the terminal running node 0, then keep sending next_timestamp requests to the two remaining nodes:

43:28.998  curl 8090  ->  "2026-08-11T14:43:28.998630Z"                        [200]
43:29      kill node 0
43:29.3    curl 8091  ->  has to forward request to: Some(0), ...:8090          [503]
   ...        (for about five seconds, both 8091 and 8092 return the same 503)
43:34.8    curl 8091  ->  has to forward request to: Some(17), ...:8092         [503]
43:34.8    curl 8092  ->  no reserved timestamp is currently available          [503]
43:35.169  curl 8092  ->  "2026-08-11T14:43:35.169756Z"                        [200]

The timeline shows two stages of recovery. At first the followers still point clients at node 0. Node 17 is then elected leader, but its first Reserve entry has not committed, so its in-memory cache is still empty. As soon as that reservation commits, the service resumes.

The timestamps stay monotonic. The first value from the new leader, 14:43:35.169756, is greater than the last value from the old leader, 14:43:28.998630. The unused interval between them is a harmless gap. /api/metrics confirms that the cluster has advanced from term 1 to term 2:

{"id":17,"current_term":2,"state":"Leader","current_leader":17}
{"id":8,"current_term":2,"state":"Follower","current_leader":17}

What Raft Owns

The consensus layer does not need to see every timestamp. It needs to see the ceiling, and it needs to say who may spend below that ceiling. Once those two facts are durable and current, next_timestamp is a local increment.

That split is the whole service. EzRaft handles replication, election, and failover. The application code persists the reservation boundary, tags each cached range with a term, and advances a cursor.

EzRaft: Build a Distributed KV Store in 100 Lines

2026-08-01T00:00:00+00:00

EzRaft: Build a Distributed KV Store in 100 Lines

For the past few years, I have spent most of my time working on OpenRaft.

OpenRaft is an open-source project that runs in production at companies around the world. To support such a wide range of applications, it has to be highly flexible. Log entries, log IDs, state machines, and storage are all expressed as Rust traits, so users can adapt them to their own needs.

Today, OpenRaft’s API and performance are both fairly mature. The trade-off is that its model of the Raft algorithm remains quite abstract.

The OpenRaft Learning Curve

That abstraction creates a steep learning curve. There are three main parts:

Understanding the Raft consensus algorithm: Before you can build a distributed store on OpenRaft, you need to understand what Raft guarantees—and what it does not.
Understanding Rust: OpenRaft is written in Rust. Reading the code and building an application requires some familiarity with Rust’s abstractions and generics.
Understanding OpenRaft’s own abstractions: Abstraction makes software easier to use once you understand the model. It gives experienced users convenience and flexibility. But when you are learning the system from the bottom up, the abstraction itself becomes another thing you must first understand.

This is a common problem with abstractions: before the whole system makes sense, you must understand how all of its abstract pieces fit together. Most people learn more naturally in the other direction—from concrete examples to general ideas.

Give someone an abstraction and ask them to imagine a concrete example, and the idea can be hard to grasp. Give them a concrete example first, and the underlying pattern is much easier to see. OpenRaft’s abstractions make it a powerful, stable tool, but not necessarily an easy place to start learning.

Why I Built EzRaft

As the maintainer of OpenRaft, I regularly hear from developers who are new to distributed systems, Rust, or OpenRaft itself. They want to build something of their own—perhaps a distributed storage system—but the amount they need to learn first can be overwhelming.

I began to wonder: could I build a layer that lets people focus on their business logic first, without requiring them to master Raft, Rust, and OpenRaft up front?

That question led to EzRaft.

EzRaft is a streamlined distributed storage library built on top of OpenRaft. It hides the details you do not need at the beginning, so you can concentrate on the application rather than the machinery underneath it.

Here is a rough comparison:

	OpenRaft	EzRaft
Traits to implement	7+ (`RaftLogStorage`, `RaftStateMachine`, …)	2 (`EzApp`, `EzStorage`)
Methods to implement	21+	4
Types to define	12 generic parameters	2 (`Request`, `Response`)
Network code	About 100 lines, written by the user	Built in, 0 lines

The Four Things Users Define

EzRaft asks you to define just four things:

The shape of a request: Request.
The shape of a response: Response. Requests and responses represent your application’s operations, so only you can define them.
How a request changes the state machine: apply().
How to read from the state machine: read(). You can think of the state machine as a key-value store and use a key to retrieve one part of it.

Together, these four pieces define the boundary of your application. EzRaft handles everything below that boundary: storage, network communication, service APIs, and more. When building directly on OpenRaft, you must provide these pieces yourself.

With EzRaft, you can build the business logic first and explore the lower-level details later, when you actually need them.

trait EzApp: Serialize + DeserializeOwned {
    type Request;
    type Response;

    async fn apply(&mut self, req: Self::Request) -> Self::Response;
    fn read(&self, key: &str) -> Option<serde_json::Value>;
}

That is the complete application-facing API. Define those four pieces, add a few lines of boilerplate to connect them, and let EzRaft handle the rest.

The EzApp value holds the application’s entire state. EzRaft persists and restores that state through serialization and deserialization, which means a snapshot is simply the serialized form of the whole application.

This deliberate simplification makes EzRaft a good fit for applications whose state fits in memory. We will return to that constraint later.

Build a Distributed KV Store in Three Steps

Building our KV store takes three steps:

Implement EzApp to define the business model.
Assemble a service from the components EzRaft provides.
Start three nodes and try the cluster for yourself.

Step One: Define the Application

Let us start with the application itself. The service supports two write operations: set a key and delete a key.

The complete application state lives in a KV map, backed here by a B-tree map.

apply takes a set or delete request and applies it to the map.

read looks up a value by key and returns it.

That is all the application needs to do. EzRaft takes care of everything else.

enum Request {
    Set { key: String, value: String },
    Delete { key: String },
}

struct KvApp {
    data: BTreeMap<String, String>,
}

impl EzApp for KvApp {
    type Request = Request;
    type Response = Option<String>;

    async fn apply(&mut self, req: Request) -> Option<String> {
        match req {
            Request::Set { key, value } => self.data.insert(key, value),
            Request::Delete { key } => self.data.remove(&key),
        }
    }

    fn read(&self, key: &str) -> Option<serde_json::Value> {
        self.data.get(key).cloned().map(serde_json::Value::String)
    }
}

Both insert and remove return the key’s previous value, so apply returns it as well. Each write tells the caller exactly what it replaced or removed, without an extra read.

To keep the example readable, I omitted the derive attributes. A real implementation needs them. Request travels over the network and is written to the Raft log, so it must support serialization through serde.

OpenRaft also prints requests in its diagnostic logs, which requires Debug and Display. EzRaft may need to retain a copy while forwarding a request to the leader, so Request also needs Clone. Finally, KvApp is the state itself, and serde turns that state into snapshots.

The repository includes a complete, runnable version in examples/kvstore.rs, with all derives and imports in place. You can run it directly with cargo run.

It differs from the code in this article in two small ways. First, it uses clap to parse --addr and --seed instead of positional arguments. Second, it returns a Response struct rather than Option, so the HTTP response is {"value":"world"} instead of "world".

Step Two: Assemble a Working Service

Now we can turn the business logic into a working server. The server accepts two arguments at startup:

This node’s address. Other EzRaft nodes use it for cluster communication, and clients use it to send requests.
A seed address. A new node with uninitialized storage contacts the seed to join an existing cluster.

The first node does not need a seed. It creates a single-node cluster and can begin serving requests immediately.

The second and third nodes use the first node as their seed. Each asks to join the existing cluster, and together they form a three-node cluster.

EzRaft and OpenRaft handle the entire process: admitting each node, assigning its ID, and synchronizing its data.

The server code is short: read the command-line arguments, then initialize storage.

This example uses EzRaft’s built-in FileStorage. It demonstrates every responsibility of a storage implementation, with clarity taking priority over performance.

A production system should replace it with a proper storage backend. FileStorage does not call fsync, so a power failure could lose a write that the cluster has already acknowledged to the client. That can break Raft’s correctness guarantees.

Fortunately, EzStorage has only three methods. The FileStorage source provides a complete template for implementing them.

After initializing storage, create the KvApp instance and pass it, the storage backend, and a configuration to EzRaft. The server is then ready to run.

#[tokio::main]
async fn main() -> io::Result<()> {
    // Usage: kvstore [addr] [seed-addr]
    let addr = std::env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
    let seed = std::env::args().nth(2);

    let storage = FileStorage::new(format!("./data/{}", addr.replace(':', "-"))).await?;
    let (app, config) = (KvApp::default(), EzConfig::default());

    let raft = match seed {
        Some(seed) => EzRaft::join(&addr, seed, app, storage, config).await?,
        None => EzRaft::create(&addr, app, storage, config).await?,
    };

    raft.serve().await
}

create and join are deliberately separate methods rather than a single method with an Option parameter. Creating a cluster and joining an existing one are fundamentally different operations.

Suppose a node is meant to join a cluster, but its seed is accidentally omitted. If it calls create, it forms a second, independent single-node cluster—and the two clusters will never merge. Requiring the caller to choose explicitly is safer than letting an empty configuration value make that decision.

Step Three: Start a Three-Node Cluster

Open three terminals and start one node in each:

cargo run -- 127.0.0.1:8080
cargo run -- 127.0.0.1:8081 127.0.0.1:8080
cargo run -- 127.0.0.1:8082 127.0.0.1:8080

Open a fourth terminal for client requests.

First, write a value. A write returns the key’s previous value. Because hello does not exist yet, the response is null:

curl -X POST 127.0.0.1:8080/api/write \
    -H 'Content-Type: application/json' \
    -d '{"Set": {"key": "hello", "value": "world"}}'
# null

Now read the value from another node. The request goes to 8082, even though no client has written to that node directly:

curl '127.0.0.1:8082/api/read?key=hello'
# "world"

Every node holds a complete copy of the application state. A read can therefore go straight to local memory, without contacting the leader or running a round of consensus.

This local read is fast, but it can be slightly stale. The node may not yet have received a write that the cluster just acknowledged. If you need a strictly linearizable read, call linearizable() first. It confirms the leader’s identity and waits for the local state to catch up; the following read() will then see every earlier acknowledged write.

Send Writes to Any Node

In Raft, only the leader can coordinate a client write, but EzRaft hides that restriction from the client. Every node exposes the same write endpoint. A follower forwards the request to the leader, waits for it to commit, and then returns the result to the caller.

That is why the following request succeeds when sent to 8081. The response, "world", is the old value replaced by the write:

curl -X POST 127.0.0.1:8081/api/write \
    -H 'Content-Type: application/json' \
    -d '{"Set": {"key": "hello", "value": "again"}}'
# "world"

The client never needs to discover or track the leader; it can connect to any node. A delete works the same way and returns the value it removed:

curl -X POST 127.0.0.1:8080/api/write \
    -H 'Content-Type: application/json' \
    -d '{"Delete": {"key": "hello"}}'
# "again"

After the deletion, the key no longer exists, so another read returns HTTP 404:

curl '127.0.0.1:8080/api/read?key=hello'
# no value for key "hello"     [HTTP 404]

Test a Node Failure

So far, a single HashMap on one machine could provide the same behavior. The reason to run three nodes is fault tolerance: the service keeps working when one node fails.

Write a key, then stop the leader—the first node, running on 8080:

curl -X POST 127.0.0.1:8080/api/write \
    -H 'Content-Type: application/json' \
    -d '{"Set": {"key": "k", "value": "before-failover"}}'
# null

# Press Ctrl-C in the first terminal, or kill the process.

The two remaining nodes still form a majority, so they elect a new leader within a few heartbeats. The data written before the failure remains available, and the cluster continues to accept writes:

curl '127.0.0.1:8081/api/read?key=k'
# "before-failover"

curl -X POST 127.0.0.1:8081/api/write \
    -H 'Content-Type: application/json' \
    -d '{"Set": {"key": "k", "value": "after-failover"}}'
# "before-failover"

curl 127.0.0.1:8081/api/metrics | jq -c '{id, state, current_leader}'
# {"id":2,"state":"Follower","current_leader":6}

Node 6 is now the leader. The cluster completed the failover without any manual intervention.

Now restart the failed node with exactly the same command:

cargo run -- 127.0.0.1:8080

Notice that the command does not include a seed. The node ID is already stored in ./data/127.0.0.1-8080/, so the node loads it during startup instead of joining again.

The original seed is no longer needed, even if it has since left the cluster. Once the restarted node is running, it automatically catches up on the write it missed:

curl '127.0.0.1:8080/api/read?key=k'
# "after-failover"

A three-node cluster tolerates one failure; a five-node cluster tolerates two. Even-numbered clusters provide no additional fault tolerance: four nodes, like three, can still tolerate only one failure.

Cluster State

EzRaft includes a default endpoint for inspecting the current state of a node and its cluster:

curl '127.0.0.1:8080/api/metrics' | jq

The response comes directly from OpenRaft and exposes its full set of metrics:

{
  "running_state": { "Ok": null },
  "id": 0,
  "current_term": 1,
  "vote": {
    "leader_id": { "term": 1, "voted_for": 0 },
    "committed": true
  },
  "last_log_index": 10,
  "local_committed": { "leader_id": 1, "index": 10 },
  "cluster_committed": { "leader_id": 1, "index": 10 },
  "committed": { "leader_id": 1, "index": 10 },
  "last_applied": { "leader_id": 1, "index": 10 },
  "snapshot": null,
  "purged": null,
  "state": "Leader",
  "current_leader": 0,
  "millis_since_quorum_ack": 3,
  "last_quorum_acked": 1785595777894002000,
  "membership_config": {
    "log_id": { "leader_id": 1, "index": 9 },
    "membership": {
      "configs": [ [ 0, 2, 6 ] ],
      "nodes": {
        "0": { "addr": "127.0.0.1:8080" },
        "2": { "addr": "127.0.0.1:8081" },
        "6": { "addr": "127.0.0.1:8082" }
      }
    }
  },
  "committed_membership_config": {
    "log_id": { "leader_id": 1, "index": 9 },
    "membership": {
      "configs": [ [ 0, 2, 6 ] ],
      "nodes": {
        "0": { "addr": "127.0.0.1:8080" },
        "2": { "addr": "127.0.0.1:8081" },
        "6": { "addr": "127.0.0.1:8082" }
      }
    }
  },
  "heartbeat": {
    "0": 1785595777894002750,
    "2": 1785595777894002209,
    "6": 1785595777894002125
  },
  "replication": {
    "0": { "leader_id": 1, "index": 10 },
    "2": { "leader_id": 1, "index": 10 },
    "6": { "leader_id": 1, "index": 10 }
  }
}

The three node IDs are 0, 2, and 6—not 0, 1, and 2. That is because the cluster assigns node IDs instead of reading them from configuration.

When a node joins, the leader first writes a blank log entry, then uses that entry’s index as the new node’s ID. Log indexes are unique, so the resulting node IDs are unique as well. They are not consecutive because other entries are written between two joins: one adds the new node as a learner, and a membership change promotes it to a voter. The gaps between IDs are a visible trace of those steps.

A new node first joins as a learner. It receives log entries, but it does not vote and does not count toward the cluster’s majority. Once it has caught up, EzRaft automatically promotes it to a voter.

From the user’s perspective, joining the node is enough. EzRaft handles synchronization and promotion, so the node begins contributing to fault tolerance automatically.

When to Use EzRaft

EzRaft is designed for applications whose state fits in memory but must never be lost or allowed to diverge across nodes. Configuration services, service registries, discovery systems, metadata stores, distributed locks, and task schedulers all fit this pattern.

The data is usually small, but it is critical: losing it or allowing replicas to disagree can disrupt every system built on top of it. ZooKeeper and etcd serve this same class of workload.

This focus follows directly from EzRaft’s snapshot model. A snapshot is the serialized form of the complete EzApp, and installing one simply deserializes that state back into the application.

As a result, snapshot support shrinks from “implement an entire snapshot mechanism” to “add one derive annotation to a struct.” Applications whose state fits in memory get the full benefit of that simplification.

EzRaft also works well for read-heavy workloads with relatively few writes. Every node holds a complete copy of the state, and reads go straight to local memory without contacting the leader or running consensus. Clients can read from any node, so adding nodes increases read throughput.

EzRaft is also a practical way to see Raft working end to end. The roughly one hundred lines above already exercise leader election, log replication, membership changes, snapshots, and failover.

That makes the example easy to experiment with. Stop a node and watch the cluster elect a new leader. Follow the log index in the metrics. Change the heartbeat interval in EzConfig and see how it affects election time. Once these ideas are concrete, moving down a layer to OpenRaft is much easier than starting with a set of trait definitions.

To turn this example into a production system, you need to replace two pieces. The first is FileStorage, which should become a production-grade storage backend.

The second is the service API. The built-in HTTP endpoints are designed for the example and provide neither authentication nor encryption. Use EzServer as a reference, define routes that fit your application, and call EzRaft::write and EzRaft::read underneath. The rest of the stack can remain unchanged.

Do Not Read: Raf Is a Useless and Valueless Failed Experiment

2026-05-11T00:00:00+00:00

Summary: raf: Raft without [T]erm is an experimental Raft variant. It does not persist currentTerm as a separate piece of state. Instead, a candidate reserves a log index when it starts an election, and that index becomes the leader term. This does not remove Raft’s logical time model. It only changes where the term is derived from in storage.

Declaration: This approach is just the same as saving terms in the separate first extra slot in the terms array. So it actually still stores the term and has no value at all. Please do not read this as a useful design; it is only a failed personal experiment.

Note: The idea in this article came from Zhang Yanpo. The code was implemented by Zhang Yanpo by hand. This article was drafted and refined with Codex.

Repository: raf (v0.1.1).

Introduction

I have seen a few interesting proposals that try to remove the term from Raft. The idea is appealing: if a consensus protocol can maintain one less piece of persistent state, perhaps both the model and the implementation become simpler. But Raft’s term is not just a counter. It represents logical time, lets nodes distinguish old leaders from new ones, and participates in Raft’s commit safety rule. So the real question is not whether we can simply delete the term. The useful question is whether we can express it in a different way.

The name raf comes from Raft without [T]erm. Here, “without term” does not mean the protocol has no term at all. It means currentTerm is no longer persisted as an independent field. The project turns this idea into a small but serious implementation: avoid storing currentTerm separately, while still giving every part of Raft that needs a term a reliable source of logical time.

This article explains that core idea. The term still exists as a concept. Logs are still compared by (term, index). What changes is the source of the term: it no longer comes from a separately incremented persistent counter; it comes from a log index reserved by an election. The goal of this implementation is not to prove that it is a drop-in replacement for standard Raft in every engineering detail. The goal is to see whether this storage representation preserves the most important safety intuition behind Raft.

We will walk through the storage model, election, replication, commit, and the three-node example in the repository. I assume the reader is already familiar with the basic Raft flow: leader election, AppendEntries, quorum commit, and log ids in the form (term, index).

Why Term Cannot Disappear

In a consensus algorithm, the log records events that have been chosen or are being proposed. The term tells us which logical time those events belong to.

Standard Raft uses the term for several jobs:

Leader election advances the term before choosing a leader. The term gives leaders an ordering.
Because of that, log freshness is compared by term first, then by index.
A leader may directly commit only entries from its own term.

This role is similar to the ballot number in Paxos. It lets the system decide which history is newer and which candidate is eligible to become leader, even when a node does not know every other node’s complete log.

In short: a log index is a local event position; a term is the logical time used to compare histories across nodes.

raf keeps the concept of term, but removes its separate storage.

Core Idea

Standard Raft usually persists state shaped roughly like this:

struct StandardRaftStorage {
    current_term: Term,
    voted_for: Option<NodeId>,
    log: Vec<LogEntry>,
}

struct LogEntry {
    term: Term,
    cmd: Cmd,
}

raf represents the persistent state as two Vecs aligned by log index:

struct RafStorage {
    terms: Vec<Term>,
    cmds: Vec<Cmd>,
}

Here:

terms[i] is the leader term of the log entry at index i.
cmds[i] is the application command of the log entry at index i.
log_id(i) is still (terms[i], i).

The following is one possible storage state. ø means empty command. cmds only reaches index 7, so indexes 8 and 9 are not complete log entries yet.

Index by index, this state means:

Index 0: the fixed default entry. terms[0] = 0, and cmds[0] is an empty command.
Index 1: a successful election for term 1. The leader reserved index 1 when it was elected and wrote its first empty command there.
Index 2: a successful election for term 2. The new leader reserved index 2; its first log entry is also an empty command.
Index 3: a user log entry C3 written by the leader of term 2, so terms[3] = 2 and cmds[3] = C3.
Index 4: this position used to be an election attempt for term 4, but it did not produce an established leader. Later, when the leader of term 6 was established, this position was filled as an empty command owned by term 6, so now terms[4] = 6.
Index 5: this position used to be another failed election attempt for term 5. It was also later filled as an empty command owned by term 6, so now terms[5] = 6.
Index 6: a successful election for term 6. The leader reserved index 6 when it was elected, and wrote its first empty command there.
Index 7: a user log entry C7 written by the leader of term 6, so terms[7] = 6 and cmds[7] = C7.
Index 8: a new election attempt for term 8. So far we have only seen terms[8] = 8; there is no command for this index yet, so it is not a complete log entry.
Index 9: another election attempt for term 9. Like index 8, it currently has only a term record and no command. From this state alone, we cannot tell whether it will eventually become a leader.

Standard Raft persists a separate current_term and an array of (term, command) log entries. raf is similar, but splits the term and command at each index into two aligned Vecs.

Storage Model

Index 0 is a fixed default entry. This keeps the types simple and avoids using Option for the initial position:

terms[0] = 0
cmds[0]  = empty

When both Vecs have a value at the same index, that index is a complete log entry. Otherwise, the index has a term but no command. That state represents an election in progress: the term has been observed, but no log entry has been written at that position yet.

log[index] = (terms[index], cmds[index])
log_id     = (terms[index], index)

terms and cmds share the same log index. During election, terms may be ahead of cmds.

During an election, terms can temporarily be longer than cmds. A candidate first reserves an index as its term. Multiple failed elections may reserve multiple indexes. Only after some candidate becomes an established leader are the positions with missing commands filled with empty commands.

So this implementation maintains a few basic facts:

cmds should never be longer than terms.
When an election first reserves a term slot, terms[term] = term.
When an established leader fills positions that still miss commands, those positions are rewritten to the current leader.term.

Therefore terms[i] <= i is not a protocol invariant. A backfilled empty entry may have terms[i] > i. That is not a problem by itself; standard Raft terms can also be greater than log indexes. The important property is different: every complete log entry must carry the term of an established leader, except for the fixed default entry at index 0.

Why Split

terms and cmds

This implementation separates leader terms and application commands into two Vecs. Raft’s protocol semantics do not require this layout. It is mainly a storage design choice that creates room for cleaner optimization.

For example, a leader may write many consecutive log entries during one term, and all of those entries have the same term. A storage engine could compress long runs of identical terms into a compact representation, while storing commands according to application needs. Once the two streams are separated, term compression, command persistence, and payload encoding can evolve independently.

This is the experimental value of the project. It expresses the relationship between “logical time” and “log position” in Raft more directly, then asks whether that expression can simplify persistent state.

Starting an Election

When a candidate starts an election, it uses the next index of the terms array as the new term:

let term = terms.len();
terms.push(term);

In this code:

The candidate declares that it wants to use term as its leader term.
The local persistent state records that this index has been reserved by an election.

The candidate then sends RequestVote. This part is the same as standard Raft. The request carries two important pieces of information:

term: the leader term the candidate wants to use.
last_log_id: the (term, index) of the candidate’s last complete log entry.

last_log_id is computed from the last index in cmds:

let last_log_index = cmds.len() - 1;
let last_log_id = (terms[last_log_index], last_log_index);

The meaning of last_log_id is the same as in standard Raft: the voter uses it to check whether the candidate’s log is at least as up to date as its own. Notice the separation here. The new election term comes from terms.len(), while last_log_id comes from cmds.len() - 1. These can point to different indexes.

In the following example, the complete log currently reaches only index 3, so last_log_id = (2, 3). When the candidate starts a new election, it uses terms.len() = 4 as the new term and first writes index 4 into terms. At this point, cmds still reaches only index 3, because this candidate has not become an established leader yet.

If the election for term 4 does not reach a quorum, it leaves only an observed term index in terms; it does not create a new command. The next election will use terms.len() again, which is now term 5.

At this point last_log_id is still (2, 3), because cmds has not moved beyond index 3. What changed is the candidate term: it advanced from 4 to 5. Only after an election succeeds and a leader is established does the system rewrite the positions with missing commands to this leader’s term and fill cmds with empty commands.

How a Voter Handles RequestVote

When a voter receives RequestVote, it checks three things:

Whether the requested term is greater than the last term observed locally.
Whether the requested term slot does not already exist locally.
Whether the candidate’s log, represented by last_log_id, is fresh enough. It must not be older than the voter’s own log.

Aside from the term check, the rest of the logic is standard Raft. The subtle difference is that the term is no longer stored independently, so term freshness is expressed through terms.

The core condition looks like this:

let local_last_log_index = cmds.len() - 1;
let local_last_log_id = (terms[local_last_log_index], local_last_log_index);
let local_last_term = terms[terms.len() - 1];

let can_vote =
    req.term > local_last_term
        && req.term >= terms.len()
        && req.last_log_id >= local_last_log_id;

The next diagram sends the same RequestVote { term: 5, last_log_id: (2, 3) } to voters with three different local states. The candidate’s own state is at the top. The three branches below it show how each voter makes its decision.

The three outcomes are:

granted: the voter’s terms reaches only index 3, and cmds also reaches only index 3. Therefore req.term = 5 is greater than the last locally observed term, names a term slot that has not appeared locally, and req.last_log_id = (2, 3) is not behind the voter. The vote can be granted.
rejected: term=7: the voter has already observed a later term 7. Since req.term = 5 is not greater than the last locally observed term, the candidate’s requested term is stale from this voter’s perspective, so the vote is rejected.
rejected: last log id = (4,4): the voter’s last complete log entry is (4, 4), which is newer than the candidate’s (2, 3). Even if the requested term could be recorded, the log freshness check still fails, so the vote is rejected.

If the request is valid, the voter records the term in local terms. If local terms is shorter than req.term, it fills the missing positions with default indexes until local state contains index req.term:

if can_vote {
    while terms.len() <= req.term {
        let index = terms.len();
        terms.push(index);
    }
}

These default items have term values equal to their own indexes. They mean the node has observed the corresponding term indexes. They do not mean those indexes are complete log entries, because the matching cmds may not exist yet. If a later leader fills those positions as complete empty entries, it rewrites their terms to its own leader.term. On the final iteration, index == req.term, so the voter has observed and accepted that term. Later, it will not accept an older term or a term index that already exists locally.

This replaces the role of standard Raft’s persisted currentTerm, but it is not fully equivalent to standard Raft’s votedFor. The current implementation does not persist “which candidate this term was granted to.” As a result, RequestVote retry and restart behavior are more conservative. We will return to this trade-off in “Current Boundaries.”

The candidate chooses terms.len() as its term. Other nodes record that term in their local terms when they grant the vote.

After the candidate receives granted replies from a quorum, it becomes an established leader. It first rewrites the local cmds.len()..terms.len() range to its own leader.term, then appends empty commands so that cmds.len() catches up with terms.len(). The index reserved by the leader’s election becomes this leader’s first complete log entry.

Establishing Leader State

After a candidate becomes an established leader, it keeps the core state of this leadership in memory. You can think of it as the following structure:

struct LeaderState {
    term: Term,
    granted_nodes: Vec<NodeId>,
    replications: BTreeMap<NodeId, ReplicationState>,
}

struct ReplicationState {
    matched: LogIndex,
    end: LogIndex,
    inflight: bool,
}

The important fields are:

term: the log index reserved when this leader was elected. All later log entries produced by this leader write this term into the corresponding positions of the terms array.
granted_nodes: the node ids that granted this leadership. This proves the leader was chosen by a quorum.
replications: the replication progress of each node from the leader’s point of view. matched is the largest index known to match on that node; end is the upper bound used for further probing or replication; inflight prevents sending multiple Append requests to the same node at the same time.

The leader also has its own replication state. This makes commit calculation uniform: look at which nodes have matched covering an index, then check whether those nodes form a quorum.

After the leader is established, every position that already exists in local terms but is still missing from cmds is taken over by the current leader: the term at that position is rewritten to leader.term, and the command is filled with an empty command. After that, every local index on the leader has a corresponding command, and new application writes can start at the next index.

In this example, index 4 used to be a term slot left behind by a failed election, and term 5 is the index reserved by the current leader. When the candidate for term 5 becomes an established leader, indexes 4 and 5 are both rewritten as term 5 empty log entries. The ø at index 5 is the entry reserved by this leader’s election; the ø at index 4 is the backfilled entry that keeps the log prefix contiguous.

Appending a Log Entry

After a node becomes leader, each new application write appends a log entry. The term does not change. It is still the term chosen when the leader was elected:

terms.push(leader.term);
cmds.push(user_cmd);

So within one leader term, all later log entries have the same terms[i]. This matches standard Raft behavior. Only the source of the term is different.

Log Replication

The leader sends Append requests to the other nodes. Conceptually, each request first names a previous log id that is already known to match, then carries a contiguous segment of log entries after that position:

struct Append {
    term: Term,
    prev_log_id: LogId,
    terms: Vec<Term>,
    cmds: Vec<Cmd>,
}

Here prev_log_id is the matching point. terms and cmds are the real entries after that point. They must have the same length, and their first item corresponds to prev_log_id.index + 1. The follower first checks whether its local log has the same LogId at prev_log_id.index. Only if that previous position matches does it accept the entries that follow.

This design is close to standard Raft’s AppendEntries. Standard Raft carries prevLogIndex and prevLogTerm separately; raf combines them into a single prev_log_id. That is clearer than using the first entry in the request as the matching point, because the request’s terms and cmds represent only the real entries to replicate.

The following diagram shows one Append request applied to several follower states. The request has term = 5, prev_log_id = (2, 3), and carries the consecutive log entries for indexes 4..=5.

The three outcomes are:

accepted: the follower matches the leader at prev_log_id = (2, 3), so it can accept this Append. {...} marks the terms range overwritten by this request, as well as the command range appended because it was missing locally. Here the term at index 4 is updated from an old value to 5, and command c5 is appended at index 5.
conflict at prev_log_id: * marks the conflict position. The follower’s term at index 3 is 3, while the request’s prev_log_id is (2, 3). Since the previous log id does not match, the follower returns a conflict index immediately. The leader must try again with an earlier prev_log_id.
rejected: follower has newer term: the follower has already observed term 6 at index 6, while the Append request is from term 5. This request comes from a stale leader, so the follower rejects it without modifying the log.

The handling logic is:

If the request term is older than the last observed local term, reject it.
Check the follower’s local log at prev_log_id. If it does not match, return the conflict index.
If prev_log_id matches, process the real entries starting at prev_log_id.index + 1.
If later local commands diverge from the leader, truncate the local commands.
Overwrite the local terms range covered by this request.
Append only the commands that are missing locally.

Append first finds the common prefix with prev_log_id, then truncates the follower’s conflicting suffix, and finally copies the leader’s entries that the follower is missing.

This is still Raft’s core replication model: the leader finds a shared log prefix, then replaces the follower’s divergent suffix with its own.

Advancing Commit

Replication to a quorum does not mean every historical entry can be committed immediately. Standard Raft has an important rule: a leader may only directly commit log entries from its own current term. Entries from older terms become committed only as a consequence of committing an entry from the current term.

raf keeps this rule. Although an established leader rewrites earlier gap slots to its own term, the leader still only directly commits matched indexes that are not smaller than its election index. Empty entries backfilled before leader.term are committed only indirectly, together with the leader’s election index or a later entry.

Intuitively:

if quorum_has_matched(index) && index >= leader.term {
    commit(index);
}

The reason is the same as in standard Raft: once an index is committed, every future valid leader must contain it and must not overwrite it.

The following state shows why “replicated to a quorum” is not enough. The leader of term 6 has replicated the old log entries at indexes 4 and 5 to quorum A+B, but that quorum has not yet matched the leader’s own term index 6. Therefore indexes 4 and 5 still cannot be committed:

If a new leader for term 7 appears later, and its last_log_id=(5,6) is newer, it can overwrite those uncommitted log entries. In the diagram, {x} marks the range replaced by the new leader.

A leader only directly commits an index that is both covered by a quorum and inside the current leader’s term range.

Example

The repository includes a three-node in-process example that demonstrates the basic flow described in this article. It creates three Raf nodes, connects them with InProcessNetwork, explicitly triggers an election on node 1, and then writes a few log entries through the leader. Metrics show the role, term, commit index, and replication progress.

The example source is here:

https://github.com/drmingdrmer/raf/blob/v0.1.1/examples/three_node.rs

Run it from the repository root:

cargo run --example three_node

This example is not a production deployment template. It is a minimal demonstration for observing the core protocol state transitions. It logs to stderr and does not include an election timer, heartbeat, snapshots, or membership changes.

Current Boundaries

The current implementation is intentionally small. It leaves out several features that a complete production system would usually need:

Automatic election triggering.
Snapshots and log compaction.
Membership changes.
Heartbeats.
RequestVote retry logic.
Persistence semantics for application payloads.

Automatic election triggering corresponds to the election timer in standard Raft. A node periodically checks whether it has gone too long without seeing a valid leader. If it times out, it starts a new election. This can be implemented by an external timer that calls Raf::elect(). It does not need to live inside the core raf state machine, so the current implementation leaves it out.

RequestVote retry has a subtler boundary. Suppose a target node successfully handles a RequestVote, but the reply is lost in the network. If the candidate retries the same request, the target node has already recorded this term in terms[req.term]. Under the current rule, it rejects the retry because that term index already exists.

One optional fix is to add an in-memory voted_for field that records which candidate owns a term. Then a retry from the same candidate for the same term can be recognized and granted again. This field does not necessarily need to be persisted. If a node restarts and loses voted_for, it can conservatively reject every RequestVote that uses a term already present locally. That creates a small availability issue, but only after restart; it does not change the persisted relationship between logs and terms.

If a RequestVote reply is lost, the retry sees an already existing term. An optional in-memory voted_for field can improve availability in that case.

These features can all be added around the core model. This article focuses on the central question: if the term comes from the log index, can Raft election, replication, and commit still be expressed in the familiar way?

Summary

raf is not “Raft without any term.” It still has terms, and it still compares logs by (term, index). What it removes is the independently persisted currentTerm; the leader term is bound to the log index reserved by an election.

This change turns the storage state into two Vecs aligned by index:

struct RafStorage {
    terms: Vec<Term>,
    cmds: Vec<Cmd>,
}

During election, a candidate chooses terms.len() as its term. After it becomes leader, both the missing-command gap slots and later log entries use this leader term. Replication and commitment still follow the basic Raft rules.

That is the core of this experimental implementation: keep Raft’s logical time model, but change where that logical time comes from in persistent state.

Repository: raf (v0.1.1).

Reference:

raf : https://github.com/drmingdrmer/raf/tree/v0.1.1

Histogram Done Right: 2KB Memory, 0.2% Error

2026-04-02T00:00:00+00:00

The Problem: Tracking Request Latency Without Slowing Things Down

When we were building Databend and OpenRaft, we ran into a familiar need: we wanted to see how request latency was distributed across the system, in real time, without burning CPU or memory to do it. This article explains the design behind base2histogram, the library we built to solve it.

Consider the life of a single Raft log entry. It passes through several stages, and each one has its own latency profile:

Received → written to storage
Persisted to local disk
Replicated to remote nodes
Acknowledged by a majority quorum
Committed → applied to the state machine

A histogram is the natural tool here — plot latency on the x-axis, request count on the y-axis, and you get an immediate picture of where time is being spent.

This kind of visibility is what lets you find bottlenecks and fix the right thing.

But there’s a catch: collecting metrics can’t get in the way of doing actual work. So the histogram needs to be:

O(1) to record — no sorting, no rebalancing, nothing that can stall a hot path
Tiny in memory — the system may run hundreds or thousands of these at once
Queryable for percentiles — P50, P95, P99

Let’s walk through how we designed one that hits all three.

Recording: Getting Samples Into Buckets

Why Log-Scale Buckets

Most requests cluster around some typical latency, with a few outliers on both ends. This is a log-normal distribution — take the log of the latency values, and the shape becomes a classic bell curve.

The signature look: a peak at lower values, then a gradual long tail stretching to the right.

To build a histogram, we divide the x-axis into buckets and count how many samples land in each one.

The key question is how to size those buckets. Equal-width buckets work great for a normal distribution, but latency is log-normal — the data only looks uniform on a logarithmic scale. So the buckets need to grow on a log scale, not a linear one.

The simplest version of this: each bucket is twice as wide as the one before it.

[0,1), [1,2), [2,4), [4,8), [8,16), ...

Why powers of 2? Because multiplying by 2 is free on a CPU, and mapping a value to its bucket takes a single leading zero count instruction.

Simulate a log-normal workload, plot the bucket counts with the bucket index on the x-axis (effectively a log transform), and the result is a clean bell curve:

Storage-wise, this is great — 65 buckets cover the entire u64 range. Resolution-wise, not so much. The last bucket spans half of all possible values. Everything that lands there is a blur.

A Tempting Fix We Passed On

An obvious improvement: use a smaller growth factor, like 1.1× instead of 2×. More buckets, finer resolution:

The problem is cost. Finding the right bucket for a value l means solving for the smallest x where 1 + 1.1 + 1.1^2 + ... + 1.1^x >= l — and that requires floating-point logarithms. That’s real overhead on a hot path.

We wanted to stay in the world of integers and bit operations.

The Trick: Float-Like Encoding

Here’s the idea that makes everything work. We keep the roughly exponential bucket sizes, but we encode each bucket using a fixed number of bits — a parameter we call WIDTH.

Think of a bucket’s lower bound as a tiny floating-point number. The MSB position gives you the exponent (which group of buckets you’re in), and the next few bits give you the offset within that group.

With WIDTH=3 (the default), a bucket boundary looks like this in binary:

00..00 1 xx 00..00
       |
       MSB
<- significant

The position of the leading 1 picks the group. The two bits that follow pick the bucket within the group.

Here’s what the first few groups look like — each bucket is fully described by just 3 bits:

WIDTH = 3:

range     bucket index        bucket size
[0, 1)     0  0b0 ..... 000    1
[1, 2)     1  0b0 ..... 001    1
[2, 3)     2  0b0 ..... 010    1
[3, 4)     3  0b0 ..... 011    1

[4, 5)     4  0b0 ..... 100    1
[5, 6)     5  0b0 ..... 101    1
[6, 7)     6  0b0 ..... 110    1
[7, 8)     7  0b0 ..... 111    1

[8, 10)    8  0b0 .... 1000    2
[10, 12)   9  0b0 .... 1010    2
[12, 14)  10  0b0 .... 1100    2
[14, 16)  11  0b0 .... 1110    2

[16, 20)  12  0b0 ... 10000    4
[20, 24)  13  0b0 ... 10100    4
[24, 28)  14  0b0 ... 11000    4
[28, 32)  15  0b0 ... 11100    4

[32, 40)  16  0b0 .. 100000    8
[40, 48)  17  0b0 .. 101000    8
[48, 56)  18  0b0 .. 110000    8
[56, 64)  19  0b0 .. 111000    8

The pattern is clean:

Each group contains 2^(WIDTH-1) = 4 buckets
The 2 bits after the MSB select the bucket within the group
It’s a 3-bit float: 1 implicit leading bit + 2 fractional bits

Bucket sizes grow roughly logarithmically, and computing the bucket index is just a matter of extracting the top WIDTH bits — a handful of integer and bit ops. Recording a sample is O(1).

Walk-through with latency = 42:

value = 42 (binary: 0b101010)
  MSB position: 5
  group: 5 - 2 = 3
  2 bits after MSB: 01 (from 1[01]010)
  offset in group: 1
  Bucket index: 4 + (3 × 4) + 1 = 17

Tuning WIDTH: The Precision–Memory Knob

WIDTH sets how many buckets each group gets (2^(WIDTH-1)). Groups top out at 64 (they still double in size, covering the full u64 range).

Here’s how the trade-off plays out:

WIDTH	Buckets	Mem/slot	Buckets per group
1	65	520 B	1
2	128	1.0 KB	2
3	252	2.0 KB	4 (default)
4	496	3.9 KB	8
5	976	7.6 KB	16
6	1920	15.0 KB	32

At the default WIDTH=3, one histogram costs 2 KB and records every sample in O(1).

That’s the write side sorted. Now for the read side.

Percentile Estimation: Getting Answers Out

Once we’ve collected the counts, we want percentiles: at what latency have 50% of requests finished (P50)? 90% (P90)? 99% (P99)?

Locating the Right Bucket

The basic idea is simple. For P50: count the total samples, take 50% to get a target rank p, then walk through the buckets from the start, accumulating counts until you pass p. That’s your bucket.

But a bucket spans a range, not a point. We still need to estimate where inside the bucket the percentile actually falls.

Here are a few ways to do that, from rough to precise. All error numbers below come from a log-normal distribution (API latency scenario), WIDTH=3, 1,000,000 samples.

Midpoint: just return (min + max) / 2. Many histogram libraries do this (e.g., iopsystems/histogram). It’s a blind guess — it ignores everything about how samples are distributed within the bucket.

	P50	P95	P99
midpoint	5.018%	7.732%	4.861%

Uniform interpolation: assume samples are spread evenly across the bucket (a flat rectangle), then interpolate linearly: estimate = min + (max - min) × rank / count.

Better than midpoint — at least it uses where in the bucket the target rank falls. But “evenly spread” is a rough assumption. Log-normal data is skewed even within a single bucket.

Trapezoid Interpolation (Our Approach)

Uniform interpolation treats the density inside a bucket as flat. In reality, it’s sloped — denser on the side closer to the peak of the distribution.

If we know which way the density tilts, we can swap the rectangle for a trapezoid and land much closer to the true value.

Each bucket stores only a count — we don’t want to add extra fields. So where does the slope information come from? The neighbors.

The densities of the left and right buckets tell us how the density slopes through the current bucket.

Here’s the recipe. Compute the average density of the left bucket: d0 = c0/(x1-x0), and treat it as the density at that bucket’s midpoint m0. Do the same for the right bucket: d2 = c2/(x3-x2) at midpoint m2. Assume density varies linearly from m0 to m2 — over this short range, that’s a reasonable approximation. This gives us the slope k.

Inside the target bucket, the density now forms a trapezoid: a sloped line with slope k, pinned so that the density at the bucket’s midpoint (x1+x2)/2 equals the bucket’s own average density d1 = c1/(x2-x1) (the midpoint of a linear function always equals its average).

To find the percentile, we solve for the x-position where the trapezoid’s area from x1 equals the target rank.

Same distribution, same buckets — here’s how it stacks up:

	P50	P95	P99
midpoint	5.018%	7.732%	4.861%
trapezoid	0.000%	0.080%	0.086%

Two orders of magnitude better, with zero additional storage.

The three-bucket layout:

Variable	Meaning
`x0, x1, x2, x3`	Boundaries of the three adjacent buckets
`w0, w1, w2`	Bucket widths: `w0 = x1-x0`, `w1 = x2-x1`, `w2 = x3-x2`
`c0, c1, c2`	Sample counts in each bucket
`rank`	How many samples into the target bucket the percentile falls

d0 = c0 / w0       -- left bucket density
d1 = c1 / w1       -- target bucket density
d2 = c2 / w2       -- right bucket density

Midpoints of the left and right buckets: m0 = (x0+x1)/2, m2 = (x2+x3)/2. Slope:

k = (d2 - d0) / (m2 - m0)

Then solve for the x-position where the trapezoid’s cumulative area from x1 equals the rank.

The whole thing runs on three counts and their bucket boundaries. Nothing else stored, nothing else needed.

Benchmarks: Seven Distributions, Six WIDTH Settings

We tested across 7 representative distributions with 1,000,000 samples each, using trapezoid interpolation.

The rows to watch are LN-API and LN-DB at W=3 — these are the real-world latency cases, running on the default 2 KB configuration:

|                   W=1      W=2      W=3      W=4      W=5      W=6
| ------------------------------------------------------------------
| Uniform P50    0.108%   0.028%   0.012%   0.018%   0.019%   0.002%
|         P95    2.317%   1.988%   1.035%   0.475%   0.005%   0.005%
|         P99    4.290%   4.129%   3.706%   1.486%   0.298%   0.162%
| 
| LN-API  P50    2.281%   0.182%   0.000%   0.000%   0.000%   0.000%
|         P95   20.256%   3.963%   0.080%   0.040%   0.040%   0.000%
|         P99   11.951%   3.594%   0.086%   0.000%   0.029%   0.000%
| 
| Bimodal P50    1.381%   0.394%   0.394%   0.197%   0.197%   0.197%
|         P95    3.918%   0.172%   0.012%   0.028%   0.038%   0.008%
|         P99    1.521%   1.344%   0.543%   0.078%   0.016%   0.014%
| 
| Expon   P50    1.012%   0.000%   0.145%   0.145%   0.145%   0.000%
|         P95   10.989%   0.200%   0.000%   0.000%   0.033%   0.033%
|         P99   18.665%   4.574%   0.824%   0.022%   0.022%   0.022%
| 
| LN-DB   P50    2.018%   0.034%   0.000%   0.000%   0.000%   0.034%
|         P95    2.027%   0.368%   0.039%   0.006%   0.019%   0.026%
|         P99    3.764%   1.066%   0.187%   0.007%   0.003%   0.062%
| 
| Sequent P50    0.095%   0.000%   0.000%   0.000%   0.000%   0.000%
|         P95    2.271%   1.967%   1.011%   0.496%   0.000%   0.000%
|         P99    4.272%   4.118%   3.696%   1.521%   0.305%   0.169%
| 
| Pareto  P50   10.127%   1.899%   0.633%   0.633%   0.633%   0.000%
|         P95    9.239%   0.272%   0.000%   0.136%   0.000%   0.000%
|         P99    3.517%   0.879%   0.231%   0.093%   0.046%   0.046%
| 
| ------------------------------------------------------------------
| Buckets            65      128      252      496      976     1920
| Mem/slot        520 B   1.0 KB   2.0 KB   3.9 KB   7.6 KB  15.0 KB
| Mem total      1.0 KB   2.0 KB   3.9 KB   7.8 KB  15.2 KB  30.0 KB

What each distribution models:

Uniform (uniform distribution): synthetic benchmarks
LN-API (log-normal σ=0.5): API and microservice latency
Bimodal (bimodal distribution): cache hit/miss — 90% fast path ~500μs, 10% slow path ~50ms
Expon (exponential distribution): network and I/O waits
LN-DB (log-normal σ=1.0): database query latency, with a wider tail
Sequent (sequential): adversarial worst case
Pareto (Pareto distribution α=1.5): heavy-tailed workloads like request sizes

For the latency distributions we care about most — LN-API and LN-DB — WIDTH=3 delivers sub-0.2% error on 2 KB of memory.

Summary

2 KB memory (WIDTH=3, 252 buckets of u64), P50/P95/P99 error under 0.2% for log-normal latency
O(1) recording, O(buckets) querying
Trapezoid interpolation is what makes it work — over 10× more accurate than midpoint, with zero extra storage
WIDTH is tunable: 520 B for bare-minimum tracking, up to 15 KB for maximum precision
Code: base2histogram

Reference:

Databend : https://github.com/databendlabs/databend
MSB : https://en.wikipedia.org/wiki/Bit_numbering#Most_significant_bit
OpenRaft : https://github.com/databendlabs/openraft
Pareto distribution : https://en.wikipedia.org/wiki/Pareto_distribution
base2histogram : https://github.com/drmingdrmer/base2histogram
bimodal distribution : https://en.wikipedia.org/wiki/Multimodal_distribution
exponential distribution : https://en.wikipedia.org/wiki/Exponential_distribution
floating-point number : https://en.wikipedia.org/wiki/Floating-point_arithmetic
histogram : https://en.wikipedia.org/wiki/Histogram
iopsystems/histogram : https://github.com/iopsystems/histogram
leading zero counting : https://en.wikipedia.org/wiki/Find_first_set#CLZ
log scale : https://en.wikipedia.org/wiki/Logarithmic_scale
log-normal distribution : https://en.wikipedia.org/wiki/Log-normal_distribution
long tail : https://en.wikipedia.org/wiki/Long_tail
normal distribution : https://en.wikipedia.org/wiki/Normal_distribution
percentile : https://en.wikipedia.org/wiki/Percentile
uniform distribution : https://en.wikipedia.org/wiki/Continuous_uniform_distribution

xp 的 AI 工作流

2026-01-04T00:00:00+00:00

目前我的日常开发基本只需要”动嘴”就能完成。这篇文章分享我在用的工具组合。

偏见声明：我更希望把精力放在问题分析上，所以市面上流行的工具并没有一一尝试。只要满足需要就不会替换，除非发现明显的效率瓶颈。

工作流概览

分析与决策 — 在 IDE 中浏览代码，理解项目，决定要做什么
描述需求 — 用语音输入把想法描述出来
代码生成 — 交给 AI 命令行工具生成代码
审核与提交 — 用 tig 逐行 review，增量提交到 Git

注意上面的几个步骤一般是穿插进行的, 一个这样的修改看做一个 session 的话, 一般我会同时做 2, 3 个工作流 session, 这是因为大模型输出代码比我 review 的速度慢一些, 我自己经常会有 IO 等待, 所以同时开 2,3 个 session 可以把我的 CPU 占满. 但是切换任务也会让自己脑袋里的 context 频繁切换, 降低效率, 所以一般我开 1 个需要我仔细思考的困难任务(例如增加新 feature), 和 1 或 2 个不太需要深度思考的简单任务(例如代码重构), 就不会产生大量颅内 context 切换. 而且忙起来似乎有助于多巴胺分泌.

第一步：分析与决策 — Rust Rover

Rust Rover 是 JetBrains 开发的 Rust IDE。JetBrains 的 IDE 系列（IntelliJ IDEA、PyCharm、WebStorm 等）在 AI 时代之前一直是开发者的首选，以代码跳转、重构、调试等功能著称。

虽然 Rust Rover 在 AI 功能上相对保守，但传统的代码跳转、全局搜索等功能依然扎实，足以快速建立对项目的整体认知。

在这个阶段，我主要根据要做的事情回顾代码结构，确定具体的实现思路；或者在没有具体任务时浏览项目，寻找需要改进的地方。

第二步：描述需求 — 语音输入

口述比打字快，语音输入法是重要补充。

语音输入的错误可被大模型理解能力覆盖——只要大致表达清晰，就能完成任务。因此我对语音输入的准确率要求不高，即使口述转文字时有些错误也没关系。只要描述足够详细、提供足够的冗余信息，大模型就能准确理解需求。

以下是几种我尝试过、都能满足需要的语音输入法：

闪电说

（本地模型）

识别准确率中等
离线可用，响应快

豆包桌面版

在线识别更精准
网络延迟所以稍慢

智谱语音输入法

AI 矫正能力强，准确率高
网络延迟所以稍慢

第三步：代码生成 — Claude Code + GLM 4.7

Claude Code 是我的主力工具，CLI 框架成熟，即便不用内置模型，作为命令行交互载体也很高效。

日常开发中，我直接口述需求给 Claude Code，全程不需要键盘打字。

为了简化开发流程（也为了好玩），我买了一个只有三个键的小键盘：一个键触发语音输入，一个键是删除，一个键是回车。目前正在尝试只用这三个键完成日常开发。

模型用的是 GLM 4.7（智谱 AI 发布）。需求描述清晰的话，完成”文字到代码”的转化没问题。与顶尖模型相比，处理抽象需求时有差距，但性价比高，国内网络环境下响应快。

就算说的乱 78 糟, 大模型也知道我说的 IOarrow 是 io::Error:

我已完全摒弃交互式辅助模式，改用”全量需求描述 + AI 独立实现”。

第四步：审核与提交 — tig

我发现最高效的工作方式不是等大模型写完 review, 而是：在大模型生成代码的过程中就开始 review 产生的变更，逐段将变更加入版本控制。确认一个变更正确且符合要求后，立即加入 Git stage(git add)。这样后续修改时, review 工作区变化可以忽略已确认(git add)的部分，效率很高。

因此这个阶段最需要的工具，是能够高效的逐段将代码加入 Git 的交互式工具。git add -p 是最基本的选择，我使用的是 tig。

tig 是基于 ncurses 的 Git 文本界面工具。

为什么用 tig 而非

git add -p

git add -p 只能显示固定的 2-3 行上下文，有时不够理解修改(我的 context 太小 🤔)。

tig 通过 tig status 后进入交互界面, 可以：

逐行查看差异，像 Vim 一样导航(j/k)
快捷键[ 和 ] 随时调整 diff 上下文大小
逐行(1)或逐块(u)将修改加入 Git stage(stage 部分也叫 cached 或 index)
R(shift-r) 刷新页面, 显示最新修改

一般我习惯在 tmux 里左右分两屏, 右边 pua 大模型干活, 左边逐行确认修改. 因为有时大模型修改的滚动太快了, 看不清它到底做了什么. tig 的逐行交互式 review 容许我异步查看每一行修改, 一旦看到有问题的修改, 就可以及时切到右边叫停努力的大模型, 重新调整:

可以看到 tig 的上下文可以用[和]任意宽度展开

Vim 的 Git fugitive 插件也可以提供类似功能但需要先启动 Vim，而且操作稍繁琐。

增量 Review

确认 OK 的部分立即加入 Git，后续再次让大模型做出修改后只 review not-staged 部分, 不再重复审查上一步已经加入 stage, 确认 OK 的代码。AI 增量修改时只需关注变化的部分。

Vibe Coding 的核心理念

AI 能极大提升效率——前提是: 你了解这个领域。否则无法判断对错，错误会吃掉所有效率提升。用 AI 开发的前提是我比 AI 更了解这个项目。

工具清单

工具	用途
Rust Rover	代码分析
闪电说 / 豆包 / 智谱语音输入法	语音输入
Claude Code	AI 编程
GLM 4.7	后端模型
tig	代码审核，增量提交

这篇文章是口述完成的。

Reference:

Claude Code : https://github.com/anthropics/claude-code
GLM 4.7 : https://open.bigmodel.cn/
Rust Rover : https://www.jetbrains.com/rust/
tig : https://github.com/jonas/tig
智谱 AI : https://www.zhipuai.cn/
智谱语音输入法 : https://autoglm.zhipuai.cn/autotyper/
豆包 : https://www.doubao.com/
闪电说 : https://shandianshuo.cn/

Raft Node Rejoin Bug

2025-11-20T00:00:00+00:00

In Raft cluster operations, there’s an easily overlooked bug: when a node is removed and then re-added to the cluster within the same term, delayed AppendEntries responses from the old membership configuration can corrupt the leader’s replication progress tracking for that node, causing the leader to enter an infinite retry loop.

The root cause is the lack of a replication session isolation mechanism. When the same node joins the cluster at different times, these should be treated as different replication sessions. However, without explicit session identifiers, the leader cannot distinguish which session a response belongs to. The result is that delayed responses from old sessions incorrectly update the progress records of new sessions.

While this creates operational challenges—continuous resource consumption and nodes unable to catch up with the cluster—the good news is that Raft’s commit protocol ensures data safety remains intact.

Analyzed Raft libs:

Implementation	Stars	Language	Status	Analysis
Apache Ratis	1,418	Java	✓ PROTECTED	Report
NuRaft	1,140	C++	✓ PROTECTED	Report
OpenRaft	1,700	Rust	✓ PROTECTED	Report
RabbitMQ Ra	908	Erlang	✓ PROTECTED	Report
braft	4,174	C++	✓ PROTECTED	Report
canonical/raft	954	C	✓ PROTECTED	Report
sofa-jraft	3,762	Java	✓ PROTECTED	Report
LogCabin	1,945	C++	✗ VULNERABLE	Report
PySyncObj	738	Python	✗ VULNERABLE	Report
dragonboat	5,262	Go	✗ VULNERABLE	Report
etcd-io/raft	943	Go	✗ VULNERABLE	Report
hashicorp/raft	8,826	Go	✗ VULNERABLE	Report
raft-java	1,234	Java	✗ VULNERABLE	Report
raft-rs (TiKV)	3,224	Rust	✗ VULNERABLE	Report
redisraft	841	C	✗ VULNERABLE	Report
willemt/raft	1,160	C	✗ VULNERABLE	Report
eliben/raft	1,232	Go	N/A	Report

This article uses raft-rs, the Raft implementation used by TiKV, as a case study to analyze this bug’s trigger conditions, impact, and potential solutions.

Complete analysis and survey of other Raft implementations can be found in the Raft Rejoin Bug Survey

Raft Log Replication Basics

In Raft, the leader replicates log entries to followers through AppendEntries RPC calls, while maintaining a replication state machine for each follower to track replication progress.

AppendEntries Request-Response Flow

Here’s how it works: The leader sends AppendEntries requests with the current term, the prev_log_index and prev_log_term pointing to the position just before the new entries, the entries[] array to replicate, and the leader’s leader_commit index. The follower responds with its own term, the highest log index it replicated, and whether the operation succeeded.

Progress Tracking

The leader relies on these responses to track each follower’s replication status. It uses matched to record the highest log index confirmed to be replicated on that follower, and next_idx to mark where to send next. When a successful response comes back with index=N, the leader updates matched=N and calculates next_idx=N+1 for the next round.

This tracking mechanism has an implicit assumption: responses correspond to the current replication session.

If this assumption isn’t handled properly, when a node rejoins the cluster, the leader can get stuck in an infinite retry loop. It keeps sending AppendEntries requests, the node keeps rejecting them, and the cycle repeats endlessly while that node never manages to catch up with the cluster.

raft-rs Progress Tracking

raft-rs tracks replication progress using a Progress structure for each follower node:

// From raft-rs/src/tracker/progress.rs
pub struct Progress {
    pub matched: u64,      // Highest log index known to be replicated
    pub next_idx: u64,     // Next log index to send
    pub state: ProgressState,
    // ... other fields
}

The matched field records the highest log index successfully replicated to this follower. Whenever the leader receives a successful AppendEntries response, it updates this field:

// From raft-rs/src/tracker/progress.rs
pub fn maybe_update(&mut self, n: u64) -> bool {
    let need_update = self.matched < n;  // Only check monotonicity
    if need_update {
        self.matched = n;  // Accept the update!
        self.resume();
    }
    need_update
}

Notice the update logic is quite simple: as long as the new index is higher than the current matched, it accepts the update. When a node gets removed from the cluster, its Progress record is deleted. When it rejoins, a brand new Progress record is created with matched = 0.

Bug Reproduction Sequence

Let’s walk through a concrete timeline to see how this bug unfolds. Pay special attention to the fact that all events happen within a single term (term=5)—this is key to understanding why term-based validation fails.

Event Timeline

| Time | Event                                         | Progress State
|------|-----------------------------------------------|----------------
| T1   | log=1, members={a,b,c}                        | C: matched=0
|      | Leader sends AppendEntries(index=1) to C      |
|      | (Network delay causes slow delivery)          |
|      |                                               |
| T2   | log=5, members={a,b}                          | C: [deleted]
|      | Node C removed from cluster                   |
|      | Progress[C] deleted from leader's tracker     |
|      |                                               |
| T3   | log=100, members={a,b,c}                      | C: matched=0 (new)
|      | Node C rejoins the cluster                    |
|      | New Progress[C] created with matched=0        |
|      |                                               |
| T4   | Delayed response arrives from T1:             |
|      | {from: C, index: 1, success: true}            |
|      | Leader finds Progress[C] (the new one!)       |
|      | maybe_update(1) called: 0 < 1, so update!     | C: matched=1 ❌
|      |                                               |
| T5   | Leader calculates next_idx = matched + 1 = 2  |
|      | Sends AppendEntries(prev_index=1)             |
|      | Node C rejects (doesn't have index 1!)        |
|      | Leader can't decrement (matched == rejected)  |
|      | Infinite loop begins...                       |

Response Handling at T4

At time T4, that response sent at T1 and delayed in the network finally arrives. Here’s how the leader handles it:

// From raft-rs/src/raft.rs
fn handle_append_response(&mut self, m: &Message) {
    // Find the progress record
    let pr = match self.prs.get_mut(m.from) {
        Some(pr) => pr,
        None => {
            debug!(self.logger, "no progress available for {}", m.from);
            return;
        }
    };

    // Update progress if the index is higher
    if !pr.maybe_update(m.index) {
        return;
    }
    // ...
}

Here’s where things go wrong: The leader does find a Progress record for node C, but it’s the new one created at T3. Since the message’s term matches the current term, it passes the term check in the step() function, and the leader updates progress with this stale index value.

Root Cause Analysis

The root of this bug is that request-response messages lack replication session identification. When node C gets removed at T2 and rejoins at T3, these should be two distinct replication sessions—but the leader has no way to distinguish between responses from requests sent at T1 versus responses from requests sent after T3.

Look at raft-rs’s Message structure:

File: proto/proto/eraftpb.proto:71-98

message Message {
    MessageType msg_type = 1;
    uint64 to = 2;
    uint64 from = 3;
    uint64 term = 4;        // Only term, no session identifier!
    uint64 log_term = 5;
    uint64 index = 6;
    // ...
}

The Message only has a from field identifying the sending node, but the same node ID joining the cluster at different times should be treated as different replication sessions. The leader needs to distinguish: is this response from node C’s first session or its second session? But the current Message structure provides no way to tell.

Impact Analysis

Infinite Retry Loop

Once the leader incorrectly sets matched=1, trouble begins. Here’s what happens:

// From raft-rs/src/tracker/progress.rs
pub fn maybe_decr_to(&mut self, rejected: u64, match_hint: u64, ...) -> bool {
    if self.state == ProgressState::Replicate {
        // Can't decrement if rejected <= matched
        if rejected < self.matched
            || (rejected == self.matched && request_snapshot == INVALID_INDEX) {
            return false;  // Ignore the rejection!
        }
        // ...
    }
}

The leader sends AppendEntries with prev_log_index=1, but node C’s log is empty—it doesn’t have index 1. Node C rejects the request. The leader wants to decrement next_idx to retry an earlier position, but here’s the problem: because rejected (1) == matched (1), the decrement logic returns false and refuses to decrement. So the leader just sends the same request again, node C rejects it again, and this cycle continues forever.

Operational Impact

This bug creates a series of operational problems. First, there’s resource exhaustion: the continuous AppendEntries-rejection cycle keeps consuming CPU and network bandwidth.

Why Data Remains Safe

Despite all the operational chaos, there’s good news: data integrity remains intact. Raft’s safety properties ensure that even with corrupted progress tracking, the cluster won’t lose any committed data.

The reason is that commit index calculation still works correctly. Even if the leader mistakenly thinks node C has matched=1, it calculates the commit index based on the actual majority. For example, node A has matched=100, node B has matched=100, and node C has matched=1 (which is wrong, but doesn’t matter). The majority looks at A and B with matched=100, so the commit index is correctly calculated as 100. Combined with Raft’s overlapping majorities property, any newly elected leader will necessarily have all committed entries, keeping data safe.

Solutions

Solution 1: Add Membership Version (Recommended)

The most straightforward fix is to add a membership configuration version to messages:

message Message {
    // ... existing fields
    uint64 membership_log_id = 17;  // New field
}

Then validate it when processing responses:

fn handle_append_response(&mut self, m: &Message) {
    let pr = self.prs.get_mut(m.from)?;

    // Check membership version
    if m.membership_log_id != self.current_membership_log_id {
        debug!("stale message from different membership");
        return;
    }

    pr.maybe_update(m.index);
}

This directly fixes the root cause—the leader can now tell which membership configuration a message comes from.

Solution 2: Generation Counters

Another approach is to add a generation counter to Progress that increments each time a node rejoins:

pub struct Progress {
    pub matched: u64,
    pub next_idx: u64,
    pub generation: u64,  // Incremented on each rejoin
    // ...
}

Include the generation in messages and validate it when responses arrive. This is lighter weight than solution 1, but you need to carefully manage the generation lifecycle.

Summary

This bug shows us that when membership changes happen within the same term, relying on term-based validation alone isn’t enough to ensure message freshness. Without explicit session isolation, delayed responses from old membership configurations can corrupt progress tracking.

Fortunately, because Raft’s commit index calculation and overlapping quorum mechanisms provide strong guarantees, this bug doesn’t compromise data safety. The main impact is operational—the symptoms look like data corruption, which can send operations teams down the rabbit hole investigating a data loss problem that doesn’t actually exist.

For production Raft implementations, it’s recommended to introduce explicit session management mechanisms. This can be achieved through membership versioning or generation counters. The most recommended approach is to add a membership_log_id field to messages, which lets the leader clearly distinguish which membership configuration a response comes from.

Complete analysis and survey of other Raft implementations can be found in the Raft Rejoin Bug Survey

Reference:

Raft 中的 IO 执行顺序：内存状态与持久化状态的陷阱

2025-10-09T00:00:00+00:00

前言

在 Raft 实现中，处理 appendEntries 请求时需要持久化两类数据：term 和 log entries。Raft 论文要求”在响应 RPC 之前必须更新持久化状态”，但并未明确说明这两类数据的持久化顺序。这个看似无关紧要的细节，却可能导致已提交数据的丢失。

问题的根源在于：Raft 论文描述的是一个简单的抽象模型（只有磁盘状态），而实际实现为了性能会分离内存状态和持久化状态。这种状态分离引入了论文中未定义的行为，当 IO 操作允许重排序时，就可能破坏 Raft 的安全性保证。

本文将深入分析这个问题是如何产生的，以及主流实现（TiKV、HashiCorp Raft、SOFAJRaft）如何避免这个陷阱。

内存状态与持久化状态的陷阱

在实际的 Raft 实现中，为了提升性能，通常会分离内存状态(current_term)和磁盘状态(persisted_term)。处理 appendEntries 请求的典型流程是：

收到 appendEntries，如果 req.term > current_term，立即更新 current_term
异步提交 save-term IO
IO 完成后更新 persisted_term（有些实现中可能没有显式的 persisted_term）

这种状态分离引入了 Raft 论文中没有定义的行为（Raft 论文只关注磁盘状态）：

struct RaftState {
    // In-memory term, updated immediately when receiving higher term
    current_term: u64,

    // Persisted term on disk, updated only after IO completes
    persisted_term: u64,
}

上面描述的流程是常见的 Raft 实现的流程, 在没有 IO-reorder 时, 它是正确的。但当 IO 操作可以重排序时，就会出现严重的安全问题。

问题场景

用一个具体的时间线来展示 IO-reorder 如何导致数据丢失：

Legend:
Ni:   Node i
Vi:   RequestVote, term=i
Li:   Establish Leader, term=i
Ei-j: Log entry, term=i, index=j

N5 |          V5  L5     E5-1     E5-2
N4 |          V5         E5-1     E5-2
N3 |  V1              V5,E5-1  V5,E5-2  E1-1
N2 |  V1      V5                        E1-1
N1 |  V1  L1                            E1-1
------+---+---+---+------+--------+-----+------> time
      t1  t2  t3  t4     t5       t6    t7

t1-t4: 两次选举，N1（term=1）和 N5（term=5）先后成为 leader
t5: L5 复制 E5-1 到 N3（N3 的 current_term=1 < req.term=5）
- N3 需要执行两个 IO：持久化 term=5 和 E5-1
- 等待两个 IO 完成才返回成功
t6: L5 复制 E5-2 到 N3（关键时刻）
- N3 可能还在处理 t5 的 IO
- 这时是否存在 IO-reorder 至关重要
t7: L1 尝试复制 E1-1（term=1, index=1）

关键在于 t6 时刻的第二个 AppendEntries 请求。让我们看看 N3 的内部状态变化。

t5 时刻：第一个 AppendEntries

N3 收到 appendEntries(term=5, entries=[E5-1])：

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check: RPC term > in-memory term?
    if req.term > self.current_term {
        self.current_term = req.term;           // Update memory immediately: 5
        self.submit_io(save_term(req.term));    // Submit IO request
    }

    self.submit_io(save_entries(req.entries));  // Submit IO request

    // Wait for both IOs to complete
    wait_for_both_ios();
    return success();
}

N3 的状态：

current_term = 5（内存已更新）
persisted_term = 1（磁盘还未更新，IO 进行中）
IO 队列：save_term(5), save_entries(E5-1)

这个请求本身是正确的，问题出现在下一个时刻。

t6 时刻：第二个 AppendEntries

N3 还没完成 t5 的 IO，就收到了 appendEntries(term=5, entries=[E5-2])。

如果代码只检查内存 current_term（大多数实现的做法）, 并提交 save-entries IO：

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check: 5 > 5? No
    if req.term > self.current_term {
        // Won't enter this branch
    }

    // Only submit save_entries(E5-2)
    self.submit_io(save_entries(req.entries));

    // Only wait for save_entries to complete
    wait_for_io(save_entries);
    return success();  // Return success!
}

问题出现：在允许 IO-reorder 的时候,

save_entries(E5-2) 完成
但 save_term(5) 可能还没完成（如果存在 IO 重排序）
N3 向 Leader 返回成功

如果 N3 此时崩溃重启，磁盘状态可能是：

persisted_term = 1（save_term(5) 未完成）
entries = [E5-1, E5-2]（都完成了）
Leader L5 认为 E5-2 已提交

t7 时刻：数据丢失

重启后 N3 的磁盘状态：term=1, entries=[E5-1, E5-2]

当 L1 发送 appendEntries(term=1, entries=[E1-1])：

N3 检查：RPC term (1) == 本地 term (1)，接受
E1-1 覆盖 index=1
已向 L5 确认提交的 E5-1 和 E5-2 被覆盖

注意, 如果不允许 IO-reorder, 那么 t6 的 save_entries(E5-2) 的完成就暗示了 save_term(5) 的完成, 满足了 appendEntries 成功的条件, 不会出现问题.

问题的本质

如果允许 IO-reorder，必须检查 persisted_term 来判断是否下发 save-term IO；如果不允许 IO-reorder，检查 current_term 即可。

Raft 论文不区分内存状态和持久化状态，这是实现相关的陷阱。论文要求 “Before responding to RPCs, a server must update its persistent state”，在实现中需要更精确的表述： 必须等待所有使 persisted_term >= req.term 的 IO 完成后，才能返回成功。

正确的做法

检查持久化的 term 而不是内存 term：

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check persisted term, not in-memory term!
    let need_save_term = req.term > self.persisted_term;

    if need_save_term {
        self.current_term = req.term;
        self.submit_io(save_term(req.term));
    }

    self.submit_io(save_entries(req.entries));

    if need_save_term {
        wait_for_both_ios();  // Must wait for save_term to complete
    } else {
        wait_for_io(save_entries);
    }

    return success();
}

注意：这种实现可能多次提交 save-term IO，需要在实现中谨慎优化。

主流实现的方案

主流实现（TiKV、HashiCorp Raft、SOFAJRaft）通过限制 save-term 和 save-entries 不能 reorder，因此只检查 current_term 也是安全的：

原子批处理（TiKV）：将 save-term 和 save-entries 放到一个 IO 请求里，一次性提交。这样根本不存在”第二个 appendEntries 只提交 save_entries”的情况。
有序分离（HashiCorp Raft）：save-term 和 save-entries 顺序执行，不会重排序。先完成 term 的 fsync（失败则 panic），再写 log。
混合顺序（SOFAJRaft）：term 同步写入（阻塞等待 fsync），log 异步批处理。保证了 save_term 完成后才会入队 save_entries。

总结

Raft 论文的抽象模型（只关注持久化状态）和实际实现（内存状态 + 持久化状态）之间存在微妙的映射关系。

关键不变式：log entry (term=T) 在磁盘 → persisted_term ≥ T 也必须在磁盘

维护此不变式的两种方式：

消除 IO-reorder：原子批处理、有序执行或混合方式（主流实现）
处理 IO-reorder：检查持久化状态，等待必要的 IO 完成

Raft Configuration Change with Single Log Entry

2025-10-07T00:00:00+00:00

Preface

TL;DR

Standard Raft configuration changes use two log entries with multi-phase commits and careful state management. Can we complete a configuration change with just one log entry? We’ll introduce effective-config, prove its correctness, then discover why the simple approach isn’t so simple after all. The standard Joint Consensus method wins for good reasons.

What We’ll Cover

How Raft’s Joint Consensus works (the two-phase approach)
The single-log-entry idea and its mechanics
Why it’s theoretically correct
Why it’s practically problematic (and the patches we’d need)
Why we should stick with Joint Consensus

Introduction to Raft Joint Consensus: 2 Config Log Entries

Changing cluster membership in Raft is tricky. Switching from the old configuration {a,b,c} to a new one {x,y,z} in one step is dangerous.

Nodes can’t all switch configurations at the exact same moment. During the transition, some nodes (say a,b) might still be using C_old while others (x,y,z) have moved to C_new. If these two groups don’t overlap—meaning a quorum from C_old (like {a,b}) and a quorum from C_new (like {x,y}) share no common nodes—we could elect two leaders in the same term, violating Raft’s fundamental safety guarantee.

The Raft paper solves this with a two-phase protocol called Joint Consensus:

Phase 1: Enter the Joint phase (C_old_new) When the leader receives a configuration change request, it writes a log entry containing C_old_new—a joint configuration that includes both old and new members. In this state, any decision (like committing a log entry) needs approval from a quorum of C_old and a quorum of C_new. The leader starts using C_old_new as soon as it writes this entry to its own log.
Phase 2: Move to the new configuration (C_new) Once C_old_new commits, the leader writes a second log entry containing just C_new. From this point forward, the leader uses only C_new, and all subsequent log entries need only commit on a C_new quorum. When this second entry commits, the configuration change is complete.

The intermediate joint phase ensures that any two quorums—whether based on C_old, C_new, or C_old_new—must overlap, preventing split brain. This requires two log entries for each configuration change.

Can We Do It With Just One Log Entry?

Can we do this safely with just one log entry?

We need a new concept: effective-config. This is the configuration the leader actually uses to determine if log entries are committed. It might not match any specific configuration stored in a log entry—it’s a runtime state that changes as the configuration change progresses.

Terminology

effective-config: The runtime configuration the leader uses to determine if entries are committed
Joint config: A configuration containing both old and new members, like C_old_new = [{a,b,c}, {x,y,z}]
Uniform config: A configuration with just one set of members, like C_new = {x,y,z}
Barrier entry: A marker log entry that signals the joint phase has safely ended

How It Works

Starting point: The cluster is running with C_old = {a,b,c}, and that configuration has been committed. The effective-config is C_old.
Propose the change: To change to C_new = {x,y,z}, the leader writes a single log entry entry-i containing just C_new.
Enter joint mode immediately: The moment the leader appends entry-i to its own log—before it commits, before it replicates—the leader switches its effective-config to the joint configuration C_old_new = [{a,b,c}, {x,y,z}]. Now entry-i and all subsequent entries must commit on a quorum from both {a,b,c} and {x,y,z}.
Normal operation continues: The cluster keeps processing requests. Every entry commits using the joint quorum rules.
Exit joint mode: Once entry-i commits under C_old_new, the leader switches effective-config to C_new = {x,y,z}. All subsequent entries need only a C_new quorum.

With one log entry, the system transitions through three states: C_old → C_old_new → C_new.

Correctness Proof

We need to show that we can’t elect two leaders—neither during the configuration change nor afterward.

Assume leader t is doing the configuration change (writing entry-i). Later, some candidate u tries to get elected in term u > t. We prove t and u can’t both be leaders.

Analyzing candidate u’s election

Candidate u either has entry-i in its log or it doesn’t.

Case 1: u has entry-i

Then u’s effective-config includes {x,y,z}. Leader t’s effective-config is either C_old_new = [{a,b,c}, {x,y,z}] (still in joint mode) or C_new = {x,y,z} (finished). Either way, it includes {x,y,z}.

Since u needs a quorum from {x,y,z} to get elected, and t needs a quorum from {x,y,z} to stay leader, these quorums must overlap. No split brain.
Case 2: u doesn’t have entry-i

Then u’s effective-config is C_old = {a,b,c}. Now we consider where leader t is:
- If t’s effective-config is C_old_new, then t needs a quorum from {a,b,c} and u needs a quorum from {a,b,c}. These must overlap. No split brain.
- If t’s effective-config is C_new = {x,y,z}, that means entry-i committed under C_old_new. So entry-i must exist on a quorum of {a,b,c}. Those nodes have logs at least as long as index i.
  
  But u doesn’t have entry-i, so its log is shorter than i. When u requests votes from nodes in {a,b,c}, they’ll reject it because their logs are more up-to-date. The election fails.

In every case, we can’t have both t and u as leaders. The algorithm is safe.

However, although theoretically correct, it introduces problems in actual implementation:

Problem 1: The Memory-Only Transition

When we move from C_old_new to C_new, we only change the in-memory effective-config. Nothing hits disk. This creates trouble.

Nodes from C_old can still initiate elections and compete with C_new nodes, because C_old logs are as long as C_new logs. Even after the configuration change completes, C_old nodes can steal leadership from C_new nodes. The root cause is that the state change is not recorded on the persistent layer. This is problematic because nodes intended for removal can still become leaders.

Compare this to standard Joint Consensus: it writes a second log entry containing C_new. That entry acts as a barrier. Nodes from C_old have shorter logs and lose elections. The single-entry approach has no such barrier—the transition from C_old_new to C_new is invisible on disk.

Look at the diagram below. The cluster transitions from C_old_new to C_new, but no logs change. Leadership moves to node x in {x,y,z}. But nodes from C_old can still start elections and steal leadership from x.

Patch-1: After entering C_new, immediately append a no-op entry. This lengthens the logs of C_new nodes, blocking elections from C_old nodes.

Problem 2: The Restart Ambiguity

When a node restarts, it can’t tell if the cluster is in joint mode or has finished the change.

The restarting node reads its log. It sees entry-i containing C_old and entry-j containing C_new.
We know entry-i is committed (Raft requires it before starting a new change).
But what about entry-j? The node can’t tell just from its local log:
- If entry-j isn’t committed yet, the cluster is in joint mode with effective-config C_old_new
- If entry-j is committed, the cluster is using C_new

Without talking to other nodes, there’s no way to know.

In the diagram above, even if entry-3 has committed, the restarting nodes b, c, x, y can’t tell whether the cluster is in joint mode or using the new configuration. (Nodes a and z never received entry-3 and are still using {a,b,c}.)

Patch-2: Always start in joint mode after a restart.

When a node starts up, it sets effective-config to the joint configuration formed from the last two config entries in its log
It uses this joint config for elections and normal operation
Only after confirming that the latest config entry has committed under the joint configuration can it switch to the new configuration

Example: A node sees configs {a,b,c} and {u,v,w} in its log. It starts with effective-config [{a,b,c}, {u,v,w}]. To become leader, it needs quorums from both groups. Only after it confirms the new config committed under the joint rules can it switch to just {u,v,w}.

Problem 3: Calling Home to Dead Nodes

Patch-2 solves the ambiguity problem but creates a worse one: nodes might try to contact old cluster members that no longer exist, making elections impossible.

Example:

The cluster changes from {a,b,c} to {x,y,z}
The config entry commits under C_old_new
The cluster transitions to C_new = {x,y,z}
Nodes a, b, c are no longer members. They get shut down, their data gets wiped, and they’re gone
Then something happens and all remaining nodes restart
Node x restarts and follows Patch-2: it sees configs {a,b,c} and {x,y,z} in its log, so it sets effective-config to [{a,b,c}, {x,y,z}]
Node x tries to run an election, but b and c don’t exist anymore! It can’t get a quorum from both groups. The election fails. The cluster is stuck.

This is state regression. The transition from C_old_new to C_new wasn’t persisted, so after a restart, the system rolls back to needing C_old.

Adding a Barrier to Prevent Regression

Restarting nodes need to know for certain that the joint phase has ended—proof that it’s safe to use C_new without calling back to C_old.

Patch-3: Add a barrier entry

After entry-j (containing C_new) commits under C_old_new, append a special barrier entry to mark that entry-j has committed.

Important: The barrier must come after entry-j commits. Otherwise it can’t serve as proof of the commit.

When a restarting node sees this barrier, it knows the joint phase ended successfully. It can safely use C_new for elections without trying to contact old nodes that might not exist anymore.

In the diagram below, when entry-3 commits under C_old_new, we add barrier entry-4:

Now when all nodes restart, there’s no regression. Nodes x and y see the barrier, so they use C_new = {x,y,z} directly. Even though b and c are gone, x or y can still get elected:

Alternative: Persisting commit-index

Instead of a barrier entry, we could persist the commit-index—an idea from Ma Jianjiang.

The rule: joint consensus ends when commit-index reaches a quorum of C_new. To make this work, we’d need to persist commit-index (standard Raft doesn’t require this).

When a node restarts, it checks: if the persisted commit-index covers the config change entry, it knows C_old_new finished and can safely use C_new. No need to contact old nodes.

But this still has Problem 1—C_old and C_new nodes competing for leadership. Here’s why: C_new nodes don’t have extra log entries, and committing commit-index to just C_new doesn’t guarantee C_old nodes see it. This is the classic distributed systems dilemma of at-least-once vs at-most-once delivery:

At-least-once (commit on C_old_new): commit-index might succeed, then C_old nodes get decommissioned, then we can’t commit it again to reach them. We’re stuck.

At-most-once (commit on C_new only): commit-index reaches C_new but might not reach C_old. Those nodes don’t know the cluster moved on, so they keep trying to run elections.

Either way, we can still end up with C_old and C_new nodes competing for leadership.

So here’s what the patched single-log approach looks like:

Start with effective-config = C_old = {a,b,c}
Leader writes entry-j containing C_new = {x,y,z} and immediately switches effective-config to C_old_new = [{a,b,c}, {x,y,z}]
All entries from index j onward replicate and commit under C_old_new
Critical step: Once entry-j commits under C_old_new, the leader writes a special barrier entry. This entry has no configuration data—it just marks “the joint phase is done.” The leader can switch to effective-config = C_new and use C_new to replicate the barrier.
When the barrier entry commits, the configuration change is complete

Restart behavior:

When a node restarts, it reads its log. It sees entry-i (C_old) and entry-j (C_new). It checks: is there a barrier after entry-j?

Barrier present: Joint phase ended. Set effective-config = C_new. No need to contact old nodes.
No barrier: Joint phase might still be active. Set effective-config = C_old_new.

Patch-3 adds a second log entry. We’re no longer doing “one log entry” configuration changes. We need “one config entry + one barrier entry.”

Conclusion

Configuration changes must pass through three states—C_old → C_old_new → C_new. One log entry gives us one bit of persistent information: C_old or C_new. That’s only two states. We can’t represent three states with two values.

To safely handle all three states, we need at least two log entries. That gives us two bits of information and up to four possible states, which is enough to encode the three states we actually need.

The “single-log-entry” approach, after all the patches, ends up needing two entries anyway—one for the configuration and one for the barrier. And it’s more complex than standard Joint Consensus, with trickier edge cases around restarts and state transitions.

Stick with Joint Consensus. It’s cleaner, simpler, and solves the problem directly without patches.

References

Diego Ongaro & John Ousterhout. In Search of an Understandable Consensus Algorithm (Raft paper): https://raft.github.io/raft.pdf
OpenRaft(rust): https://github.com/databendlabs/openraft
etcd/raft source code: https://github.com/etcd-io/raft
Hashicorp Raft implementation: https://github.com/hashicorp/raft

Reference:

马健将 : https://weibo.com/u/1516609505

Raft IO Execution Order (Revised)

2025-10-04T00:00:00+00:00

Preface

I need to come clean about something. In my previous article on IO ordering in Raft, I tried to demonstrate the dangers of “writing log entries before term” using a committed data loss scenario. The problem? That example was fundamentally flawed—it didn’t actually capture the real issue with IO reordering at all.

So let’s fix that. This article walks through what I got wrong and, more importantly, presents a correct understanding of when and why IO reordering becomes dangerous in Raft implementations.

What Went Wrong in My Original Analysis

Let me show you the timeline I used in the previous article:

Legend:
Ni:   Node i
Vi:   RequestVote, term=i
Li:   Establish Leader, term=i
Ei-j: Log entry, term=i, index=j

N5 |          V5  L5       E5-1
N4 |          V5           E5-1
N3 |  V1                V5,E5-1  E1-1
N2 |  V1      V5                 E1-1
N1 |  V1  L1                     E1-1
------+---+---+---+--------+-----+---------> time
      t1  t2  t3  t4       t5    t6
Here’s what I claimed would happen:

At t5: N3 receives entry E5-1 from leader L5 (term=5) and needs to persist both term=5 and E5-1

I argued: “If N3 writes E5-1 first but crashes before writing term=5, it could restart with term=1, entries=[E5-1]”

At t6: The old leader L1 (term=1) could then overwrite E5-1, causing data loss

Here’s the flaw in my reasoning: Raft’s protocol explicitly requires that both the term update and log entries must be successfully persisted before a follower responds with success. If either IO fails or is incomplete, the leader never receives confirmation and therefore never considers the entry committed. The Raft paper’s design is actually bulletproof here.

So if Raft’s design is correct, where does the IO ordering problem actually come from? The answer lies in a subtle gap between theory and implementation—specifically, how real Raft systems separate in-memory state from on-disk state.

The Real Culprit: In-Memory vs Persisted State

Here’s where things get interesting. The Raft paper describes a beautifully simple world where a server has just one state: what’s on disk. But real implementations need to be fast, so they introduce an optimization—they split their state into two layers:

In-memory state: The “optimistic” view that updates immediately when receiving RPCs Persisted state: The “durable” view that updates only after IO completes

Here’s how a typical implementation handles an appendEntries request:

Receive appendEntries RPC with req.term
If req.term > current_term, immediately update current_term to req.term
Asynchronously submit a save-term IO operation
Eventually update persisted_term when the IO completes

In code, this looks like:

struct RaftState {
    // In-memory term - may be ahead of what's on disk
    current_term: u64,

    // Persisted term on disk - the durable truth
    persisted_term: u64,
}

This separation is where the danger lurks. The Raft paper assumes only one “term” variable—what’s persisted on disk. But implementations now have two term values, and this introduces a behavior the paper never defined or analyzed.

The pattern above is ubiquitous in Raft implementations. And here’s the kicker: without IO reordering, it works perfectly fine. The bug only surfaces when IOs can complete out of order.

A Concrete Example: Where IO Reordering Breaks Raft

Let’s build a scenario that actually exposes the bug. I’ll walk you through it step by step:

Legend:
Ni:   Node i
Vi:   RequestVote, term=i
Li:   Establish Leader, term=i
Ei-j: Log entry, term=i, index=j

N5 |          V5  L5     E5-1     E5-2
N4 |          V5         E5-1     E5-2
N3 |  V1              V5,E5-1  V5,E5-2  E1-1
N2 |  V1      V5                        E1-1
N1 |  V1  L1                            E1-1
------+---+---+---+------+--------+-----+------> time
      t1  t2  t3  t4     t5       t6    t7

Here’s the sequence of events:

t1-t4: Two elections occur. First N1 becomes leader (term=1), then N5 becomes leader (term=5)
t5: Leader L5 sends its first entry E5-1 to follower N3
- N3’s current state: current_term=1, persisted_term=1
- N3 receives appendEntries(term=5, entries=[E5-1])
- N3 must persist both term=5 and entry E5-1
- N3 responds “success” only after both IOs complete
t6: Leader L5 sends a second entry E5-2 to N3 ← This is the critical moment
- N3 might still be waiting for t5’s IOs to complete
- Whether IO reordering can occur makes all the difference
t7: The old leader L1 (term=1) attempts to replicate E1-1 to N3

The bug manifests in what happens at t6—when the second AppendEntries arrives while the first one’s IOs are still in flight. Let’s zoom into N3’s internal state at each step.

At t5: The First AppendEntries

When N3 receives appendEntries(term=5, entries=[E5-1]), here’s what happens inside:

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check: Is the RPC term newer than our in-memory term?
    if req.term > self.current_term {
        self.current_term = req.term;           // Update memory immediately: 1 → 5
        self.submit_io(save_term(req.term));    // Queue IO to persist term=5
    }

    self.submit_io(save_entries(req.entries));  // Queue IO to persist E5-1

    // Wait for both IOs to complete before responding
    wait_for_both_ios();
    return success();
}

After this call executes, N3’s state looks like:

current_term = 5 (memory updated immediately)
persisted_term = 1 (disk not yet updated—IO still in flight)
IO queue: [save_term(5), save_entries(E5-1)] waiting to complete

So far, so good. This request is handled correctly—N3 won’t respond until both IOs finish. The trouble starts at the next moment.

At t6: The Second AppendEntries—Where Everything Goes Wrong

Now here’s the critical moment. Before t5’s IOs have completed, N3 receives a second request: appendEntries(term=5, entries=[E5-2]).

Most implementations check only the in-memory current_term to decide whether to persist the term. Watch what happens:

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check: Is 5 > 5? Nope!
    if req.term > self.current_term {
        // We skip this branch entirely
    }

    // We only queue the entries IO
    self.submit_io(save_entries(req.entries));

    // We only wait for the entries IO to complete
    wait_for_io(save_entries);
    return success();  // We're done!
}

See the problem? N3 returns success as soon as save_entries(E5-2) completes. But here’s the dangerous part: if IO reordering is allowed, the system might have:

✅ Completed save_entries(E5-2)
✅ Completed save_entries(E5-1)
❌ NOT completed save_term(5) (still in flight from t5)

N3 happily returns success to Leader L5, which then considers E5-2 replicated and potentially committed.

Now imagine N3 crashes. When it restarts, its disk state is:

persisted_term = 1 (the save_term(5) never finished)
entries = [E5-1, E5-2] (both successfully written)

This is an inconsistent state that Raft’s protocol assumes can never exist. And it’s about to cause data loss.

At t7: The Data Loss Materializes

After N3 restarts with term=1, entries=[E5-1, E5-2], the old leader L1 (from term=1) sends an appendEntries request: appendEntries(term=1, entries=[E1-1]).

N3’s logic:

Check: RPC term (1) == my local term (1) ✅
Accept the request
Write E1-1 at index=1, overwriting E5-1

The disaster: Entries E5-1 and E5-2, which Leader L5 believed were successfully replicated and possibly committed, have just been silently destroyed. We’ve lost committed data.

Important note: If IO reordering were not allowed, this bug wouldn’t occur. Here’s why: when save_entries(E5-2) completes at t6, it would guarantee that save_term(5) (queued earlier) has also completed. The sequential ordering ensures that N3’s disk state remains consistent, and the AppendEntries success response would be legitimate.

The Root Cause: A Mismatch Between Theory and Practice

Let’s crystallize what we’ve learned:

The core issue: When deciding whether to persist the term, should we check current_term or persisted_term?

If IO reordering is not allowed → checking current_term is safe
If IO reordering is allowed → we must check persisted_term

This isn’t obvious because the Raft paper never talks about in-memory vs persisted state—it only knows about one kind of state: what’s on disk. The paper says: “Before responding to RPCs, a server must update its persistent state.”

But in real implementations with in-memory and persisted state split, this requirement needs to be more precise:

Before returning success, we must ensure all IOs that make persisted_term >= req.term have completed.

Checking only current_term creates a window where we might respond successfully while the required disk updates are still in flight. If those updates can complete out of order, we’ve violated Raft’s safety guarantees.

How to Fix It: Check Persisted State, Not In-Memory State

If you need to support IO reordering, the fix is conceptually simple—check the on-disk term, not the in-memory term:

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check against disk state, not memory!
    let need_save_term = req.term > self.persisted_term;

    if need_save_term {
        self.current_term = req.term;
        self.submit_io(save_term(req.term));
    }

    self.submit_io(save_entries(req.entries));

    // Wait for the right IOs based on what we actually need
    if need_save_term {
        wait_for_both_ios();  // Must wait for term update to complete
    } else {
        wait_for_io(save_entries);  // Only need to wait for entries
    }

    return success();
}

By checking persisted_term instead of current_term, we correctly detect when the term IO is still in flight and wait for it to complete.

Caveat: This approach might submit multiple save_term(T) IOs for the same term T (if multiple AppendEntries arrive in quick succession). You’ll need to handle this carefully—either make the IO layer idempotent or add deduplication logic.

How Production Systems Solve This

Here’s the interesting part: most mature Raft implementations don’t actually support IO reordering. Instead, they eliminate the problem entirely by ensuring save-term and save-entries execute in order. This lets them safely check current_term without the bug we just analyzed.

Let’s look at three different approaches from production systems:

1. Atomic Batching (TiKV)

Strategy: Bundle save-term and save-entries into a single atomic IO operation.

When an AppendEntries requires both a term update and log writes, TiKV combines them into one batch and submits it as a single IO request. This makes it impossible for the entries to persist without the term—they’re literally the same operation.

This elegantly sidesteps the entire reordering problem. There’s no “second AppendEntries that only submits save_entries” scenario because term and entries are always written together.

2. Ordered Separation (HashiCorp Raft)

Strategy: Persist term and entries separately, but enforce strict ordering.

HashiCorp’s Raft implementation writes the term first (with fsync, panicking on failure), then writes the log entries. The key is that these operations execute sequentially—save_entries can’t start until save_term completes.

This guarantees that if entries reach disk, the term has definitely reached disk first. Sequential ordering prevents the reordering bug.

3. Hybrid Ordering (SOFAJRaft)

Strategy: Synchronous term writes, asynchronous batched log writes.

SOFAJRaft writes the term synchronously (blocking the current thread for fsync) but batches log entries for asynchronous writing. The crucial property: save_term always completes before save_entries is even enqueued.

This hybrid approach gets you most of the performance benefits of async IO while maintaining the ordering guarantee that prevents the bug.

Summary: Bridging Theory and Practice

The IO ordering bug in Raft implementations stems from a subtle gap between the paper’s abstract model and real-world code. The Raft paper assumes a single state: what’s on disk. Real implementations optimize with in-memory and persisted state split, introducing behaviors the paper never analyzed.

The invariant we must maintain:

If a log entry with term=T is on disk, then persisted_term ≥ T must also be on disk.

Violating this invariant—having entries from term T on disk while persisted_term < T—breaks Raft’s safety guarantees and can cause committed data loss.

Two ways to maintain the invariant:

Eliminate IO reordering (mainstream approach)
- Atomic batching: Write term and entries together
- Ordered execution: Guarantee term persists before entries
- Hybrid ordering: Synchronous term, async entries
Handle IO reordering explicitly
- Check persisted_term instead of current_term when deciding whether to persist the term
- Wait for all required IOs to complete before responding

Most production systems choose option 1—it’s simpler to reason about and avoids the complexity of tracking multiple in-flight term updates. But if you do need to support IO reordering, now you know where the dragons are hiding.

Reference:

OpenRaft docs: io-ordering : https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/protocol/io_ordering.md
hashicorp/raft : https://github.com/hashicorp/raft
The Hidden Danger in Raft Why IO Ordering Matters : https://blog.openacid.com/algo/raft-io-order/
sofastack/sofa-jraft : https://github.com/sofastack/sofa-jraft
tikv/tikv : https://github.com/tikv/tikv

OpenACID Blog

The Pitfalls of Raft Membership Change

The Problem

Looking at the Problem Through Quorum Sets

A First Patch: Let

Why Majority Runs Out of Room

How to Expand a Majority

What the Middle State Really Needs

The Correctness Conditions

Joint Consensus Gives Us Exactly That

The Bug in Single-Server Change

Look Closely at That Fix

Closing Thoughts

Build a Distributed Timestamp Oracle with EzRaft

What a Timestamp Oracle Must Guarantee

Safe Baseline: One Raft Write per Timestamp

Fast Path: Reserve Timestamps in Batches

Preserve Monotonicity Across Leader Failover

Reject a Stale Leader with a Lease

Serve

Bind Cached Ranges to a Leader Term

Run the Service

Start a Three-Node Cluster

Request Timestamps

Stop the Leader

What Raft Owns

Links

EzRaft: Build a Distributed KV Store in 100 Lines

EzRaft: Build a Distributed KV Store in 100 Lines

The OpenRaft Learning Curve

Why I Built EzRaft

The Four Things Users Define

Build a Distributed KV Store in Three Steps

Step One: Define the Application

Step Two: Assemble a Working Service

Step Three: Start a Three-Node Cluster

Send Writes to Any Node

Test a Node Failure

Cluster State

When to Use EzRaft

Links

Do Not Read: Raf Is a Useless and Valueless Failed Experiment

Introduction

Why Term Cannot Disappear

Core Idea

Storage Model

Why Split

Starting an Election

How a Voter Handles RequestVote

Establishing Leader State

Appending a Log Entry

Log Replication

Advancing Commit

Example

Current Boundaries

Summary

Histogram Done Right: 2KB Memory, 0.2% Error

The Problem: Tracking Request Latency Without Slowing Things Down

Recording: Getting Samples Into Buckets

Why Log-Scale Buckets

A Tempting Fix We Passed On

The Trick: Float-Like Encoding

Tuning WIDTH: The Precision–Memory Knob

Percentile Estimation: Getting Answers Out

Locating the Right Bucket

Trapezoid Interpolation (Our Approach)

Benchmarks: Seven Distributions, Six WIDTH Settings

Summary

xp 的 AI 工作流

工作流概览

第一步：分析与决策 — Rust Rover

第二步：描述需求 — 语音输入

闪电说

豆包桌面版

智谱语音输入法

第三步：代码生成 — Claude Code + GLM 4.7

第四步：审核与提交 — tig

为什么用 tig 而非

增量 Review

Vibe Coding 的核心理念