Summary: raf: Raft without [T]erm is an experimental Raft variant. It does not persist currentTerm as a separate piece of state. Instead, a candidate reserves a log index when it starts an election, and that index becomes the leader term. This does not remove Raft’s logical time model. It only changes where the term is derived from in storage.

Declaration: This approach is just the same as saving terms in the separate first extra slot in the terms array. So it actually still stores the term and has no value at all. Please do not read this as a useful design; it is only a failed personal experiment.

Note: The idea in this article came from Zhang Yanpo. The code was implemented by Zhang Yanpo by hand. This article was drafted and refined with Codex.

Repository: raf (v0.1.1).

Introduction

I have seen a few interesting proposals that try to remove the term from Raft. The idea is appealing: if a consensus protocol can maintain one less piece of persistent state, perhaps both the model and the implementation become simpler. But Raft’s term is not just a counter. It represents logical time, lets nodes distinguish old leaders from new ones, and participates in Raft’s commit safety rule. So the real question is not whether we can simply delete the term. The useful question is whether we can express it in a different way.

The name raf comes from Raft without [T]erm. Here, “without term” does not mean the protocol has no term at all. It means currentTerm is no longer persisted as an independent field. The project turns this idea into a small but serious implementation: avoid storing currentTerm separately, while still giving every part of Raft that needs a term a reliable source of logical time.

This article explains that core idea. The term still exists as a concept. Logs are still compared by (term, index). What changes is the source of the term: it no longer comes from a separately incremented persistent counter; it comes from a log index reserved by an election. The goal of this implementation is not to prove that it is a drop-in replacement for standard Raft in every engineering detail. The goal is to see whether this storage representation preserves the most important safety intuition behind Raft.

We will walk through the storage model, election, replication, commit, and the three-node example in the repository. I assume the reader is already familiar with the basic Raft flow: leader election, AppendEntries, quorum commit, and log ids in the form (term, index).

Why Term Cannot Disappear

In a consensus algorithm, the log records events that have been chosen or are being proposed. The term tells us which logical time those events belong to.

Standard Raft uses the term for several jobs:

• Leader election advances the term before choosing a leader. The term gives leaders an ordering.
• Because of that, log freshness is compared by term first, then by index.
• A leader may directly commit only entries from its own term.

This role is similar to the ballot number in Paxos. It lets the system decide which history is newer and which candidate is eligible to become leader, even when a node does not know every other node’s complete log.

In short: a log index is a local event position; a term is the logical time used to compare histories across nodes.

raf keeps the concept of term, but removes its separate storage.

Core Idea

Standard Raft usually persists state shaped roughly like this:

struct StandardRaftStorage {
    current_term: Term,
    voted_for: Option<NodeId>,
    log: Vec<LogEntry>,
}

struct LogEntry {
    term: Term,
    cmd: Cmd,
}

raf represents the persistent state as two Vecs aligned by log index:

struct RafStorage {
    terms: Vec<Term>,
    cmds: Vec<Cmd>,
}

Here:

• terms[i] is the leader term of the log entry at index i.
• cmds[i] is the application command of the log entry at index i.
• log_id(i) is still (terms[i], i).

The following is one possible storage state. ø means empty command. cmds only reaches index 7, so indexes 8 and 9 are not complete log entries yet.

Index by index, this state means:

• Index 0: the fixed default entry. terms[0] = 0, and cmds[0] is an empty command.
• Index 1: a successful election for term 1. The leader reserved index 1 when it was elected and wrote its first empty command there.
• Index 2: a successful election for term 2. The new leader reserved index 2; its first log entry is also an empty command.
• Index 3: a user log entry C3 written by the leader of term 2, so terms[3] = 2 and cmds[3] = C3.
• Index 4: this position used to be an election attempt for term 4, but it did not produce an established leader. Later, when the leader of term 6 was established, this position was filled as an empty command owned by term 6, so now terms[4] = 6.
• Index 5: this position used to be another failed election attempt for term 5. It was also later filled as an empty command owned by term 6, so now terms[5] = 6.
• Index 6: a successful election for term 6. The leader reserved index 6 when it was elected, and wrote its first empty command there.
• Index 7: a user log entry C7 written by the leader of term 6, so terms[7] = 6 and cmds[7] = C7.
• Index 8: a new election attempt for term 8. So far we have only seen terms[8] = 8; there is no command for this index yet, so it is not a complete log entry.
• Index 9: another election attempt for term 9. Like index 8, it currently has only a term record and no command. From this state alone, we cannot tell whether it will eventually become a leader.

Standard Raft persists a separate current_term and an array of (term, command) log entries. raf is similar, but splits the term and command at each index into two aligned Vecs.

Storage Model

Index 0 is a fixed default entry. This keeps the types simple and avoids using Option for the initial position:

terms[0] = 0
cmds[0]  = empty

When both Vecs have a value at the same index, that index is a complete log entry. Otherwise, the index has a term but no command. That state represents an election in progress: the term has been observed, but no log entry has been written at that position yet.

log[index] = (terms[index], cmds[index])
log_id     = (terms[index], index)

terms and cmds share the same log index. During election, terms may be ahead of cmds.

During an election, terms can temporarily be longer than cmds. A candidate first reserves an index as its term. Multiple failed elections may reserve multiple indexes. Only after some candidate becomes an established leader are the positions with missing commands filled with empty commands.

So this implementation maintains a few basic facts:

• cmds should never be longer than terms.
• When an election first reserves a term slot, terms[term] = term.
• When an established leader fills positions that still miss commands, those positions are rewritten to the current leader.term.

Therefore terms[i] <= i is not a protocol invariant. A backfilled empty entry may have terms[i] > i. That is not a problem by itself; standard Raft terms can also be greater than log indexes. The important property is different: every complete log entry must carry the term of an established leader, except for the fixed default entry at index 0.

Why Split

terms and cmds

This implementation separates leader terms and application commands into two Vecs. Raft’s protocol semantics do not require this layout. It is mainly a storage design choice that creates room for cleaner optimization.

For example, a leader may write many consecutive log entries during one term, and all of those entries have the same term. A storage engine could compress long runs of identical terms into a compact representation, while storing commands according to application needs. Once the two streams are separated, term compression, command persistence, and payload encoding can evolve independently.

This is the experimental value of the project. It expresses the relationship between “logical time” and “log position” in Raft more directly, then asks whether that expression can simplify persistent state.

Starting an Election

When a candidate starts an election, it uses the next index of the terms array as the new term:

let term = terms.len();
terms.push(term);

In this code:

• The candidate declares that it wants to use term as its leader term.
• The local persistent state records that this index has been reserved by an election.

The candidate then sends RequestVote. This part is the same as standard Raft. The request carries two important pieces of information:

• term: the leader term the candidate wants to use.
• last_log_id: the (term, index) of the candidate’s last complete log entry.

last_log_id is computed from the last index in cmds:

let last_log_index = cmds.len() - 1;
let last_log_id = (terms[last_log_index], last_log_index);

The meaning of last_log_id is the same as in standard Raft: the voter uses it to check whether the candidate’s log is at least as up to date as its own. Notice the separation here. The new election term comes from terms.len(), while last_log_id comes from cmds.len() - 1. These can point to different indexes.

In the following example, the complete log currently reaches only index 3, so last_log_id = (2, 3). When the candidate starts a new election, it uses terms.len() = 4 as the new term and first writes index 4 into terms. At this point, cmds still reaches only index 3, because this candidate has not become an established leader yet.

If the election for term 4 does not reach a quorum, it leaves only an observed term index in terms; it does not create a new command. The next election will use terms.len() again, which is now term 5.

At this point last_log_id is still (2, 3), because cmds has not moved beyond index 3. What changed is the candidate term: it advanced from 4 to 5. Only after an election succeeds and a leader is established does the system rewrite the positions with missing commands to this leader’s term and fill cmds with empty commands.

How a Voter Handles RequestVote

When a voter receives RequestVote, it checks three things:

1. Whether the requested term is greater than the last term observed locally.
2. Whether the requested term slot does not already exist locally.
3. Whether the candidate’s log, represented by last_log_id, is fresh enough. It must not be older than the voter’s own log.

Aside from the term check, the rest of the logic is standard Raft. The subtle difference is that the term is no longer stored independently, so term freshness is expressed through terms.

The core condition looks like this:

let local_last_log_index = cmds.len() - 1;
let local_last_log_id = (terms[local_last_log_index], local_last_log_index);
let local_last_term = terms[terms.len() - 1];

let can_vote =
    req.term > local_last_term
        && req.term >= terms.len()
        && req.last_log_id >= local_last_log_id;

The next diagram sends the same RequestVote { term: 5, last_log_id: (2, 3) } to voters with three different local states. The candidate’s own state is at the top. The three branches below it show how each voter makes its decision.

The three outcomes are:

• granted: the voter’s terms reaches only index 3, and cmds also reaches only index 3. Therefore req.term = 5 is greater than the last locally observed term, names a term slot that has not appeared locally, and req.last_log_id = (2, 3) is not behind the voter. The vote can be granted.
• rejected: term=7: the voter has already observed a later term 7. Since req.term = 5 is not greater than the last locally observed term, the candidate’s requested term is stale from this voter’s perspective, so the vote is rejected.
• rejected: last log id = (4,4): the voter’s last complete log entry is (4, 4), which is newer than the candidate’s (2, 3). Even if the requested term could be recorded, the log freshness check still fails, so the vote is rejected.

If the request is valid, the voter records the term in local terms. If local terms is shorter than req.term, it fills the missing positions with default indexes until local state contains index req.term:

if can_vote {
    while terms.len() <= req.term {
        let index = terms.len();
        terms.push(index);
    }
}

These default items have term values equal to their own indexes. They mean the node has observed the corresponding term indexes. They do not mean those indexes are complete log entries, because the matching cmds may not exist yet. If a later leader fills those positions as complete empty entries, it rewrites their terms to its own leader.term. On the final iteration, index == req.term, so the voter has observed and accepted that term. Later, it will not accept an older term or a term index that already exists locally.

This replaces the role of standard Raft’s persisted currentTerm, but it is not fully equivalent to standard Raft’s votedFor. The current implementation does not persist “which candidate this term was granted to.” As a result, RequestVote retry and restart behavior are more conservative. We will return to this trade-off in “Current Boundaries.”

The candidate chooses terms.len() as its term. Other nodes record that term in their local terms when they grant the vote.

After the candidate receives granted replies from a quorum, it becomes an established leader. It first rewrites the local cmds.len()..terms.len() range to its own leader.term, then appends empty commands so that cmds.len() catches up with terms.len(). The index reserved by the leader’s election becomes this leader’s first complete log entry.

Establishing Leader State

After a candidate becomes an established leader, it keeps the core state of this leadership in memory. You can think of it as the following structure:

struct LeaderState {
    term: Term,
    granted_nodes: Vec<NodeId>,
    replications: BTreeMap<NodeId, ReplicationState>,
}

struct ReplicationState {
    matched: LogIndex,
    end: LogIndex,
    inflight: bool,
}

The important fields are:

• term: the log index reserved when this leader was elected. All later log entries produced by this leader write this term into the corresponding positions of the terms array.
• granted_nodes: the node ids that granted this leadership. This proves the leader was chosen by a quorum.
• replications: the replication progress of each node from the leader’s point of view. matched is the largest index known to match on that node; end is the upper bound used for further probing or replication; inflight prevents sending multiple Append requests to the same node at the same time.

The leader also has its own replication state. This makes commit calculation uniform: look at which nodes have matched covering an index, then check whether those nodes form a quorum.

After the leader is established, every position that already exists in local terms but is still missing from cmds is taken over by the current leader: the term at that position is rewritten to leader.term, and the command is filled with an empty command. After that, every local index on the leader has a corresponding command, and new application writes can start at the next index.

In this example, index 4 used to be a term slot left behind by a failed election, and term 5 is the index reserved by the current leader. When the candidate for term 5 becomes an established leader, indexes 4 and 5 are both rewritten as term 5 empty log entries. The ø at index 5 is the entry reserved by this leader’s election; the ø at index 4 is the backfilled entry that keeps the log prefix contiguous.

Appending a Log Entry

After a node becomes leader, each new application write appends a log entry. The term does not change. It is still the term chosen when the leader was elected:

terms.push(leader.term);
cmds.push(user_cmd);

So within one leader term, all later log entries have the same terms[i]. This matches standard Raft behavior. Only the source of the term is different.

Log Replication

The leader sends Append requests to the other nodes. Conceptually, each request first names a previous log id that is already known to match, then carries a contiguous segment of log entries after that position:

struct Append {
    term: Term,
    prev_log_id: LogId,
    terms: Vec<Term>,
    cmds: Vec<Cmd>,
}

Here prev_log_id is the matching point. terms and cmds are the real entries after that point. They must have the same length, and their first item corresponds to prev_log_id.index + 1. The follower first checks whether its local log has the same LogId at prev_log_id.index. Only if that previous position matches does it accept the entries that follow.

This design is close to standard Raft’s AppendEntries. Standard Raft carries prevLogIndex and prevLogTerm separately; raf combines them into a single prev_log_id. That is clearer than using the first entry in the request as the matching point, because the request’s terms and cmds represent only the real entries to replicate.

The following diagram shows one Append request applied to several follower states. The request has term = 5, prev_log_id = (2, 3), and carries the consecutive log entries for indexes 4..=5.

The three outcomes are:

• accepted: the follower matches the leader at prev_log_id = (2, 3), so it can accept this Append. {...} marks the terms range overwritten by this request, as well as the command range appended because it was missing locally. Here the term at index 4 is updated from an old value to 5, and command c5 is appended at index 5.
• conflict at prev_log_id: * marks the conflict position. The follower’s term at index 3 is 3, while the request’s prev_log_id is (2, 3). Since the previous log id does not match, the follower returns a conflict index immediately. The leader must try again with an earlier prev_log_id.
• rejected: follower has newer term: the follower has already observed term 6 at index 6, while the Append request is from term 5. This request comes from a stale leader, so the follower rejects it without modifying the log.

The handling logic is:

1. If the request term is older than the last observed local term, reject it.
2. Check the follower’s local log at prev_log_id. If it does not match, return the conflict index.
3. If prev_log_id matches, process the real entries starting at prev_log_id.index + 1.
4. If later local commands diverge from the leader, truncate the local commands.
5. Overwrite the local terms range covered by this request.
6. Append only the commands that are missing locally.

Append first finds the common prefix with prev_log_id, then truncates the follower’s conflicting suffix, and finally copies the leader’s entries that the follower is missing.

This is still Raft’s core replication model: the leader finds a shared log prefix, then replaces the follower’s divergent suffix with its own.

Advancing Commit

Replication to a quorum does not mean every historical entry can be committed immediately. Standard Raft has an important rule: a leader may only directly commit log entries from its own current term. Entries from older terms become committed only as a consequence of committing an entry from the current term.

raf keeps this rule. Although an established leader rewrites earlier gap slots to its own term, the leader still only directly commits matched indexes that are not smaller than its election index. Empty entries backfilled before leader.term are committed only indirectly, together with the leader’s election index or a later entry.

Intuitively:

if quorum_has_matched(index) && index >= leader.term {
    commit(index);
}

The reason is the same as in standard Raft: once an index is committed, every future valid leader must contain it and must not overwrite it.

The following state shows why “replicated to a quorum” is not enough. The leader of term 6 has replicated the old log entries at indexes 4 and 5 to quorum A+B, but that quorum has not yet matched the leader’s own term index 6. Therefore indexes 4 and 5 still cannot be committed:

If a new leader for term 7 appears later, and its last_log_id=(5,6) is newer, it can overwrite those uncommitted log entries. In the diagram, {x} marks the range replaced by the new leader.

A leader only directly commits an index that is both covered by a quorum and inside the current leader’s term range.

Example

The repository includes a three-node in-process example that demonstrates the basic flow described in this article. It creates three Raf nodes, connects them with InProcessNetwork, explicitly triggers an election on node 1, and then writes a few log entries through the leader. Metrics show the role, term, commit index, and replication progress.

The example source is here:

https://github.com/drmingdrmer/raf/blob/v0.1.1/examples/three_node.rs

Run it from the repository root:

cargo run --example three_node

This example is not a production deployment template. It is a minimal demonstration for observing the core protocol state transitions. It logs to stderr and does not include an election timer, heartbeat, snapshots, or membership changes.

Current Boundaries

The current implementation is intentionally small. It leaves out several features that a complete production system would usually need:

• Automatic election triggering.
• Snapshots and log compaction.
• Membership changes.
• Heartbeats.
• RequestVote retry logic.
• Persistence semantics for application payloads.

Automatic election triggering corresponds to the election timer in standard Raft. A node periodically checks whether it has gone too long without seeing a valid leader. If it times out, it starts a new election. This can be implemented by an external timer that calls Raf::elect(). It does not need to live inside the core raf state machine, so the current implementation leaves it out.

RequestVote retry has a subtler boundary. Suppose a target node successfully handles a RequestVote, but the reply is lost in the network. If the candidate retries the same request, the target node has already recorded this term in terms[req.term]. Under the current rule, it rejects the retry because that term index already exists.

One optional fix is to add an in-memory voted_for field that records which candidate owns a term. Then a retry from the same candidate for the same term can be recognized and granted again. This field does not necessarily need to be persisted. If a node restarts and loses voted_for, it can conservatively reject every RequestVote that uses a term already present locally. That creates a small availability issue, but only after restart; it does not change the persisted relationship between logs and terms.

If a RequestVote reply is lost, the retry sees an already existing term. An optional in-memory voted_for field can improve availability in that case.

These features can all be added around the core model. This article focuses on the central question: if the term comes from the log index, can Raft election, replication, and commit still be expressed in the familiar way?

Summary

raf is not “Raft without any term.” It still has terms, and it still compares logs by (term, index). What it removes is the independently persisted currentTerm; the leader term is bound to the log index reserved by an election.

This change turns the storage state into two Vecs aligned by index:

struct RafStorage {
    terms: Vec<Term>,
    cmds: Vec<Cmd>,
}

During election, a candidate chooses terms.len() as its term. After it becomes leader, both the missing-command gap slots and later log entries use this leader term. Replication and commitment still follow the basic Raft rules.

That is the core of this experimental implementation: keep Raft’s logical time model, but change where that logical time comes from in persistent state.

Repository: raf (v0.1.1).

Reference:

• raf : https://github.com/drmingdrmer/raf/tree/v0.1.1

The Problem: Tracking Request Latency Without Slowing Things Down

When we were building Databend and OpenRaft, we ran into a familiar need: we wanted to see how request latency was distributed across the system, in real time, without burning CPU or memory to do it. This article explains the design behind base2histogram, the library we built to solve it.

Consider the life of a single Raft log entry. It passes through several stages, and each one has its own latency profile:

• Received → written to storage
• Persisted to local disk
• Replicated to remote nodes
• Acknowledged by a majority quorum
• Committed → applied to the state machine

A histogram is the natural tool here — plot latency on the x-axis, request count on the y-axis, and you get an immediate picture of where time is being spent.

This kind of visibility is what lets you find bottlenecks and fix the right thing.

But there’s a catch: collecting metrics can’t get in the way of doing actual work. So the histogram needs to be:

• O(1) to record — no sorting, no rebalancing, nothing that can stall a hot path
• Tiny in memory — the system may run hundreds or thousands of these at once
• Queryable for percentiles — P50, P95, P99

Let’s walk through how we designed one that hits all three.

Recording: Getting Samples Into Buckets

Why Log-Scale Buckets

Most requests cluster around some typical latency, with a few outliers on both ends. This is a log-normal distribution — take the log of the latency values, and the shape becomes a classic bell curve.

The signature look: a peak at lower values, then a gradual long tail stretching to the right.

To build a histogram, we divide the x-axis into buckets and count how many samples land in each one.

The key question is how to size those buckets. Equal-width buckets work great for a normal distribution, but latency is log-normal — the data only looks uniform on a logarithmic scale. So the buckets need to grow on a log scale, not a linear one.

The simplest version of this: each bucket is twice as wide as the one before it.

[0,1), [1,2), [2,4), [4,8), [8,16), ...

Why powers of 2? Because multiplying by 2 is free on a CPU, and mapping a value to its bucket takes a single leading zero count instruction.

Simulate a log-normal workload, plot the bucket counts with the bucket index on the x-axis (effectively a log transform), and the result is a clean bell curve:

Storage-wise, this is great — 65 buckets cover the entire u64 range. Resolution-wise, not so much. The last bucket spans half of all possible values. Everything that lands there is a blur.

A Tempting Fix We Passed On

An obvious improvement: use a smaller growth factor, like 1.1× instead of 2×. More buckets, finer resolution:

The problem is cost. Finding the right bucket for a value l means solving for the smallest x where 1 + 1.1 + 1.1^2 + ... + 1.1^x >= l — and that requires floating-point logarithms. That’s real overhead on a hot path.

We wanted to stay in the world of integers and bit operations.

The Trick: Float-Like Encoding

Here’s the idea that makes everything work. We keep the roughly exponential bucket sizes, but we encode each bucket using a fixed number of bits — a parameter we call WIDTH.

Think of a bucket’s lower bound as a tiny floating-point number. The MSB position gives you the exponent (which group of buckets you’re in), and the next few bits give you the offset within that group.

With WIDTH=3 (the default), a bucket boundary looks like this in binary:

00..00 1 xx 00..00
       |
       MSB
<- significant

The position of the leading 1 picks the group. The two bits that follow pick the bucket within the group.

Here’s what the first few groups look like — each bucket is fully described by just 3 bits:

WIDTH = 3:

range     bucket index        bucket size
[0, 1)     0  0b0 ..... 000    1
[1, 2)     1  0b0 ..... 001    1
[2, 3)     2  0b0 ..... 010    1
[3, 4)     3  0b0 ..... 011    1

[4, 5)     4  0b0 ..... 100    1
[5, 6)     5  0b0 ..... 101    1
[6, 7)     6  0b0 ..... 110    1
[7, 8)     7  0b0 ..... 111    1

[8, 10)    8  0b0 .... 1000    2
[10, 12)   9  0b0 .... 1010    2
[12, 14)  10  0b0 .... 1100    2
[14, 16)  11  0b0 .... 1110    2

[16, 20)  12  0b0 ... 10000    4
[20, 24)  13  0b0 ... 10100    4
[24, 28)  14  0b0 ... 11000    4
[28, 32)  15  0b0 ... 11100    4

[32, 40)  16  0b0 .. 100000    8
[40, 48)  17  0b0 .. 101000    8
[48, 56)  18  0b0 .. 110000    8
[56, 64)  19  0b0 .. 111000    8

The pattern is clean:

• Each group contains 2^(WIDTH-1) = 4 buckets
• The 2 bits after the MSB select the bucket within the group
• It’s a 3-bit float: 1 implicit leading bit + 2 fractional bits

Bucket sizes grow roughly logarithmically, and computing the bucket index is just a matter of extracting the top WIDTH bits — a handful of integer and bit ops. Recording a sample is O(1).

Walk-through with latency = 42:

value = 42 (binary: 0b101010)
  MSB position: 5
  group: 5 - 2 = 3
  2 bits after MSB: 01 (from 1[01]010)
  offset in group: 1
  Bucket index: 4 + (3 × 4) + 1 = 17

Tuning WIDTH: The Precision–Memory Knob

WIDTH sets how many buckets each group gets (2^(WIDTH-1)). Groups top out at 64 (they still double in size, covering the full u64 range).

Here’s how the trade-off plays out:

At the default WIDTH=3, one histogram costs 2 KB and records every sample in O(1).

That’s the write side sorted. Now for the read side.

Percentile Estimation: Getting Answers Out

Once we’ve collected the counts, we want percentiles: at what latency have 50% of requests finished (P50)? 90% (P90)? 99% (P99)?

Locating the Right Bucket

The basic idea is simple. For P50: count the total samples, take 50% to get a target rank p, then walk through the buckets from the start, accumulating counts until you pass p. That’s your bucket.

But a bucket spans a range, not a point. We still need to estimate where inside the bucket the percentile actually falls.

Here are a few ways to do that, from rough to precise. All error numbers below come from a log-normal distribution (API latency scenario), WIDTH=3, 1,000,000 samples.

Midpoint: just return (min + max) / 2. Many histogram libraries do this (e.g., iopsystems/histogram). It’s a blind guess — it ignores everything about how samples are distributed within the bucket.

Uniform interpolation: assume samples are spread evenly across the bucket (a flat rectangle), then interpolate linearly: estimate = min + (max - min) × rank / count.

Better than midpoint — at least it uses where in the bucket the target rank falls. But “evenly spread” is a rough assumption. Log-normal data is skewed even within a single bucket.

Trapezoid Interpolation (Our Approach)

Uniform interpolation treats the density inside a bucket as flat. In reality, it’s sloped — denser on the side closer to the peak of the distribution.

If we know which way the density tilts, we can swap the rectangle for a trapezoid and land much closer to the true value.

Each bucket stores only a count — we don’t want to add extra fields. So where does the slope information come from? The neighbors.

The densities of the left and right buckets tell us how the density slopes through the current bucket.

Here’s the recipe. Compute the average density of the left bucket: d0 = c0/(x1-x0), and treat it as the density at that bucket’s midpoint m0. Do the same for the right bucket: d2 = c2/(x3-x2) at midpoint m2. Assume density varies linearly from m0 to m2 — over this short range, that’s a reasonable approximation. This gives us the slope k.

Inside the target bucket, the density now forms a trapezoid: a sloped line with slope k, pinned so that the density at the bucket’s midpoint (x1+x2)/2 equals the bucket’s own average density d1 = c1/(x2-x1) (the midpoint of a linear function always equals its average).

To find the percentile, we solve for the x-position where the trapezoid’s area from x1 equals the target rank.

Same distribution, same buckets — here’s how it stacks up:

Two orders of magnitude better, with zero additional storage.

The three-bucket layout:

d0 = c0 / w0       -- left bucket density
d1 = c1 / w1       -- target bucket density
d2 = c2 / w2       -- right bucket density

Midpoints of the left and right buckets: m0 = (x0+x1)/2, m2 = (x2+x3)/2. Slope:

k = (d2 - d0) / (m2 - m0)

Then solve for the x-position where the trapezoid’s cumulative area from x1 equals the rank.

The whole thing runs on three counts and their bucket boundaries. Nothing else stored, nothing else needed.

Benchmarks: Seven Distributions, Six WIDTH Settings

We tested across 7 representative distributions with 1,000,000 samples each, using trapezoid interpolation.

The rows to watch are LN-API and LN-DB at W=3 — these are the real-world latency cases, running on the default 2 KB configuration:

|                   W=1      W=2      W=3      W=4      W=5      W=6
| ------------------------------------------------------------------
| Uniform P50    0.108%   0.028%   0.012%   0.018%   0.019%   0.002%
|         P95    2.317%   1.988%   1.035%   0.475%   0.005%   0.005%
|         P99    4.290%   4.129%   3.706%   1.486%   0.298%   0.162%
| 
| LN-API  P50    2.281%   0.182%   0.000%   0.000%   0.000%   0.000%
|         P95   20.256%   3.963%   0.080%   0.040%   0.040%   0.000%
|         P99   11.951%   3.594%   0.086%   0.000%   0.029%   0.000%
| 
| Bimodal P50    1.381%   0.394%   0.394%   0.197%   0.197%   0.197%
|         P95    3.918%   0.172%   0.012%   0.028%   0.038%   0.008%
|         P99    1.521%   1.344%   0.543%   0.078%   0.016%   0.014%
| 
| Expon   P50    1.012%   0.000%   0.145%   0.145%   0.145%   0.000%
|         P95   10.989%   0.200%   0.000%   0.000%   0.033%   0.033%
|         P99   18.665%   4.574%   0.824%   0.022%   0.022%   0.022%
| 
| LN-DB   P50    2.018%   0.034%   0.000%   0.000%   0.000%   0.034%
|         P95    2.027%   0.368%   0.039%   0.006%   0.019%   0.026%
|         P99    3.764%   1.066%   0.187%   0.007%   0.003%   0.062%
| 
| Sequent P50    0.095%   0.000%   0.000%   0.000%   0.000%   0.000%
|         P95    2.271%   1.967%   1.011%   0.496%   0.000%   0.000%
|         P99    4.272%   4.118%   3.696%   1.521%   0.305%   0.169%
| 
| Pareto  P50   10.127%   1.899%   0.633%   0.633%   0.633%   0.000%
|         P95    9.239%   0.272%   0.000%   0.136%   0.000%   0.000%
|         P99    3.517%   0.879%   0.231%   0.093%   0.046%   0.046%
| 
| ------------------------------------------------------------------
| Buckets            65      128      252      496      976     1920
| Mem/slot        520 B   1.0 KB   2.0 KB   3.9 KB   7.6 KB  15.0 KB
| Mem total      1.0 KB   2.0 KB   3.9 KB   7.8 KB  15.2 KB  30.0 KB

What each distribution models:

• Uniform (uniform distribution): synthetic benchmarks
• LN-API (log-normal σ=0.5): API and microservice latency
• Bimodal (bimodal distribution): cache hit/miss — 90% fast path ~500μs, 10% slow path ~50ms
• Expon (exponential distribution): network and I/O waits
• LN-DB (log-normal σ=1.0): database query latency, with a wider tail
• Sequent (sequential): adversarial worst case
• Pareto (Pareto distribution α=1.5): heavy-tailed workloads like request sizes

For the latency distributions we care about most — LN-API and LN-DB — WIDTH=3 delivers sub-0.2% error on 2 KB of memory.

Summary

• 2 KB memory (WIDTH=3, 252 buckets of u64), P50/P95/P99 error under 0.2% for log-normal latency
• O(1) recording, O(buckets) querying
• Trapezoid interpolation is what makes it work — over 10× more accurate than midpoint, with zero extra storage
• WIDTH is tunable: 520 B for bare-minimum tracking, up to 15 KB for maximum precision
• Code: base2histogram

Reference:

• Databend : https://github.com/databendlabs/databend

• MSB : https://en.wikipedia.org/wiki/Bit_numbering#Most_significant_bit

• OpenRaft : https://github.com/databendlabs/openraft

• Pareto distribution : https://en.wikipedia.org/wiki/Pareto_distribution

• base2histogram : https://github.com/drmingdrmer/base2histogram

• bimodal distribution : https://en.wikipedia.org/wiki/Multimodal_distribution

• exponential distribution : https://en.wikipedia.org/wiki/Exponential_distribution

• floating-point number : https://en.wikipedia.org/wiki/Floating-point_arithmetic

• histogram : https://en.wikipedia.org/wiki/Histogram

• iopsystems/histogram : https://github.com/iopsystems/histogram

• leading zero counting : https://en.wikipedia.org/wiki/Find_first_set#CLZ

• log scale : https://en.wikipedia.org/wiki/Logarithmic_scale

• log-normal distribution : https://en.wikipedia.org/wiki/Log-normal_distribution

• long tail : https://en.wikipedia.org/wiki/Long_tail

• normal distribution : https://en.wikipedia.org/wiki/Normal_distribution

• percentile : https://en.wikipedia.org/wiki/Percentile

• uniform distribution : https://en.wikipedia.org/wiki/Continuous_uniform_distribution

偏见声明：我更希望把精力放在问题分析上，所以市面上流行的工具并没有一一尝试。只要满足需要就不会替换，除非发现明显的效率瓶颈。

工作流概览

1. 分析与决策 — 在 IDE 中浏览代码，理解项目，决定要做什么
2. 描述需求 — 用语音输入把想法描述出来
3. 代码生成 — 交给 AI 命令行工具生成代码
4. 审核与提交 — 用 tig 逐行 review，增量提交到 Git

注意上面的几个步骤一般是穿插进行的, 一个这样的修改看做一个 session 的话, 一般我会同时做 2, 3 个工作流 session, 这是因为大模型输出代码比我 review 的速度慢一些, 我自己经常会有 IO 等待, 所以同时开 2,3 个 session 可以把我的 CPU 占满. 但是切换任务也会让自己脑袋里的 context 频繁切换, 降低效率, 所以一般我开 1 个需要我仔细思考的困难任务(例如增加新 feature), 和 1 或 2 个不太需要深度思考的简单任务(例如代码重构), 就不会产生大量颅内 context 切换. 而且忙起来似乎有助于多巴胺分泌.

第一步：分析与决策 — Rust Rover

Rust Rover 是 JetBrains 开发的 Rust IDE。JetBrains 的 IDE 系列（IntelliJ IDEA、PyCharm、WebStorm 等）在 AI 时代之前一直是开发者的首选，以代码跳转、重构、调试等功能著称。

虽然 Rust Rover 在 AI 功能上相对保守，但传统的代码跳转、全局搜索等功能依然扎实，足以快速建立对项目的整体认知。

在这个阶段，我主要根据要做的事情回顾代码结构，确定具体的实现思路；或者在没有具体任务时浏览项目，寻找需要改进的地方。

第二步：描述需求 — 语音输入

口述比打字快，语音输入法是重要补充。

语音输入的错误可被大模型理解能力覆盖——只要大致表达清晰，就能完成任务。因此我对语音输入的准确率要求不高，即使口述转文字时有些错误也没关系。只要描述足够详细、提供足够的冗余信息，大模型就能准确理解需求。

以下是几种我尝试过、都能满足需要的语音输入法：

闪电说

（本地模型）

• 识别准确率中等
• 离线可用，响应快

豆包桌面版

• 在线识别更精准
• 网络延迟所以稍慢

智谱语音输入法

• AI 矫正能力强，准确率高
• 网络延迟所以稍慢

第三步：代码生成 — Claude Code + GLM 4.7

Claude Code 是我的主力工具，CLI 框架成熟，即便不用内置模型，作为命令行交互载体也很高效。

日常开发中，我直接口述需求给 Claude Code，全程不需要键盘打字。

为了简化开发流程（也为了好玩），我买了一个只有三个键的小键盘：一个键触发语音输入，一个键是删除，一个键是回车。目前正在尝试只用这三个键完成日常开发。

模型用的是 GLM 4.7（智谱 AI 发布）。需求描述清晰的话，完成”文字到代码”的转化没问题。与顶尖模型相比，处理抽象需求时有差距，但性价比高，国内网络环境下响应快。

就算说的乱 78 糟, 大模型也知道我说的 IOarrow 是 io::Error:

我已完全摒弃交互式辅助模式，改用”全量需求描述 + AI 独立实现”。

第四步：审核与提交 — tig

我发现最高效的工作方式不是等大模型写完 review, 而是：在大模型生成代码的过程中就开始 review 产生的变更，逐段将变更加入版本控制。确认一个变更正确且符合要求后，立即加入 Git stage(git add)。这样后续修改时, review 工作区变化可以忽略已确认(git add)的部分，效率很高。

因此这个阶段最需要的工具，是能够高效的逐段将代码加入 Git 的交互式工具。git add -p 是最基本的选择，我使用的是 tig。

tig 是基于 ncurses 的 Git 文本界面工具。

为什么用 tig 而非

git add -p

git add -p 只能显示固定的 2-3 行上下文，有时不够理解修改(我的 context 太小 🤔)。

tig 通过 tig status 后进入交互界面, 可以：

• 逐行查看差异，像 Vim 一样导航(j/k)
• 快捷键[ 和 ] 随时调整 diff 上下文大小
• 逐行(1)或逐块(u)将修改加入 Git stage(stage 部分也叫 cached 或 index)
• R(shift-r) 刷新页面, 显示最新修改

一般我习惯在 tmux 里左右分两屏, 右边 pua 大模型干活, 左边逐行确认修改. 因为有时大模型修改的滚动太快了, 看不清它到底做了什么. tig 的逐行交互式 review 容许我异步查看每一行修改, 一旦看到有问题的修改, 就可以及时切到右边叫停努力的大模型, 重新调整:

可以看到 tig 的上下文可以用[和]任意宽度展开

Vim 的 Git fugitive 插件也可以提供类似功能但需要先启动 Vim，而且操作稍繁琐。

增量 Review

确认 OK 的部分立即加入 Git，后续再次让大模型做出修改后只 review not-staged 部分, 不再重复审查上一步已经加入 stage, 确认 OK 的代码。AI 增量修改时只需关注变化的部分。

Vibe Coding 的核心理念

AI 能极大提升效率——前提是: 你了解这个领域。否则无法判断对错，错误会吃掉所有效率提升。用 AI 开发的前提是我比 AI 更了解这个项目。

工具清单

这篇文章是口述完成的。

Reference:

• Claude Code : https://github.com/anthropics/claude-code

• GLM 4.7 : https://open.bigmodel.cn/

• Rust Rover : https://www.jetbrains.com/rust/

• tig : https://github.com/jonas/tig

• 智谱 AI : https://www.zhipuai.cn/

• 智谱语音输入法 : https://autoglm.zhipuai.cn/autotyper/

• 豆包 : https://www.doubao.com/

• 闪电说 : https://shandianshuo.cn/

In Raft cluster operations, there’s an easily overlooked bug: when a node is removed and then re-added to the cluster within the same term, delayed AppendEntries responses from the old membership configuration can corrupt the leader’s replication progress tracking for that node, causing the leader to enter an infinite retry loop.

The root cause is the lack of a replication session isolation mechanism. When the same node joins the cluster at different times, these should be treated as different replication sessions. However, without explicit session identifiers, the leader cannot distinguish which session a response belongs to. The result is that delayed responses from old sessions incorrectly update the progress records of new sessions.

While this creates operational challenges—continuous resource consumption and nodes unable to catch up with the cluster—the good news is that Raft’s commit protocol ensures data safety remains intact.

Analyzed Raft libs:

This article uses raft-rs, the Raft implementation used by TiKV, as a case study to analyze this bug’s trigger conditions, impact, and potential solutions.

Complete analysis and survey of other Raft implementations can be found in the Raft Rejoin Bug Survey

Raft Log Replication Basics

In Raft, the leader replicates log entries to followers through AppendEntries RPC calls, while maintaining a replication state machine for each follower to track replication progress.

AppendEntries Request-Response Flow

Here’s how it works: The leader sends AppendEntries requests with the current term, the prev_log_index and prev_log_term pointing to the position just before the new entries, the entries[] array to replicate, and the leader’s leader_commit index. The follower responds with its own term, the highest log index it replicated, and whether the operation succeeded.

Progress Tracking

The leader relies on these responses to track each follower’s replication status. It uses matched to record the highest log index confirmed to be replicated on that follower, and next_idx to mark where to send next. When a successful response comes back with index=N, the leader updates matched=N and calculates next_idx=N+1 for the next round.

This tracking mechanism has an implicit assumption: responses correspond to the current replication session.

If this assumption isn’t handled properly, when a node rejoins the cluster, the leader can get stuck in an infinite retry loop. It keeps sending AppendEntries requests, the node keeps rejecting them, and the cycle repeats endlessly while that node never manages to catch up with the cluster.

raft-rs Progress Tracking

raft-rs tracks replication progress using a Progress structure for each follower node:

// From raft-rs/src/tracker/progress.rs
pub struct Progress {
    pub matched: u64,      // Highest log index known to be replicated
    pub next_idx: u64,     // Next log index to send
    pub state: ProgressState,
    // ... other fields
}

The matched field records the highest log index successfully replicated to this follower. Whenever the leader receives a successful AppendEntries response, it updates this field:

// From raft-rs/src/tracker/progress.rs
pub fn maybe_update(&mut self, n: u64) -> bool {
    let need_update = self.matched < n;  // Only check monotonicity
    if need_update {
        self.matched = n;  // Accept the update!
        self.resume();
    }
    need_update
}

Notice the update logic is quite simple: as long as the new index is higher than the current matched, it accepts the update. When a node gets removed from the cluster, its Progress record is deleted. When it rejoins, a brand new Progress record is created with matched = 0.

Bug Reproduction Sequence

Let’s walk through a concrete timeline to see how this bug unfolds. Pay special attention to the fact that all events happen within a single term (term=5)—this is key to understanding why term-based validation fails.

Event Timeline

| Time | Event                                         | Progress State
|------|-----------------------------------------------|----------------
| T1   | log=1, members={a,b,c}                        | C: matched=0
|      | Leader sends AppendEntries(index=1) to C      |
|      | (Network delay causes slow delivery)          |
|      |                                               |
| T2   | log=5, members={a,b}                          | C: [deleted]
|      | Node C removed from cluster                   |
|      | Progress[C] deleted from leader's tracker     |
|      |                                               |
| T3   | log=100, members={a,b,c}                      | C: matched=0 (new)
|      | Node C rejoins the cluster                    |
|      | New Progress[C] created with matched=0        |
|      |                                               |
| T4   | Delayed response arrives from T1:             |
|      | {from: C, index: 1, success: true}            |
|      | Leader finds Progress[C] (the new one!)       |
|      | maybe_update(1) called: 0 < 1, so update!     | C: matched=1 ❌
|      |                                               |
| T5   | Leader calculates next_idx = matched + 1 = 2  |
|      | Sends AppendEntries(prev_index=1)             |
|      | Node C rejects (doesn't have index 1!)        |
|      | Leader can't decrement (matched == rejected)  |
|      | Infinite loop begins...                       |

Response Handling at T4

At time T4, that response sent at T1 and delayed in the network finally arrives. Here’s how the leader handles it:

// From raft-rs/src/raft.rs
fn handle_append_response(&mut self, m: &Message) {
    // Find the progress record
    let pr = match self.prs.get_mut(m.from) {
        Some(pr) => pr,
        None => {
            debug!(self.logger, "no progress available for {}", m.from);
            return;
        }
    };

    // Update progress if the index is higher
    if !pr.maybe_update(m.index) {
        return;
    }
    // ...
}

Here’s where things go wrong: The leader does find a Progress record for node C, but it’s the new one created at T3. Since the message’s term matches the current term, it passes the term check in the step() function, and the leader updates progress with this stale index value.

Root Cause Analysis

The root of this bug is that request-response messages lack replication session identification. When node C gets removed at T2 and rejoins at T3, these should be two distinct replication sessions—but the leader has no way to distinguish between responses from requests sent at T1 versus responses from requests sent after T3.

Look at raft-rs’s Message structure:

File: proto/proto/eraftpb.proto:71-98

message Message {
    MessageType msg_type = 1;
    uint64 to = 2;
    uint64 from = 3;
    uint64 term = 4;        // Only term, no session identifier!
    uint64 log_term = 5;
    uint64 index = 6;
    // ...
}

The Message only has a from field identifying the sending node, but the same node ID joining the cluster at different times should be treated as different replication sessions. The leader needs to distinguish: is this response from node C’s first session or its second session? But the current Message structure provides no way to tell.

Impact Analysis

Infinite Retry Loop

Once the leader incorrectly sets matched=1, trouble begins. Here’s what happens:

// From raft-rs/src/tracker/progress.rs
pub fn maybe_decr_to(&mut self, rejected: u64, match_hint: u64, ...) -> bool {
    if self.state == ProgressState::Replicate {
        // Can't decrement if rejected <= matched
        if rejected < self.matched
            || (rejected == self.matched && request_snapshot == INVALID_INDEX) {
            return false;  // Ignore the rejection!
        }
        // ...
    }
}

The leader sends AppendEntries with prev_log_index=1, but node C’s log is empty—it doesn’t have index 1. Node C rejects the request. The leader wants to decrement next_idx to retry an earlier position, but here’s the problem: because rejected (1) == matched (1), the decrement logic returns false and refuses to decrement. So the leader just sends the same request again, node C rejects it again, and this cycle continues forever.

Operational Impact

This bug creates a series of operational problems. First, there’s resource exhaustion: the continuous AppendEntries-rejection cycle keeps consuming CPU and network bandwidth.

Why Data Remains Safe

Despite all the operational chaos, there’s good news: data integrity remains intact. Raft’s safety properties ensure that even with corrupted progress tracking, the cluster won’t lose any committed data.

The reason is that commit index calculation still works correctly. Even if the leader mistakenly thinks node C has matched=1, it calculates the commit index based on the actual majority. For example, node A has matched=100, node B has matched=100, and node C has matched=1 (which is wrong, but doesn’t matter). The majority looks at A and B with matched=100, so the commit index is correctly calculated as 100. Combined with Raft’s overlapping majorities property, any newly elected leader will necessarily have all committed entries, keeping data safe.

Solutions

Solution 1: Add Membership Version (Recommended)

The most straightforward fix is to add a membership configuration version to messages:

message Message {
    // ... existing fields
    uint64 membership_log_id = 17;  // New field
}

Then validate it when processing responses:

fn handle_append_response(&mut self, m: &Message) {
    let pr = self.prs.get_mut(m.from)?;

    // Check membership version
    if m.membership_log_id != self.current_membership_log_id {
        debug!("stale message from different membership");
        return;
    }

    pr.maybe_update(m.index);
}

This directly fixes the root cause—the leader can now tell which membership configuration a message comes from.

Solution 2: Generation Counters

Another approach is to add a generation counter to Progress that increments each time a node rejoins:

pub struct Progress {
    pub matched: u64,
    pub next_idx: u64,
    pub generation: u64,  // Incremented on each rejoin
    // ...
}

Include the generation in messages and validate it when responses arrive. This is lighter weight than solution 1, but you need to carefully manage the generation lifecycle.

Summary

This bug shows us that when membership changes happen within the same term, relying on term-based validation alone isn’t enough to ensure message freshness. Without explicit session isolation, delayed responses from old membership configurations can corrupt progress tracking.

Fortunately, because Raft’s commit index calculation and overlapping quorum mechanisms provide strong guarantees, this bug doesn’t compromise data safety. The main impact is operational—the symptoms look like data corruption, which can send operations teams down the rabbit hole investigating a data loss problem that doesn’t actually exist.

For production Raft implementations, it’s recommended to introduce explicit session management mechanisms. This can be achieved through membership versioning or generation counters. The most recommended approach is to add a membership_log_id field to messages, which lets the leader clearly distinguish which membership configuration a response comes from.

Complete analysis and survey of other Raft implementations can be found in the Raft Rejoin Bug Survey

Reference:

前言

在 Raft 实现中，处理 appendEntries 请求时需要持久化两类数据：term 和 log entries。Raft 论文要求”在响应 RPC 之前必须更新持久化状态”，但并未明确说明这两类数据的持久化顺序。这个看似无关紧要的细节，却可能导致已提交数据的丢失。

问题的根源在于：Raft 论文描述的是一个简单的抽象模型（只有磁盘状态），而实际实现为了性能会分离内存状态和持久化状态。这种状态分离引入了论文中未定义的行为，当 IO 操作允许重排序时，就可能破坏 Raft 的安全性保证。

本文将深入分析这个问题是如何产生的，以及主流实现（TiKV、HashiCorp Raft、SOFAJRaft）如何避免这个陷阱。

内存状态与持久化状态的陷阱

在实际的 Raft 实现中，为了提升性能，通常会分离内存状态(current_term)和磁盘状态(persisted_term)。处理 appendEntries 请求的典型流程是：

1. 收到 appendEntries，如果 req.term > current_term，立即更新 current_term
2. 异步提交 save-term IO
3. IO 完成后更新 persisted_term（有些实现中可能没有显式的 persisted_term）

这种状态分离引入了 Raft 论文中没有定义的行为（Raft 论文只关注磁盘状态）：

struct RaftState {
    // In-memory term, updated immediately when receiving higher term
    current_term: u64,

    // Persisted term on disk, updated only after IO completes
    persisted_term: u64,
}

上面描述的流程是常见的 Raft 实现的流程, 在没有 IO-reorder 时, 它是正确的。但当 IO 操作可以重排序时，就会出现严重的安全问题。

问题场景

用一个具体的时间线来展示 IO-reorder 如何导致数据丢失：

Legend:
Ni:   Node i
Vi:   RequestVote, term=i
Li:   Establish Leader, term=i
Ei-j: Log entry, term=i, index=j

N5 |          V5  L5     E5-1     E5-2
N4 |          V5         E5-1     E5-2
N3 |  V1              V5,E5-1  V5,E5-2  E1-1
N2 |  V1      V5                        E1-1
N1 |  V1  L1                            E1-1
------+---+---+---+------+--------+-----+------> time
      t1  t2  t3  t4     t5       t6    t7

• t1-t4: 两次选举，N1（term=1）和 N5（term=5）先后成为 leader
• t5: L5 复制 E5-1 到 N3（N3 的 current_term=1 < req.term=5）
  • N3 需要执行两个 IO：持久化 term=5 和 E5-1
  • 等待两个 IO 完成才返回成功
• t6: L5 复制 E5-2 到 N3（关键时刻）
  • N3 可能还在处理 t5 的 IO
  • 这时是否存在 IO-reorder 至关重要
• t7: L1 尝试复制 E1-1（term=1, index=1）

关键在于 t6 时刻的第二个 AppendEntries 请求。让我们看看 N3 的内部状态变化。

t5 时刻：第一个 AppendEntries

N3 收到 appendEntries(term=5, entries=[E5-1])：

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check: RPC term > in-memory term?
    if req.term > self.current_term {
        self.current_term = req.term;           // Update memory immediately: 5
        self.submit_io(save_term(req.term));    // Submit IO request
    }

    self.submit_io(save_entries(req.entries));  // Submit IO request

    // Wait for both IOs to complete
    wait_for_both_ios();
    return success();
}

N3 的状态：

• current_term = 5（内存已更新）
• persisted_term = 1（磁盘还未更新，IO 进行中）
• IO 队列：save_term(5), save_entries(E5-1)

这个请求本身是正确的，问题出现在下一个时刻。

t6 时刻：第二个 AppendEntries

N3 还没完成 t5 的 IO，就收到了 appendEntries(term=5, entries=[E5-2])。

如果代码只检查内存 current_term（大多数实现的做法）, 并提交 save-entries IO：

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check: 5 > 5? No
    if req.term > self.current_term {
        // Won't enter this branch
    }

    // Only submit save_entries(E5-2)
    self.submit_io(save_entries(req.entries));

    // Only wait for save_entries to complete
    wait_for_io(save_entries);
    return success();  // Return success!
}

问题出现：在允许 IO-reorder 的时候,

• save_entries(E5-2) 完成
• 但 save_term(5) 可能还没完成（如果存在 IO 重排序）
• N3 向 Leader 返回成功

如果 N3 此时崩溃重启，磁盘状态可能是：

• persisted_term = 1（save_term(5) 未完成）
• entries = [E5-1, E5-2]（都完成了）
• Leader L5 认为 E5-2 已提交

t7 时刻：数据丢失

重启后 N3 的磁盘状态：term=1, entries=[E5-1, E5-2]

当 L1 发送 appendEntries(term=1, entries=[E1-1])：

• N3 检查：RPC term (1) == 本地 term (1)，接受
• E1-1 覆盖 index=1
• 已向 L5 确认提交的 E5-1 和 E5-2 被覆盖

注意, 如果不允许 IO-reorder, 那么 t6 的 save_entries(E5-2) 的完成就暗示了 save_term(5) 的完成, 满足了 appendEntries 成功的条件, 不会出现问题.

问题的本质

如果允许 IO-reorder，必须检查 persisted_term 来判断是否下发 save-term IO；如果不允许 IO-reorder，检查 current_term 即可。

Raft 论文不区分内存状态和持久化状态，这是实现相关的陷阱。论文要求 “Before responding to RPCs, a server must update its persistent state”，在实现中需要更精确的表述： 必须等待所有使 persisted_term >= req.term 的 IO 完成后，才能返回成功。

正确的做法

检查持久化的 term 而不是内存 term：

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check persisted term, not in-memory term!
    let need_save_term = req.term > self.persisted_term;

    if need_save_term {
        self.current_term = req.term;
        self.submit_io(save_term(req.term));
    }

    self.submit_io(save_entries(req.entries));

    if need_save_term {
        wait_for_both_ios();  // Must wait for save_term to complete
    } else {
        wait_for_io(save_entries);
    }

    return success();
}

注意：这种实现可能多次提交 save-term IO，需要在实现中谨慎优化。

主流实现的方案

主流实现（TiKV、HashiCorp Raft、SOFAJRaft）通过限制 save-term 和 save-entries 不能 reorder，因此只检查 current_term 也是安全的：

1. 原子批处理（TiKV）：将 save-term 和 save-entries 放到一个 IO 请求里，一次性提交。这样根本不存在”第二个 appendEntries 只提交 save_entries”的情况。

2. 有序分离（HashiCorp Raft）：save-term 和 save-entries 顺序执行，不会重排序。先完成 term 的 fsync（失败则 panic），再写 log。

3. 混合顺序（SOFAJRaft）：term 同步写入（阻塞等待 fsync），log 异步批处理。保证了 save_term 完成后才会入队 save_entries。

总结

Raft 论文的抽象模型（只关注持久化状态）和实际实现（内存状态 + 持久化状态）之间存在微妙的映射关系。

关键不变式：log entry (term=T) 在磁盘 → persisted_term ≥ T 也必须在磁盘

维护此不变式的两种方式：

1. 消除 IO-reorder：原子批处理、有序执行或混合方式（主流实现）
2. 处理 IO-reorder：检查持久化状态，等待必要的 IO 完成

相关资源

• OpenRaft docs: io-ordering
• tikv/tikv
• hashicorp/raft
• sofastack/sofa-jraft

Reference:

• OpenRaft docs: io-ordering : https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/protocol/io_ordering.md

• hashicorp/raft : https://github.com/hashicorp/raft

• sofastack/sofa-jraft : https://github.com/sofastack/sofa-jraft

• tikv/tikv : https://github.com/tikv/tikv

Preface

TL;DR

Standard Raft configuration changes use two log entries with multi-phase commits and careful state management. Can we complete a configuration change with just one log entry? We’ll introduce effective-config, prove its correctness, then discover why the simple approach isn’t so simple after all. The standard Joint Consensus method wins for good reasons.

What We’ll Cover

1. How Raft’s Joint Consensus works (the two-phase approach)
2. The single-log-entry idea and its mechanics
3. Why it’s theoretically correct
4. Why it’s practically problematic (and the patches we’d need)
5. Why we should stick with Joint Consensus

Introduction to Raft Joint Consensus: 2 Config Log Entries

Changing cluster membership in Raft is tricky. Switching from the old configuration {a,b,c} to a new one {x,y,z} in one step is dangerous.

Nodes can’t all switch configurations at the exact same moment. During the transition, some nodes (say a,b) might still be using C_old while others (x,y,z) have moved to C_new. If these two groups don’t overlap—meaning a quorum from C_old (like {a,b}) and a quorum from C_new (like {x,y}) share no common nodes—we could elect two leaders in the same term, violating Raft’s fundamental safety guarantee.

The Raft paper solves this with a two-phase protocol called Joint Consensus:

1. Phase 1: Enter the Joint phase (C_old_new) When the leader receives a configuration change request, it writes a log entry containing C_old_new—a joint configuration that includes both old and new members. In this state, any decision (like committing a log entry) needs approval from a quorum of C_old and a quorum of C_new. The leader starts using C_old_new as soon as it writes this entry to its own log.

2. Phase 2: Move to the new configuration (C_new) Once C_old_new commits, the leader writes a second log entry containing just C_new. From this point forward, the leader uses only C_new, and all subsequent log entries need only commit on a C_new quorum. When this second entry commits, the configuration change is complete.

The intermediate joint phase ensures that any two quorums—whether based on C_old, C_new, or C_old_new—must overlap, preventing split brain. This requires two log entries for each configuration change.

Can We Do It With Just One Log Entry?

Can we do this safely with just one log entry?

We need a new concept: effective-config. This is the configuration the leader actually uses to determine if log entries are committed. It might not match any specific configuration stored in a log entry—it’s a runtime state that changes as the configuration change progresses.

Terminology

• effective-config: The runtime configuration the leader uses to determine if entries are committed
• Joint config: A configuration containing both old and new members, like C_old_new = [{a,b,c}, {x,y,z}]
• Uniform config: A configuration with just one set of members, like C_new = {x,y,z}
• Barrier entry: A marker log entry that signals the joint phase has safely ended

How It Works

• Starting point: The cluster is running with C_old = {a,b,c}, and that configuration has been committed. The effective-config is C_old.

  

• Propose the change: To change to C_new = {x,y,z}, the leader writes a single log entry entry-i containing just C_new.

• Enter joint mode immediately: The moment the leader appends entry-i to its own log—before it commits, before it replicates—the leader switches its effective-config to the joint configuration C_old_new = [{a,b,c}, {x,y,z}]. Now entry-i and all subsequent entries must commit on a quorum from both {a,b,c} and {x,y,z}.

  

• Normal operation continues: The cluster keeps processing requests. Every entry commits using the joint quorum rules.

• Exit joint mode: Once entry-i commits under C_old_new, the leader switches effective-config to C_new = {x,y,z}. All subsequent entries need only a C_new quorum.

With one log entry, the system transitions through three states: C_old → C_old_new → C_new.

Correctness Proof

We need to show that we can’t elect two leaders—neither during the configuration change nor afterward.

Assume leader t is doing the configuration change (writing entry-i). Later, some candidate u tries to get elected in term u > t. We prove t and u can’t both be leaders.

Analyzing candidate u’s election

Candidate u either has entry-i in its log or it doesn’t.

• Case 1: u has entry-i

  Then u’s effective-config includes {x,y,z}. Leader t’s effective-config is either C_old_new = [{a,b,c}, {x,y,z}] (still in joint mode) or C_new = {x,y,z} (finished). Either way, it includes {x,y,z}.

  Since u needs a quorum from {x,y,z} to get elected, and t needs a quorum from {x,y,z} to stay leader, these quorums must overlap. No split brain.

• Case 2: u doesn’t have entry-i

  Then u’s effective-config is C_old = {a,b,c}. Now we consider where leader t is:

  • If t’s effective-config is C_old_new, then t needs a quorum from {a,b,c} and u needs a quorum from {a,b,c}. These must overlap. No split brain.

  • If t’s effective-config is C_new = {x,y,z}, that means entry-i committed under C_old_new. So entry-i must exist on a quorum of {a,b,c}. Those nodes have logs at least as long as index i.

    But u doesn’t have entry-i, so its log is shorter than i. When u requests votes from nodes in {a,b,c}, they’ll reject it because their logs are more up-to-date. The election fails.

In every case, we can’t have both t and u as leaders. The algorithm is safe.

However, although theoretically correct, it introduces problems in actual implementation:

Problem 1: The Memory-Only Transition

When we move from C_old_new to C_new, we only change the in-memory effective-config. Nothing hits disk. This creates trouble.

Nodes from C_old can still initiate elections and compete with C_new nodes, because C_old logs are as long as C_new logs. Even after the configuration change completes, C_old nodes can steal leadership from C_new nodes. The root cause is that the state change is not recorded on the persistent layer. This is problematic because nodes intended for removal can still become leaders.

Compare this to standard Joint Consensus: it writes a second log entry containing C_new. That entry acts as a barrier. Nodes from C_old have shorter logs and lose elections. The single-entry approach has no such barrier—the transition from C_old_new to C_new is invisible on disk.

Look at the diagram below. The cluster transitions from C_old_new to C_new, but no logs change. Leadership moves to node x in {x,y,z}. But nodes from C_old can still start elections and steal leadership from x.

Patch-1: After entering C_new, immediately append a no-op entry. This lengthens the logs of C_new nodes, blocking elections from C_old nodes.

Problem 2: The Restart Ambiguity

When a node restarts, it can’t tell if the cluster is in joint mode or has finished the change.

• The restarting node reads its log. It sees entry-i containing C_old and entry-j containing C_new.

• We know entry-i is committed (Raft requires it before starting a new change).

• But what about entry-j? The node can’t tell just from its local log:

  • If entry-j isn’t committed yet, the cluster is in joint mode with effective-config C_old_new
  • If entry-j is committed, the cluster is using C_new

Without talking to other nodes, there’s no way to know.

In the diagram above, even if entry-3 has committed, the restarting nodes b, c, x, y can’t tell whether the cluster is in joint mode or using the new configuration. (Nodes a and z never received entry-3 and are still using {a,b,c}.)

Patch-2: Always start in joint mode after a restart.

1. When a node starts up, it sets effective-config to the joint configuration formed from the last two config entries in its log
2. It uses this joint config for elections and normal operation
3. Only after confirming that the latest config entry has committed under the joint configuration can it switch to the new configuration

Example: A node sees configs {a,b,c} and {u,v,w} in its log. It starts with effective-config [{a,b,c}, {u,v,w}]. To become leader, it needs quorums from both groups. Only after it confirms the new config committed under the joint rules can it switch to just {u,v,w}.

Problem 3: Calling Home to Dead Nodes

Patch-2 solves the ambiguity problem but creates a worse one: nodes might try to contact old cluster members that no longer exist, making elections impossible.

Example:

1. The cluster changes from {a,b,c} to {x,y,z}

2. The config entry commits under C_old_new

3. The cluster transitions to C_new = {x,y,z}

4. Nodes a, b, c are no longer members. They get shut down, their data gets wiped, and they’re gone

5. Then something happens and all remaining nodes restart

6. Node x restarts and follows Patch-2: it sees configs {a,b,c} and {x,y,z} in its log, so it sets effective-config to [{a,b,c}, {x,y,z}]

7. Node x tries to run an election, but b and c don’t exist anymore! It can’t get a quorum from both groups. The election fails. The cluster is stuck.

This is state regression. The transition from C_old_new to C_new wasn’t persisted, so after a restart, the system rolls back to needing C_old.

Adding a Barrier to Prevent Regression

Restarting nodes need to know for certain that the joint phase has ended—proof that it’s safe to use C_new without calling back to C_old.

Patch-3: Add a barrier entry

After entry-j (containing C_new) commits under C_old_new, append a special barrier entry to mark that entry-j has committed.

Important: The barrier must come after entry-j commits. Otherwise it can’t serve as proof of the commit.

When a restarting node sees this barrier, it knows the joint phase ended successfully. It can safely use C_new for elections without trying to contact old nodes that might not exist anymore.

In the diagram below, when entry-3 commits under C_old_new, we add barrier entry-4:

Now when all nodes restart, there’s no regression. Nodes x and y see the barrier, so they use C_new = {x,y,z} directly. Even though b and c are gone, x or y can still get elected:

Alternative: Persisting commit-index

Instead of a barrier entry, we could persist the commit-index—an idea from Ma Jianjiang.

The rule: joint consensus ends when commit-index reaches a quorum of C_new. To make this work, we’d need to persist commit-index (standard Raft doesn’t require this).

When a node restarts, it checks: if the persisted commit-index covers the config change entry, it knows C_old_new finished and can safely use C_new. No need to contact old nodes.

But this still has Problem 1—C_old and C_new nodes competing for leadership. Here’s why: C_new nodes don’t have extra log entries, and committing commit-index to just C_new doesn’t guarantee C_old nodes see it. This is the classic distributed systems dilemma of at-least-once vs at-most-once delivery:

• At-least-once (commit on C_old_new): commit-index might succeed, then C_old nodes get decommissioned, then we can’t commit it again to reach them. We’re stuck.
• At-most-once (commit on C_new only): commit-index reaches C_new but might not reach C_old. Those nodes don’t know the cluster moved on, so they keep trying to run elections.

Either way, we can still end up with C_old and C_new nodes competing for leadership.

So here’s what the patched single-log approach looks like:

1. Start with effective-config = C_old = {a,b,c}

2. Leader writes entry-j containing C_new = {x,y,z} and immediately switches effective-config to C_old_new = [{a,b,c}, {x,y,z}]

3. All entries from index j onward replicate and commit under C_old_new

4. Critical step: Once entry-j commits under C_old_new, the leader writes a special barrier entry. This entry has no configuration data—it just marks “the joint phase is done.” The leader can switch to effective-config = C_new and use C_new to replicate the barrier.

5. When the barrier entry commits, the configuration change is complete

Restart behavior:

When a node restarts, it reads its log. It sees entry-i (C_old) and entry-j (C_new). It checks: is there a barrier after entry-j?

• Barrier present: Joint phase ended. Set effective-config = C_new. No need to contact old nodes.

• No barrier: Joint phase might still be active. Set effective-config = C_old_new.

Patch-3 adds a second log entry. We’re no longer doing “one log entry” configuration changes. We need “one config entry + one barrier entry.”

Conclusion

Configuration changes must pass through three states—C_old → C_old_new → C_new. One log entry gives us one bit of persistent information: C_old or C_new. That’s only two states. We can’t represent three states with two values.

To safely handle all three states, we need at least two log entries. That gives us two bits of information and up to four possible states, which is enough to encode the three states we actually need.

The “single-log-entry” approach, after all the patches, ends up needing two entries anyway—one for the configuration and one for the barrier. And it’s more complex than standard Joint Consensus, with trickier edge cases around restarts and state transitions.

Stick with Joint Consensus. It’s cleaner, simpler, and solves the problem directly without patches.

References

• Diego Ongaro & John Ousterhout. In Search of an Understandable Consensus Algorithm (Raft paper): https://raft.github.io/raft.pdf
• OpenRaft(rust): https://github.com/databendlabs/openraft
• etcd/raft source code: https://github.com/etcd-io/raft
• Hashicorp Raft implementation: https://github.com/hashicorp/raft

Reference:

• 马健将 : https://weibo.com/u/1516609505

Preface

I need to come clean about something. In my previous article on IO ordering in Raft, I tried to demonstrate the dangers of “writing log entries before term” using a committed data loss scenario. The problem? That example was fundamentally flawed—it didn’t actually capture the real issue with IO reordering at all.

So let’s fix that. This article walks through what I got wrong and, more importantly, presents a correct understanding of when and why IO reordering becomes dangerous in Raft implementations.

What Went Wrong in My Original Analysis

Let me show you the timeline I used in the previous article:

Legend:
Ni:   Node i
Vi:   RequestVote, term=i
Li:   Establish Leader, term=i
Ei-j: Log entry, term=i, index=j

N5 |          V5  L5       E5-1
N4 |          V5           E5-1
N3 |  V1                V5,E5-1  E1-1
N2 |  V1      V5                 E1-1
N1 |  V1  L1                     E1-1
------+---+---+---+--------+-----+---------> time
      t1  t2  t3  t4       t5    t6

Here’s what I claimed would happen:

• At t5: N3 receives entry E5-1 from leader L5 (term=5) and needs to persist both term=5 and E5-1
• I argued: “If N3 writes E5-1 first but crashes before writing term=5, it could restart with term=1, entries=[E5-1]”
• At t6: The old leader L1 (term=1) could then overwrite E5-1, causing data loss

Here’s the flaw in my reasoning: Raft’s protocol explicitly requires that both the term update and log entries must be successfully persisted before a follower responds with success. If either IO fails or is incomplete, the leader never receives confirmation and therefore never considers the entry committed. The Raft paper’s design is actually bulletproof here.

So if Raft’s design is correct, where does the IO ordering problem actually come from? The answer lies in a subtle gap between theory and implementation—specifically, how real Raft systems separate in-memory state from on-disk state.

The Real Culprit: In-Memory vs Persisted State

Here’s where things get interesting. The Raft paper describes a beautifully simple world where a server has just one state: what’s on disk. But real implementations need to be fast, so they introduce an optimization—they split their state into two layers:

In-memory state: The “optimistic” view that updates immediately when receiving RPCs Persisted state: The “durable” view that updates only after IO completes

Here’s how a typical implementation handles an appendEntries request:

1. Receive appendEntries RPC with req.term
2. If req.term > current_term, immediately update current_term to req.term
3. Asynchronously submit a save-term IO operation
4. Eventually update persisted_term when the IO completes

In code, this looks like:

struct RaftState {
    // In-memory term - may be ahead of what's on disk
    current_term: u64,

    // Persisted term on disk - the durable truth
    persisted_term: u64,
}

This separation is where the danger lurks. The Raft paper assumes only one “term” variable—what’s persisted on disk. But implementations now have two term values, and this introduces a behavior the paper never defined or analyzed.

The pattern above is ubiquitous in Raft implementations. And here’s the kicker: without IO reordering, it works perfectly fine. The bug only surfaces when IOs can complete out of order.

A Concrete Example: Where IO Reordering Breaks Raft

Let’s build a scenario that actually exposes the bug. I’ll walk you through it step by step:

Legend:
Ni:   Node i
Vi:   RequestVote, term=i
Li:   Establish Leader, term=i
Ei-j: Log entry, term=i, index=j

N5 |          V5  L5     E5-1     E5-2
N4 |          V5         E5-1     E5-2
N3 |  V1              V5,E5-1  V5,E5-2  E1-1
N2 |  V1      V5                        E1-1
N1 |  V1  L1                            E1-1
------+---+---+---+------+--------+-----+------> time
      t1  t2  t3  t4     t5       t6    t7

Here’s the sequence of events:

• t1-t4: Two elections occur. First N1 becomes leader (term=1), then N5 becomes leader (term=5)
• t5: Leader L5 sends its first entry E5-1 to follower N3
  • N3’s current state: current_term=1, persisted_term=1
  • N3 receives appendEntries(term=5, entries=[E5-1])
  • N3 must persist both term=5 and entry E5-1
  • N3 responds “success” only after both IOs complete
• t6: Leader L5 sends a second entry E5-2 to N3 ← This is the critical moment
  • N3 might still be waiting for t5’s IOs to complete
  • Whether IO reordering can occur makes all the difference
• t7: The old leader L1 (term=1) attempts to replicate E1-1 to N3

The bug manifests in what happens at t6—when the second AppendEntries arrives while the first one’s IOs are still in flight. Let’s zoom into N3’s internal state at each step.

At t5: The First AppendEntries

When N3 receives appendEntries(term=5, entries=[E5-1]), here’s what happens inside:

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check: Is the RPC term newer than our in-memory term?
    if req.term > self.current_term {
        self.current_term = req.term;           // Update memory immediately: 1 → 5
        self.submit_io(save_term(req.term));    // Queue IO to persist term=5
    }

    self.submit_io(save_entries(req.entries));  // Queue IO to persist E5-1

    // Wait for both IOs to complete before responding
    wait_for_both_ios();
    return success();
}

After this call executes, N3’s state looks like:

• current_term = 5 (memory updated immediately)
• persisted_term = 1 (disk not yet updated—IO still in flight)
• IO queue: [save_term(5), save_entries(E5-1)] waiting to complete

So far, so good. This request is handled correctly—N3 won’t respond until both IOs finish. The trouble starts at the next moment.

At t6: The Second AppendEntries—Where Everything Goes Wrong

Now here’s the critical moment. Before t5’s IOs have completed, N3 receives a second request: appendEntries(term=5, entries=[E5-2]).

Most implementations check only the in-memory current_term to decide whether to persist the term. Watch what happens:

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check: Is 5 > 5? Nope!
    if req.term > self.current_term {
        // We skip this branch entirely
    }

    // We only queue the entries IO
    self.submit_io(save_entries(req.entries));

    // We only wait for the entries IO to complete
    wait_for_io(save_entries);
    return success();  // We're done!
}

See the problem? N3 returns success as soon as save_entries(E5-2) completes. But here’s the dangerous part: if IO reordering is allowed, the system might have:

• ✅ Completed save_entries(E5-2)
• ✅ Completed save_entries(E5-1)
• ❌ NOT completed save_term(5) (still in flight from t5)

N3 happily returns success to Leader L5, which then considers E5-2 replicated and potentially committed.

Now imagine N3 crashes. When it restarts, its disk state is:

• persisted_term = 1 (the save_term(5) never finished)
• entries = [E5-1, E5-2] (both successfully written)

This is an inconsistent state that Raft’s protocol assumes can never exist. And it’s about to cause data loss.

At t7: The Data Loss Materializes

After N3 restarts with term=1, entries=[E5-1, E5-2], the old leader L1 (from term=1) sends an appendEntries request: appendEntries(term=1, entries=[E1-1]).

N3’s logic:

1. Check: RPC term (1) == my local term (1) ✅
2. Accept the request
3. Write E1-1 at index=1, overwriting E5-1

The disaster: Entries E5-1 and E5-2, which Leader L5 believed were successfully replicated and possibly committed, have just been silently destroyed. We’ve lost committed data.

Important note: If IO reordering were not allowed, this bug wouldn’t occur. Here’s why: when save_entries(E5-2) completes at t6, it would guarantee that save_term(5) (queued earlier) has also completed. The sequential ordering ensures that N3’s disk state remains consistent, and the AppendEntries success response would be legitimate.

The Root Cause: A Mismatch Between Theory and Practice

Let’s crystallize what we’ve learned:

The core issue: When deciding whether to persist the term, should we check current_term or persisted_term?

• If IO reordering is not allowed → checking current_term is safe
• If IO reordering is allowed → we must check persisted_term

This isn’t obvious because the Raft paper never talks about in-memory vs persisted state—it only knows about one kind of state: what’s on disk. The paper says: “Before responding to RPCs, a server must update its persistent state.”

But in real implementations with in-memory and persisted state split, this requirement needs to be more precise:

Before returning success, we must ensure all IOs that make persisted_term >= req.term have completed.

Checking only current_term creates a window where we might respond successfully while the required disk updates are still in flight. If those updates can complete out of order, we’ve violated Raft’s safety guarantees.

How to Fix It: Check Persisted State, Not In-Memory State

If you need to support IO reordering, the fix is conceptually simple—check the on-disk term, not the in-memory term:

fn handle_append_entries(&mut self, req: AppendEntries) {
    // Check against disk state, not memory!
    let need_save_term = req.term > self.persisted_term;

    if need_save_term {
        self.current_term = req.term;
        self.submit_io(save_term(req.term));
    }

    self.submit_io(save_entries(req.entries));

    // Wait for the right IOs based on what we actually need
    if need_save_term {
        wait_for_both_ios();  // Must wait for term update to complete
    } else {
        wait_for_io(save_entries);  // Only need to wait for entries
    }

    return success();
}

By checking persisted_term instead of current_term, we correctly detect when the term IO is still in flight and wait for it to complete.

Caveat: This approach might submit multiple save_term(T) IOs for the same term T (if multiple AppendEntries arrive in quick succession). You’ll need to handle this carefully—either make the IO layer idempotent or add deduplication logic.

How Production Systems Solve This

Here’s the interesting part: most mature Raft implementations don’t actually support IO reordering. Instead, they eliminate the problem entirely by ensuring save-term and save-entries execute in order. This lets them safely check current_term without the bug we just analyzed.

Let’s look at three different approaches from production systems:

1. Atomic Batching (TiKV)

Strategy: Bundle save-term and save-entries into a single atomic IO operation.

When an AppendEntries requires both a term update and log writes, TiKV combines them into one batch and submits it as a single IO request. This makes it impossible for the entries to persist without the term—they’re literally the same operation.

This elegantly sidesteps the entire reordering problem. There’s no “second AppendEntries that only submits save_entries” scenario because term and entries are always written together.

2. Ordered Separation (HashiCorp Raft)

Strategy: Persist term and entries separately, but enforce strict ordering.

HashiCorp’s Raft implementation writes the term first (with fsync, panicking on failure), then writes the log entries. The key is that these operations execute sequentially—save_entries can’t start until save_term completes.

This guarantees that if entries reach disk, the term has definitely reached disk first. Sequential ordering prevents the reordering bug.

3. Hybrid Ordering (SOFAJRaft)

Strategy: Synchronous term writes, asynchronous batched log writes.

SOFAJRaft writes the term synchronously (blocking the current thread for fsync) but batches log entries for asynchronous writing. The crucial property: save_term always completes before save_entries is even enqueued.

This hybrid approach gets you most of the performance benefits of async IO while maintaining the ordering guarantee that prevents the bug.

Summary: Bridging Theory and Practice

The IO ordering bug in Raft implementations stems from a subtle gap between the paper’s abstract model and real-world code. The Raft paper assumes a single state: what’s on disk. Real implementations optimize with in-memory and persisted state split, introducing behaviors the paper never analyzed.

The invariant we must maintain:

If a log entry with term=T is on disk, then persisted_term ≥ T must also be on disk.

Violating this invariant—having entries from term T on disk while persisted_term < T—breaks Raft’s safety guarantees and can cause committed data loss.

Two ways to maintain the invariant:

1. Eliminate IO reordering (mainstream approach)

   • Atomic batching: Write term and entries together
   • Ordered execution: Guarantee term persists before entries
   • Hybrid ordering: Synchronous term, async entries

2. Handle IO reordering explicitly

   • Check persisted_term instead of current_term when deciding whether to persist the term
   • Wait for all required IOs to complete before responding

Most production systems choose option 1—it’s simpler to reason about and avoids the complexity of tracking multiple in-flight term updates. But if you do need to support IO reordering, now you know where the dragons are hiding.

Related Resources

• The Hidden Danger in Raft: Why IO Ordering Matters
• OpenRaft docs: io-ordering
• tikv/tikv
• hashicorp/raft
• sofastack/sofa-jraft

Reference:

• OpenRaft docs: io-ordering : https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/protocol/io_ordering.md

• hashicorp/raft : https://github.com/hashicorp/raft

• The Hidden Danger in Raft Why IO Ordering Matters : https://blog.openacid.com/algo/raft-io-order/

• sofastack/sofa-jraft : https://github.com/sofastack/sofa-jraft

• tikv/tikv : https://github.com/tikv/tikv

IO Reordering Breaks Committed Data

In Raft, if you write log entries before the term when persisting AppendEntries, you risk losing committed data.

This article explores how this happens, how production systems handle it, and how to prevent it.

Background

When a follower receives an AppendEntries RPC in Raft, it must persist two critical pieces: metadata (HardState, containing term and vote) and log entries (the actual application data). Only after both are safely on disk can the follower respond to the leader. Here’s the catch: persistence order matters tremendously.

How Data Loss Happens

Let’s walk through a concrete timeline to see how this plays out:

Legend:
Ni:   Node i
Vi:   RequestVote, term=i
Li:   Establish Leader, term=i
Ei-j: Log entry, term=i, index=j

N5 |          V5  L5       E5-1
N4 |          V5           E5-1
N3 |  V1                V5,E5-1  E1-1
N2 |  V1      V5                 E1-1
N1 |  V1  L1                     E1-1
------+---+---+---+--------+-----+-------------------------------------------> time
      t1  t2  t3  t4       t5    t6

• t1: N1 starts an election (term=1), receives votes from N1, N2, N3
• t2: N1 becomes leader L1
• t3: N5 starts an election (term=5), receives votes from N5, N4, N2
• t4: N5 becomes leader L5
• t5: L5 replicates its first log entry E5-1 to N4 and N3. Key point: N3’s stored term (1) is stale compared to the RPC’s term (5), so N3 must perform two sequential IO operations: persist term=5, then persist E5-1
• t6: L1 attempts to replicate E1-1 (term=1, index=1)

The critical moment is t5, where N3’s behavior determines everything:

If IO operations is not reordered (correct):

N3 executes sequentially: first persist term=5, then persist E5-1. This guarantees that whenever E5-1 is on disk, term=5 is already there too.

If IO operations can be reordered (wrong):

IO operations might complete out of order: E5-1 hits disk first, then term=5.

Here’s where disaster strikes: if the server crashes after writing E5-1 but before persisting term=5, N3’s stored term stays at 1 while E5-1 sits in the log.

When N3 recovers and receives L1’s replication request for E1-1 (term=1, index=1), it accepts it—the terms match! E1-1 overwrites E5-1.

The damage is done: E5-1 was already replicated to 3 nodes (N5, N4, N3) and considered committed by L5, but now it’s gone, replaced by stale data. Committed data has vanished.

At its core, this problem breaks a critical invariant:

If a log entry E (term=T) exists on disk → the stored term must be ≥T

Proper IO ordering preserves this invariant, guaranteeing that whenever a log entry hits disk, its term is already there.

What Raft’s Paper Doesn’t Say

The Raft paper states: “Before responding to RPCs, a server must update its persistent state.”

The paper assumes persistence is atomic without explicitly spelling out the ordering requirements between term and log.

The trap most implementations fall into: when a follower receives an AppendEntries RPC, it needs to persist two types of data—metadata (term, vote, etc., in MetaStore) and log entries (in LogStore).

For performance and clean separation of concerns, many implementations store these separately and submit IO requests in parallel:

fn handle_append_entries(&mut self, req: AppendEntries) -> Response {
    self.meta_store.save_term_async(req.term);  // Async submit
    self.log_store.append_async(req.entries);   // Async submit

    self.log_store.sync();  // Only wait for log persistence!
    return Response::success();  // Ignore whether term is persisted
}

The trap is subtle: developers focus on persisting logs (the “real” application data) while treating term as mere “metadata” that can wait. The result? Logs hit disk while term is still in memory—or worse, in a write queue. When the server crashes, the invariant shatters.

Production Implementations

I examined 4 production Raft implementations to see how they tackle this:

Three Safe Solutions

From successful production implementations, three safe patterns emerge:

Atomic Batching (TiKV)

TiKV bundles term and log entries into a single atomic batch. The code adds both to a batch, then calls write_batch(sync=true) to commit everything at once with checksum verification.

The beauty: all-or-nothing. Order within the batch doesn’t matter, making correctness reasoning trivial.

The trade-off? You need atomic batch support, but you only pay one fsync. Perfect for custom storage engines or when you want the simplest possible safety guarantees.

batch.put_term(new_term);
batch.put_entries(entries);
storage.write_batch(batch, sync=true); // Atomic write + checksum verification

Sequential Writes (HashiCorp Raft)

HashiCorp Raft keeps it simple: write term first, then log—both synchronously.

Looking at raft.go:1414,1922, setCurrentTerm includes an fsync that panics on failure before StoreLogs even runs. Once term is on disk, the higher term acts as a shield against stale leader requests.

The upside? Dead simple to implement, works with any storage backend, and embraces fail-fast philosophy. The price? Two fsyncs mean slightly higher latency. Great for general use with standard storage like files or BoltDB.

// raft.go:1414,1922
r.setCurrentTerm(a.Term)  // Includes fsync, panics on failure
r.logs.StoreLogs(entries) // Includes fsync

Hybrid Approach (SOFAJRaft)

SOFAJRaft splits the difference: synchronous term writes, asynchronous log batching.

In NodeImpl.java:1331,2079, setTermAndVotedFor blocks until fsync completes, while appendEntries just enqueues the log and returns instantly—background threads handle the batch writes.

The key: logs queue only after term’s fsync completes, guaranteeing term persists first. This delivers peak performance because term changes are rare (only during leader switches), making sync acceptable, while log writes are constant (every client request), where async batching shines.

The catch? Complex implementation needing a bulletproof async pipeline (SOFAJRaft uses LMAX Disruptor). Ideal when you’re pushing >10K writes/sec.

// NodeImpl.java:1331,2079
this.metaStorage.setTermAndVotedFor(req.term, null); // Sync fsync, blocks
this.logManager.appendEntries(entries, closure);     // Async enqueue, returns immediately

Async IO Scheduling

All three approaches guarantee safety by sacrificing IO concurrency: either serial execution (wait for one to finish before starting the next) or atomic batching.

For higher performance, OpenRaft is exploring an async IO scheduler: the Raft core fires all IO requests into an execution queue, which schedules them and signals completion via callbacks. This maximizes IO parallelism and throughput but surfaces a fundamental question: which IOs can be reordered safely, and which absolutely cannot?

Summary

Term must be persisted before (or at the same time as) log

Invariant: If log(term=T) is on disk → term≥T must also be on disk

I like thinking about distributed consensus through time and history. Consensus algorithms create a virtual timeline, and the Raft log is simply the sequence of events on that timeline. In this view: term is time itself, and log entries are the events happening in that time.

When IO lets term roll back, you’re letting time itself rewind. But here’s the paradox: rewinding time doesn’t erase what happened—the system can rewrite history at an earlier point, letting new events overwrite the old. That’s data loss at its core.

Choose your approach: Atomic batching for simplicity, ordered writes for compatibility, hybrid for maximum throughput.

Related Resources

• OpenRaft docs: io-ordering
• tikv/tikv
• hashicorp/raft
• sofastack/sofa-jraft

Reference:

前言

我的好友 fuzhe 在阅读 LevelDB 源码时，发现了一个有趣的细节：系统在存储 CRC 校验码时，并不直接使用计算出的值，而是要先做一个看似”多余”的 mask 操作。这个操作包括右旋转 15 位和加上一个神秘的常数 0xa282ead8ul。

代码注释提到这是为了解决”对包含嵌入式 CRC 的字符串计算 CRC 是有问题的”，但到底什么问题？为什么位操作能解决它？这个设计背后隐藏着什么原理？

今天我们带着这些疑问，开始一场CRC的探索之旅。一步步揭开 CRC 的数学本质，理解”自包含退化”现象，最终看到 LevelDB 工程师们如何用优雅的数学技巧解决了这个深刻的问题。

什么是 CRC：从自然数除法到校验码

传输数字 1234 时，如何检测错误？用除法取余数：

1234 ÷ 97 = 12 余 70

余数 70 就是 1234 的指纹。

校验原理：

• 发送：数据=1234, 校验=70
• 接收：重新计算 received % 97 的余数
• 余数不是 70 → 检测到错误

判断收到的消息正确性的方式是：

接收到：数据=1234, 校验=70

1. 计算 1234 % 97 = 70
2. 比较：计算余数(70) == 收到的校验码(70) ？
3. 相等 → 数据很可能正确
4. 不等 → 数据肯定有错误

这里有个重要的点：余数相等不能 100%保证数据正确（可能有碰撞），但余数不等就可以 100%确定数据有错误。

例子：

1234 % 97 = 70
1235 % 97 = 71  ← 数据变了，余数也变了

另外，为了防止消息体和校验值一样，CRC 在定义的时候还提出了一点，就是先要乘以一个比较大的数字，来保证所得余数跟消息本身不一样: 对于所有输入数字，先放大：

先乘 100：42 → 4200
4200 % 97 = 29  → 发送：数据=42, 校验=29

碰撞概率：使用质数 97 作除数，碰撞概率约 1/97 ≈ 1%。

以上就是 CRC 校验的核心思想：

• 用固定除数计算余数作为”指纹”

这些原理和标准 CRC 完全一致。唯一区别是 CRC 使用二进制多项式代替十进制数字，使用异或代替普通加减法法。理解了这个基础，接下来看看如何正式的定义 CRC 算法.

从自然数四则运算到 GF(2)多项式的四则运算

GF(2) 运算：二进制世界的数学

在二进制世界里，我们就只有两个数字：0 和 1, 即 GF(2), 或 Galois Field of 2 elements. 因为只有 0 和 1, 这里的运算规则可以很简单, 但又和我们日常使用的加减乘除非常相似(符合交换律分配律结合律等)：

01 的加法规则（相当于异或 XOR）：

0 + 0 = 0
0 + 1 = 1
1 + 0 = 1
1 + 1 = 0  ← 没有进位！

01 的减法规则（同样是异或）：

0 - 0 = 0
1 - 1 = 0
1 - 0 = 1
0 - 1 = 1  ← 没有借位！

可以看出, 01 的世界里加法和减法相同: x+1 == x-1, 都对应异或操作: a + b → a ^ b

01 的乘法规则：

0 × 0 = 0
0 × 1 = 0
1 × 0 = 0
1 × 1 = 1  ← 和普通乘法一样

可以看出 01 世界中的乘法就是 AND 操作: a × b = a & b

01 的除法规则：

0 ÷ 1 = 0
1 ÷ 1 = 1  ← 和普通除法一样

看到这里我们发现：GF(2) 也是四则运算，和整数的计算规则基本一样，就是加法变成了异或。

还有一个有趣的地方：GF(2) 计算是封闭的 - 不管怎么算，结果永远只有 0 或 1，绝不会蹦出别的数字。一个完美的封闭系统。

多项式的四则运算

而一个关于 x 的多项式, 它的加减乘除运算是定义为:

• 加减法: 等幂项的系数相加相减;
• 乘法: 一个多项式的每一项跟另一个多项式的每一项相乘, 再把结果中的等幂项合并;
• 除法是乘法的逆运算;

从这里我们看出, 要让多项式的四则运算成立, 前提是它的系数必须可以进行 加减法和乘法运算, 换句话说, 任何满足加减乘的东西都可以作为多项式的系数. GF(2)就可以.

不论是整数还是 GF(2) 做多项式的系数, 多项式的四则运算都满足下面这些要求:

• 加法交换律：P(x) + Q(x) = Q(x) + P(x)
• 乘法交换律：P(x) × Q(x) = Q(x) × P(x)
• 加法结合律：(P + Q) + R = P + (Q + R)
• 乘法结合律：(P × Q) × R = P × (Q × R)
• 分配律：P × (Q + R) = P×Q + P×R

GF(2) 多项式运算示例：

分配律验证：
(x² + 1) × (x + 1) = (x² + 1)×x + (x² + 1)×1
                   = x³ + x + x² + 1    ← 展开后有重复项

体现 GF(2)特性的例子：
(x + 1) + (x + 1) = x + 1 + x + 1 = (x + x) + (1 + 1)
                  = 0 + 0 = 0       ← 相同项抵消！

多项式加法：
(x² + x + 1) + (x² + 1) = x² + x + 1 + x² + 1
                        = (x² + x²) + x + (1 + 1)
                        = 0 + x + 0 = x    ← 相同系数的项抵消

到这里你可能已经注意到：GF(2) 多项式和二进制数之间有着完美的一一对应关系。这意味着什么呢？我们可以把二进制数当作多项式来算！说白了，我们是借用多项式的四则运算，给二进制数定义了一套全新的运算规则。

多位运算示例：

加法/减法：
  1011
⊕ 1101
------
  0110

乘法（多项式乘法）：
  101  × 11 = 101 × (10 + 1) = 101 × 10 + 101 × 1
            = 1010 ⊕ 101 = 1111

进一步, 我们之前说的 CRC 计算，其实我们不需要基于整数的除法来定义 CRC, 我们真正需要的是: 四则运算. 它压根不在乎你用的是整数、自然数还是别的什么。只要满足这些运算规律，CRC 就能工作。

使用 GF(2)多项式的四则运算是为了充分利用值域空间

这里, 我们把二进制数看作一个 GF(2)为系数的多项式, 然后按照多项式的四则运算来定义这个二进制数的四则运算.

这样做是因为能完全使用二进制数的值空间.

例如, 如果我们还用整数定义的四则运算, 那么要找一个质数作为 CRC 的除数, 例如 97, 那么每个校验值的值域是[0, 97), 要是用一个 byte 来存, 那么 97 到 255 的数字用不到, 校验能力就降低了.

但是 GF(2)的多项式的 prime 多项式, 例如 x³ + x + 1 (对应二进制 1011), 那么用它做除数多项式, 余数多项式可以是全部 2³个 2 次多项式, 增加了校验值的值域空间, 也就是最大化了检测能力. 标准的 CRC32 则是全部利用了2³²个值.

二进制数和多项式的对应关系

每个二进制数都能对应一个多项式：

多项式表示：

二进制：1011 → 多项式：1·x³ + 0·x² + 1·x¹ + 1·x⁰ = x³ + x + 1
二进制：1101 → 多项式：1·x³ + 1·x² + 0·x¹ + 1·x⁰ = x³ + x² + 1

现在我们熟悉的除法思想就可以用到二进制数据上了！

CRC定义为：

数据多项式 * 100..000  % 生成多项式 = 余数(CRC 值)

就像我们选择质数 97 作为除数一样，CRC-32 使用的生成多项式也必须是 不可约的多项式（irreducible polynomial）。这保证了最好的错误检测性能。

让我们用一个 4 位的不可约多项式来演示 CRC 计算：

数据多项式：x² + 1 (对应二进制 101)
生成多项式：x³ + x + 1 (对应二进制 1011，这是一个不可约多项式)

为了计算 CRC，我们需要计算：
(x² + 1) · x³ % (x³ + x + 1) 的余数
数据后面补 3 个 0(相当于乘以 x³，补 0 个数=生成多项式度数)

多项式除法过程：
被除数：(x² + 1) · x³ = x⁵ + x³ (对应二进制 101000)

                                  x²
              ___________________________________
 x³ + x + 1  √ x⁵ + 0·x⁴ + x³ + 0·x² + 0·x + 0
               x⁵ + 0·x⁴ + x³ +   x²
               ----------------------------------
                                  x² + 0·x + 0   ← 余数

结果：数据 (x² + 1) 的 CRC 是 100
发送 消息M | CRC(M)：101 | 100

这和我们第一章介绍的用整数除法在概念上完全一致，只是现在用的是多项式除法和异或运算。

基于整数除法的CRC是CRC32的核心原理, 在此基础上再把四则运算替换为 GF(2) 多项式的四则运算, 则充分利用了值的空间(配合计算机中2⁸宽的场景)

CRC 在 LevelDB 中一个看似多余的设计

LevelDB 的 Mask 操作

在 LevelDB 的代码中（源码链接），系统计算出 CRC-32 校验值后，并不直接使用，而是要做一个变换：

// https://github.com/google/leveldb/blob/main/util/crc32c.h#L26
static const uint32_t kMaskDelta = 0xa282ead8ul;

// Return a masked representation of crc.
//
// Motivation: it is problematic to compute the CRC of a string that
// contains embedded CRCs.  Therefore we recommend that CRCs stored
// somewhere (e.g., in files) should be masked before being stored.
inline uint32_t Mask(uint32_t crc) {
  // Rotate right by 15 bits and add a constant.
  return ((crc >> 15) | (crc << 17)) + kMaskDelta;
}

这个操作包含两个步骤：

1. 右旋 15 位：(crc >> 15) | (crc << 17)
2. 加上常数：+ kMaskDelta

问题

代码注释指出了关键问题：

“it is problematic to compute the CRC of a string that contains embedded CRCs” 对包含嵌入式 CRC 的字符串计算 CRC 是有问题的

Yes, But, 到底什么东西 problematic 了?

实际上它所指的这个问题出现在一个常见场景中： 当 CRC 校验码本身也成为被校验数据的一部分时。

考虑这样的嵌套数据结构, 我们先有了消息 M, 然后为 M 做一次 CRC 得到CRC(M), 然后把 M 和CRC(M) 拼到一起, 然后传输协议的下一层又对整个M | CRC(M), 当做一个消息, 又做一次 CRC：

第一层：
消息 M  →  M | CRC(M)

第二层：
M | CRC(M)  →  M | CRC(M) | CRC(M | CRC(M))

这里问题出现了：如果两层使用相同的 CRC 算法，那么不管消息 M 是什么内容，第二层的 CRC 值都会是同一个固定常数！

这意味着外层 CRC 完全失去了检错能力 - 无论内层数据如何变化，外层看到的校验结果永远相同。

LevelDB 的 mask 操作正是为了解决这个问题而设计的。通过对 CRC 值进行可逆的变换，避免了退化现象的发生，保证了多层校验系统的有效性。

CRC 的”自包含退化”现象

第一层：
消息 M  →  M | CRC(M)

第二层：
M | CRC(M)  →  M | CRC(M) | CRC(M | CRC(M))

用简单 CRC 推导退化现象

为了清楚展示退化过程，我们定义一个简化的 CRC 算法, 除数多项式是111₂：

CRC = (消息 M × 1000₂) % 111₂

用二进制表示：

• 1000₂ = 8 (即 2³，对应多项式 x³)
• 111₂ = 7 (对应多项式 x² + x + 1，这是一个 3 次不可约多项式)

这相当于 3 位的 CRC。

完整的数学推导

∵ CRC(M) = (M × 1000₂) % 111₂

∴ M × 1000₂ = k × 111₂ + CRC(M) （其中 k 为某个数）

二进制连接操作: 在二进制中，要把 CRC 追加到消息 M 后面，需要先为 CRC 空出位置。由于我们的 CRC 是 3 位（111₂有 3 位），所以需要将 M 左移 3 位：

M × 1000₂   (左移 3 位，为 3 位 CRC 空出位置)

然后把 CRC 值放入这 3 位中：

M | CRC(M) = M × 1000₂ + CRC(M)

(注意实际使用时, CRC32 是 4 个 byte, 所以把 CRC32 追加到消息 M 的操作是: M 后面空出 4 byte, 即M * 2³², 然后追加 M 的 CRC32)

例如：M = 101₂，CRC(M) = 101₂

M × 1000₂ = 101₂ × 1000₂ = 101000₂  (为 CRC 空出 3 个位置)
M | CRC(M) = 101000₂ + 101₂ = 101101₂  (把 CRC 填入空出的位置)

然后我们将关系代入到追加后的数据得到：

M | CRC(M)  =  M × 1000₂ + CRC(M)
            =  (k × 111₂ + CRC(M)) + CRC(M)
            =  k × 111₂ + CRC(M) + CRC(M)

在 GF(2)域中，任何数与自己相加都等于 0：

CRC(M) + CRC(M) = 0  (GF(2)加法性质)

因此：

M | CRC(M)  =  k × 111₂

于是我们发现：M | CRC(M) 正好是 111₂ 的倍数！

计算第二层 CRC

CRC(M | CRC(M))  =  CRC(k × 111₂)
                 =  (k × 111₂ × 1000₂) % 111₂
                 =  (k × 111₂ × 1000₂) % 111₂
                 =  0   // 任何 111₂的倍数模 111₂都等于 0

这就是数学上退化现象的本质： 无论 M 是什么值，CRC(M | CRC(M)) 总是等于同一个固定常数(0)！

实际系统中的表现

在真实的 CRC 实现中，由于存在初始值、最终异或等处理步骤，这个固定值通常不是 0，而是某个特定常数（称为 residue）。但核心问题不变：

CRC(任意消息 | 该消息的 CRC) = 固定常数（与消息内容无关）

Mask 机制的影响分析

所有 LeveldDB 中的注释就是在说这个问题, 并且试图用 Mask 对 CRC 值做一个变换来消除这种退化特性. 显然, ((crc >> 15) | (crc << 17)) + kMaskDelta 操作之后就破坏了上面的数学关系, 使得外层 CRC 不再是一个消息无关的常数.

然后我们来分析下 Mask 能提供哪些特性:

对随机数据损坏的防护能力：无变化

从数学角度分析，mask 机制对随机 bit 错误的检测能力既不会提升，也不会降低。

我们来看看这两种方案的区别。 设原始消息为 M，两种方案的映射关系是：

• 无 mask 方案：M → (CRC₁, CRC₂) = (CRC(M), 固定常数)

• 有 mask 方案：M → (CRC₁, CRC₂’) = (CRC(M), CRC(M

  mask(CRC(M))))

   CRC(M | mask(CRC(M)))
=  CRC( M * 1000₂         + mask(CRC(M)) )
=  CRC( k * 111₂ + CRC(M) + mask(CRC(M)) )
=     ( k * 111₂ + CRC(M) + mask(CRC(M)) ) * 1000₂ % 111₂
=     ( CRC(M) + mask(CRC(M)) ) * 1000₂ % 111₂

从这个推导可以看出，有 mask 的方案中，最终结果也只与 CRC(M) 相关。由于 CRC(M) 的取值只有 2³个，所有的消息 M 最终映射到平面上的 2³个点，纠错能力也只有 1/2³。

有意思的是，两种方案都将任意消息 M 映射到一个 2 维空间中的特定点。虽然无 mask 时外层 CRC 是固定常数，但实际上这并不减少整体的错误检测能力：

1. 两种方案的值域大小相同（都是 2³个可能的组合）
2. 对于随机的 bit 翻转，两种方案检测错误的概率完全相等
3. 外层 CRC 的”退化”不影响内层 CRC 的有效性

对人为攻击的防护能力：显著提升

Mask 机制的真正价值在于防范恶意攻击，而非随机错误。

我们来比较一下两种情况下攻击者面临的难度：

如果没有 mask： 攻击者知道外层 CRC 永远是固定常数，就可以任意构造 (M’, CRC(M’)) 对。只要保证 CRC 计算正确，外层校验必然通过，攻击成本很低。

如果有了 mask： 攻击者不知道 mask 后的外层 CRC 值，无法直接构造能通过外层校验的伪造数据，必须破解 mask 算法或进行暴力搜索，攻击成本大大增加。

这里有个重要的区别：

• 抗损坏：防御随机的 bit 变化，主要看值域大小
• 抗攻击：防御人为的 bit 变化，主要看构造难度

Mask 机制巧妙地提升了攻击的构造难度，让系统更安全，同时对随机错误的检测能力完全不受影响。这就是 LevelDB 为什么要加入这个看起来”多余”的 mask 操作的原因。

实践验证：用代码重现退化现象

Python实现

• Python 示例代码地址 – https://gist.github.com/drmingdrmer/2b49106b225ca7bf361fd21454f693da

让我们用CRC-32C（LevelDB使用的算法）来验证自包含退化现象：

import os
import random

# CRC-32C (Castagnoli) 参数
POLY_REV = 0x82F63B78

def _make_crc32c_table():
    tbl = []
    for i in range(256):
        crc = i
        for _ in range(8):
            if crc & 1:
                crc = (crc >> 1) ^ POLY_REV
            else:
                crc >>= 1
        tbl.append(crc & 0xFFFFFFFF)
    return tbl

_CRC32C_TABLE = _make_crc32c_table()

def crc32c(data: bytes) -> int:
    """标准 CRC-32C 实现"""
    crc = 0xFFFFFFFF
    for b in data:
        crc = _CRC32C_TABLE[(crc ^ b) & 0xFF] ^ (crc >> 8)
    return (crc ^ 0xFFFFFFFF) & 0xFFFFFFFF

验证嵌套CRC退化现象

def demonstrate_nested_crc_degradation():
    # 模拟不同的LevelDB数据块
    leveldb_blocks = [
        b"user_data_1",
        b"user_data_2",
        b"important_record_12345",
        b"transaction_log_abcdef",
        bytes(range(50)),  # 二进制数据
    ]

    # 添加一些随机数据块
    for _ in range(5):
        leveldb_blocks.append(os.urandom(random.randint(10, 100)))

    print("=== 嵌套CRC退化验证 ===")
    outer_crcs = []

    for i, block_data in enumerate(leveldb_blocks):
        # 第一层：LevelDB内部CRC
        inner_crc = crc32c(block_data)
        leveldb_block = block_data + inner_crc.to_bytes(4, "little")

        # 第二层：外层协议CRC（如网络传输）
        outer_crc = crc32c(leveldb_block)
        complete_packet = leveldb_block + outer_crc.to_bytes(4, "little")

        outer_crcs.append(outer_crc)

        print(f"块 #{i+1:2d}: 数据长度={len(block_data):3d}, "
              f"内层CRC=0x{inner_crc:08X}, "
              f"外层CRC=0x{outer_crc:08X}")

    # 检查外层CRC是否都相同
    unique_outer_crcs = set(outer_crcs)
    print(f"\n不同外层CRC的数量: {len(unique_outer_crcs)}")
    print(f"所有外层CRC都是: 0x{list(unique_outer_crcs)[0]:08X}")
    print(">>> 外层CRC完全退化！无论内层数据如何变化，外层CRC都是固定值")

运行结果

=== 嵌套CRC退化验证 ===
块 # 1: 数据长度= 11, 内层CRC=0x12AB34CD, 外层CRC=0x48674BC7
块 # 2: 数据长度= 11, 内层CRC=0x56EF78AB, 外层CRC=0x48674BC7
块 # 3: 数据长度= 21, 内层CRC=0x9A2B3C4D, 外层CRC=0x48674BC7
块 # 4: 数据长度= 22, 内层CRC=0xDEF01234, 外层CRC=0x48674BC7
块 # 5: 数据长度= 50, 内层CRC=0x567890AB, 外层CRC=0x48674BC7
...

不同外层CRC的数量: 1
所有外层CRC都是: 0x48674BC7
>>> 外层CRC完全退化！无论内层数据如何变化，外层CRC都是固定值

我们看到, 尽管：

• 每个LevelDB数据块的内容完全不同
• 每个数据块的内层CRC也完全不同（0x12AB34CD, 0x56EF78AB等）
• 但是所有的外层CRC都是同一个值：0x48674BC7

这意味着外层协议完全失去了检错能力！不管LevelDB内部的数据如何变化，外层看到的”校验码”永远是同一个常数。

LevelDB的Mask方案

现在让我们看看mask如何打破这种退化：

# LevelDB的mask函数
K_MASK_DELTA = 0xA282EAD8

def mask(crc: int) -> int:
    # 右旋15位 + 加常数
    rotated = ((crc >> 15) | (crc << 17)) & 0xFFFFFFFF
    return (rotated + K_MASK_DELTA) & 0xFFFFFFFF

def demonstrate_mask_effect():
    # 使用前面相同的LevelDB数据块
    leveldb_blocks = [
        b"user_data_1",
        b"user_data_2",
        b"important_record_12345",
        b"transaction_log_abcdef",
    ]

    print("\n=== 使用LevelDB mask后的结果 ===")
    outer_crcs_masked = []

    for i, block_data in enumerate(leveldb_blocks):
        # 第一层：LevelDB内部CRC + mask
        inner_crc = crc32c(block_data)
        masked_crc = mask(inner_crc)  # 关键：存储前mask
        leveldb_block_masked = block_data + masked_crc.to_bytes(4, "little")

        # 第二层：外层协议CRC
        outer_crc = crc32c(leveldb_block_masked)

        outer_crcs_masked.append(outer_crc)

        print(f"块 #{i+1}: 原CRC=0x{inner_crc:08X}, "
              f"Mask后=0x{masked_crc:08X}, "
              f"外层CRC=0x{outer_crc:08X}")

    unique_masked = set(outer_crcs_masked)
    print(f"\n使用mask后，不同外层CRC的数量: {len(unique_masked)}")
    print(">>> Mask成功！外层CRC现在随数据内容变化")

输出：

=== 使用LevelDB mask后的结果 ===
块 #1: 原CRC=0x12AB34CD, Mask后=0xE5F2A891, 外层CRC=0x3C7B2A15
块 #2: 原CRC=0x56EF78AB, Mask后=0xC4D8B672, 外层CRC=0x8F4E1B29
块 #3: 原CRC=0x9A2B3C4D, Mask后=0x7B3EA254, 外层CRC=0x1D6C9F38
块 #4: 原CRC=0xDEF01234, Mask后=0x2A8C5E91, 外层CRC=0x94E7C2B6

使用mask后，不同外层CRC的数量: 4
>>> Mask成功！外层CRC现在随数据内容变化

我们也看到了区别. 使用mask后，每个 LevelDB 数据块都产生了不同的外层 CRC 值，外层协议重新获得了检错能力！

现实应用：分层存储系统中的校验挑战

存储系统的分层结构

现代存储系统通常采用多层架构，每层都有自己的完整性校验：

应用层数据
├── 应用层 CRC (例：数据库记录校验)
├── 存储引擎层 CRC (例：LevelDB block 校验)
├── 文件系统层 CRC (例：ext4/ZFS 校验)
└── 硬件层 CRC (例：磁盘扇区校验)

在这种架构中，上层数据经常包含下层的校验码，形成嵌套结构。

问题场景举例

场景 1：数据库备份

原始记录: [user_data | record_crc]
备份文件: [backup_header | [user_data | record_crc] | backup_crc]

如果 backup_crc 采用相同的 CRC 算法，就会遇到自包含退化。

场景 2：网络传输

LevelDB block: [block_data | block_crc(masked)]
网络包:        [packet_header | [block_data | block_crc] | packet_crc]

LevelDB 的 mask 确保了 packet_crc 不会退化。

场景 3：RAID 系统

磁盘扇区: [user_data | user_crc | sector_crc]
RAID 校验: [扇区 1 | 扇区 2 | ... | raid_parity]

其他系统的应对策略

第一种方案是使用不同的 CRC 多项式，让各层使用不同的生成多项式。这种方法简单直接，但需要在系统各层之间进行协调，增加了实现复杂度。第二种是加入层标识，在数据前加入层标识符，例如 [layer_id | data | crc]，这样能够明确分层，但会增加存储开销。第三种方案是使用不同的校验算法，让不同层使用 CRC、MD5、SHA 等不同算法，虽然能完全避免算法层面的冲突，但带来了性能和复杂度的开销。第四种就是 LevelDB 式的 mask 方案，对存储的校验码进行可逆变换，这种方法开销最小，效果最好，但需要理解数学原理才能正确实现。

设计选择的权衡

从这个表格可以看出，LevelDB 选择 mask 方案是有道理的 - 既保持了高性能，又解决了根本问题。

适用范围和限制

mask 方案特别适用于高性能存储系统、嵌套数据结构频繁的场景以及需要保持算法一致性的系统。不过在一些场景下不太适用，比如对校验码有加密要求的场景，因为 mask 操作是可逆的，不提供额外的安全保护。在极端安全敏感的环境中，可能需要更复杂的校验方案。另外，如果系统已有成熟的分层校验方案，引入 mask 可能得不偿失。

实施建议

如果你的系统也遇到了类似问题，建议可以这样考虑：先确认问题确实存在，检查是否真的有嵌套 CRC 的场景。然后选择合适的 mask 参数，可以参考 LevelDB 的做法，选择旋转位数和常数。别忘了实现 unmask 函数，读取时需要能还原出原始 CRC。充分测试验证很重要，要确保 mask 真的解决了退化问题。最后记录设计决策，给未来的维护者留个说明，解释为什么要这么做。

所以下次看到这种看似”多余”的代码时，不妨先想想是否有什么深层的原因。很多时候，这些操作都是经过深思熟虑的。

结论：从数学到工程的完美结合

我们从一个简单的观察开始：LevelDB 在存储 CRC 时要做一个看似多余的 mask 操作。通过一步步分析，我们发现这个操作其实解决了一个很深刻的数学问题。从数学层面看，CRC 基于 GF(2)多项式除法，当余数被附加回原消息时，形成的码字能被生成多项式整除，导致对这种”自包含”结构再次计算 CRC 时总是得到固定常数。从工程层面看，在分层存储系统中，这种退化会导致上层校验失效，无法检测到下层数据的损坏或篡改。

LevelDB 的 mask 方案体现了优秀的工程设计思路。它采用最小干预原则，不改变 CRC 算法本身，只在存储时做简单变换，对性能影响微乎其微。同时保持数学严密性，右旋转提供线性置换，加法常数引入非线性变换，彻底破坏 GF(2)域的代数结构。在工程实用性方面，操作可逆保证数据完整性，实现简单减少出错概率，向后兼容便于系统演进。

这个案例让我们看到系统软件设计的深层道理。数学基础的重要性在于，只有深入理解 CRC 的数学原理，才能发现问题的根源，进而设计出优雅的解决方案。细节往往是关键，一个看起来微不足道的操作，可能就是系统完整性的关键保护。简单往往最有效，面对复杂的数学问题，最好的工程解决方案通常是最简单的那个。

这种思路可以应用到其他地方。Hash 函数的嵌套使用中，其他校验算法可能也存在类似的退化问题。在密码学协议设计中，要小心协议消息的自包含可能带来的安全隐患。在编码理论应用中，纠错码分层使用时也要注意避免类似的陷阱。

作为开发者，这个案例给了我们启发：别急着删”多余”的代码，先搞清楚为什么要这样写，再考虑要不要”优化”。数学基础真的有用，很多看起来是工程问题的，其实根源都在数学。测试要考虑边界情况，特别是一些嵌套、组合的场景。好的注释很重要，给后来的人解释清楚”为什么这样做”。

LevelDB 的这个 mask 操作，表面上看就是几行简单的代码，但背后其实体现了对数学原理的深刻理解和对工程质量的严格要求。这也许就是优秀系统软件的特点吧：用最简单的方法解决最复杂的问题，在正确性和性能之间找到最佳的平衡。所以下次当你在代码中看到一些”奇怪”的操作时，不妨先停下来想想：这里是不是隐藏着什么有趣的设计想法呢？

参考资料

源码资料

• 原文链接 – https://blog.openacid.com/algo/crc
• Python 示例代码地址 – https://gist.github.com/drmingdrmer/2b49106b225ca7bf361fd21454f693da
• fuzhe at x – https://x.com/fuzhe19

LevelDB CRC 实现：

• LevelDB CRC32C 头文件 - Mask 函数的完整实现
• LevelDB CRC32C 实现文件 - CRC-32C 计算的完整代码
• LevelDB Block 格式文档 - 数据块存储格式说明

数学理论基础

CRC 算法原理：

• Wikipedia: Cyclic redundancy check - CRC 算法的数学基础
• Wikipedia: Finite field arithmetic - GF(2) 有限域运算
• Wikipedia: Irreducible polynomial - 不可约多项式的数学定义

多项式运算：

• Wikipedia: Polynomial long division - 多项式长除法
• Wikipedia: Galois field - 伽罗华域 GF(2) 的详细解释

CRC 标准和应用

CRC-32C (Castagnoli)：

• RFC 3720 - Internet Small Computer Systems Interface (iSCSI) - CRC-32C 在 iSCSI 中的应用
• Wikipedia: Cyclic redundancy check - CRC-32C - CRC-32C 算法详细说明

其他 CRC 实现：

• zlib CRC-32 实现 - 广泛使用的 CRC-32 实现

系统设计相关

存储系统设计：

• LevelDB 设计文档 - LevelDB 整体架构设计
• LSM-Tree 论文 - Log-Structured Merge Trees 原理

数据完整性：

• End-to-end data integrity - 端到端数据完整性保护

工程实践案例

类似的 Mask 技术：

• PostgreSQL CRC implementation - PostgreSQL 的 CRC 实现
• ZFS checksum algorithms - ZFS 文件系统的校验算法

错误检测技术：

• Reed-Solomon codes - 纠错码理论
• Hamming codes - 汉明码原理

实用工具

CRC 计算工具：

• Online CRC Calculator - 在线 CRC 计算器，支持多种 CRC 算法
• RevEng CRC Catalogue - 各种 CRC 算法的参数表

Python 实现：

• crcmod Python library - Python CRC 计算库
• zlib.crc32 文档 - Python 内置 CRC-32 函数

学术资料

密码学相关：

• Handbook of Applied Cryptography - 应用密码学手册

Reference:

前言

TL;DR

原生 Raft 配置变更需要两条日志条目，涉及多阶段提交与状态管理。本文探索单条日志完成配置变更的可行性，提出effective-config概念，分析其正确性与实际问题。 最终比较不同方案，解释为何标准 Joint Consensus 仍是推荐选择。

文章大纲

1. Raft Joint Consensus 简介
2. 单条日志变更的基本思路
3. 正确性证明
4. 单条日志方案的局限性及补丁
5. 结论

Raft Joint Consensus 简介, 2 条 Config log entry

在 Raft 共识算法中, 集群成员的变更是一个关键且复杂的操作. 直接从旧 config(例如 {a,b,c})切换到新 config(例如 {x,y,z})是不安全的. 因为集群中的 Node 不可能在同一时刻原子性地完成切换. 在切换过程中, 可能存在一个时间点, 一部分 Node(如 a,b)仍在使用旧 config C_old, 而另一部分 Node(如 x,y,z)已经切换到新 config C_new. 如果 C_old 的 Quorum(例如 {a,b})和 C_new 的 Quorum(例如 {x,y})没有交集, 就可能在同一任期(Term)内选出两个不同的 Leader, 这破坏了 Raft 的核心安全保证, 导致”split brain”.

为了安全地进行 config change, Raft 论文提出了一种名为 Joint Consensus(联合共识) 的两阶段方法：

1. 第一阶段：进入 Joint state (C_old_new):

Leader 收到变更请求时, 创建包含 Joint config C_old_new 的 log entry, 在 C_old_new 状态下, 任何决策都需同时获得 C_old 和 C_new 各自 Quorum 的支持. 一旦 Leader 添加 Joint config 到日志, 立即使用 C_old_new.

1. 第二阶段：切换到 Uniform state (C_new):

C_old_new 被提交后, Leader 创建第二个 config log entry, 仅包含新 config C_new. 一旦 Leader 看到这个新的 config entry, 立即切换为开始使用 C_new. 从这条日志开始, 这个 config entry 和所有后续的 log entry 只需要在 C_new 定义的 quorum 中提交. 当这个包含 C_new 的日志条目被提交后, config change 过程正式完成.

这种两阶段方法通过引入一个中间的”Joint state”, 保证了在整个 config change 过程中, 任意两个可能形成的 Quorum(无论是基于 C_old, C_new 还是 C_old_new)都必然存在交集, 从而防止了”split brain”问题, 确保了安全性. 然而, 这个过程需要两条config log entry 完成一次 config change.

希望只用一个 log entry 完成 安全的 Raft config change.

Joint Consensus 需要两个 log entry 和两个阶段, 略显复杂. 一个自然的想法是：能否只用一个config log entry 就安全地完成 config change?

Single log entry 的 config change 的想法在社区被先后提出过很多次, 下面我们将详细讨论这种 Single log entry 方法的工作原理, 并与标准的 Joint Consensus 方法进行对比分析, 看看它如何在保证安全性的同时简化实现流程.

这里我们引入一个概念 effective-config: 即 Leader 当前实际使用的、用它指定的 quorum 来判断一个 log entry 是否完成 commit 的 config. 这个 effective-config 可能跟 Leader 日志中任何一个已存储的 config log entry 中的 config 都不同. 它是一个动态的概念, 反映了 Leader 在 config change 过程中的”实时”决策依据. 在下面的算法执行步骤中我们会详细介绍它.

术语说明

• effective-config: Leader 当前用于判断 log entry 是否提交的动态配置集合.
• Joint config (联合配置): 同时包含旧 config 和新 config 的配置集, 例如 C_old_new.
• Uniform config (单一配置): 仅包含新 config 的配置集, 例如 C_new.
• Barrier entry (屏障条目): 用于标记 joint state 安全结束的特殊日志条目.

实现方法是:

使用单个 config log entry 完成变更的方法:

• 初始状态: 假设当前集群 config 为 {a,b,c}, 对应的 config log entry 已经提交, 记为 C_old. 这时 effective-config 也为 C_old. Raft 要求前一个 config log entry 必须提交后才能 propose 新的 config change, 这一点保持不变.

  

• 发起变更: 当需要将 config 更改为 {x,y,z} (记为 C_new) 时, Leader 提议 (propose) 一个新的 config log entry entry-i(log index 为 i), 该条目内包含 config C_new ({x,y,z}).

• 进入 Joint state (关键步骤): 当 Leader 看到 entry-i 时(即, 仅仅是追加到自己的日志中, 不需要等待它被提交), Leader 立即 将其 effective-config 变更为Joint config (联合配置): [{a,b,c}, {x,y,z}], 记为 C_old_new.

  • 这意味着从 entry-i 这条日志开始(包括 entry-i 自身), 后续的所有 log entry 都必须被复制到 C_old_new 所定义的 quorum 上(例如 [{a,b}, {x,y}]), 才能被认为是 committed.

  

• 允许写入: 在此期间, 集群仍然可以处理其他的 log entry(例如客户端请求), 但这些条目的提交同样需要满足 C_old_new 的 Joint Consensus 规则.

• 完成变更: 一旦 entry-i 在 C_old_new 的 quorum 中被成功 committed, Leader 就将 effective-config 切换回Uniform config (单一配置), 即新的 config C_new ({x,y,z}).

  • 从这个时间点开始, 后续的所有 log entry(entry-i 之后的条目)的提交, 只需提交到 config C_new 的 Quorum 即可.

通过这种方式, 理论上只用了一个 config log entry entry-i 就驱动了 C_old -> C_old_new -> C_new 的 config change.

正确性证明

我们需要证明这种单条目变更方法不会破坏 Raft 的安全性, 即, 不会在当前 Term 和 后续 Term 中产生两个 Leader(split brain).

假设执行 config change(提议 entry-i)的 Leader 的任期是 t. 考虑一个可能在之后当选的新 Candidate, 其任期为 u, 且 u > t. 我们需要证明 Leader t 和 要发起选举的 Candidate u 不会同时成为 Leader.

情况：新 Candidate u 的选举

新 Candidate u 的日志中可能包含 entry-i, 也可能不包含 entry-i, 分两种情况讨论:

• 如果 Candidate u 的日志中包含 entry-i, 则它的 effective-config 一定包含 {x,y,z}.

  旧 Leader t 的 effective-config 可能是 C_old_new: [{a,b,c}, {x,y,z}](config change 进行中), 或 C_new: {x,y,z}(config change 完成), 不论哪种情况, 它都包含 {x,y,z}. 而因为 Candidate u 必须需要获得 C_new: {x,y,z} 的 Quorum 的选票. 那么 u 和 t 无法同时成为 Leader;

• 如果 Candidate u 的日志中不包含 entry-i, 则它的 effective-config 一定包含 C_old: {a,b,c}. 这时要分两种情况讨论 Leader t 的 effective-config:

  • 如果 Leader t 的 effective-config 是 C_old_new: [{a,b,c}, {x,y,z}], 因为 Candidate u 需要获得 C_old 的 Quorum 的选票. 那么 u 和 t 无法同时成为 Leader;

  • 如果 t 的 effective-config 是 C_new: {x,y,z}, 那么说明 entry-i 已经在 C_old_new 上的一个 Quorum 被 commit, 那么就说明, entry-i 一定存在在 {a,b,c} 的一个 Quorum 的每个 Node 中. 因此这个 Quorum 里每个 Node 的 log index 已经大于等于 entry-i 的 log index: i.

    但是 Candidate u 不包含 entry-i, 即它的最大 log index 小于 i, 它在选举过程中会因为在 {a,b,c} 中的一个 Quorum 中看到大于 i 的 log entry 从而放弃选举.

所以在各种情况下, 在 Leader t 完成 config change 的整个过程, Candidate u 都不会成为 Leader, 从而避免了 split brain 的发生.

由此可见, 我们这种只需要单条 log entry 去进行 config change 的算法是正确的. 这种方法简化了配置变更过程, 减少了状态转换的复杂性.

然而, 尽管理论上正确, 但在实际实现中它会引入一些问题:

第 1 个问题: 持久层无法区分 Joint 和 Uniform

第一个问题是：节点从 Joint 状态切换到 Uniform 状态时(C_old_new 到 C_new), 变化仅在内存中发生(修改 effective-config), 没有写入持久层. 这导致：

C_old 中的节点仍可以发起选举, 与 C_new 中的节点竞争, 因为 C_old 的 log 和 C_new 的 log 一样多, 即使配置变更已完成, C_old 中的节点仍能抢走 C_new 中的节点的 Leadership. 这个问题根源是状态变化未记录在持久层上.

相比之下, 标准 Raft Joint Consensus 会将写一条新的 Uniform config (C_new) 日志, 形成屏障. C_old 中的节点在选举时会因日志不够新而被拒绝. 单条目配置变更对系统从 C_old_new 到 C_new 缺少这种持久层状态记录, 导致状态表达缺失, 也可以认为是一种状态回退.

例如下图中, 集群从 C_old_new 切换到 C_new 时, 任何 Node 的 log 都没有发生变化, 假设 Leadership 已经 transfer 到 C_new: {x,y,z}中的一个节点, 例如 x, 但C_old中的节点仍会发起选举, 并且可以抢走x的 Leadership.

补丁-1: 可以在进入 Uniform config 后, 在 C_new 的日志中添加一个 Noop log entry, 用来屏蔽 C_old 中的节点发起的选举.

第 2 个问题: 新启动 Node 必须先进入 Joint config

这种单条目 config change 方法虽然简洁, 但也引入了一个棘手的问题：一个新启动(重启)的 Node, 无法确定当前集群是处于 Joint state 还是 Uniform config 阶段:

• 启动后, 该 Node 在其持久化的日志存储中, 可以看到一系列的 config log entry. 假设它看到的最后两条 config log entry(按时间顺序)分别是 entry-i (包含 config C_old) 和 entry-j (包含 config C_new): i < j.

• 根据 Raft 规则, 可以确认 entry-i 一定是已提交的(因为 Leader 必须等前一个 config 提交后才能提议下一个).

• 但是, 对于最新的 entry-j, 该 Node 无法仅凭本地信息判断它是否已经被集群提交:

  • 如果 entry-j 未提交, 集群的 effective-config 应该是 Joint config C_old_new;
  • 如果 entry-j 已提交, 则 effective-config 应该是 Uniform config C_new.

这是 single log entry 做 config change 的第2个问题: 新启动的 Node 无法确定当前集群是处于 Joint state 还是 Uniform config 阶段.

例如上图中, 即使 entry-3 已经提交, 但新启动的 Node b,c,x,y 无法确定当前集群是处于 Joint state 还是 Uniform config 阶段(另外 2 个 Node a, z, 它们没收到 Joint config entry, 仍然停留在旧 config {a,b,c}).

了解了这个问题后, 它的解决方法也很直接: 补丁-2: 重启后回到 Joint state:

1. 新启动的 Node 在启动后, 必须将初始的 effective-config 设置为由最后两条 config log entry 组成的 Joint config C_old_new, 并使用此 Joint config 参与选举.
2. 只有在确认最后一个 config log entry(包含 C_new 的 log

entry) 在 C_old_new 上被提交后, 该 Node 才能安全地将其 effective-config 切换为 Uniform config C_new.

示例: 新启动的 Node 看到最后两条 config 为 {a,b,c} 和 {u,v,w} 时, 必须先使用 Joint config [{a,b,c}, {u,v,w}] 作为 effective-config, 需要获得两个 config 各自的 Quorum 投票才能当选 Leader, 确认新 config 在 [{a,b,c}, {u,v,w}] 上被提交后, 才能切换到仅使用 {u,v,w} 的 Uniform config.

但这个解决方案又引入了另一个问题:

第 3 个问题: 依赖旧 config 可能无法完成选举

补丁-2(新 Node 启动时默认使用 Joint config)解决了新 Node 状态不确定的问题, 但它自身又引入了另一个问题： 可能导致 Node 尝试联系已经被移除且不存在的旧 Node, 从而无法完成选举.

例子:

1. 集群 config 从 C_old ({a,b,c}) 变更到 C_new ({x,y,z}).

2. 包含 C_new 的 entry-j 已经被成功在 C_old_new 上提交.

3. 集群现在切换 config 到 C_new ({x,y,z}) 下.

4. 根据 C_new config 的定义, Node a, b, c 不再是集群成员. 它们可能已经被安全地关闭、数据被清理、并彻底从集群中移除.

5. 之后, 发生了某种情况导致集群中所有的 Node 全部重启.

6. Node x 重启后, 根据 补丁-2, 它检查日志, 发现最后两个 config 是 C_old ({a,b,c}) 和 C_new ({x,y,z}), 并将其 effective-config 设置为 Joint config ([{a,b,c}, {x,y,z}]).

7. Node x 决定发起选举. 但 Node b, c 已经不存在了！ 因此, Node x 无法获得 Joint config C_old_new 所需的选票, 选举失败, 集群可能无法选出 Leader 而永远无法恢复.

因为从 C_old_new 切换到 C_new 时, 没有持久化任何信息, 所以这个问题可以看做是系统在重启后发生的一次”状态回退”(C_old_new -> C_old)而造成的.

缓解措施: 可以通过外部的集群管理工具来延迟或协调旧 Node 的下线清理操作, 确保它们在被彻底移除前, 集群已经稳定运行在新 config 下一段时间. 但这只能”缓解”, 不能从根本上解决重启 Node 在特定时刻仍可能尝试联系旧 Node 的风险.

真正的解决方案还需要从 Raft 内部来解决:

添加 barrier 防止用旧 config 选举

为了解决 补丁-2 引入的”尝试联系已移除旧 Node 进行选举”的问题, 我们需要一种机制, 让重启的 Node 能够确切地知道 Joint state 是否已经安全结束, 从而避免使用包含旧 config 的 Joint config 进行选举.

补丁-3：引入 Barrier entry (屏障条目)

解决方案: 在 config log entry entry-j (C_new) 在 C_old_new 上完成 commit 后, 再 commit 一个特殊类型的 log entry, 我们称之为 Barrier entry, 表示 entry-j 已经完成 commit.

注意: 这个 Barrier entry 必须在 entry-j commit 后再 append 到 log 中, 否则它不能作为 entry-j 已经完成 commit 的根据.

这样, 新启动的 Node 如果看到这条 Barrier entry, 那么它就能确定之前的 Joint config C_old_new 状态已经安全结束, 可以直接使用新的 Uniform config C_new 来进行选举和其他操作, 而无需再尝试联系可能已经不存在的旧集群成员.

例如下图中, 当 entry-3 在 C_old_new 下被提交后, 添加一个 Barrier entry-4, 表示 entry-3 已经完成 commit:

这样在所有节点再次重启后, 也不会出现状态回退, Node x, y 看到 Barrier entry 后, 直接使用 C_new {x,y,z} 进行选举, 它们已经进入 C_new 的状态不会回退 这时即使 Node b, c 下线, Node x 或 y , 也能顺利当选 Leader:

Plan-B: 如果不用 log entry 做屏障, 另一个选择是 记录 commit-index, 这个方案最初由 马健将 提出.

这个方案中, 要求 Joint Consensus 结束的标准为：commit-index 同步到一个 C_new 的 quorum, 并且要求 Raft 的实现 持久化 commit-index 的值.

虽然标准 Raft 是不需要记录 commit-index 的值的, 但是记录 commit-index 几个题外话的好处, 在这里的好处就是可以当做一个状态存储的补充, 来区分 Joint 和 Uniform 状态.

因为 commit-index 记录了哪些 log entry 已经被提交, 如果 commit-index 包含 config change entry 的 index, 就可以知道 C_old_new 已经提交, 可以直接进入 C_new 状态, 从而避免出现向已清理节点请求选票的问题;

但是这个方案仍会出现 C_old_new 与 C_new 争夺 Leadership 的问题(问题-1), 因为 C_new 没有更多的 log, 而且 commit-index 只在 C_new 上提交无法保证一定会同步给 C_old 中的节点. 这是分布式中消息到底是投递至少一次还是投递至多一次的选择的问题：

• 如果选择至少一次, 即要求 commit-index 在 C_old_new 上 commit, 则可能出现 commit-index commit 成功了, C_old 节点被清理了, 然后再次尝试 commit commit-index 时无法完成的情况.
• 所以只能选择至多一次, 即 commit-index 只在 C_new 上 commit, 在 C_old 上尽量投递, 但这会导致 C_old 中节点可能没有收到 commit-index 更新的情况, 使它们不知道集群已经变化, 继续发起选举.

所以还有几率出现 C_old_new 与 C_new 争夺 Leadership 的问题(问题-1).

最后, 修改后的 Single log config change 流程如下:

1. 初始状态同上, effective-config 为 C_old ({a,b,c}).

2. Leader 提议一个包含 C_new = {x,y,z} 的 config log entry: entry-j, 并立即将 effective-config 切换为 Joint config C_old_new = [{a,b,c}, {x,y,z}].

3. 所有 index >= j 的日志在 C_old_new 下进行复制和提交.

4. 关键步骤: 一旦 entry-j 在 C_old_new 下成功提交, Leader 立即提议一个特殊的 Barrier entry. 这个 Barrier entry 不包含任何 config 信息, 它的存在仅仅是为了标记”Joint state 的安全结束点”. Leader 在提议 Barrier entry 的同时, 可以立即将 effective-config 切换为 C_new = {x,y,z}, 并使用这个新 config 来复制和提交 Barrier entry.

5. 当 Barrier entry 被提交后, config change 过程结束.

对应的, 新 Node 启动时的行为:

一个新启动的 Node 检查日志. 它看到最后两个 config 日志是 entry-i (C_old) 和 entry-j (C_new). 它首先检查在 entry-j 之后, 是否存在一个Barrier entry:

• 如果存在 Barrier entry: 这意味着 Joint state C_old_new 已经安全结束. 该 Node 应该直接将 effective-config 设置为C_new. 不需要去联系可能已经被移除的旧 Node.

• 如果不存在 Barrier entry: 这意味着 Joint state 可能仍在进行中. 那么该 Node 应该将 effective-config 设置为 Joint config C_old_new.

但是, 到这里细心的你一定已经发现: 这个 补丁-3 引入了第二条 log entry, 不再是严格意义上的”单条日志条目”完成, 而是需要”一条 config 条目 + 一条 Barrier 条目”.

总结

因为 config change 必然让系统经历 3 个阶段: C_old -> C_old_new -> C_new, 如果只用一条 log, 那么在持久层层面, 它只能表示 1 bit 的信息, 即最多表示 2 个状态: C_old 或 C_new, 这是问题的根本所在, 一个通用且安全的 config change 算法, 至少需要 2 条 log entry 参与, 才能提供 2 bit 的信息, 最多能表达 4 个状态, 来支持系统所需的 3 个状态.

所以, 还是老老实实回到 Joint Consensus 吧, 它更简洁.

参考资料

• Diego Ongaro & John Ousterhout. In Search of an Understandable Consensus Algorithm (Raft 原论文): https://raft.github.io/raft.pdf
• OpenRaft(rust): https://github.com/databendlabs/openraft
• etcd/raft 项目源码: https://github.com/etcd-io/raft
• Hashicorp Raft 实现: https://github.com/hashicorp/raft

Reference:

• 马健将 : https://weibo.com/u/1516609505