<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <link href="https://blog.openacid.com/feed.xml" rel="self" type="application/atom+xml"/>
  <link href="https://blog.openacid.com/" rel="alternate" type="text/html"/>
  <updated>2026-10-24T15:17:47&#43;00:00</updated>
  <id>https://blog.openacid.com/feed.xml</id>
  <title type="html">OpenACID Blog</title>
  <subtitle>Distributed systems research notes.</subtitle>
  <author><name>OpenACID</name></author>
  <entry>
    <title type="html">Adaptive Membership Changes in OpenRaft: Skip Joint Consensus When Quorums Overlap</title>
    <link href="https://blog.openacid.com/distributed/raft-adaptive-config-change/" rel="alternate" type="text/html" title="Adaptive Membership Changes in OpenRaft: Skip Joint Consensus When Quorums Overlap"/>
    <published>2026-10-21T00:00:00&#43;00:00</published>
    <updated>2026-10-21T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/distributed/raft-adaptive-config-change</id>
    <content type="html" xml:base="https://blog.openacid.com/distributed/raft-adaptive-config-change/">&lt;p&gt;&lt;img src=&#34;/post-res/raft-adaptive-config-change/3ca421a19971e5c0-raft-adaptive-config-change-banner-small.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;OpenRaft now skips joint consensus whenever the old and new configurations have quorum overlap: every old quorum shares at least one node with every new quorum.&lt;/p&gt;&#xA;&lt;p&gt;For &lt;code&gt;Raft::change_membership()&lt;/code&gt;, that means two possible paths:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;With quorum overlap, write one entry containing the new configuration.&lt;/li&gt;&#xA;&lt;li&gt;Without quorum overlap, write a joint configuration first, then the new configuration: two entries in total.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The idea follows from the purpose of joint consensus. A joint configuration lets us cross from the old configuration to the new one in two steps, each with the required quorum overlap. If the old and new quorums already overlap, we can make that transition directly.&lt;/p&gt;&#xA;&lt;p&gt;For configurations that use majority quorums, OpenRaft can test this condition exactly in linear time. Raft&amp;rsquo;s familiar &lt;a href=&#34;https://github.com/ongardie/dissertation&#34; title=&#34;Diego Ongaro: Consensus: Bridging Theory and Practice, Chapter 4&#34;&gt;single-server change&lt;/a&gt; rule—adding or removing one node at a time—is a simpler way to guarantee the same property. The exact test also admits a few changes involving two nodes.&lt;/p&gt;&#xA;&lt;h2 id=&#34;1-joint-consensus-provides-quorum-overlap-at-each-step&#34;&gt;1. Joint consensus provides quorum overlap at each step&lt;/h2&gt;&#xA;&lt;p&gt;Let &lt;code&gt;C0&lt;/code&gt; be the old configuration, &lt;code&gt;C1&lt;/code&gt; the new one, and &lt;code&gt;C01 = [C0,C1]&lt;/code&gt; their joint configuration. Write &lt;code&gt;C0 ~ C1&lt;/code&gt; when every quorum of &lt;code&gt;C0&lt;/code&gt; intersects every quorum of &lt;code&gt;C1&lt;/code&gt;. A direct transition from &lt;code&gt;C0&lt;/code&gt; to &lt;code&gt;C1&lt;/code&gt; requires this property; Section 3 of &lt;a href=&#34;https://blog.openacid.com/distributed/raft-membership-log-barrier/&#34; title=&#34;Raft Membership Changes: When a Leader Must Commit a Current-Term Entry First&#34;&gt;Raft Membership Changes: When a Leader Must Commit a Current-Term Entry First&lt;/a&gt; explains why.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Joint consensus splits a configuration change into two transitions that each preserve quorum overlap.&lt;/strong&gt; Every quorum of &lt;code&gt;C01&lt;/code&gt; contains a quorum of &lt;code&gt;C0&lt;/code&gt; and a quorum of &lt;code&gt;C1&lt;/code&gt;. As a result, &lt;code&gt;C0 ~ C01&lt;/code&gt; and &lt;code&gt;C01 ~ C1&lt;/code&gt; always hold, even when &lt;code&gt;C0 ~ C1&lt;/code&gt; does not.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Joint change, two entries:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|     C0 -----&amp;gt; C01 -----&amp;gt; C1&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|        C0 ~ C01   C01 ~ C1&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| When C0 ~ C1, one entry:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|     C0 ----------------&amp;gt; C1&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;When &lt;code&gt;C0 ~ C1&lt;/code&gt; already holds, we can go straight to &lt;code&gt;C1&lt;/code&gt;. The joint step would add an entry and another wait for commitment without providing any additional overlap.&lt;/p&gt;&#xA;&lt;p&gt;There is one other requirement: the leader must first commit an entry from its own term. Skipping that barrier allows the bug described in &lt;a href=&#34;https://blog.openacid.com/distributed/raft-bug-cn/&#34; title=&#34;Raft 成员变更上踩的坑&#34;&gt;The Pitfalls of Raft Membership Change&lt;/a&gt;; the &lt;a href=&#34;https://blog.openacid.com/distributed/raft-membership-log-barrier/&#34; title=&#34;Raft Membership Changes: When a Leader Must Commit a Current-Term Entry First&#34;&gt;article on membership log barriers&lt;/a&gt; explains the failure in detail.&lt;/p&gt;&#xA;&lt;p&gt;OpenRaft already enforces this barrier for every membership change. It returns &lt;code&gt;UncommittedLeaderLog&lt;/code&gt; until the first entry the leader writes after its election has committed. With that requirement satisfied, the choice between a direct change and joint consensus comes down to &lt;code&gt;C0 ~ C1&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;2-majority-sizes-give-us-an-exact-overlap-test&#34;&gt;2. Majority sizes give us an exact overlap test&lt;/h2&gt;&#xA;&lt;p&gt;For a configuration &lt;code&gt;C&lt;/code&gt; that uses majority quorums, define its majority size as &lt;code&gt;maj(C) = ⌊|C|/2⌋ + 1&lt;/code&gt;. Two such configurations have quorum overlap exactly when their majority sizes add up to more than the number of distinct nodes across both configurations:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;code&gt;C0 ~ C1&lt;/code&gt; if and only if &lt;code&gt;maj(C0) + maj(C1) &amp;gt; |C0 ∪ C1|&lt;/code&gt;.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;We only need to compute one set union. The work is linear in the total number of nodes in the two configurations. Enumerating all possible quorums to check the definition directly would take exponential time.&lt;/p&gt;&#xA;&lt;p&gt;To see why the test works, suppose we can pick an old quorum and a new quorum that share no nodes. Together, they occupy at least &lt;code&gt;maj(C0) + maj(C1)&lt;/code&gt; distinct nodes, all within &lt;code&gt;C0 ∪ C1&lt;/code&gt;. That requires &lt;code&gt;maj(C0) + maj(C1) ≤ |C0 ∪ C1|&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The converse holds too. If &lt;code&gt;maj(C0) + maj(C1) ≤ |C0 ∪ C1|&lt;/code&gt;, build each quorum from nodes unique to its configuration first. Then fill any remaining places with shared nodes, assigning each shared node to only one quorum. The inequality guarantees enough nodes to finish both quorums without overlap.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Adding or removing one node always passes the overlap test.&lt;/strong&gt; When we add one node, &lt;code&gt;C1&lt;/code&gt; contains all of &lt;code&gt;C0&lt;/code&gt;, so &lt;code&gt;|C0 ∪ C1| = |C1| = |C0| + 1&lt;/code&gt;. The two majority sizes add up to:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;maj(C0) + maj(C1) = ⌊|C0|/2⌋ + ⌊(|C0|+1)/2⌋ + 2 = |C0| + 2&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;That is one more than the size of the union, so quorum overlap is guaranteed. Removing one node gives the same result with &lt;code&gt;C0&lt;/code&gt; and &lt;code&gt;C1&lt;/code&gt; exchanged. The exact test also tells us when a change involving two nodes can skip joint consensus:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Change&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Exact test&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;One-node-at-a-time rule&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;{a,b,c} → {a,b,c,d}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Overlap&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Allowed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;{a,b,c} → {b,c,d}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No overlap: quorum &lt;code&gt;{a,b}&lt;/code&gt; of &lt;code&gt;C0&lt;/code&gt; and quorum &lt;code&gt;{c,d}&lt;/code&gt; of &lt;code&gt;C1&lt;/code&gt; share no nodes&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Not allowed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;{a,b,c,d} → {a,b,c,e}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Overlap&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Not allowed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;The exact test admits three kinds of changes beyond the one-node-at-a-time rule. All three start and end with an even number of voters:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Replace one voter: &lt;code&gt;{a,b,c,d} → {a,b,c,e}&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Add two voters: &lt;code&gt;{a,b,c,d} → {a,b,c,d,e,f}&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Remove two voters: &lt;code&gt;{a,b,c,d} → {a,b}&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Checking the even and odd cases in the formula shows that these are the only additional possibilities. Production clusters usually have an odd number of voters. Starting from an odd-sized configuration, the exact test permits the same changes as the one-node-at-a-time rule.&lt;/p&gt;&#xA;&lt;h2 id=&#34;3-openraft-chooses-the-next-configuration-with-bridge_to&#34;&gt;3. OpenRaft chooses the next configuration with bridge_to()&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&#34;https://github.com/databendlabs/openraft/commit/c90c6b5311654f83a9095187923bc102420b36a8&#34; title=&#34;feat: membership: skip the joint step when quorums intersect&#34;&gt;Commit c90c6b53&lt;/a&gt; brings this test to &lt;code&gt;Raft::change_membership()&lt;/code&gt;. It uses &lt;code&gt;bridge_to()&lt;/code&gt; from &lt;a href=&#34;https://github.com/drmingdrmer/quorum-set&#34; title=&#34;quorum-set&#34;&gt;quorum-set&lt;/a&gt; 0.3.0, the library OpenRaft uses for quorum calculations, to choose the next configuration.&lt;/p&gt;&#xA;&lt;p&gt;When the current configuration is not joint, &lt;code&gt;bridge_to()&lt;/code&gt; applies the exact test from Section 2. If the quorums overlap, it returns the target configuration, completing the change with one entry. Otherwise, it returns the joint configuration &lt;code&gt;C01&lt;/code&gt;. &lt;code&gt;change_membership()&lt;/code&gt; waits for that entry to commit before appending the target configuration.&lt;/p&gt;&#xA;&lt;p&gt;Adding or removing one voter therefore takes a single entry. So do the three additional cases involving even-sized configurations. Replacing all three voters in &lt;code&gt;{a,b,c} → {d,e,f}&lt;/code&gt; still goes through joint consensus: the old and new quorums have no overlap.&lt;/p&gt;&#xA;&lt;p&gt;Before this change, &lt;code&gt;change_membership()&lt;/code&gt; wrote a joint configuration whenever the voters changed. It now takes the direct path whenever the overlap test permits it.&lt;/p&gt;&#xA;&lt;h2 id=&#34;4-how-21-other-raft-implementations-choose-the-transition&#34;&gt;4. How 21 other Raft implementations choose the transition&lt;/h2&gt;&#xA;&lt;p&gt;The 21 implementations below fall into three groups. Each code link is pinned to the commit examined.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Six implementations use the number of changes to decide whether to skip joint consensus:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Implementation&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Condition for skipping joint consensus&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Code&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;etcd-io/raft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The request uses &lt;code&gt;ConfChangeTransitionAuto&lt;/code&gt; and contains exactly one change&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/etcd-io/raft/blob/1931ff5beacc1e2539003c0c70f096285f04c425/raftpb/confchange.go#L85-L91&#34; title=&#34;etcd-io/raft: confchange.go&#34;&gt;raftpb/confchange.go:85-91&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;tikv/raft-rs&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The same conditions as etcd-io/raft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/tikv/raft-rs/blob/ad13f3d90780f53aea2488c6a4b76c0d334bf136/proto/src/confchange.rs#L134&#34; title=&#34;tikv/raft-rs: confchange.rs&#34;&gt;proto/src/confchange.rs:134&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;TiKV&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The request contains at most one change, in which case TiKV sends &lt;code&gt;ConfChangeTransitionAuto&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/tikv/tikv/blob/134cdb2ce9852aa30e878ade3849f877dd921018/components/raftstore/src/store/util.rs#L982&#34; title=&#34;TiKV: util.rs&#34;&gt;util.rs:982&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;CockroachDB&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The request contains exactly one change, and that change does not demote a voter&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/cockroachdb/cockroach/blob/d30c905fff79ef825adc96bcc647f1872a90f2ff/pkg/kv/kvserver/replica_command.go#L1978&#34; title=&#34;CockroachDB: replica_command.go&#34;&gt;replica_command.go:1978&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;braft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The total number of nodes added and removed is at most one&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/baidu/braft/blob/ab0017f0b98d429138d83a04d3ed351197d671a9/src/braft/node.cpp#L3296&#34; title=&#34;braft: node.cpp&#34;&gt;node.cpp:3296&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Redpanda&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Each learner is promoted to voter individually, skipping joint consensus. Removing a voter always uses joint consensus&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/redpanda-data/redpanda/blob/4254992f6115e92c1dbd86f90bb0aee0b869f15e/src/v/raft/consensus.cc#L550&#34; title=&#34;Redpanda: consensus.cc&#34;&gt;consensus.cc:550&lt;/a&gt;, &lt;a href=&#34;https://github.com/redpanda-data/redpanda/blob/4254992f6115e92c1dbd86f90bb0aee0b869f15e/src/v/raft/group_configuration.cc#L1391&#34; title=&#34;Redpanda: group_configuration.cc&#34;&gt;group_configuration.cc:1391&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;strong&gt;Six implementations use joint consensus whenever the voters change:&lt;/strong&gt; &lt;a href=&#34;https://github.com/sofastack/sofa-jraft/blob/cb1a0c7ac16ff146ad8abb04b1bcb035270bb234/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java#L510&#34; title=&#34;SOFAJRaft: NodeImpl.java&#34;&gt;SOFAJRaft&lt;/a&gt;, &lt;a href=&#34;https://github.com/apache/ratis/blob/5b52180ff0867fbf879832469051eaec2657953c/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderStateImpl.java#L632&#34; title=&#34;Apache Ratis: LeaderStateImpl.java&#34;&gt;Apache Ratis&lt;/a&gt;, &lt;a href=&#34;https://github.com/logcabin/logcabin/blob/ee6c55ae9744b82b451becd9707d26c7c1b6bbfb/Server/RaftConsensus.cc#L1677&#34; title=&#34;LogCabin: RaftConsensus.cc&#34;&gt;LogCabin&lt;/a&gt;, &lt;a href=&#34;https://github.com/scylladb/scylladb/blob/5e0d3eb88f8e6b7a88623b8e26816258c68a2c66/raft/fsm.cc#L191&#34; title=&#34;ScyllaDB: fsm.cc&#34;&gt;ScyllaDB&lt;/a&gt;, &lt;a href=&#34;https://github.com/camunda/camunda/blob/e60d9d54786f77a755fdbb7f4af0200388ab7714/zeebe/atomix/cluster/src/main/java/io/atomix/raft/roles/LeaderRole.java#L360&#34; title=&#34;Zeebe: LeaderRole.java&#34;&gt;Zeebe (Camunda&amp;rsquo;s Atomix Raft)&lt;/a&gt;, and &lt;a href=&#34;https://github.com/rethinkdb/rethinkdb/blob/a2cf1308b57dca2e05d744f225a082881bab3fca/src/clustering/generic/raft_core.tcc#L237&#34; title=&#34;RethinkDB: raft_core.tcc&#34;&gt;RethinkDB&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Nine implementations support only one-node-at-a-time changes, with no joint consensus:&lt;/strong&gt; &lt;a href=&#34;https://github.com/hashicorp/raft/blob/a5b469295ddc9924541477eafe34bf59a8571063/configuration.go#L224&#34; title=&#34;hashicorp/raft: configuration.go&#34;&gt;hashicorp/raft&lt;/a&gt;, &lt;a href=&#34;https://github.com/lni/dragonboat/blob/076c7f6497dcc18880aed6323246d5079661942c/raftpb/configchange.go#L11&#34; title=&#34;Dragonboat: configchange.go&#34;&gt;Dragonboat&lt;/a&gt;, &lt;a href=&#34;https://github.com/eBay/NuRaft/blob/46c73ad37cf5750342dd7b3b222422993f638251/src/handle_join_leave.cxx#L78&#34; title=&#34;NuRaft: handle_join_leave.cxx&#34;&gt;NuRaft&lt;/a&gt;, &lt;a href=&#34;https://github.com/canonical/raft/blob/c17fecbd4841f302b648fee26139ed29efbb2717/include/raft.h#L1091&#34; title=&#34;canonical/raft: raft.h&#34;&gt;canonical/raft&lt;/a&gt;, &lt;a href=&#34;https://github.com/rabbitmq/ra/blob/2d81d2126c03aaba50d5b6ac850853e2a77ba3c8/README.md?plain=1#L26&#34; title=&#34;RabbitMQ Ra: README.md&#34;&gt;RabbitMQ Ra&lt;/a&gt;, &lt;a href=&#34;https://github.com/apache/kafka/blob/9fbcd3f76cecd04015e5d893976d9849f407af30/raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java#L51&#34; title=&#34;Kafka KRaft: AddVoterHandler.java&#34;&gt;Kafka KRaft&lt;/a&gt;, &lt;a href=&#34;https://github.com/apache/kudu/blob/a0409299d44318756156ca8760812b1fdfadf291/src/kudu/consensus/raft_consensus.cc#L2074&#34; title=&#34;Kudu: raft_consensus.cc&#34;&gt;Kudu&lt;/a&gt;, &lt;a href=&#34;https://github.com/yugabyte/yugabyte-db/blob/d2e9935d89e003240174b045199d9d8242939c71/src/yb/consensus/consensus.proto#L89-L93&#34; title=&#34;YugabyteDB: consensus.proto&#34;&gt;YugabyteDB&lt;/a&gt;, and &lt;a href=&#34;https://github.com/mongodb/mongo/blob/ffc52d7f3327dffdd92e5142a893e079527088c0/src/mongo/db/repl/repl_set_config_checks.cpp#L115&#34; title=&#34;MongoDB: repl_set_config_checks.cpp&#34;&gt;MongoDB&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;etcd&amp;rsquo;s choice to count changes is deliberate. The comment above the check in &lt;a href=&#34;https://github.com/etcd-io/raft/blob/1931ff5beacc1e2539003c0c70f096285f04c425/raftpb/confchange.go#L85-L91&#34; title=&#34;etcd-io/raft: confchange.go&#34;&gt;&lt;code&gt;raftpb/confchange.go&lt;/code&gt;&lt;/a&gt; explains why it uses joint consensus even when a simpler transition might be safe:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;NB: in theory, more config changes could qualify for the &amp;ldquo;simple&amp;rdquo; protocol but it depends on the config on top of which the changes apply. For example, adding two learners is not OK if both nodes are part of the base config (i.e. two voters are turned into learners in the process of applying the conf change). In practice, these distinctions should not matter, so we keep it simple and use Joint Consensus liberally.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;MongoDB explicitly connects the one-node-at-a-time rule to quorum overlap. Its &lt;code&gt;validateSingleNodeChange&lt;/code&gt; rejects reconfigurations that add or remove more than one node (&lt;a href=&#34;https://github.com/mongodb/mongo/blob/ffc52d7f3327dffdd92e5142a893e079527088c0/src/mongo/db/repl/repl_set_config_checks.cpp#L115&#34; title=&#34;MongoDB: repl_set_config_checks.cpp&#34;&gt;repl_set_config_checks.cpp:115&lt;/a&gt;). The comment at the call site gives the reason (&lt;a href=&#34;https://github.com/mongodb/mongo/blob/ffc52d7f3327dffdd92e5142a893e079527088c0/src/mongo/db/repl/repl_set_config_checks.cpp#L544-L546&#34; title=&#34;MongoDB: repl_set_config_checks.cpp&#34;&gt;lines 544–546&lt;/a&gt;):&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;For non-force reconfigs, verify that the reconfig only adds or removes a single node. This ensures that all quorums of the new config overlap with all quorums of the old config.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;William Schultz, first author of the &lt;a href=&#34;https://arxiv.org/abs/2102.11960&#34; title=&#34;Design and Analysis of a Logless Dynamic Reconfiguration Protocol&#34;&gt;paper on MongoDB&amp;rsquo;s reconfiguration protocol&lt;/a&gt;, proposed using the more general &lt;code&gt;QuorumsOverlap&lt;/code&gt; condition in a &lt;a href=&#34;https://will62794.github.io/distributed-systems/2024/02/09/note-on-raft-reconfig-bug.html&#34; title=&#34;William Schultz: On Raft&#39;s Reconfiguration Bug&#34;&gt;2024 note&lt;/a&gt;. This is exactly &lt;code&gt;C0 ~ C1&lt;/code&gt;. His TLA+ model, &lt;a href=&#34;https://github.com/will62794/logless-reconfig/blob/05178a700a05e3a3e7f1808c39a5f61f2e6cece0/MongoLoglessDynamicRaft.tla#L95-L99&#34; title=&#34;MongoLoglessDynamicRaft.tla: Reconfig&#34;&gt;MongoLoglessDynamicRaft.tla&lt;/a&gt;, already uses &lt;code&gt;QuorumsOverlap&lt;/code&gt; as a precondition for reconfiguration. MongoDB&amp;rsquo;s implementation still enforces the one-node-at-a-time restriction.&lt;/p&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;braft: node.cpp : &lt;a href=&#34;https://github.com/baidu/braft/blob/ab0017f0b98d429138d83a04d3ed351197d671a9/src/braft/node.cpp#L3296&#34;&gt;https://github.com/baidu/braft/blob/ab0017f0b98d429138d83a04d3ed351197d671a9/src/braft/node.cpp#L3296&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;canonical/raft: raft.h : &lt;a href=&#34;https://github.com/canonical/raft/blob/c17fecbd4841f302b648fee26139ed29efbb2717/include/raft.h#L1091&#34;&gt;https://github.com/canonical/raft/blob/c17fecbd4841f302b648fee26139ed29efbb2717/include/raft.h#L1091&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;CockroachDB: replica_command.go : &lt;a href=&#34;https://github.com/cockroachdb/cockroach/blob/d30c905fff79ef825adc96bcc647f1872a90f2ff/pkg/kv/kvserver/replica_command.go#L1978&#34;&gt;https://github.com/cockroachdb/cockroach/blob/d30c905fff79ef825adc96bcc647f1872a90f2ff/pkg/kv/kvserver/replica_command.go#L1978&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Dragonboat: configchange.go : &lt;a href=&#34;https://github.com/lni/dragonboat/blob/076c7f6497dcc18880aed6323246d5079661942c/raftpb/configchange.go#L11&#34;&gt;https://github.com/lni/dragonboat/blob/076c7f6497dcc18880aed6323246d5079661942c/raftpb/configchange.go#L11&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;etcd-io/raft: confchange.go : &lt;a href=&#34;https://github.com/etcd-io/raft/blob/1931ff5beacc1e2539003c0c70f096285f04c425/raftpb/confchange.go#L85-L91&#34;&gt;https://github.com/etcd-io/raft/blob/1931ff5beacc1e2539003c0c70f096285f04c425/raftpb/confchange.go#L85-L91&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;hashicorp/raft: configuration.go : &lt;a href=&#34;https://github.com/hashicorp/raft/blob/a5b469295ddc9924541477eafe34bf59a8571063/configuration.go#L224&#34;&gt;https://github.com/hashicorp/raft/blob/a5b469295ddc9924541477eafe34bf59a8571063/configuration.go#L224&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Kafka KRaft: AddVoterHandler.java : &lt;a href=&#34;https://github.com/apache/kafka/blob/9fbcd3f76cecd04015e5d893976d9849f407af30/raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java#L51&#34;&gt;https://github.com/apache/kafka/blob/9fbcd3f76cecd04015e5d893976d9849f407af30/raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java#L51&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Kudu: raft_consensus.cc : &lt;a href=&#34;https://github.com/apache/kudu/blob/a0409299d44318756156ca8760812b1fdfadf291/src/kudu/consensus/raft_consensus.cc#L2074&#34;&gt;https://github.com/apache/kudu/blob/a0409299d44318756156ca8760812b1fdfadf291/src/kudu/consensus/raft_consensus.cc#L2074&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;LogCabin: RaftConsensus.cc : &lt;a href=&#34;https://github.com/logcabin/logcabin/blob/ee6c55ae9744b82b451becd9707d26c7c1b6bbfb/Server/RaftConsensus.cc#L1677&#34;&gt;https://github.com/logcabin/logcabin/blob/ee6c55ae9744b82b451becd9707d26c7c1b6bbfb/Server/RaftConsensus.cc#L1677&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;MongoDB: repl_set_config_checks.cpp : &lt;a href=&#34;https://github.com/mongodb/mongo/blob/ffc52d7f3327dffdd92e5142a893e079527088c0/src/mongo/db/repl/repl_set_config_checks.cpp#L115&#34;&gt;https://github.com/mongodb/mongo/blob/ffc52d7f3327dffdd92e5142a893e079527088c0/src/mongo/db/repl/repl_set_config_checks.cpp#L115&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;MongoDB: repl_set_config_checks.cpp : &lt;a href=&#34;https://github.com/mongodb/mongo/blob/ffc52d7f3327dffdd92e5142a893e079527088c0/src/mongo/db/repl/repl_set_config_checks.cpp#L544-L546&#34;&gt;https://github.com/mongodb/mongo/blob/ffc52d7f3327dffdd92e5142a893e079527088c0/src/mongo/db/repl/repl_set_config_checks.cpp#L544-L546&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;NuRaft: handle_join_leave.cxx : &lt;a href=&#34;https://github.com/eBay/NuRaft/blob/46c73ad37cf5750342dd7b3b222422993f638251/src/handle_join_leave.cxx#L78&#34;&gt;https://github.com/eBay/NuRaft/blob/46c73ad37cf5750342dd7b3b222422993f638251/src/handle_join_leave.cxx#L78&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;RabbitMQ Ra: README.md : &lt;a href=&#34;https://github.com/rabbitmq/ra/blob/2d81d2126c03aaba50d5b6ac850853e2a77ba3c8/README.md?plain=1#L26&#34;&gt;https://github.com/rabbitmq/ra/blob/2d81d2126c03aaba50d5b6ac850853e2a77ba3c8/README.md?plain=1#L26&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;tikv/raft-rs: confchange.rs : &lt;a href=&#34;https://github.com/tikv/raft-rs/blob/ad13f3d90780f53aea2488c6a4b76c0d334bf136/proto/src/confchange.rs#L134&#34;&gt;https://github.com/tikv/raft-rs/blob/ad13f3d90780f53aea2488c6a4b76c0d334bf136/proto/src/confchange.rs#L134&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Apache Ratis: LeaderStateImpl.java : &lt;a href=&#34;https://github.com/apache/ratis/blob/5b52180ff0867fbf879832469051eaec2657953c/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderStateImpl.java#L632&#34;&gt;https://github.com/apache/ratis/blob/5b52180ff0867fbf879832469051eaec2657953c/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderStateImpl.java#L632&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Redpanda: consensus.cc : &lt;a href=&#34;https://github.com/redpanda-data/redpanda/blob/4254992f6115e92c1dbd86f90bb0aee0b869f15e/src/v/raft/consensus.cc#L550&#34;&gt;https://github.com/redpanda-data/redpanda/blob/4254992f6115e92c1dbd86f90bb0aee0b869f15e/src/v/raft/consensus.cc#L550&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Redpanda: group_configuration.cc : &lt;a href=&#34;https://github.com/redpanda-data/redpanda/blob/4254992f6115e92c1dbd86f90bb0aee0b869f15e/src/v/raft/group_configuration.cc#L1391&#34;&gt;https://github.com/redpanda-data/redpanda/blob/4254992f6115e92c1dbd86f90bb0aee0b869f15e/src/v/raft/group_configuration.cc#L1391&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;RethinkDB: raft_core.tcc : &lt;a href=&#34;https://github.com/rethinkdb/rethinkdb/blob/a2cf1308b57dca2e05d744f225a082881bab3fca/src/clustering/generic/raft_core.tcc#L237&#34;&gt;https://github.com/rethinkdb/rethinkdb/blob/a2cf1308b57dca2e05d744f225a082881bab3fca/src/clustering/generic/raft_core.tcc#L237&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;MongoLoglessDynamicRaft.tla: Reconfig : &lt;a href=&#34;https://github.com/will62794/logless-reconfig/blob/05178a700a05e3a3e7f1808c39a5f61f2e6cece0/MongoLoglessDynamicRaft.tla#L95-L99&#34;&gt;https://github.com/will62794/logless-reconfig/blob/05178a700a05e3a3e7f1808c39a5f61f2e6cece0/MongoLoglessDynamicRaft.tla#L95-L99&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ScyllaDB: fsm.cc : &lt;a href=&#34;https://github.com/scylladb/scylladb/blob/5e0d3eb88f8e6b7a88623b8e26816258c68a2c66/raft/fsm.cc#L191&#34;&gt;https://github.com/scylladb/scylladb/blob/5e0d3eb88f8e6b7a88623b8e26816258c68a2c66/raft/fsm.cc#L191&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;SOFAJRaft: NodeImpl.java : &lt;a href=&#34;https://github.com/sofastack/sofa-jraft/blob/cb1a0c7ac16ff146ad8abb04b1bcb035270bb234/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java#L510&#34;&gt;https://github.com/sofastack/sofa-jraft/blob/cb1a0c7ac16ff146ad8abb04b1bcb035270bb234/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java#L510&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;TiKV: util.rs : &lt;a href=&#34;https://github.com/tikv/tikv/blob/134cdb2ce9852aa30e878ade3849f877dd921018/components/raftstore/src/store/util.rs#L982&#34;&gt;https://github.com/tikv/tikv/blob/134cdb2ce9852aa30e878ade3849f877dd921018/components/raftstore/src/store/util.rs#L982&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;YugabyteDB: consensus.proto : &lt;a href=&#34;https://github.com/yugabyte/yugabyte-db/blob/d2e9935d89e003240174b045199d9d8242939c71/src/yb/consensus/consensus.proto#L89-L93&#34;&gt;https://github.com/yugabyte/yugabyte-db/blob/d2e9935d89e003240174b045199d9d8242939c71/src/yb/consensus/consensus.proto#L89-L93&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Zeebe: LeaderRole.java : &lt;a href=&#34;https://github.com/camunda/camunda/blob/e60d9d54786f77a755fdbb7f4af0200388ab7714/zeebe/atomix/cluster/src/main/java/io/atomix/raft/roles/LeaderRole.java#L360&#34;&gt;https://github.com/camunda/camunda/blob/e60d9d54786f77a755fdbb7f4af0200388ab7714/zeebe/atomix/cluster/src/main/java/io/atomix/raft/roles/LeaderRole.java#L360&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;feat: membership: skip the joint step when quorums intersect : &lt;a href=&#34;https://github.com/databendlabs/openraft/commit/c90c6b5311654f83a9095187923bc102420b36a8&#34;&gt;https://github.com/databendlabs/openraft/commit/c90c6b5311654f83a9095187923bc102420b36a8&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Raft 成员变更上踩的坑 : &lt;a href=&#34;https://blog.openacid.com/distributed/raft-bug-cn/&#34;&gt;https://blog.openacid.com/distributed/raft-bug-cn/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Raft Membership Changes: When a Leader Must Commit a Current-Term Entry First : &lt;a href=&#34;https://blog.openacid.com/distributed/raft-membership-log-barrier/&#34;&gt;https://blog.openacid.com/distributed/raft-membership-log-barrier/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Design and Analysis of a Logless Dynamic Reconfiguration Protocol : &lt;a href=&#34;https://arxiv.org/abs/2102.11960&#34;&gt;https://arxiv.org/abs/2102.11960&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Diego Ongaro: Consensus: Bridging Theory and Practice, Chapter 4 : &lt;a href=&#34;https://github.com/ongardie/dissertation&#34;&gt;https://github.com/ongardie/dissertation&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;William Schultz: On Raft&amp;rsquo;s Reconfiguration Bug : &lt;a href=&#34;https://will62794.github.io/distributed-systems/2024/02/09/note-on-raft-reconfig-bug.html&#34;&gt;https://will62794.github.io/distributed-systems/2024/02/09/note-on-raft-reconfig-bug.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;quorum-set : &lt;a href=&#34;https://github.com/drmingdrmer/quorum-set&#34;&gt;https://github.com/drmingdrmer/quorum-set&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="distributed"/>
    <category term="raft"/>
    <category term="consensus"/>
    <category term="membership"/>
    <summary type="html">OpenRaft now skips joint consensus when every old quorum intersects every new quorum. A linear-time test identifies exactly when this is safe for majority-based configurations. It covers Raft&#39;s familiar one-node-at-a-time changes and a few additional cases, while retaining joint consensus for changes that need it.</summary>
  </entry>
  <entry>
    <title type="html">Raft Membership Changes: When a Leader Must Commit a Current-Term Entry First</title>
    <link href="https://blog.openacid.com/distributed/raft-membership-log-barrier/" rel="alternate" type="text/html" title="Raft Membership Changes: When a Leader Must Commit a Current-Term Entry First"/>
    <published>2026-10-20T00:00:00&#43;00:00</published>
    <updated>2026-10-20T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/distributed/raft-membership-log-barrier</id>
    <content type="html" xml:base="https://blog.openacid.com/distributed/raft-membership-log-barrier/">&lt;p&gt;&lt;img src=&#34;/post-res/raft-membership-log-barrier/6b92f27a3f402930-raft-membership-noop-banner-small.png&#34; alt=&#34;Raft membership changes and log barriers&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Raft&amp;rsquo;s corrected single-step membership-change protocol requires a leader to commit an entry from its current term before proposing a new config. Standard joint consensus needs no separate noop barrier: its config entries already establish mutual exclusion. If we extend joint consensus to allow rollback, however, both rollback and completion require the extra barrier.&lt;/p&gt;&#xA;&lt;p&gt;The distinction comes down to which configs can still elect leaders or commit entries. Starting from &lt;code&gt;C0&lt;/code&gt;, we need to check its overlap with each proposed successor, &lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt;. We also need to check the overlap between &lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt; for as long as both can still be used. Throughout this article, elections and commits use ordinary Raft majority quorums.&lt;/p&gt;&#xA;&lt;h2 id=&#34;1-raft-supports-single-step-changes-and-joint-consensus&#34;&gt;1. Raft supports single-step changes and joint consensus&lt;/h2&gt;&#xA;&lt;p&gt;A &lt;strong&gt;single-step config change&lt;/strong&gt; moves directly from &lt;code&gt;C0&lt;/code&gt; to &lt;code&gt;C1&lt;/code&gt; with one config entry. The usual single-server variant adds or removes one voting node at a time, ensuring that &lt;code&gt;C0&lt;/code&gt; and &lt;code&gt;C1&lt;/code&gt; intersect. For example, to move from &lt;code&gt;{a,b,c}&lt;/code&gt; to &lt;code&gt;{b,c,d}&lt;/code&gt;, we first add &lt;code&gt;d&lt;/code&gt;. Once that change commits, we remove &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;A &lt;strong&gt;joint config change&lt;/strong&gt; introduces an intermediate config, &lt;code&gt;C01 = [C0,C1]&lt;/code&gt;. Under &lt;code&gt;C01&lt;/code&gt;, every election and commit requires both a quorum of &lt;code&gt;C0&lt;/code&gt; and a quorum of &lt;code&gt;C1&lt;/code&gt;. The leader commits &lt;code&gt;C01&lt;/code&gt; before appending the target config &lt;code&gt;C1&lt;/code&gt;, as specified in &lt;a href=&#34;https://raft.github.io/raft.pdf&#34; title=&#34;In Search of an Understandable Consensus Algorithm&#34;&gt;§6 of the Raft paper&lt;/a&gt;. This intermediate step allows several nodes to change at once.&lt;/p&gt;&#xA;&lt;p&gt;The two columns show the same migration. Time runs upward in both:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;single-step                 joint&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{b,c,d}                     C1 = {b,c,d}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    ↑                           ↑  after C01 commits&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{a,b,c,d}                   C01 = [{a,b,c},{b,c,d}]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    ↑                           ↑&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{a,b,c}                      C0 = {a,b,c}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The timing of config activation is important to the examples that follow. &lt;a href=&#34;/distributed/raft-rw-quorum/&#34; title=&#34;Flexible Raft: Read and Write Quorums&#34;&gt;Flexible Raft: Read and Write Quorums&lt;/a&gt; describes the rule, which we can restate here:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Raft propagates membership through config entries in the log. A node begins using &lt;code&gt;C1&lt;/code&gt; as soon as it appends the corresponding entry, even if that entry is uncommitted.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;A leader may therefore leave an uncommitted &lt;code&gt;C1&lt;/code&gt; on just a few nodes before crashing. Those nodes can still use &lt;code&gt;C1&lt;/code&gt; in a later election. The crash alone does not rule out the proposal.&lt;/p&gt;&#xA;&lt;h2 id=&#34;2-two-quorum-sets-intersect-when-every-pair-of-quorums-overlaps&#34;&gt;2. Two quorum sets intersect when every pair of quorums overlaps&lt;/h2&gt;&#xA;&lt;p&gt;Here, &lt;strong&gt;config&lt;/strong&gt; and &lt;strong&gt;quorum set&lt;/strong&gt; mean the same thing: a collection of permitted quorums. With majority voting, the smallest quorums of &lt;code&gt;C = {a,b,c}&lt;/code&gt; are &lt;code&gt;{a,b}&lt;/code&gt;, &lt;code&gt;{a,c}&lt;/code&gt;, and &lt;code&gt;{b,c}&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Two quorum sets &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; intersect if every quorum of &lt;code&gt;A&lt;/code&gt; shares at least one node with every quorum of &lt;code&gt;B&lt;/code&gt;. Formally, &lt;code&gt;qa ∩ qb ≠ ∅&lt;/code&gt; for all &lt;code&gt;qa ∈ A&lt;/code&gt; and &lt;code&gt;qb ∈ B&lt;/code&gt;. We denote this property by &lt;code&gt;Intersect(A,B)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This definition applies whenever we say that two configs intersect. Consider &lt;code&gt;C0 = {a,b,c}&lt;/code&gt; and &lt;code&gt;C1 = {b,c,d}&lt;/code&gt;. They share nodes &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt;, but &lt;code&gt;C0&lt;/code&gt; permits quorum &lt;code&gt;{a,b}&lt;/code&gt;, while &lt;code&gt;C1&lt;/code&gt; permits quorum &lt;code&gt;{c,d}&lt;/code&gt;. Since those quorums are disjoint, &lt;code&gt;C0&lt;/code&gt; and &lt;code&gt;C1&lt;/code&gt; do not intersect.&lt;/p&gt;&#xA;&lt;h2 id=&#34;3-quorums-that-can-still-elect-or-commit-must-intersect-one-another&#34;&gt;3. Quorums that can still elect or commit must intersect one another&lt;/h2&gt;&#xA;&lt;p&gt;For membership changes to be safe, &lt;strong&gt;the quorums that can still elect leaders or commit entries must intersect&lt;/strong&gt;. The log and voting records on shared nodes give Raft the information it needs to reject conflicting decisions.&lt;/p&gt;&#xA;&lt;p&gt;Suppose &lt;code&gt;C0&lt;/code&gt; has committed. Leader &lt;code&gt;L1&lt;/code&gt; proposes &lt;code&gt;C1&lt;/code&gt; from &lt;code&gt;C0&lt;/code&gt;, then crashes before &lt;code&gt;C1&lt;/code&gt; commits. A new leader, &lt;code&gt;L2&lt;/code&gt;, has not received &lt;code&gt;C1&lt;/code&gt;. It therefore also starts from &lt;code&gt;C0&lt;/code&gt; and proposes &lt;code&gt;C2&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       C1          C2&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;          ↖      ↗&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;              C0&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This history creates two overlap requirements:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;C0&lt;/code&gt; must intersect both &lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt;.&lt;/strong&gt; As &lt;code&gt;C1&lt;/code&gt; propagates, some nodes may use &lt;code&gt;C0&lt;/code&gt; while others use &lt;code&gt;C1&lt;/code&gt;. The same is true of a transition to &lt;code&gt;C2&lt;/code&gt;. Each transition needs overlap between the old and new configs.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt; must intersect while both remain usable.&lt;/strong&gt; Nodes retaining &lt;code&gt;C1&lt;/code&gt; may elect a leader under &lt;code&gt;C1&lt;/code&gt; while &lt;code&gt;L2&lt;/code&gt; uses &lt;code&gt;C2&lt;/code&gt;. If &lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt; allow disjoint quorums, those leaders may make conflicting decisions.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;If &lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt; allow disjoint quorums, the protocol must rule out the old &lt;code&gt;C1&lt;/code&gt; proposal before enabling &lt;code&gt;C2&lt;/code&gt;. Once &lt;code&gt;C1&lt;/code&gt; can no longer elect a leader or commit entries, it no longer competes with &lt;code&gt;C2&lt;/code&gt;. A log barrier establishes that ordering.&lt;/p&gt;&#xA;&lt;h2 id=&#34;4-without-a-barrier-an-uncommitted-cu-can-overwrite-a-committed-cv&#34;&gt;4. Without a barrier, an uncommitted Cu can overwrite a committed Cv&lt;/h2&gt;&#xA;&lt;h3 id=&#34;adding-u-and-v-separately-to-the-same-c0&#34;&gt;Adding u and v separately to the same C0&lt;/h3&gt;&#xA;&lt;p&gt;Consider the counterexample from &lt;a href=&#34;/distributed/raft-bug/&#34; title=&#34;The Pitfalls of Raft Membership Change&#34;&gt;The Pitfalls of Raft Membership Change&lt;/a&gt;. The committed config is &lt;code&gt;C0 = {a,b,c,d}&lt;/code&gt;. Leader &lt;code&gt;a&lt;/code&gt; proposes &lt;code&gt;Cu&lt;/code&gt; by adding &lt;code&gt;u&lt;/code&gt;; a later leader, &lt;code&gt;d&lt;/code&gt;, proposes &lt;code&gt;Cv&lt;/code&gt; by adding &lt;code&gt;v&lt;/code&gt;. Each successor intersects &lt;code&gt;C0&lt;/code&gt;, but &lt;code&gt;Cu&lt;/code&gt; and &lt;code&gt;Cv&lt;/code&gt; permit disjoint quorums: &lt;code&gt;{a,b,u}&lt;/code&gt; and &lt;code&gt;{c,d,v}&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;   Cu = C0 + u          Cv = C0 + v&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;   quorum {a,b,u}       quorum {c,d,v}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;              ↖       ↗&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;                  C0&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;              {a,b,c,d}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;All six nodes initially have the same log through index 5, and the entry at index 5 belongs to term 0. Nodes &lt;code&gt;u&lt;/code&gt; and &lt;code&gt;v&lt;/code&gt; have caught up as nonvoting nodes. To expose the bug, suppose leaders can propose configs before committing an entry from their current term.&lt;/p&gt;&#xA;&lt;p&gt;Read the chart from bottom to top. &lt;code&gt;L1&lt;/code&gt; and &lt;code&gt;F1&lt;/code&gt; identify the leader and followers in term 1; terms 2 and 3 use the same notation. &lt;code&gt;X&lt;/code&gt; marks a crash, and &lt;code&gt;Cu&lt;/code&gt; and &lt;code&gt;Cv&lt;/code&gt; mark the config entries stored on each node.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;time ↑ |   u     a     b     c     d     v&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       | -----------------------------------&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; t8    |  Cu    Cu    Cu    Cu    Cu&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; t7    |  F3    L3    F3&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; t6    |                          X&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; t5    |                    Cv    Cv    Cv&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; t4    |              F2    F2    L2&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; t3    |        X&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; t2    |  Cu    Cu&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; t1    |        L1    F1    F1&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       | -----------------------------------&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;t1–t3: &lt;code&gt;a&lt;/code&gt; leaves an uncommitted &lt;code&gt;Cu&lt;/code&gt; in term 1.&lt;/strong&gt; Node &lt;code&gt;a&lt;/code&gt; wins votes from &lt;code&gt;{a,b,c}&lt;/code&gt; under &lt;code&gt;C0&lt;/code&gt; and appends &lt;code&gt;Cu&lt;/code&gt; at index 6. Only &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;u&lt;/code&gt; store the entry before &lt;code&gt;a&lt;/code&gt; crashes.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;t4–t6: &lt;code&gt;d&lt;/code&gt; commits &lt;code&gt;Cv&lt;/code&gt; in term 2.&lt;/strong&gt; Nodes &lt;code&gt;b&lt;/code&gt;, &lt;code&gt;c&lt;/code&gt;, and &lt;code&gt;d&lt;/code&gt; have not received &lt;code&gt;Cu&lt;/code&gt;, so they elect &lt;code&gt;d&lt;/code&gt; under &lt;code&gt;C0&lt;/code&gt;. Leader &lt;code&gt;d&lt;/code&gt; appends &lt;code&gt;Cv&lt;/code&gt; at index 6 and replicates it to &lt;code&gt;{c,d,v}&lt;/code&gt;, a quorum of &lt;code&gt;Cv&lt;/code&gt;. It commits &lt;code&gt;Cv&lt;/code&gt;, then crashes. Node &lt;code&gt;b&lt;/code&gt; votes in the election but receives no term 2 entries.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;t7: &lt;code&gt;a&lt;/code&gt; is elected under &lt;code&gt;Cu&lt;/code&gt; in term 3.&lt;/strong&gt; After recovering, &lt;code&gt;a&lt;/code&gt; collects votes from &lt;code&gt;{a,b,u}&lt;/code&gt; using its local &lt;code&gt;Cu&lt;/code&gt;. The log on &lt;code&gt;b&lt;/code&gt; still ends at index 5 in term 0. Since &lt;code&gt;a&lt;/code&gt; holds a term 1 entry, Raft&amp;rsquo;s log comparison allows &lt;code&gt;b&lt;/code&gt; to vote for &lt;code&gt;a&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;t8: &lt;code&gt;a&lt;/code&gt; commits &lt;code&gt;Cu&lt;/code&gt; and overwrites &lt;code&gt;Cv&lt;/code&gt;.&lt;/strong&gt; Leader &lt;code&gt;a&lt;/code&gt; replicates a term 3 noop to &lt;code&gt;{a,b,u}&lt;/code&gt;, committing both the noop and its prefix, including &lt;code&gt;Cu&lt;/code&gt;. It then sends &lt;code&gt;Cu&lt;/code&gt; to &lt;code&gt;c&lt;/code&gt; and the recovered &lt;code&gt;d&lt;/code&gt;, overwriting their committed &lt;code&gt;Cv&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The noop is needed because &lt;code&gt;Cu&lt;/code&gt; was written in term 1. Under Raft&amp;rsquo;s normal commit rule, &lt;code&gt;a&lt;/code&gt; commits a term 3 entry, which also commits the preceding entries. &lt;strong&gt;In term 3, &lt;code&gt;a&lt;/code&gt; commits the &lt;code&gt;Cu&lt;/code&gt; it originally wrote in term 1.&lt;/strong&gt; The chart omits the noop because it tracks only config propagation.&lt;/p&gt;&#xA;&lt;p&gt;This execution has only one leader per term, yet it loses a committed config. The failure occurs across terms: quorum &lt;code&gt;{c,d,v}&lt;/code&gt; commits &lt;code&gt;Cv&lt;/code&gt;, while the disjoint quorum &lt;code&gt;{a,b,u}&lt;/code&gt; later elects &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h3 id=&#34;establishing-mutual-exclusion-through-the-common-parent-c0&#34;&gt;Establishing mutual exclusion through the common parent C0&lt;/h3&gt;&#xA;&lt;p&gt;Both &lt;code&gt;Cu&lt;/code&gt; and &lt;code&gt;Cv&lt;/code&gt; are proposed from &lt;code&gt;C0&lt;/code&gt;, so the protocol can use &lt;code&gt;C0&lt;/code&gt; to establish mutual exclusion. It can commit an entry under &lt;code&gt;C0&lt;/code&gt;, leaving a log record that later elections must respect. It could also impose stronger election requirements to prevent &lt;code&gt;a&lt;/code&gt;, which holds &lt;code&gt;Cu&lt;/code&gt; but lacks &lt;code&gt;Cv&lt;/code&gt;, from winning again.&lt;/p&gt;&#xA;&lt;p&gt;Single-step changes use a log barrier. The correction discussed in &lt;a href=&#34;/distributed/raft-bug/&#34; title=&#34;The Pitfalls of Raft Membership Change&#34;&gt;the earlier article&lt;/a&gt; can be stated directly:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;A new leader must commit an entry from its current term before appending a membership-change entry.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;In this example, &lt;code&gt;d&lt;/code&gt; must commit a term 2 entry under &lt;code&gt;C0&lt;/code&gt; before proposing &lt;code&gt;Cv&lt;/code&gt;. A quorum of &lt;code&gt;C0&lt;/code&gt; will then hold an entry newer than the term 1 &lt;code&gt;Cu&lt;/code&gt;. Since every quorum of &lt;code&gt;Cu&lt;/code&gt; intersects that quorum, &lt;code&gt;a&lt;/code&gt; cannot win with a log that still ends at &lt;code&gt;Cu&lt;/code&gt;. This is the condition in &lt;a href=&#34;https://gist.github.com/ongardie/a11f32b70581e20d6bcd&#34; title=&#34;Safety of Raft single-server membership changes&#34;&gt;Ongaro&amp;rsquo;s correction for single-server membership changes&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;5-neither-step-of-standard-joint-consensus-needs-a-separate-noop-barrier&#34;&gt;5. Neither step of standard joint consensus needs a separate noop barrier&lt;/h2&gt;&#xA;&lt;h3 id=&#34;both-c01-and-c02-retain-a-quorum-of-c0&#34;&gt;Both C01 and C02 retain a quorum of C0&lt;/h3&gt;&#xA;&lt;p&gt;Suppose leader &lt;code&gt;L1&lt;/code&gt; proposes &lt;code&gt;C01 = [C0,C1]&lt;/code&gt; from &lt;code&gt;C0&lt;/code&gt;, then crashes before &lt;code&gt;C01&lt;/code&gt; commits. A new leader, &lt;code&gt;L2&lt;/code&gt;, has not received &lt;code&gt;C01&lt;/code&gt;, so it starts from &lt;code&gt;C0&lt;/code&gt; and proposes &lt;code&gt;C02 = [C0,C2]&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  C01 = [C0,C1]       C02 = [C0,C2]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;              ↖     ↗&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;                 C0&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Every quorum of &lt;code&gt;C01&lt;/code&gt; contains a quorum of &lt;code&gt;C0&lt;/code&gt;, and so does every quorum of &lt;code&gt;C02&lt;/code&gt;. Any two quorums of &lt;code&gt;C0&lt;/code&gt; intersect. Consequently, &lt;code&gt;C01&lt;/code&gt; and &lt;code&gt;C02&lt;/code&gt; must intersect, regardless of whether &lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt; overlap.&lt;/p&gt;&#xA;&lt;h3 id=&#34;committing-c01-fixes-c1-as-the-destination&#34;&gt;Committing C01 fixes C1 as the destination&lt;/h3&gt;&#xA;&lt;p&gt;Once &lt;code&gt;C01 = [C0,C1]&lt;/code&gt; commits, standard joint consensus permits only &lt;code&gt;C1&lt;/code&gt; as the final config for this transition. If leader &lt;code&gt;L2&lt;/code&gt; crashes while completing the change, its successor &lt;code&gt;L3&lt;/code&gt; must continue toward the same &lt;code&gt;C1&lt;/code&gt;. Their completion quorums therefore belong to &lt;code&gt;C1&lt;/code&gt; and must intersect.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Committing &lt;code&gt;C01&lt;/code&gt; provides the log barrier between the two stages.&lt;/strong&gt; Before that commit, alternative joint configs such as &lt;code&gt;C01&lt;/code&gt; and &lt;code&gt;C02&lt;/code&gt; retain a quorum of &lt;code&gt;C0&lt;/code&gt;. After it, every leader completing the transition uses &lt;code&gt;C1&lt;/code&gt;. The config entries themselves establish the required mutual exclusion, so neither stage needs a separate noop barrier.&lt;/p&gt;&#xA;&lt;p&gt;Raft&amp;rsquo;s rules for committing entries from earlier terms still apply. If &lt;code&gt;L2&lt;/code&gt; inherits an uncommitted &lt;code&gt;C01&lt;/code&gt; from &lt;code&gt;L1&lt;/code&gt;, it must commit &lt;code&gt;C01&lt;/code&gt; before appending &lt;code&gt;C1&lt;/code&gt;. That may require &lt;code&gt;L2&lt;/code&gt; to commit an entry from its own term, committing &lt;code&gt;C01&lt;/code&gt; as part of the prefix. This follows the ordinary commit rule in &lt;a href=&#34;https://raft.github.io/raft.pdf&#34; title=&#34;In Search of an Understandable Consensus Algorithm&#34;&gt;§5.4.2 of the Raft paper&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;6-allowing-joint-rollback-breaks-mutual-exclusion-and-requires-a-log-barrier&#34;&gt;6. Allowing joint rollback breaks mutual exclusion and requires a log barrier&lt;/h2&gt;&#xA;&lt;p&gt;Standard Raft completes the transition to &lt;code&gt;C1&lt;/code&gt; after committing &lt;code&gt;C01 = [C0,C1]&lt;/code&gt;. Suppose we extend that protocol to also allow a rollback from &lt;code&gt;C01&lt;/code&gt; to &lt;code&gt;C0&lt;/code&gt;. Take &lt;code&gt;C0 = {a,b,c}&lt;/code&gt; and &lt;code&gt;C1 = {x,y,z}&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  C0 = {a,b,c}          C1 = {x,y,z}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  quorum {a,b}          quorum {x,y}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;              ↖        ↗&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;                  C01&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;                [C0,C1]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Both &lt;code&gt;C0&lt;/code&gt; and &lt;code&gt;C1&lt;/code&gt; intersect &lt;code&gt;C01&lt;/code&gt;, but their own quorums can be disjoint: &lt;code&gt;{a,b}&lt;/code&gt; shares no node with &lt;code&gt;{x,y}&lt;/code&gt;. Allowing rollback removes the guarantee that every completion uses &lt;code&gt;C1&lt;/code&gt;. Leader &lt;code&gt;a&lt;/code&gt; may retain an uncommitted rollback to &lt;code&gt;C0&lt;/code&gt;, while a later leader &lt;code&gt;y&lt;/code&gt; completes the transition to &lt;code&gt;C1&lt;/code&gt;. Without a barrier, &lt;code&gt;a&lt;/code&gt; can subsequently try to commit its rollback through a quorum disjoint from the one used by &lt;code&gt;y&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h3 id=&#34;guarding-rollback-alone-can-still-allow-both-c0-and-c1-to-commit&#34;&gt;Guarding rollback alone can still allow both C0 and C1 to commit&lt;/h3&gt;&#xA;&lt;p&gt;The barrier is required before both rollback and completion. A check that applies only to rollback requests still permits the following execution:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Term 1: &lt;code&gt;a&lt;/code&gt; satisfies the barrier, then proposes rollback.&lt;/strong&gt; Leader &lt;code&gt;a&lt;/code&gt; commits its noop on joint quorum &lt;code&gt;{a,b,x,y}&lt;/code&gt;, then appends the entry for returning to &lt;code&gt;C0&lt;/code&gt;. Only &lt;code&gt;a&lt;/code&gt; stores the rollback entry before crashing.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Term 2: &lt;code&gt;y&lt;/code&gt; skips the barrier and completes the change.&lt;/strong&gt; Node &lt;code&gt;y&lt;/code&gt; holds the term 1 noop and wins votes from joint quorum &lt;code&gt;{b,c,x,y}&lt;/code&gt;. It appends &lt;code&gt;C1&lt;/code&gt; immediately and commits it on &lt;code&gt;{y,z}&lt;/code&gt;. Node &lt;code&gt;b&lt;/code&gt; receives no term 2 entries.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Term 3: &lt;code&gt;a&lt;/code&gt; commits the old rollback entry.&lt;/strong&gt; When &lt;code&gt;a&lt;/code&gt; recovers, its log has one more term 1 entry than the log on &lt;code&gt;b&lt;/code&gt;: the rollback to &lt;code&gt;C0&lt;/code&gt;. It can therefore win votes from &lt;code&gt;{a,b}&lt;/code&gt; under &lt;code&gt;C0&lt;/code&gt;. Committing a term 3 noop then also commits the rollback entry.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The missing barrier is in term 2. Before proposing &lt;code&gt;C1&lt;/code&gt;, &lt;code&gt;y&lt;/code&gt; needs to commit a term 2 entry under &lt;code&gt;C01&lt;/code&gt; to rule out the older rollback held by &lt;code&gt;a&lt;/code&gt;. When &lt;code&gt;a&lt;/code&gt; returns in term 3, it can commit its existing rollback without proposing a new one. A guard that checks only new rollback requests therefore misses this execution.&lt;/p&gt;&#xA;&lt;p&gt;The standard joint argument applies when &lt;code&gt;C01 → C1&lt;/code&gt; is the only permitted completion and no &lt;code&gt;C0&lt;/code&gt; rollback entries remain. Every leader completing the transition then uses quorums of &lt;code&gt;C1&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;7-a-leader-establishes-the-barrier-by-committing-under-the-common-config&#34;&gt;7. A leader establishes the barrier by committing under the common config&lt;/h2&gt;&#xA;&lt;p&gt;&lt;strong&gt;For single-step changes and joint consensus extended with rollback, the leader first commits a current-term entry under the common config. Only then may it propose the next config.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Suppose &lt;code&gt;L1&lt;/code&gt; proposes an uncommitted &lt;code&gt;C1&lt;/code&gt; from &lt;code&gt;C0&lt;/code&gt; in term 1, and &lt;code&gt;L2&lt;/code&gt; wants to propose &lt;code&gt;C2&lt;/code&gt; from &lt;code&gt;C0&lt;/code&gt; in term 2. By first committing a term 2 entry under &lt;code&gt;C0&lt;/code&gt;, &lt;code&gt;L2&lt;/code&gt; records its authority on a quorum of &lt;code&gt;C0&lt;/code&gt;. Every quorum of &lt;code&gt;C1&lt;/code&gt; intersects that quorum. Raft&amp;rsquo;s term checks and log comparison rules then prevent &lt;code&gt;L1&lt;/code&gt; and its old &lt;code&gt;C1&lt;/code&gt; branch from producing a conflicting commit.&lt;/p&gt;&#xA;&lt;p&gt;The entry is usually a noop, but a current-term application entry serves the same purpose. Once the leader has committed such an entry in its term, it can reuse that barrier for later membership changes.&lt;/p&gt;&#xA;&lt;h3 id=&#34;a-joint-election-would-depend-on-c0-which-may-already-be-retired&#34;&gt;A joint election would depend on C0, which may already be retired&lt;/h3&gt;&#xA;&lt;p&gt;We could also try to establish mutual exclusion through an election. Suppose a candidate&amp;rsquo;s latest config is &lt;code&gt;C_new&lt;/code&gt;, following &lt;code&gt;C0&lt;/code&gt;. Requiring the candidate to win under joint config &lt;code&gt;[C0,C_new]&lt;/code&gt; would record its term on quorums of both configs. The voting records in &lt;code&gt;C0&lt;/code&gt; would constrain leaders from earlier terms.&lt;/p&gt;&#xA;&lt;p&gt;The difficulty is that &lt;strong&gt;a candidate may not know whether &lt;code&gt;C_new&lt;/code&gt; has committed or whether a quorum of &lt;code&gt;C0&lt;/code&gt; is still available&lt;/strong&gt;. For example, a cluster may have moved from &lt;code&gt;C0 = {a,b,c}&lt;/code&gt; through joint consensus to &lt;code&gt;C_new = {x,y,z}&lt;/code&gt;. Once &lt;code&gt;C_new&lt;/code&gt; commits, &lt;code&gt;a&lt;/code&gt;, &lt;code&gt;b&lt;/code&gt;, and &lt;code&gt;c&lt;/code&gt; can all shut down, as described in &lt;a href=&#34;https://raft.github.io/raft.pdf&#34; title=&#34;In Search of an Understandable Consensus Algorithm&#34;&gt;§6 of the Raft paper&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Candidate &lt;code&gt;x&lt;/code&gt; can still win an election under &lt;code&gt;C_new&lt;/code&gt; with votes from &lt;code&gt;{x,y}&lt;/code&gt;. Requiring an additional quorum of &lt;code&gt;C0&lt;/code&gt; would make that election depend on nodes that have already left the cluster.&lt;/p&gt;&#xA;&lt;p&gt;A log barrier lets the leader work with the config that elected it. The candidate first wins under its latest local config, &lt;code&gt;C_new&lt;/code&gt;, then commits a current-term entry under &lt;code&gt;C_new&lt;/code&gt;. That commit confirms &lt;code&gt;C_new&lt;/code&gt; and its prefix before the leader proposes &lt;code&gt;C_next&lt;/code&gt;. For this next transition, the common config is &lt;code&gt;C_new&lt;/code&gt;, so the barrier has no dependency on the retired &lt;code&gt;C0&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;8-single-step-changes-and-joint-consensus-enforce-mutual-exclusion-through-committed-entries&#34;&gt;8. Single-step changes and joint consensus enforce mutual exclusion through committed entries&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&#34;/distributed/raft-bug/&#34; title=&#34;The Pitfalls of Raft Membership Change&#34;&gt;The Pitfalls of Raft Membership Change&lt;/a&gt; describes the principle shared by these protocols. In this article&amp;rsquo;s terminology:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Membership changes establish mutual exclusion through quorums of the old config. At most one of the conflicting changes proposed from that old config can commit.&lt;/p&gt;&#xA;&lt;p&gt;Single-step changes first use an application entry or a noop to establish mutual exclusion. Joint consensus does this through the config entries themselves.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;In the OpenRaft implementation described here, rollback is permitted. A membership change therefore requires the current leader&amp;rsquo;s first noop to be committed. The same check applies to direct config appends:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;cluster_committed&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Some&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;noop_log_id&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;return&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Ok&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(());&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Here, &lt;code&gt;cluster_committed&lt;/code&gt; is the commit position confirmed by a quorum, and &lt;code&gt;noop_log_id&lt;/code&gt; identifies the current leader&amp;rsquo;s first noop. This uniform guard also covers ordinary joint transitions exposed through the same API.&lt;/p&gt;&#xA;&lt;p&gt;The requirement for a separate current-term log barrier depends on the permitted transitions:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Standard joint consensus needs no separate noop barrier in either step.&lt;/strong&gt; Proposals &lt;code&gt;C01 = [C0,C1]&lt;/code&gt; and &lt;code&gt;C02 = [C0,C2]&lt;/code&gt; both retain a quorum of &lt;code&gt;C0&lt;/code&gt;, so they intersect. Committing &lt;code&gt;C01&lt;/code&gt; in the first step provides the log barrier. Every leader completing that transition must then use &lt;code&gt;C1&lt;/code&gt;, so the second step needs no extra noop condition either.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Single-step changes require a current-term commit before proposing the new config.&lt;/strong&gt; Configs &lt;code&gt;Cu&lt;/code&gt; and &lt;code&gt;Cv&lt;/code&gt; may each intersect their parent &lt;code&gt;C0&lt;/code&gt; while allowing disjoint quorums between themselves. The leader must first commit a current-term entry under &lt;code&gt;C0&lt;/code&gt; to rule out conflicting proposals from earlier terms.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;With joint rollback enabled, both completion and rollback require a current-term commit first.&lt;/strong&gt; Config &lt;code&gt;C01&lt;/code&gt; can lead to either &lt;code&gt;C0&lt;/code&gt; or &lt;code&gt;C1&lt;/code&gt;, whose quorums may be disjoint. Before appending either config, the leader must commit an entry from its current term under &lt;code&gt;C01&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;A noop is the usual way to establish the extra barrier, but an application entry from the current term works equally well. Whenever the protocol requires that barrier, &lt;strong&gt;the leader must have committed an entry from its own current term before proposing the new config&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;The Pitfalls of Raft Membership Change : &lt;a href=&#34;/distributed/raft-bug/&#34;&gt;/distributed/raft-bug/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Flexible Raft: Read and Write Quorums : &lt;a href=&#34;/distributed/raft-rw-quorum/&#34;&gt;/distributed/raft-rw-quorum/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;In Search of an Understandable Consensus Algorithm : &lt;a href=&#34;https://raft.github.io/raft.pdf&#34;&gt;https://raft.github.io/raft.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Safety of Raft single-server membership changes : &lt;a href=&#34;https://gist.github.com/ongardie/a11f32b70581e20d6bcd&#34;&gt;https://gist.github.com/ongardie/a11f32b70581e20d6bcd&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="distributed"/>
    <category term="raft"/>
    <category term="consensus"/>
    <category term="membership"/>
    <summary type="html">Standard joint consensus provides its own log barrier through config entries. Single-step changes and joint consensus extended with rollback require a committed current-term entry before a new config. Two failure scenarios show why the difference comes down to quorum overlap.</summary>
  </entry>
  <entry>
    <title type="html">Flexible Raft: Read and Write Quorums</title>
    <link href="https://blog.openacid.com/distributed/raft-rw-quorum/" rel="alternate" type="text/html" title="Flexible Raft: Read and Write Quorums"/>
    <published>2026-10-15T00:00:00&#43;00:00</published>
    <updated>2026-10-15T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/distributed/raft-rw-quorum</id>
    <content type="html" xml:base="https://blog.openacid.com/distributed/raft-rw-quorum/">&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/3d0cad3a8d43e154-raft-rw-quorum-banner-small.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Picture a five-node Raft cluster handling a write. The leader has saved the entry locally and sent it to the followers. One follower replies. Then a second. Now the leader has three copies, enough to commit.&lt;/p&gt;&#xA;&lt;p&gt;That wait is part of every write. Electing a leader also takes three votes. But elections and writes do different jobs, so we have some room to change who we wait for. If we&amp;rsquo;re willing to ask more nodes to vote in an election, we can sometimes ask fewer nodes to acknowledge each write.&lt;/p&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s give those two groups names. A quorum is a group of nodes whose replies let an operation proceed. We&amp;rsquo;ll call the group used for elections a &lt;strong&gt;&lt;code&gt;read quorum&lt;/code&gt;&lt;/strong&gt;, and the group used to commit an entry a &lt;strong&gt;&lt;code&gt;write quorum&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Raft can use different rules for these two groups. We just have to make sure those rules still let the cluster deal safely with competing leaders. &lt;strong&gt;We can either keep one elected leader per term, or give the protocol a way to order leaders within the same term.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;We&amp;rsquo;ll start with a few nodes running Paxos, see what happens when two leaders try to write, and then bring the same idea to Raft. I&amp;rsquo;ll assume you&amp;rsquo;re already familiar with the basics of both protocols.&lt;/p&gt;&#xA;&lt;h2 id=&#34;flexible-paxos-separates-election-and-replication-quorums&#34;&gt;Flexible Paxos separates election and replication quorums&lt;/h2&gt;&#xA;&lt;h3 id=&#34;every-read-quorum-must-intersect-every-write-quorum&#34;&gt;Every &lt;code&gt;read quorum&lt;/code&gt; must intersect every &lt;code&gt;write quorum&lt;/code&gt;&lt;/h3&gt;&#xA;&lt;p&gt;In the usual Paxos setup, both phases need a majority. Any two majorities share at least one node, so all the quorums overlap. That&amp;rsquo;s the arrangement in Lamport&amp;rsquo;s &lt;a href=&#34;https://lamport.azurewebsites.net/pubs/paxos-simple.pdf&#34; title=&#34;Paxos Made Simple&#34;&gt;Paxos Made Simple&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Flexible Paxos lets us be more selective about that overlap. We use a &lt;code&gt;read quorum&lt;/code&gt; in phase 1 to establish a leader, and a &lt;code&gt;write quorum&lt;/code&gt; in phase 2 to commit a command. The rule we need is this: &lt;strong&gt;every &lt;code&gt;read quorum&lt;/code&gt; must share a node with every &lt;code&gt;write quorum&lt;/code&gt;&lt;/strong&gt;. Two &lt;code&gt;read quorums&lt;/code&gt; can be disjoint. So can two &lt;code&gt;write quorums&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s put four nodes on a grid and see what that buys us.&lt;/p&gt;&#xA;&lt;h3 id=&#34;in-a-four-node-grid-a-row-elects-and-a-column-replicates&#34;&gt;In a four-node grid, a row elects and a column replicates&lt;/h3&gt;&#xA;&lt;p&gt;Here&amp;rsquo;s our cluster:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Column 1&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Column 2&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Row 1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;node-1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;node-2&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Row 2&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;node-3&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;node-4&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;We&amp;rsquo;ll use a whole row to elect a leader and a whole column to commit a command. So our &lt;code&gt;read quorums&lt;/code&gt; are &lt;code&gt;{1, 2}&lt;/code&gt; and &lt;code&gt;{3, 4}&lt;/code&gt;, and our &lt;code&gt;write quorums&lt;/code&gt; are &lt;code&gt;{1, 3}&lt;/code&gt; and &lt;code&gt;{2, 4}&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Pick any row, then any column. They always meet at one node. That gives us exactly the intersection Flexible Paxos asks for, with only two nodes in each quorum. A majority of four would need three.&lt;/p&gt;&#xA;&lt;p&gt;Node-1, for example, can finish phase 1 with &lt;code&gt;{1, 2}&lt;/code&gt;. When it wants to write, it can use either column, &lt;code&gt;{1, 3}&lt;/code&gt; or &lt;code&gt;{2, 4}&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/7b7dff2480981bb2-01-quorum-grid.png&#34; alt=&#34;Four-node grid: read rows intersect write columns&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;There&amp;rsquo;s something else to notice about those rows. They don&amp;rsquo;t overlap with each other. Node-1 can get enough votes from the top row while node-4 gets enough from the bottom row. Both can finish phase 1.&lt;/p&gt;&#xA;&lt;p&gt;That&amp;rsquo;s allowed. Paxos has another rule that settles what happens when they try to write.&lt;/p&gt;&#xA;&lt;h3 id=&#34;paxos-resolves-competing-leaders-in-phase-2&#34;&gt;Paxos resolves competing leaders in phase 2&lt;/h3&gt;&#xA;&lt;p&gt;Each Paxos election attempt has a ballot. Ballots are globally unique and totally ordered, which means we can always tell which of two different ballots is higher.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;When a node promises to honor a ballot, it&amp;rsquo;s also promising to reject later writes with a lower ballot.&lt;/strong&gt; A leader finishes phase 1 only after a &lt;code&gt;read quorum&lt;/code&gt; has made that promise. Every &lt;code&gt;write quorum&lt;/code&gt; touches that &lt;code&gt;read quorum&lt;/code&gt;, so every possible &lt;code&gt;write quorum&lt;/code&gt; contains someone who can enforce the promise.&lt;/p&gt;&#xA;&lt;p&gt;Now let&amp;rsquo;s follow node-1 and node-4. Node-1 uses &lt;code&gt;ballot_1&lt;/code&gt;, node-4 uses &lt;code&gt;ballot_4&lt;/code&gt;, and &lt;code&gt;ballot_1 &amp;lt; ballot_4&lt;/code&gt;. Both finish phase 1 before either tries to write at a new index &lt;code&gt;i&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Node-1 wants to write &lt;code&gt;a&lt;/code&gt; through &lt;code&gt;{1, 3}&lt;/code&gt;. It sends the entry to node-3. But node-3 has already promised to honor &lt;code&gt;ballot_4&lt;/code&gt;. Its answer is effectively, &amp;ldquo;I&amp;rsquo;ve promised a higher ballot. I can&amp;rsquo;t accept this write.&amp;rdquo; Node-1 is missing an acknowledgment, so it can&amp;rsquo;t commit &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Next, node-4 tries to write &lt;code&gt;b&lt;/code&gt; through &lt;code&gt;{2, 4}&lt;/code&gt;. Node-2 has promised &lt;code&gt;ballot_1&lt;/code&gt;, which is lower than &lt;code&gt;ballot_4&lt;/code&gt;, so it can accept the newer ballot. Node-4 can accept its own ballot too. That completes the &lt;code&gt;write quorum&lt;/code&gt;, and &lt;code&gt;b&lt;/code&gt; commits.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/b975de2f457cf883-02-paxos-phase-conflict.png&#34; alt=&#34;Paxos rejects the lower ballot in phase 2&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Both leaders made it through phase 1. Once they tried to write, the ballot ordering gave the nodes a way to settle the competition. In this execution, only the leader with the higher ballot can finish phase 2.&lt;/p&gt;&#xA;&lt;p&gt;That&amp;rsquo;s why the &lt;code&gt;read quorums&lt;/code&gt; can be disjoint. The overlap between an election quorum and every &lt;code&gt;write quorum&lt;/code&gt; still gives the promises somewhere to take effect. The rest of Paxos, including recovery of previously accepted values, stays in place. &lt;a href=&#34;https://arxiv.org/pdf/1608.06696&#34; title=&#34;Flexible Paxos: Quorum intersection revisited&#34;&gt;Flexible Paxos, §2, §4.1, §5&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h2 id=&#34;raft-also-needs-a-guarantee-about-competing-leaders&#34;&gt;Raft also needs a guarantee about competing leaders&lt;/h2&gt;&#xA;&lt;p&gt;It&amp;rsquo;s tempting to give Raft the same grid. Its election does the job of Paxos phase 1, and its log replication does the job of phase 2. In a nine-node grid, we could use three nodes in a row for an election and three in a column for a write. The usual majority would be five.&lt;/p&gt;&#xA;&lt;p&gt;Before we change the quorum rules, though, we need to look at an assumption Raft already makes: each term elects at most one leader. The grid can give us two leaders in the same term, and Raft&amp;rsquo;s existing replication rules aren&amp;rsquo;t prepared for that.&lt;/p&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s run the four-node example again, this time with Raft.&lt;/p&gt;&#xA;&lt;h3 id=&#34;disjoint-read-quorums-can-commit-different-commands-at-the-same-index&#34;&gt;Disjoint &lt;code&gt;read quorums&lt;/code&gt; can commit different commands at the same index&lt;/h3&gt;&#xA;&lt;p&gt;All four nodes start with the same log, and the next available index is &lt;code&gt;i&lt;/code&gt;. We&amp;rsquo;ll keep the same &lt;code&gt;read quorums&lt;/code&gt;, &lt;code&gt;{1, 2}&lt;/code&gt; and &lt;code&gt;{3, 4}&lt;/code&gt;, and &lt;code&gt;write quorums&lt;/code&gt;, &lt;code&gt;{1, 3}&lt;/code&gt; and &lt;code&gt;{2, 4}&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Node-1 and node-4 both start an election in term &lt;code&gt;t&lt;/code&gt;. Node-1 gets the votes in the top row. Node-4 gets the votes in the bottom row. Both have enough votes under our new rules.&lt;/p&gt;&#xA;&lt;p&gt;We&amp;rsquo;ll use &lt;code&gt;voted_for&lt;/code&gt; to track each node&amp;rsquo;s vote in this term. Node-2 has &lt;code&gt;voted_for = 1&lt;/code&gt;, and node-3 has &lt;code&gt;voted_for = 4&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Now node-1 writes &lt;code&gt;a&lt;/code&gt; at index &lt;code&gt;i&lt;/code&gt; and sends it to node-3. Node-3 checks the request. The term matches its own term. The log prefix matches too. So it accepts the entry, even though it voted for node-4. Node-1 now has the &lt;code&gt;write quorum&lt;/code&gt; &lt;code&gt;{1, 3}&lt;/code&gt;. It commits and executes &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Then node-4 writes &lt;code&gt;b&lt;/code&gt; at the same index and sends it to node-2. Node-2 makes the same checks: matching term, matching prefix. It accepts &lt;code&gt;b&lt;/code&gt;, despite having voted for node-1. Node-4 has the &lt;code&gt;write quorum&lt;/code&gt; &lt;code&gt;{2, 4}&lt;/code&gt;, so it commits and executes &lt;code&gt;b&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/585f00470b400a2a-03-raft-same-term-conflict.png&#34; alt=&#34;Raft with disjoint read quorums commits conflicting commands&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;We&amp;rsquo;ve now executed two different commands at the same index.&lt;/strong&gt; That&amp;rsquo;s a consensus failure.&lt;/p&gt;&#xA;&lt;p&gt;The surprising part is that neither follower rejected the other leader&amp;rsquo;s write. In standard Raft, a follower can accept AppendEntries from the elected leader even if it voted for someone else. The sender doesn&amp;rsquo;t have to match &lt;code&gt;voted_for&lt;/code&gt;; the request still has to pass the term and log matching checks. &lt;a href=&#34;https://raft.github.io/raft.pdf&#34; title=&#34;In Search of an Understandable Consensus Algorithm&#34;&gt;Raft paper, Figures 2 and 3&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;That behavior is useful when there&amp;rsquo;s one elected leader. Our new election rules have allowed two.&lt;/p&gt;&#xA;&lt;h3 id=&#34;raft-relies-on-a-unique-leader-within-each-term&#34;&gt;Raft relies on a unique leader within each term&lt;/h3&gt;&#xA;&lt;p&gt;Think about how a node tells one leader from another. For this discussion, let&amp;rsquo;s write an election identity as &lt;code&gt;(term, voted_for)&lt;/code&gt; and call it a &lt;code&gt;LeaderId&lt;/code&gt;. The &lt;code&gt;voted_for&lt;/code&gt; part names the candidate in that election.&lt;/p&gt;&#xA;&lt;p&gt;In Paxos, ballots give us a total order. Standard Raft&amp;rsquo;s rules work more like a partial order. A higher term is newer. The same term and candidate ID describe the same election. But two different candidates in the same term don&amp;rsquo;t have an ordering. OpenRaft&amp;rsquo;s &lt;a href=&#34;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/vote/leader_id/leader_id_cmp.rs#L9-L27&#34; title=&#34;OpenRaft: partial comparison of leader IDs&#34;&gt;partial comparison rules&lt;/a&gt; express that directly.&lt;/p&gt;&#xA;&lt;p&gt;Raft gets away with this because &lt;strong&gt;each term elects at most one leader&lt;/strong&gt;. With majority election quorums, two candidates would have to share at least one voter. That voter only votes once, so both candidates can&amp;rsquo;t win.&lt;/p&gt;&#xA;&lt;p&gt;Now imagine a follower that voted for the losing candidate. It still needs to follow the winner. It can keep its election vote and accept valid replication requests from the elected leader. The vote and the leader it follows serve different purposes.&lt;/p&gt;&#xA;&lt;p&gt;If two leaders win in the same term, that same follower has a problem. The term doesn&amp;rsquo;t tell it which leader should take precedence.&lt;/p&gt;&#xA;&lt;p&gt;You can see the same assumption in the log. Raft uses &lt;code&gt;(term, index)&lt;/code&gt; as a log ID. Since one leader creates the entries for a term, the same term and index identify the same entry. In our broken grid example, two leaders created different entries with that same ID.&lt;/p&gt;&#xA;&lt;p&gt;So we have two ways forward. We can change the &lt;code&gt;read quorums&lt;/code&gt; to keep one leader per term. Or we can change the identities and the protocol rules so nodes can order leaders within a term.&lt;/p&gt;&#xA;&lt;h2 id=&#34;two-ways-to-implement-read-and-write-quorums-in-raft&#34;&gt;Two ways to implement &lt;code&gt;read and write quorums&lt;/code&gt; in Raft&lt;/h2&gt;&#xA;&lt;h3 id=&#34;approach-1-read-quorums-also-intersect-one-another&#34;&gt;Approach 1: &lt;code&gt;Read quorums&lt;/code&gt; also intersect one another&lt;/h3&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s try the smaller protocol change first. We&amp;rsquo;ll keep Raft&amp;rsquo;s existing identity and voting rules, and make sure &lt;strong&gt;any two election quorums share a voter&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;We now have three quorum rules:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Every &lt;code&gt;read quorum&lt;/code&gt; intersects every &lt;code&gt;write quorum&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Every pair of &lt;code&gt;read quorums&lt;/code&gt; intersects.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Write quorums&lt;/code&gt; can still be disjoint.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;On our four-node grid, we can build a &lt;code&gt;read quorum&lt;/code&gt; from a whole row plus a whole column. That gives us three nodes. We&amp;rsquo;ll keep each &lt;code&gt;write quorum&lt;/code&gt; as one column, so a write still needs two nodes.&lt;/p&gt;&#xA;&lt;p&gt;Take &lt;code&gt;read quorums&lt;/code&gt; &lt;code&gt;{1, 2, 3}&lt;/code&gt; and &lt;code&gt;{2, 3, 4}&lt;/code&gt;. They share node-2 and node-3. Any other pair of these three-node &lt;code&gt;read quorums&lt;/code&gt; shares at least two nodes as well. And because every &lt;code&gt;read quorum&lt;/code&gt; includes a whole row, it still meets every write column.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/d47e1f6b0285941d-04-quorum-size-tradeoff.png&#34; alt=&#34;Two read quorums share nodes 2 and 3 in a four-node grid&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;We can also describe this using quorum sizes alone. Let &lt;code&gt;N&lt;/code&gt; be the number of nodes, &lt;code&gt;R&lt;/code&gt; the &lt;code&gt;read quorum&lt;/code&gt; size, and &lt;code&gt;W&lt;/code&gt; the &lt;code&gt;write quorum&lt;/code&gt; size. To make every &lt;code&gt;read quorum&lt;/code&gt; meet every &lt;code&gt;write quorum&lt;/code&gt;, we need &lt;code&gt;R ≥ N-W+1&lt;/code&gt;. To make any two &lt;code&gt;read quorums&lt;/code&gt; meet, we also need &lt;code&gt;R ≥ floor(N/2)+1&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Put those together, and the smallest &lt;code&gt;read quorum&lt;/code&gt; we can choose is &lt;strong&gt;&lt;code&gt;R = max(N-W+1, floor(N/2)+1)&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Here&amp;rsquo;s what that means for the five-node cluster we started with. Choose &lt;code&gt;R=4,W=2&lt;/code&gt;, and a leader can commit after two nodes have the entry. But electing its replacement takes four votes.&lt;/p&gt;&#xA;&lt;p&gt;Now let the leader and one other node fail. Three nodes are still running, but they can&amp;rsquo;t produce the four votes we asked for. The cluster has to wait. With the usual &lt;code&gt;R=3,W=3&lt;/code&gt; majority setup, those same three nodes could elect a leader and keep writing.&lt;/p&gt;&#xA;&lt;p&gt;So a smaller &lt;code&gt;write quorum&lt;/code&gt; has a cost. &lt;strong&gt;We wait for fewer acknowledgments while the leader is running, but we need more nodes available when it&amp;rsquo;s time to elect a replacement.&lt;/strong&gt; Majority quorums tolerate the most arbitrary node failures when we need to preserve both elections and writes.&lt;/p&gt;&#xA;&lt;h4 id=&#34;dedicated-election-nodes-can-provide-the-shared-votes&#34;&gt;Dedicated election nodes can provide the shared votes&lt;/h4&gt;&#xA;&lt;p&gt;There&amp;rsquo;s another way to make those election quorums overlap. We can add a small group of nodes whose job is to vote. They persist their term and &lt;code&gt;voted_for&lt;/code&gt;, but they don&amp;rsquo;t normally receive log entries or run for leader.&lt;/p&gt;&#xA;&lt;p&gt;Every &lt;code&gt;read quorum&lt;/code&gt; must include a majority of these election nodes. Any two of those majorities share a voter. That voter can&amp;rsquo;t vote twice in a term, so two candidates can&amp;rsquo;t both win.&lt;/p&gt;&#xA;&lt;p&gt;We still need the nodes that actually store the log. A &lt;code&gt;read quorum&lt;/code&gt; must meet every &lt;code&gt;write quorum&lt;/code&gt; at one of those data nodes. Before granting a vote, a data node checks that the candidate&amp;rsquo;s log is at least as up to date as its own. That&amp;rsquo;s how the election protects committed entries. &lt;strong&gt;The dedicated voters keep the leader unique; the data nodes keep the log safe.&lt;/strong&gt; &lt;a href=&#34;https://raft.github.io/raft.pdf&#34; title=&#34;In Search of an Understandable Consensus Algorithm&#34;&gt;Raft paper, §5.4.1, §5.4.3&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;For example, keep four data nodes in the &lt;code&gt;2×2&lt;/code&gt; grid and add three election nodes. To elect a leader, take one whole data row and any two election nodes. To commit a write, take one whole data column.&lt;/p&gt;&#xA;&lt;p&gt;Every election quorum shares a voter with every other election quorum. It also meets every &lt;code&gt;write quorum&lt;/code&gt; at a data node. We now have seven nodes in all, with four votes needed for an election and two acknowledgments for a write.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/731715acecff1158-05-election-only-nodes.png&#34; alt=&#34;Dedicated election nodes connect small read quorums&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Both versions of this first approach do the same job: they arrange for two elections to share a voter. If we want to let election quorums be disjoint, we&amp;rsquo;ll need to give the protocol another way to handle competing leaders.&lt;/p&gt;&#xA;&lt;h3 id=&#34;approach-2-use-ordered-leader-identities-throughout-the-protocol&#34;&gt;Approach 2: Use ordered leader identities throughout the protocol&lt;/h3&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s give Raft the ordering that helped Paxos earlier. Each election attempt gets a globally unique &lt;code&gt;(term, voted_for)&lt;/code&gt; identity. Compare the terms first. If the terms match, compare the candidate IDs. Now &lt;code&gt;(t, 1) &amp;lt; (t, 4)&lt;/code&gt; has a clear meaning.&lt;/p&gt;&#xA;&lt;p&gt;A candidate must use a fresh identity each time it campaigns, and each identity belongs to just one candidate. Two leaders can then belong to the same term while still having identities that every node can compare.&lt;/p&gt;&#xA;&lt;p&gt;We have to carry that comparison through the protocol. Changing the election alone would leave us with the same kind of problem we just saw. There are three places to pay attention to:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;When a vote request arrives, the voter checks the full leader identity against the highest identity it has accepted. It rejects a lower identity and checks the candidate&amp;rsquo;s log before accepting a higher one. The accepted identity is persisted, even if the term hasn&amp;rsquo;t changed.&lt;/li&gt;&#xA;&lt;li&gt;When AppendEntries arrives, the receiver makes the same full-identity comparison. It rejects a lower identity and persists a higher one when accepting it. Once it has accepted the higher identity, later writes from the lower one are rejected.&lt;/li&gt;&#xA;&lt;li&gt;The entries themselves record who created them. A log ID becomes &lt;code&gt;(term, voted_for, index)&lt;/code&gt;. That candidate ID stays with the entry, even as the receiver votes in later elections. Elections compare the last entries by leader identity first, then index. Prefix matching and conflict detection use the full log ID too.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Commitment has to follow the same rule. &lt;strong&gt;To commit an entry directly, the leader needs an entry from its own current full &lt;code&gt;LeaderId&lt;/code&gt;.&lt;/strong&gt; Committing that entry also commits the entries before it, as in standard Raft.&lt;/p&gt;&#xA;&lt;p&gt;Once all those rules agree, a node that has accepted a higher identity can stop a lower leader&amp;rsquo;s writes. We can use the Flexible Paxos quorum rule again: &lt;strong&gt;every &lt;code&gt;read quorum&lt;/code&gt; must meet every &lt;code&gt;write quorum&lt;/code&gt;, while &lt;code&gt;read quorums&lt;/code&gt; can be disjoint.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s see how that changes the story for nodes 1 and 4. We&amp;rsquo;re back on the four-node grid. The &lt;code&gt;read quorums&lt;/code&gt; are &lt;code&gt;{1, 2}&lt;/code&gt; and &lt;code&gt;{3, 4}&lt;/code&gt;, and the &lt;code&gt;write quorums&lt;/code&gt; are &lt;code&gt;{1, 3}&lt;/code&gt; and &lt;code&gt;{2, 4}&lt;/code&gt;. The logs are identical before term &lt;code&gt;t&lt;/code&gt;. Node-1 campaigns with &lt;code&gt;(t, 1)&lt;/code&gt;, node-4 with &lt;code&gt;(t, 4)&lt;/code&gt;, and node-1 wants to commit &lt;code&gt;a&lt;/code&gt; at index &lt;code&gt;i&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Suppose node-3 votes for node-4 first. It saves &lt;code&gt;(t, 4)&lt;/code&gt;. When node-1 later sends AppendEntries with &lt;code&gt;(t, 1)&lt;/code&gt;, node-3 rejects the request. Node-1 can&amp;rsquo;t get the acknowledgment it needs from &lt;code&gt;{1, 3}&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Now reverse the order. Node-1 first commits &lt;code&gt;a&lt;/code&gt; through &lt;code&gt;{1, 3}&lt;/code&gt;, so node-3 already has the entry. When node-4 asks for a vote, its log is still shorter. Node-3 rejects it on log freshness, and node-4 can&amp;rsquo;t win through &lt;code&gt;{3, 4}&lt;/code&gt;. To win that election, node-4 first needs a sufficiently up-to-date log that preserves &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/844a3ef3ce88096f-06-raft-ordered-leader.png&#34; alt=&#34;Ordered leader identities handle two message orders&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;In one order, the vote prevents the conflicting write. In the other, the committed entry prevents an election from discarding it. That&amp;rsquo;s the protection we were missing when we first gave Raft the grid.&lt;/p&gt;&#xA;&lt;p&gt;We should also try a failure while we&amp;rsquo;re looking at grids. Take nine nodes in a &lt;code&gt;3×3&lt;/code&gt; grid, with rows for elections and columns for writes. Let the three nodes on a diagonal fail.&lt;/p&gt;&#xA;&lt;p&gt;Six nodes are still alive. But look at the rows: every row is missing a node. Every column is missing one too. We can&amp;rsquo;t put together a single election quorum or &lt;code&gt;write quorum&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;With majority quorums, the nine-node cluster would need five nodes. The same six survivors could still elect a leader and commit writes. &lt;strong&gt;The grid&amp;rsquo;s availability depends on where the failures land, as well as how many nodes fail.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/8a8fed8d10b10fd4-07-quorum-failure-comparison.png&#34; alt=&#34;Three diagonal failures stop grid quorums; five of six survivors still form a majority&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;This second approach lets us use disjoint election quorums, but it changes more of the protocol. &lt;strong&gt;The log ID grows from &lt;code&gt;(term, index)&lt;/code&gt; to &lt;code&gt;(term, voted_for, index)&lt;/code&gt;.&lt;/strong&gt; Votes, replication requests, and commitment checks all need to use that full identity.&lt;/p&gt;&#xA;&lt;p&gt;If you&amp;rsquo;d like to see what those changes look like in code, OpenRaft has examples of &lt;a href=&#34;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/vote_handler/mod.rs#L101-L155&#34; title=&#34;OpenRaft: vote with and persist the full leader identity&#34;&gt;voting and persisting full leader identities&lt;/a&gt;, &lt;a href=&#34;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/following_handler/mod.rs#L120-L139&#34; title=&#34;OpenRaft: check log prefixes with full log IDs&#34;&gt;checking prefixes with full log IDs&lt;/a&gt;, and &lt;a href=&#34;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/replication_handler/mod.rs#L230-L244&#34; title=&#34;OpenRaft: check the current leader identity before committing&#34;&gt;checking the current leader identity before committing&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;client-reads-can-confirm-leadership-through-a-write-quorum&#34;&gt;Client reads can confirm leadership through a &lt;code&gt;write quorum&lt;/code&gt;&lt;/h2&gt;&#xA;&lt;p&gt;Now suppose a client sends a read instead of a write. For that read to be linearizable, it has to reflect every write that finished before the read began.&lt;/p&gt;&#xA;&lt;p&gt;ReadIndex gives the leader two jobs. It needs to confirm that it&amp;rsquo;s still allowed to act as leader, then wait until its state machine has applied enough of the log to answer the read.&lt;/p&gt;&#xA;&lt;p&gt;There&amp;rsquo;s a startup condition to keep here. &lt;strong&gt;Before serving ReadIndex requests, a leader must have committed an entry from its current leadership.&lt;/strong&gt; In standard Raft, that means its current term. In the second approach, it means its current full leader identity. Changing the quorums doesn&amp;rsquo;t remove that requirement. You can see the standard check in &lt;a href=&#34;https://github.com/etcd-io/raft/blob/v3.6.0/raft.go#L1278-L1285&#34; title=&#34;etcd/raft: serve ReadIndex only after committing in the current term&#34;&gt;etcd/raft&amp;rsquo;s ReadIndex readiness code&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;After that, &lt;strong&gt;either approach can confirm leadership through a &lt;code&gt;write quorum&lt;/code&gt;&lt;/strong&gt;. Here&amp;rsquo;s how a read proceeds:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The request arrives, and the leader records its current commit index as &lt;code&gt;read_index&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The leader sends a fresh round of heartbeats and waits for a &lt;code&gt;write quorum&lt;/code&gt; to acknowledge its current identity. It includes a round identifier so replies from an earlier round won&amp;rsquo;t count for this read.&lt;/li&gt;&#xA;&lt;li&gt;The leader waits for its local state machine to apply every entry through &lt;code&gt;read_index&lt;/code&gt;. Only then does the application read the state and return the result. &lt;a href=&#34;https://github.com/etcd-io/raft/blob/v3.6.0/node.go#L199-L205&#34; title=&#34;etcd/raft: ReadIndex interface and application requirements&#34;&gt;etcd/raft ReadIndex interface and application requirements&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The heartbeats use the same identity rules as the rest of the protocol. The first approach compares terms. The second compares the full &lt;code&gt;(term, voted_for)&lt;/code&gt; identity.&lt;/p&gt;&#xA;&lt;p&gt;To see why a &lt;code&gt;write quorum&lt;/code&gt; is enough, imagine that a newer leader was elected before this read arrived. That election used a &lt;code&gt;read quorum&lt;/code&gt;. Our old leader is now asking a &lt;code&gt;write quorum&lt;/code&gt; to acknowledge its heartbeat.&lt;/p&gt;&#xA;&lt;p&gt;Those two groups have to share a node. The shared node has already accepted the newer identity, so it rejects the old leader&amp;rsquo;s heartbeat. The old leader can&amp;rsquo;t collect the replies it needs, and it can&amp;rsquo;t return a stale read.&lt;/p&gt;&#xA;&lt;p&gt;In our five-node cluster with &lt;code&gt;R=4,W=2&lt;/code&gt;, any two nodes share at least one member with every four-node election quorum. So ReadIndex can confirm leadership with just two acknowledgments.&lt;/p&gt;&#xA;&lt;h2 id=&#34;membership-changes-must-preserve-intersections-across-configurations&#34;&gt;Membership changes must preserve intersections across configurations&lt;/h2&gt;&#xA;&lt;p&gt;So far, we&amp;rsquo;ve kept the same nodes in the cluster. Now let&amp;rsquo;s add or remove some. The quorum rules change with the membership, and the nodes won&amp;rsquo;t all learn about that change at the same instant.&lt;/p&gt;&#xA;&lt;p&gt;We&amp;rsquo;ll call the old configuration &lt;code&gt;c0&lt;/code&gt; and the new one &lt;code&gt;c1&lt;/code&gt;. Between them, we&amp;rsquo;ll use a joint configuration, &lt;code&gt;c01&lt;/code&gt;. The cluster moves through &lt;code&gt;c0 → c01 → c1&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;While &lt;code&gt;c01&lt;/code&gt; is in use, an election has to satisfy the &lt;code&gt;read quorum&lt;/code&gt; rules of both &lt;code&gt;c0&lt;/code&gt; and &lt;code&gt;c1&lt;/code&gt;. A write has to satisfy both sets of &lt;code&gt;write quorum&lt;/code&gt; rules. That gives us overlap while the cluster moves from one membership to the other.&lt;/p&gt;&#xA;&lt;p&gt;This part of the story applies to nodes that store the log. The dedicated election nodes we discussed earlier would need their own way to learn and activate membership changes.&lt;/p&gt;&#xA;&lt;h3 id=&#34;commit-the-joint-configuration-before-activating-the-new-configuration&#34;&gt;Commit the joint configuration before activating the new configuration&lt;/h3&gt;&#xA;&lt;p&gt;Raft sends membership changes through the log. In the transition we&amp;rsquo;re describing, a node uses the configuration in its latest local configuration entry. It switches as soon as it appends that entry, before the entry commits.&lt;/p&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s follow the leader through the change:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It appends and replicates &lt;code&gt;c01&lt;/code&gt;. Each node starts using &lt;code&gt;c01&lt;/code&gt; when it appends that entry locally.&lt;/li&gt;&#xA;&lt;li&gt;It waits for &lt;code&gt;c01&lt;/code&gt; to commit under &lt;code&gt;c01&lt;/code&gt;&amp;rsquo;s &lt;code&gt;write quorum&lt;/code&gt; rules. Only then does it append and replicate &lt;code&gt;c1&lt;/code&gt;. Nodes start using &lt;code&gt;c1&lt;/code&gt; as they append that entry.&lt;/li&gt;&#xA;&lt;li&gt;It commits &lt;code&gt;c1&lt;/code&gt; under &lt;code&gt;c1&lt;/code&gt;&amp;rsquo;s &lt;code&gt;write quorum&lt;/code&gt; rules. The membership change is complete.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;At first, some nodes may still be using &lt;code&gt;c0&lt;/code&gt; while others have moved to &lt;code&gt;c01&lt;/code&gt;. Later, some may be using &lt;code&gt;c01&lt;/code&gt; while others have moved to &lt;code&gt;c1&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The wait between those two stages matters. &lt;strong&gt;By committing &lt;code&gt;c01&lt;/code&gt; before introducing &lt;code&gt;c1&lt;/code&gt;, we keep the old and new configurations from making independent decisions.&lt;/strong&gt; The election freshness check and commitment rules also make later leaders retain the committed configuration entry. A change of leader therefore preserves that ordering. &lt;a href=&#34;https://raft.github.io/raft.pdf&#34; title=&#34;In Search of an Understandable Consensus Algorithm&#34;&gt;Raft paper, §6, Figure 11&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;So we check intersections across each adjacent pair: &lt;code&gt;c0&lt;/code&gt; with &lt;code&gt;c01&lt;/code&gt;, then &lt;code&gt;c01&lt;/code&gt; with &lt;code&gt;c1&lt;/code&gt;. &lt;strong&gt;Every &lt;code&gt;read quorum&lt;/code&gt; on either side must meet every &lt;code&gt;write quorum&lt;/code&gt; on the other side.&lt;/strong&gt; With the first approach, &lt;code&gt;read quorums&lt;/code&gt; must also meet across the pair, so we still elect only one leader per term. With the second approach, ordered leader identities handle the competition.&lt;/p&gt;&#xA;&lt;h3 id=&#34;the-joint-configuration-combines-read-and-write-quorums-separately&#34;&gt;The joint configuration combines &lt;code&gt;read and write quorums&lt;/code&gt; separately&lt;/h3&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s make the joint quorums concrete. Call one &lt;code&gt;read quorum&lt;/code&gt; from the old configuration &lt;code&gt;c0.read&lt;/code&gt;, and one &lt;code&gt;write quorum&lt;/code&gt; &lt;code&gt;c0.write&lt;/code&gt;. We&amp;rsquo;ll use the same notation for &lt;code&gt;c01&lt;/code&gt; and &lt;code&gt;c1&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;To build a joint &lt;code&gt;read quorum&lt;/code&gt;, take a &lt;code&gt;read quorum&lt;/code&gt; from each configuration and combine them. To build a joint &lt;code&gt;write quorum&lt;/code&gt;, do the same with a &lt;code&gt;write quorum&lt;/code&gt; from each:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;c01.read&lt;/code&gt; contains a &lt;code&gt;c0.read&lt;/code&gt; quorum and a &lt;code&gt;c1.read&lt;/code&gt; quorum.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;c01.write&lt;/code&gt; contains a &lt;code&gt;c0.write&lt;/code&gt; quorum and a &lt;code&gt;c1.write&lt;/code&gt; quorum.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raft-rw-quorum/20c0463ced3f9844-08-joint-quorums.png&#34; alt=&#34;Three phases: c0, c01, and c1 each show read and write quorums&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;The joint configuration now shares the quorums it needs with both sides. The old and new configurations can even have completely different members. They still connect through &lt;code&gt;c01&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Both pieces matter: the joint quorums give us the overlap, and committing &lt;code&gt;c01&lt;/code&gt; before introducing &lt;code&gt;c1&lt;/code&gt; makes sure we use that overlap throughout the change.&lt;/p&gt;&#xA;&lt;h2 id=&#34;quorums-and-leader-identities-must-be-designed-together&#34;&gt;Quorums and leader identities must be designed together&lt;/h2&gt;&#xA;&lt;p&gt;Go back to that five-node leader waiting for its third copy of a write. We can let it commit with two copies. One way is to ask for four votes at the next election. Another is to order full leader identities and carry that ordering through voting, replication, log matching, and commitment, so we can safely use other quorum arrangements.&lt;/p&gt;&#xA;&lt;p&gt;In either case, every &lt;code&gt;read quorum&lt;/code&gt; still has to meet every &lt;code&gt;write quorum&lt;/code&gt;. The same overlap helps us confirm leadership for reads and move safely between memberships.&lt;/p&gt;&#xA;&lt;p&gt;We can make the leader wait for fewer acknowledgments. We just need to be clear about what we&amp;rsquo;re asking the rest of the cluster to do in return, especially when a leader fails or the membership changes.&lt;/p&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;etcd/raft: ReadIndex interface and application requirements : &lt;a href=&#34;https://github.com/etcd-io/raft/blob/v3.6.0/node.go#L199-L205&#34;&gt;https://github.com/etcd-io/raft/blob/v3.6.0/node.go#L199-L205&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;etcd/raft: serve ReadIndex only after committing in the current term : &lt;a href=&#34;https://github.com/etcd-io/raft/blob/v3.6.0/raft.go#L1278-L1285&#34;&gt;https://github.com/etcd-io/raft/blob/v3.6.0/raft.go#L1278-L1285&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;OpenRaft: check the current leader identity before committing : &lt;a href=&#34;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/replication_handler/mod.rs#L230-L244&#34;&gt;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/replication_handler/mod.rs#L230-L244&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;OpenRaft: partial comparison of leader IDs : &lt;a href=&#34;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/vote/leader_id/leader_id_cmp.rs#L9-L27&#34;&gt;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/vote/leader_id/leader_id_cmp.rs#L9-L27&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;OpenRaft: check log prefixes with full log IDs : &lt;a href=&#34;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/following_handler/mod.rs#L120-L139&#34;&gt;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/following_handler/mod.rs#L120-L139&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;OpenRaft: vote with and persist the full leader identity : &lt;a href=&#34;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/vote_handler/mod.rs#L101-L155&#34;&gt;https://github.com/databendlabs/openraft/blob/728e66bba05c4305dd37016535d1c247a35d2059/openraft/src/engine/handler/vote_handler/mod.rs#L101-L155&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Flexible Paxos: Quorum intersection revisited : &lt;a href=&#34;https://arxiv.org/pdf/1608.06696&#34;&gt;https://arxiv.org/pdf/1608.06696&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Paxos Made Simple : &lt;a href=&#34;https://lamport.azurewebsites.net/pubs/paxos-simple.pdf&#34;&gt;https://lamport.azurewebsites.net/pubs/paxos-simple.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;In Search of an Understandable Consensus Algorithm : &lt;a href=&#34;https://raft.github.io/raft.pdf&#34;&gt;https://raft.github.io/raft.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="distributed"/>
    <category term="raft"/>
    <category term="consensus"/>
    <category term="distributed"/>
    <summary type="html">A Raft leader usually waits for a majority before committing a write. We can reduce that wait, but the election rules and leader identities must still protect the log. Through a few small clusters, we explore two ways to do this, then follow the consequences for reads, failures, and membership changes.</summary>
  </entry>
  <entry>
    <title type="html">The Pitfalls of Raft Membership Change</title>
    <link href="https://blog.openacid.com/distributed/raft-bug/" rel="alternate" type="text/html" title="The Pitfalls of Raft Membership Change"/>
    <published>2026-08-27T00:00:00&#43;00:00</published>
    <updated>2026-08-27T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/distributed/raft-bug</id>
    <content type="html" xml:base="https://blog.openacid.com/distributed/raft-bug/">&lt;p&gt;&lt;img src=&#34;/post-res/raft-bug/4bcb4c62d2b23134-raft-bug-en-banner-small.webp&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;h1 id=&#34;the-problem&#34;&gt;The Problem&lt;/h1&gt;&#xA;&lt;p&gt;A while back, over coffee, a friend described a problem their team had hit with &lt;a href=&#34;https://raft.github.io/&#34; title=&#34;Raft&#34;&gt;Raft&lt;/a&gt; in production.&#xA;It looks like a small detail. It cost them a whole cluster.&lt;/p&gt;&#xA;&lt;p&gt;Their implementation uses &lt;a href=&#34;https://gist.github.com/ongardie/a11f32b70581e20d6bcd&#34; title=&#34;Safety of Raft single-server membership changes&#34;&gt;single-server change&lt;/a&gt;:&#xA;to change the replica set, you add or remove one node at a time.&#xA;Moving from &lt;code&gt;abc&lt;/code&gt; to &lt;code&gt;bcd&lt;/code&gt; takes two steps.&#xA;First add &lt;code&gt;d&lt;/code&gt;, which gives &lt;code&gt;abcd&lt;/code&gt;. Then remove &lt;code&gt;a&lt;/code&gt;, which gives &lt;code&gt;bcd&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The trouble lives in the middle step.&#xA;While the cluster has four nodes, a network split of the shape &lt;code&gt;ad | bc&lt;/code&gt; leaves it unable to elect a leader.&#xA;That shape of split is easy to get when the nodes live in different datacenters.&#xA;Say &lt;code&gt;a&lt;/code&gt;, &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt; each sit in their own datacenter:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;&#xA; a      b      c&#xA;----   ----   ----&#xA;DC-1   DC-2   DC-3&#xA;&#xA;        | add `d` in DC-1&#xA;        v&#xA;&#xA; a      b      c     partitioned     a   |  b      c&#xA; d                   no leader!      d   |&#xA;----   ----   ----   ------------&amp;gt;  ---- | ----   ----&#xA;DC-1   DC-2   DC-3                  DC-1 | DC-2   DC-3&#xA;&#xA;        | remove `a`,&#xA;        | healthy again&#xA;        v&#xA;&#xA;        b      c&#xA; d&#xA;----   ----   ----&#xA;DC-1   DC-2   DC-3&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;In the steady state, the cluster has three nodes. If any one datacenter loses contact with the outside,&#xA;the other two still hold a majority. They elect a leader and keep serving.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;In the middle state, DC-1 holds two nodes, &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;d&lt;/code&gt;.&#xA;A majority of four nodes needs three nodes.&#xA;DC-1 has only two, so it cannot elect a leader on its own.&#xA;DC-2 and DC-3 together also have only two, so they cannot either.&#xA;One datacenter going quiet now stops the whole cluster.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Look at what happened to DC-1. In the four-node middle state, every majority has to include a node from DC-1.&#xA;DC-1 became a &lt;strong&gt;single point of failure&lt;/strong&gt;, and only because a membership change was in progress.&lt;/p&gt;&#xA;&lt;p&gt;The root cause is that single-server change is rigid about one thing:&#xA;a quorum is always a majority, and never anything else.&#xA;Once we let go of that rule, the problem dissolves.&#xA;So we will look at membership change through quorum sets,&#xA;and that view leads us straight to joint consensus.&lt;/p&gt;&#xA;&lt;p&gt;If quorum sets are new to you, my earlier article&#xA;&lt;a href=&#34;https://blog.openacid.com/algo/quorum/&#34; title=&#34;A Minority Implementation of Majority Read/Write&#34;&gt;A Minority Implementation of Majority Read/Write&lt;/a&gt; introduces the way of thinking we use below.&lt;/p&gt;&#xA;&lt;h1 id=&#34;looking-at-the-problem-through-quorum-sets&#34;&gt;Looking at the Problem Through Quorum Sets&lt;/h1&gt;&#xA;&lt;p&gt;Instead of describing a cluster by its nodes, describe it by its quorums.&#xA;A quorum is a group of nodes that is allowed to commit something.&#xA;The list of all such groups is the quorum set.&lt;/p&gt;&#xA;&lt;p&gt;One rule keeps a quorum set safe: &lt;strong&gt;any two groups in it must share at least one node.&lt;/strong&gt;&#xA;That shared node is what stops two different values from both being committed,&#xA;and it is the only thing Paxos and Raft really need from a quorum.&lt;/p&gt;&#xA;&lt;p&gt;Here are the quorum sets in our story:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;The starting state &lt;code&gt;abc&lt;/code&gt; uses every majority of &lt;code&gt;abc&lt;/code&gt;: M(abc) = {ab, ac, bc}.&#xA;The full group &lt;code&gt;abc&lt;/code&gt; is a quorum too, but it already contains &lt;code&gt;ab&lt;/code&gt;,&#xA;and any group that contains a quorum is a quorum.&#xA;So listing the bigger groups adds nothing, and we list only the smallest ones.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;The final state &lt;code&gt;bcd&lt;/code&gt; uses M(bcd) = {bc, cd, bd}.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;The middle state &lt;code&gt;abcd&lt;/code&gt; of a single-server change is again a majority set:&#xA;M(abcd) = {abc, abd, acd, bcd}.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;So a single-server change is a walk through three quorum sets:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;M(abc) → M(abcd) → M(bcd)&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Now the availability problem has a one-line explanation.&#xA;Every quorum in the middle set needs three nodes.&#xA;When the network splits into &lt;code&gt;ad | bc&lt;/code&gt;, neither side has three nodes.&#xA;Neither side can elect a leader, and the cluster stops.&lt;/p&gt;&#xA;&lt;h2 id=&#34;a-first-patch-let-bc-commit&#34;&gt;A First Patch: Let &lt;code&gt;bc&lt;/code&gt; Commit&lt;/h2&gt;&#xA;&lt;p&gt;The middle set is where it hurts, so let us change the middle set.&#xA;Suppose we also allow &lt;code&gt;bc&lt;/code&gt; to commit. The middle quorum set becomes:&lt;/p&gt;&#xA;&lt;p&gt;Q(abcd) = M(abcd) ∪ {bc}&lt;/p&gt;&#xA;&lt;p&gt;In plain words: &lt;strong&gt;an entry is committed once it reaches &lt;code&gt;bc&lt;/code&gt;, or any three of the four nodes.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;This is safe, and the check is quick.&#xA;&lt;code&gt;bc&lt;/code&gt; shares a node with every group in M(abcd), because &lt;code&gt;bc&lt;/code&gt; has two nodes,&#xA;each three-node majority leaves out only one node, and it cannot leave out both &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt;.&#xA;So the sharing rule still holds across the whole set.&#xA;Paxos and Raft run on this middle state exactly as before, and consistency is untouched.&lt;/p&gt;&#xA;&lt;p&gt;The change now reads: M(abc) → M(abcd) ∪ {bc} → M(bcd).&lt;/p&gt;&#xA;&lt;p&gt;It is also still a legal membership change.&#xA;Raft&amp;rsquo;s safety argument for a single-server change asks for one thing:&#xA;every quorum of the old node set must share a node with every quorum of the new one.&#xA;Our added group &lt;code&gt;bc&lt;/code&gt; meets &lt;code&gt;ab&lt;/code&gt; at &lt;code&gt;b&lt;/code&gt;, meets &lt;code&gt;ac&lt;/code&gt; at &lt;code&gt;c&lt;/code&gt;, and meets &lt;code&gt;bc&lt;/code&gt; at both.&#xA;So &lt;strong&gt;if Raft is safe for M(abc) → M(abcd), it is equally safe for M(abc) → M(abcd) ∪ {bc}&lt;/strong&gt;.&#xA;The same check passes for the second step, M(abcd) ∪ {bc} → M(bcd).&lt;/p&gt;&#xA;&lt;p&gt;That treats the symptom. The cluster now survives the &lt;code&gt;ad | bc&lt;/code&gt; split while the change is running.&lt;/p&gt;&#xA;&lt;h2 id=&#34;why-majority-runs-out-of-room&#34;&gt;Why Majority Runs Out of Room&lt;/h2&gt;&#xA;&lt;p&gt;The four-node state has this weakness for a reason worth naming:&#xA;&lt;strong&gt;M(abcd) is not the largest safe quorum set for four nodes.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;With an odd number of nodes, majority is the best you can do.&#xA;Take three nodes &lt;code&gt;abc&lt;/code&gt;: you cannot add any smaller group to {ab, ac, bc},&#xA;because a single node such as &lt;code&gt;a&lt;/code&gt; fails to meet &lt;code&gt;bc&lt;/code&gt;.&#xA;Majority is already maximal, so nothing is lost.&lt;/p&gt;&#xA;&lt;p&gt;With an even number of nodes, majority leaves quorums on the table.&#xA;A four-node system has four three-node majorities.&#xA;On top of those, it can safely hold three more two-node groups:&lt;/p&gt;&#xA;&lt;p&gt;Q&amp;rsquo;(abcd) = M(abcd) ∪ &lt;strong&gt;{ab, bc, ac}&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Every pair in Q&amp;rsquo;(abcd) shares a node.&#xA;&lt;code&gt;ab&lt;/code&gt; and &lt;code&gt;bc&lt;/code&gt; share &lt;code&gt;b&lt;/code&gt;. &lt;code&gt;ab&lt;/code&gt; and &lt;code&gt;ac&lt;/code&gt; share &lt;code&gt;a&lt;/code&gt;. &lt;code&gt;bc&lt;/code&gt; and &lt;code&gt;ac&lt;/code&gt; share &lt;code&gt;c&lt;/code&gt;.&#xA;Each two-node group also meets each three-node group, because two plus three is more than four.&#xA;Paxos and Raft run on Q&amp;rsquo;(abcd) with no changes at all, and it tolerates strictly more failures than M(abcd).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Majority is the first weak spot in Raft&amp;rsquo;s design.&lt;/strong&gt;&#xA;By writing majority into the algorithm, Raft gives away availability that an even-sized cluster could have had.&lt;/p&gt;&#xA;&lt;h2 id=&#34;how-to-expand-a-majority&#34;&gt;How to Expand a Majority&lt;/h2&gt;&#xA;&lt;p&gt;Here is the general recipe. Let the node set be C, for example C = {a,b,c}.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;For an odd node count, n = 2k+1, keep the majorities. They are already maximal:&lt;/p&gt;&#xA;\[Q_{odd}(C) = M(C) = \{ q : q \subseteq C,  |q| &amp;gt; |C|/2 \}\]&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;For an even node count, n = 2k, notice that &lt;strong&gt;any n/2 nodes must share a node with any n/2+1 nodes&lt;/strong&gt;:&#xA;together they count n+1 nodes in a cluster of only n.&#xA;So we may add groups of size n/2 to M(C).&#xA;The only extra thing to check is that the added groups share nodes with each other.&lt;/p&gt;&#xA;&lt;p&gt;In our four-node example:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Q&amp;rsquo; = M(abcd) ∪ {ab, bc, ca} works: the three added groups pairwise share a node.&lt;/li&gt;&#xA;&lt;li&gt;Q&amp;rsquo; = M(abcd) ∪ {bc, cd, bd} works for the same reason.&lt;/li&gt;&#xA;&lt;li&gt;Q&amp;rsquo; = M(abcd) ∪ {ab, bc, cd} does not work: &lt;code&gt;ab&lt;/code&gt; and &lt;code&gt;cd&lt;/code&gt; share nothing,&#xA;so two leaders could be elected at the same time.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;There is an easy way to produce a good one.&#xA;Treat the even cluster as an odd cluster C plus one extra node x:&lt;/p&gt;&#xA;\[D = C \cup \{x\}\]&#xA;&lt;p&gt;Then the quorum set for the even cluster can be an expansion of M(D):&lt;/p&gt;&#xA;\[Q_{even}(D)_x = M(D) \cup M(D \setminus \{x\})\]&#xA;&lt;p&gt;In words: keep every majority of the four nodes, and also accept every majority of the three nodes left when you ignore x.&#xA;Picking x = d produces the first example above, and picking x = a produces the second.&#xA;Both hold more quorums than M(abcd), so both are more available, and both survive the datacenter split we started with.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h1 id=&#34;what-the-middle-state-really-needs&#34;&gt;What the Middle State Really Needs&lt;/h1&gt;&#xA;&lt;p&gt;Those examples make one thing clear.&#xA;The middle state of a membership change does not have to be a majority set.&#xA;It only has to be safe, and for our datacenter problem it has to contain &lt;code&gt;bc&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Several middle states qualify:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;M(abcd) ∪ {ab, bc, ac},&lt;/li&gt;&#xA;&lt;li&gt;{abc, abd, acd, bcd, bc},&lt;/li&gt;&#xA;&lt;li&gt;and even {abd, acd, bcd, bc}, with &lt;code&gt;abc&lt;/code&gt; dropped.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Joint consensus qualifies too. It looks complicated on paper, and it turns out to be the simplest of all.&lt;/p&gt;&#xA;&lt;h2 id=&#34;the-correctness-conditions&#34;&gt;The Correctness Conditions&lt;/h2&gt;&#xA;&lt;p&gt;Before comparing algorithms, let us write down what a membership change has to guarantee.&#xA;Describe each state by its quorum set, the way we have been doing, and let the change go from Q₁ to Q₂.&#xA;It has to meet three conditions:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;&lt;strong&gt;A committed change stays visible.&lt;/strong&gt;&#xA;If one change is already committed, every uncommitted change must be recognizable as uncommitted.&#xA;Otherwise a new leader cannot tell which of them to keep.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;&lt;strong&gt;Concurrent changes exclude each other.&lt;/strong&gt;&#xA;Only one of several concurrent changes may succeed,&#xA;so every process proposing a change must commit it against the same quorum set.&#xA;The only thing all processes already agree on is Q₁.&#xA;So a change must be committed to Q₁, or to an expansion of Q₁ that every process derives in the same way.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;&lt;strong&gt;The change reaches the new configuration.&lt;/strong&gt;&#xA;It must also be committed to a quorum of Q₂.&#xA;Otherwise a leader elected under Q₂ may never see it.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Raft&amp;rsquo;s original single-server change misses the first condition. The author fixed it later, and we will come back to that.&lt;/p&gt;&#xA;&lt;h2 id=&#34;joint-consensus-gives-us-exactly-that&#34;&gt;Joint Consensus Gives Us Exactly That&lt;/h2&gt;&#xA;&lt;p&gt;Joint consensus meets all three conditions.&#xA;It also handles our datacenter problem, without anyone designing it for that.&lt;/p&gt;&#xA;&lt;p&gt;In a change from &lt;code&gt;abc&lt;/code&gt; to &lt;code&gt;bcd&lt;/code&gt;, the joint middle state is the product of the two majority sets:&lt;/p&gt;&#xA;&lt;p&gt;Q = M(abc) x M(bcd)&lt;/p&gt;&#xA;&lt;p&gt;A joint quorum is any group that contains one quorum of M(abc) and one quorum of M(bcd) at the same time.&#xA;With M(abc) = {ab, bc, ca} and M(bcd) = {bc, cd, bd}, the product is:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;M(abc) x M(bcd) = {&#xA;    ab ∪ bc,&#xA;    ab ∪ cd,&#xA;    ab ∪ bd,&#xA;    bc ∪ bc,&#xA;    bc ∪ cd,&#xA;    bc ∪ bd,&#xA;    ac ∪ bc,&#xA;    ac ∪ cd,&#xA;    ac ∪ bd,&#xA;} = {&#xA;    abc,&#xA;    abcd,&#xA;    abd,&#xA;    acd,&#xA;    bc,&#xA;    bcd,&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;That is exactly M(abcd) ∪ {bc}&lt;/strong&gt; — the very quorum set we built by hand a few sections ago.&lt;/p&gt;&#xA;&lt;p&gt;So joint consensus hands us everything we were after:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It tolerates one node failure.&lt;/li&gt;&#xA;&lt;li&gt;It always contains &lt;code&gt;bc&lt;/code&gt;, so it survives the &lt;code&gt;ad | bc&lt;/code&gt; split that started this article.&lt;/li&gt;&#xA;&lt;li&gt;The whole change finishes with two committed log entries, whether or not the leader changes along the way.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h1 id=&#34;the-bug-in-single-server-change&#34;&gt;The Bug in Single-Server Change&lt;/h1&gt;&#xA;&lt;p&gt;Single-server change has a second problem, and this one is heavier than availability.&#xA;As first published, it was simply incorrect.&lt;/p&gt;&#xA;&lt;p&gt;The bug appears when a leader change and a membership change run at the same time.&#xA;The author announced it in 2015:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Unfortunately, I need to announce a bug in the dissertation version of&#xA;membership changes (the single-server changes, not joint consensus). The bug is&#xA;potentially severe, but the fix I&amp;rsquo;m proposing is easy to implement.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Here is how it goes wrong.&#xA;The cluster starts with the four nodes &lt;code&gt;abcd&lt;/code&gt;.&#xA;One process wants to add &lt;code&gt;u&lt;/code&gt;, another wants to add &lt;code&gt;v&lt;/code&gt;.&#xA;A leader change in the middle loses a committed entry:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;C₀ = {a, b, c, d}&#xA;Cᵤ = C₀ ∪ {u}&#xA;Cᵥ = C₀ ∪ {v}&#xA;&#xA;Lᵢ: Leader in term `i`&#xA;Fᵢ: Follower in term `i`&#xA;☒ : crash&#xA;&#xA;    |&#xA; u  |         Cᵤ                  F₂  Cᵤ&#xA;--- | ----------------------------------&#xA; a  | C₀  L₀  Cᵤ  ☒               L₂  Cᵤ&#xA; b  | C₀  F₀          F₁          F₂  Cᵤ&#xA; c  | C₀  F₀          F₁  Cᵥ          Cᵤ&#xA; d  | C₀              L₁  Cᵥ  ☒       Cᵤ&#xA;--- | ----------------------------------&#xA; v  |                     Cᵥ                  time&#xA;    +--------------------------------------------&amp;gt;&#xA;          t₁  t₂  t₃  t₄  t₅  t₆  t₇  t₈&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;&#xA;&lt;li&gt;t₁: the four nodes &lt;code&gt;abcd&lt;/code&gt; elect &lt;code&gt;a&lt;/code&gt; as leader in term 0, with followers &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;t₂: &lt;code&gt;a&lt;/code&gt; appends a change entry &lt;code&gt;Cᵤ&lt;/code&gt; and switches to the new config &lt;code&gt;Cᵤ&lt;/code&gt; right away. The entry reaches only &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;u&lt;/code&gt;, so it is not committed.&lt;/li&gt;&#xA;&lt;li&gt;t₃: &lt;code&gt;a&lt;/code&gt; crashes.&lt;/li&gt;&#xA;&lt;li&gt;t₄: &lt;code&gt;d&lt;/code&gt; is elected leader in term 1, with followers &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;t₅: &lt;code&gt;d&lt;/code&gt; appends another change entry &lt;code&gt;Cᵥ&lt;/code&gt; and switches to &lt;code&gt;Cᵥ&lt;/code&gt;. The entry reaches &lt;code&gt;c&lt;/code&gt;, &lt;code&gt;d&lt;/code&gt; and &lt;code&gt;v&lt;/code&gt;, which is a majority of the five nodes in &lt;code&gt;Cᵥ&lt;/code&gt;, so it is committed.&lt;/li&gt;&#xA;&lt;li&gt;t₆: &lt;code&gt;d&lt;/code&gt; crashes.&lt;/li&gt;&#xA;&lt;li&gt;t₇: &lt;code&gt;a&lt;/code&gt; comes back and is elected leader in term 2. It runs under &lt;code&gt;Cᵤ&lt;/code&gt;, the config it sees in its own log, and collects votes from &lt;code&gt;u&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;t₈: &lt;code&gt;a&lt;/code&gt; replicates its own log to everyone, and the committed &lt;code&gt;Cᵥ&lt;/code&gt; is gone.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Read t₅ and t₈ together, because that is where the damage is.&#xA;&lt;code&gt;Cᵥ&lt;/code&gt; was committed under every rule Raft gives us, and then it was overwritten.&#xA;The reason is that &lt;code&gt;a&lt;/code&gt; was allowed to run an election under a configuration nobody else had ever committed.&lt;/p&gt;&#xA;&lt;p&gt;The author&amp;rsquo;s fix is short, and it echoes a rule Raft already has for ordinary entries:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The solution I&amp;rsquo;m proposing is exactly like the dissertation describes except&#xA;that a leader may not append a new configuration entry until it has committed&#xA;an entry from its current term.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;In our timeline, &lt;code&gt;d&lt;/code&gt; must commit a no-op entry in term 1 before it may append &lt;code&gt;Cᵥ&lt;/code&gt;.&#xA;Once &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt; hold that no-op, &lt;code&gt;a&lt;/code&gt; can no longer win the term-2 election:&#xA;&lt;code&gt;b&lt;/code&gt; sees that &lt;code&gt;a&lt;/code&gt;&amp;rsquo;s log is behind and refuses to vote for it.&#xA;So &lt;code&gt;a&lt;/code&gt; never becomes L₂, and the committed &lt;code&gt;Cᵥ&lt;/code&gt; survives.&lt;/p&gt;&#xA;&lt;h2 id=&#34;look-closely-at-that-fix&#34;&gt;Look Closely at That Fix&lt;/h2&gt;&#xA;&lt;p&gt;The fix quietly turns single-server change into joint consensus.&lt;/p&gt;&#xA;&lt;p&gt;Both end up doing the same job.&#xA;A change has to pass through a quorum of the old configuration first,&#xA;so that only one change out of several concurrent ones can be considered committed.&#xA;Single-server change reaches that point with an extra entry:&#xA;an ordinary application entry if one is handy, or a no-op if not.&#xA;Joint consensus reaches it directly, because its middle state already is&#xA;the old configuration and the new configuration at the same time.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A correct single-server change costs two log commits, every time.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Single-server change was proposed to make things simpler, and it does not.&#xA;Changing &lt;code&gt;abc&lt;/code&gt; to &lt;code&gt;bcd&lt;/code&gt; costs 2 to 4 log entries with single-server change.&#xA;With joint consensus it costs 2.&lt;/p&gt;&#xA;&lt;p&gt;There is a fair objection here: single-server change often needs only 2 entries,&#xA;since the leader usually has a committed entry of its own term already and no no-op is required.&#xA;That is true, and it does not help.&#xA;Code is not a bet on probability.&#xA;Every branch that can run has to be written, tested and maintained,&#xA;including the one that fires once in ten thousand changes.&#xA;So a correct single-server change carries almost the same logic as joint consensus,&#xA;implements a two-step change anyway, and wins nothing at runtime.&lt;/p&gt;&#xA;&lt;h1 id=&#34;closing-thoughts&#34;&gt;Closing Thoughts&lt;/h1&gt;&#xA;&lt;p&gt;Raft is a beautiful bridge from theory to working code,&#xA;and that beauty is exactly why one design mistake in it travelled so far.&lt;/p&gt;&#xA;&lt;p&gt;If you are building or maintaining a Raft implementation, the advice is short: use joint consensus.&#xA;It closes the availability hole in the middle state,&#xA;it removes the bug, and it is less code than a correct single-server change.&lt;/p&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;A Minority Implementation of Majority Read/Write : &lt;a href=&#34;https://blog.openacid.com/algo/quorum/&#34;&gt;https://blog.openacid.com/algo/quorum/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Raft : &lt;a href=&#34;https://raft.github.io/&#34;&gt;https://raft.github.io/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Safety of Raft single-server membership changes : &lt;a href=&#34;https://gist.github.com/ongardie/a11f32b70581e20d6bcd&#34;&gt;https://gist.github.com/ongardie/a11f32b70581e20d6bcd&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="distributed"/>
    <category term="raft"/>
    <category term="paxos"/>
    <category term="consensus"/>
    <category term="membership"/>
    <summary type="html">Raft&#39;s single-server membership change carries two problems: its middle state can turn one datacenter into a single point of failure, and the original algorithm could lose a committed entry. Let us look at membership change through quorum sets, and see why joint consensus is the answer.</summary>
  </entry>
  <entry>
    <title type="html">Build a Distributed Timestamp Oracle with EzRaft</title>
    <link href="https://blog.openacid.com/algo/ez-ts-oracle/" rel="alternate" type="text/html" title="Build a Distributed Timestamp Oracle with EzRaft"/>
    <published>2026-08-11T00:00:00&#43;00:00</published>
    <updated>2026-08-11T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/algo/ez-ts-oracle</id>
    <content type="html" xml:base="https://blog.openacid.com/algo/ez-ts-oracle/">&lt;p&gt;&lt;img src=&#34;/post-res/ez-ts-oracle/b43c2a98b06d02ef-ez-time-server-banner.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;A timestamp oracle issues one globally increasing sequence. Transaction systems built on &lt;a href=&#34;https://en.wikipedia.org/wiki/Multiversion_concurrency_control&#34; title=&#34;Multiversion concurrency control&#34;&gt;MVCC&lt;/a&gt; consume two of those values on every commit, so the oracle sits on the critical path of every write. The naive way to keep the sequence safe is one Raft commit per request. That design is correct. It is also too slow.&lt;/p&gt;&#xA;&lt;p&gt;This article builds the other design: reserve a range with one Raft write, then allocate from memory. The service is &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs&#34; title=&#34;ezraft timestamp_oracle example&#34;&gt;timestamp_oracle&lt;/a&gt;, about 200 lines on &lt;a href=&#34;https://github.com/drmingdrmer/ezraft&#34; title=&#34;ezraft&#34;&gt;EzRaft&lt;/a&gt;. It runs as a three-node cluster, stays available after any one node fails, and answers &lt;code&gt;next_timestamp&lt;/code&gt; without waiting for a Raft commit.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&#34;https://github.com/drmingdrmer/ezraft&#34; title=&#34;ezraft&#34;&gt;EzRaft&lt;/a&gt;, from the &lt;a href=&#34;https://blog.openacid.com/algo/ezraft/&#34; title=&#34;EzRaft: Build a Distributed KV Store in 100 Lines&#34;&gt;previous article&lt;/a&gt;, is a small &lt;a href=&#34;https://www.rust-lang.org/&#34; title=&#34;Rust&#34;&gt;Rust&lt;/a&gt; API over &lt;a href=&#34;https://github.com/databendlabs/openraft&#34; title=&#34;OpenRaft&#34;&gt;OpenRaft&lt;/a&gt;. It hides the &lt;a href=&#34;https://raft.github.io/&#34; title=&#34;Raft&#34;&gt;Raft&lt;/a&gt;, networking, and storage machinery, so you can start with the service itself.&lt;/p&gt;&#xA;&lt;p&gt;Three calls carry the design:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;EzRaft::write()&lt;/code&gt; commits the upper bound of a reserved range and applies it to the state machine.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;EzRaft::wait_metrics()&lt;/code&gt; lets a background task sleep until its node becomes the leader.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;EzRaft::linearizable(ReadPolicy::LeaseRead)&lt;/code&gt; checks the leader lease and returns the current term, so each cached range is bound to one term.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The idea underneath those calls is simple. Raft owns a ceiling. The leader spends from below that ceiling in memory. A lease and a term tag decide whether it is still allowed to spend.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/ez-ts-oracle/0830bc2afe95fe2a-time-server-architecture.png&#34; alt=&#34;Architecture overview: next_timestamp requests allocate timestamps from the leader&amp;rsquo;s memory, while the reservation task connects the in-memory cache to a three-node EzRaft cluster&#34;&gt;&lt;/p&gt;&#xA;&lt;h2 id=&#34;what-a-timestamp-oracle-must-guarantee&#34;&gt;What a Timestamp Oracle Must Guarantee&lt;/h2&gt;&#xA;&lt;p&gt;The interface is one call: a client sends &lt;code&gt;next_timestamp&lt;/code&gt;, and the service returns a timestamp.&lt;/p&gt;&#xA;&lt;p&gt;Those values usually order distributed transactions. Models such as &lt;a href=&#34;https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Peng.pdf&#34; title=&#34;Large-scale Incremental Processing Using Distributed Transactions and Notifications&#34;&gt;Percolator&lt;/a&gt; give each transaction a start timestamp and a commit timestamp. Those two numbers decide which transaction came first. TiDB&amp;rsquo;s &lt;a href=&#34;https://docs.pingcap.com/tidb/stable/glossary/#timestamp-oracle-tso&#34; title=&#34;TiDB: Timestamp Oracle (TSO)&#34;&gt;TSO (Timestamp Oracle)&lt;/a&gt; is the same component.&lt;/p&gt;&#xA;&lt;p&gt;The service has three hard requirements:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Monotonicity. Timestamps increase in the order requests reach the oracle, and they never go backward after a failover. Every timestamp from the new leader must be greater than every timestamp the old leader already returned.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Fault tolerance. The service stays available while a majority of the cluster is alive. OpenRaft already provides that, so a three-node cluster can lose any one node and keep issuing timestamps.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Low latency. &lt;code&gt;next_timestamp&lt;/code&gt; sits on the critical path of every operation that needs an ordering point, so each request has to be cheap.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;There is a softer requirement as well. The values should stay close to wall-clock time. Correctness only requires the sequence to increase. Closeness to wall-clock time is a quality goal, not a safety property.&lt;/p&gt;&#xA;&lt;h2 id=&#34;safe-baseline-one-raft-write-per-timestamp&#34;&gt;Safe Baseline: One Raft Write per Timestamp&lt;/h2&gt;&#xA;&lt;p&gt;The cluster only has to agree on one durable value: &lt;code&gt;reserved_end&lt;/code&gt;, the exclusive end of all timestamp space reserved so far. Every value below &lt;code&gt;reserved_end&lt;/code&gt; is permanently consumed, whether or not a client ever received it.&lt;/p&gt;&#xA;&lt;p&gt;The obvious safe design writes one Raft log entry for every &lt;code&gt;next_timestamp&lt;/code&gt; request. Once that entry is replicated and applied, the state machine returns the timestamp it claimed. Every leader walks the same log, so a failover cannot reuse an earlier value.&lt;/p&gt;&#xA;&lt;p&gt;The state machine is correspondingly small:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Reserve&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;reserve_upto_us: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Interval&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;start: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;end: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;TimeState&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;reserved_end: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;impl&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;EzApp&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;for&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;TimeState&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;type&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Request&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;Reserve&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;type&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Response&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;Interval&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;apply&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;req: &lt;span style=&#34;color:#1f2328&#34;&gt;Reserve&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Interval&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;start&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;reserved_end&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;end&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;start&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;max&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;req&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;reserve_upto_us&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;reserved_end&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;end&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;Interval&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;start&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;end&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;Reserve&lt;/code&gt; asks the state machine to move &lt;code&gt;reserved_end&lt;/code&gt; to at least &lt;code&gt;reserve_upto_us&lt;/code&gt;. The response is the newly claimed half-open interval &lt;code&gt;[start, end)&lt;/code&gt;. Those types and &lt;code&gt;apply()&lt;/code&gt; are all that &lt;a href=&#34;https://docs.rs/ezraft/latest/ezraft/app/trait.EzApp.html&#34; title=&#34;EzApp&#34;&gt;&lt;code&gt;EzApp&lt;/code&gt;&lt;/a&gt; needs. See &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L66-L112&#34; title=&#34;Reserve / Interval / TimeState / impl EzApp&#34;&gt;timestamp_oracle.rs:66-112&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;max&lt;/code&gt; in &lt;code&gt;apply()&lt;/code&gt; makes the transition idempotent. Applying the same upper bound again leaves &lt;code&gt;reserved_end&lt;/code&gt; unchanged, so a timed-out reservation can be retried safely.&lt;/p&gt;&#xA;&lt;p&gt;This design is safe. It is also slow: every request pays for Raft replication and state-machine application. The rest of the article takes both of those costs off the request path.&lt;/p&gt;&#xA;&lt;h2 id=&#34;fast-path-reserve-timestamps-in-batches&#34;&gt;Fast Path: Reserve Timestamps in Batches&lt;/h2&gt;&#xA;&lt;p&gt;One Raft entry can reserve a wide range instead of a single value. By default the service reserves one second of timestamp space, then the leader hands out individual values from that range in memory.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;code&gt;reserved_end&lt;/code&gt; is durable. The leader&amp;rsquo;s position inside the reserved range is not: it lives only in memory.&#xA;If the leader fails, its successor skips the unused suffix and reserves a new range.&#xA;The sequence may contain gaps. It never goes backward.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;A background reservation task keeps the leader&amp;rsquo;s cache filled. Each successful reservation is stored in a &lt;code&gt;Reserved&lt;/code&gt; value with three fields: the term that owns the range, the next available timestamp, and the exclusive upper bound.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Reserved&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;term: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;next: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;end: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;run_reserver()&lt;/code&gt; uses &lt;code&gt;wait_metrics()&lt;/code&gt; to wait until the node becomes the leader. Once it is, the task calls &lt;code&gt;refill()&lt;/code&gt; and repeats every half-reservation interval. See &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L170-L186&#34; title=&#34;TimeService::run_reserver&#34;&gt;timestamp_oracle.rs:170-186&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;TimeService&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// ...&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;reserved: &lt;span style=&#34;color:#1f2328&#34;&gt;Mutex&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;Reserved&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;impl&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;TimeService&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;run_reserver&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;refresh_interval&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;Duration::from_micros&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;reservation_width&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;get&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;/&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;2&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;loop&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;leads&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;raft&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;wait_metrics&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;None&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;|&lt;/span&gt;m&lt;span style=&#34;color:#0550ae&#34;&gt;|&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;m&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;state&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;==&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;ServerState::Leader&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;reserve timestamps&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Err&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;error&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;leads&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;                &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;return&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Err&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;error&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;refill&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;                &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;warn!&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;{error}&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;tokio::time::sleep&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;refresh_interval&lt;span style=&#34;color:#1f2328&#34;&gt;).&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;refill()&lt;/code&gt; first obtains a valid leader term. It then computes &lt;code&gt;now + width&lt;/code&gt;, commits that upper bound through Raft, and stores the returned interval in &lt;code&gt;TimeService.reserved&lt;/code&gt;. See &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L188-L205&#34; title=&#34;TimeService::refill&#34;&gt;timestamp_oracle.rs:188-205&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;refill&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#1f2328&#34;&gt;io&lt;/span&gt;::&lt;span style=&#34;color:#6639ba&#34;&gt;Result&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// ensure leader lease and get leader term.&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;leader_term&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;now&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;unix_timestamp_micros&lt;span style=&#34;color:#1f2328&#34;&gt;();&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;width&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;reservation_width&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;get&lt;span style=&#34;color:#1f2328&#34;&gt;();&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;upto&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;now&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;saturating_add&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;width&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;interval&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;raft&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;write&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;Reserve&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;reserve_upto_us: &lt;span style=&#34;color:#1f2328&#34;&gt;upto&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}).&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;reserved&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;lock&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;install&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;term&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;interval&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Ok&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(())&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The Raft write rate is now independent of the request rate. For a reservation width of &lt;code&gt;width&lt;/code&gt;, the task writes once every &lt;code&gt;width / 2&lt;/code&gt;. A single distributed commit can serve 100,000 &lt;code&gt;next_timestamp&lt;/code&gt; requests.&lt;/p&gt;&#xA;&lt;h2 id=&#34;preserve-monotonicity-across-leader-failover&#34;&gt;Preserve Monotonicity Across Leader Failover&lt;/h2&gt;&#xA;&lt;p&gt;Batch allocation stays safe because the committed upper bound, not the in-memory cursor, is the handoff point. When a new leader is elected, it already has the log. After it applies those entries, &lt;code&gt;reserved_end&lt;/code&gt; is at least as large as anything the old leader reserved. The new leader never starts from the old leader&amp;rsquo;s in-memory cursor, because that cursor was never replicated.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/ez-ts-oracle/2df814a86a1bcc27-time-server-failover-safety.png&#34; alt=&#34;Three-node leader failover: Raft log replication and state-machine replay keep reserved_end consistent across nodes, so the new leader begins from a safe timestamp range&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;That committed bound protects the new leader. It does not stop the old one. A network partition can isolate the old leader long enough for the other nodes to elect a replacement, while the old leader still believes it owns the cluster.&lt;/p&gt;&#xA;&lt;p&gt;Once the new leader issues a timestamp from a later range, any later allocation from the old leader&amp;rsquo;s range would move time backward. The old leader must refuse requests as soon as it can no longer prove that it still holds leadership.&lt;/p&gt;&#xA;&lt;h2 id=&#34;reject-a-stale-leader-with-a-lease&#34;&gt;Reject a Stale Leader with a Lease&lt;/h2&gt;&#xA;&lt;p&gt;A lease is a time-bounded promise: as long as the leader has heard from a &lt;a href=&#34;https://blog.openacid.com/algo/quorum/&#34; title=&#34;Quorum Reads and Writes with a Minority&#34;&gt;quorum&lt;/a&gt; recently, no other node can win an election. Checking that promise is a local read of heartbeat state, so it adds no network round trip to a timestamp request. &lt;code&gt;timestamp_oracle&lt;/code&gt; performs that check with a &lt;a href=&#34;https://blog.openacid.com/algo/linearizable/&#34; title=&#34;Linearizable Transactions in Distributed Systems&#34;&gt;linearizable&lt;/a&gt; read using &lt;code&gt;ReadPolicy::LeaseRead&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;OpenRaft refreshes the lease from heartbeat acknowledgments. A recent acknowledgment from a quorum opens a window during which another leader cannot be elected. The current leader can allocate timestamps for the length of that window.&lt;/p&gt;&#xA;&lt;p&gt;Once the latest quorum acknowledgment is older than the lease timeout, the node can no longer prove it is the only leader. &lt;code&gt;linearizable()&lt;/code&gt; returns an error, and &lt;code&gt;next_timestamp()&lt;/code&gt; stops before it touches the cache.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/ez-ts-oracle/cac807cbb92f1fc1-time-server-lease-handoff.png&#34; alt=&#34;Three-node timeline: heartbeat acknowledgments keep the old leader&amp;rsquo;s lease valid; after the lease expires, a local check rejects next_timestamp requests, and the cluster then elects a new leader&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/src/raft.rs#L343-L365&#34; title=&#34;EzRaft::linearizable&#34;&gt;&lt;code&gt;EzRaft::linearizable(ReadPolicy::LeaseRead)&lt;/code&gt;&lt;/a&gt; reads the most recent heartbeat state in memory. It does not contact the quorum again. Internally it calls OpenRaft&amp;rsquo;s &lt;a href=&#34;https://docs.rs/openraft/0.10.0-alpha.33/openraft/raft/struct.Raft.html#method.ensure_linearizable&#34; title=&#34;Raft::ensure_linearizable&#34;&gt;&lt;code&gt;Raft::ensure_linearizable()&lt;/code&gt;&lt;/a&gt; and returns &lt;code&gt;(term, index)&lt;/code&gt;. The term identifies the current leadership; the allocator uses it to validate the cached range:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;leader_term&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#1f2328&#34;&gt;io&lt;/span&gt;::&lt;span style=&#34;color:#6639ba&#34;&gt;Result&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;term&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;_index&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;raft&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;linearizable&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;ReadPolicy::LeaseRead&lt;span style=&#34;color:#1f2328&#34;&gt;).&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Ok&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;term&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;serve-next_timestamp-from-memory&#34;&gt;Serve &lt;code&gt;next_timestamp&lt;/code&gt; from Memory&lt;/h2&gt;&#xA;&lt;p&gt;With a reserved range in place, the request path has three steps:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Call &lt;code&gt;leader_term()&lt;/code&gt; to validate the lease and obtain the current term.&lt;/li&gt;&#xA;&lt;li&gt;Allocate one timestamp from the cache and advance the cursor.&lt;/li&gt;&#xA;&lt;li&gt;Return the timestamp to the client.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The HTTP handler, the request path, and the cache allocation fit in a few lines. See &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L124-L133&#34; title=&#34;Reserved::take&#34;&gt;timestamp_oracle.rs:124-133&lt;/a&gt; and &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L162-L168&#34; title=&#34;TimeService::next_timestamp&#34;&gt;timestamp_oracle.rs:162-168&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;get_time&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;service: &lt;span style=&#34;color:#1f2328&#34;&gt;web&lt;/span&gt;::Data&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;TimeService&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#6639ba&#34;&gt;Result&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;web::Json&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;actix_web::Error&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;micros&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;service&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;next_timestamp&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;map_err&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;actix_web::error::ErrorServiceUnavailable&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Ok&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;web::Json&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;format_timestamp&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;micros&lt;span style=&#34;color:#1f2328&#34;&gt;)))&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;next_timestamp&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#1f2328&#34;&gt;io&lt;/span&gt;::&lt;span style=&#34;color:#6639ba&#34;&gt;Result&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;reserved&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;reserved&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;lock&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;leader_term&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;timestamp&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;reserved&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;take&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;term&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;unix_timestamp_micros&lt;span style=&#34;color:#1f2328&#34;&gt;());&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;timestamp&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;ok_or_else&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;||&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;io::Error::other&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;no reserved timestamp is currently available&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;))&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;take&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;leader_term: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;now: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#6639ba&#34;&gt;Option&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;timestamp&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;now&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;max&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;next&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;!=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;leader_term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;||&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;timestamp&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;end&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;return&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;None&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;next&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;timestamp&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;+&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Some&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;timestamp&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;now.max(self.next)&lt;/code&gt; keeps timestamps close to wall-clock time while preserving local monotonicity. If the clock advances, the allocator follows it. If several requests arrive in the same microsecond—or the wall clock steps backward—&lt;code&gt;self.next&lt;/code&gt; keeps every result unique and increasing.&lt;/p&gt;&#xA;&lt;p&gt;The term comparison in &lt;code&gt;take()&lt;/code&gt; is not part of the lease check. The next section is why that field exists. &lt;code&gt;leader_term()&lt;/code&gt; keeps the term and discards the log index; see &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L207-L210&#34; title=&#34;TimeService::leader_term&#34;&gt;timestamp_oracle.rs:207-210&lt;/a&gt;. The &lt;a href=&#34;https://docs.rs/openraft/0.10.0-alpha.33/openraft/raft/enum.ReadPolicy.html&#34; title=&#34;OpenRaft ReadPolicy&#34;&gt;OpenRaft documentation&lt;/a&gt; describes the other &lt;code&gt;ReadPolicy&lt;/code&gt; options. The underlying read optimization is covered in &lt;a href=&#34;https://blog.openacid.com/algo/openraft-read/&#34; title=&#34;How OpenRaft Optimizes ReadIndex&#34;&gt;How OpenRaft Optimizes ReadIndex&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;bind-cached-ranges-to-a-leader-term&#34;&gt;Bind Cached Ranges to a Leader Term&lt;/h2&gt;&#xA;&lt;p&gt;A valid lease proves that this node is the leader now. It does not prove that the cached range belongs to the current term.&lt;/p&gt;&#xA;&lt;p&gt;Suppose node A reserved a range in term 1, lost leadership, and won it back in term 3. The leftover cache is still in the same process, so a restart is not there to wipe it. In between, node B may have been the leader in term 2 and may already have issued later timestamps. Node A&amp;rsquo;s leftover cache now sits behind time. Using it would go backward.&lt;/p&gt;&#xA;&lt;p&gt;That is why &lt;code&gt;Reserved&lt;/code&gt; stores the term next to &lt;code&gt;next&lt;/code&gt; and &lt;code&gt;end&lt;/code&gt;. A range reserved in term 1 is valid only in term 1. The term-3 leader must commit a new reservation before it can serve requests again.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/ez-ts-oracle/5d18bdfd9b3d7222-time-server-reserved-lifecycle.png&#34; alt=&#34;Cross-term timeline for three nodes: node A is the leader in terms 1 and 3, but its old reservation cache is invalidated; next_timestamp succeeds only after refill installs a new range for term 3&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;take()&lt;/code&gt; enforces both cache invariants in one guard. A term mismatch rejects a range from an earlier leadership. &lt;code&gt;timestamp &amp;gt;= self.end&lt;/code&gt; rejects an exhausted range. Later requests succeed only after the background task installs a valid range.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;!=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;leader_term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;||&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;timestamp&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;end&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;return&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;None&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;With the lease check and the term check in place, the in-memory path is safe across leader changes.&lt;/p&gt;&#xA;&lt;h2 id=&#34;run-the-service&#34;&gt;Run the Service&lt;/h2&gt;&#xA;&lt;h3 id=&#34;start-a-three-node-cluster&#34;&gt;Start a Three-Node Cluster&lt;/h3&gt;&#xA;&lt;p&gt;Open three terminals and start one node in each. The first command creates the cluster; the other two use &lt;code&gt;--seed&lt;/code&gt; to find the first node and join it:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo run --example timestamp_oracle -- --addr 127.0.0.1:8090&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo run --example timestamp_oracle -- --addr 127.0.0.1:8091 --seed 127.0.0.1:8090&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo run --example timestamp_oracle -- --addr 127.0.0.1:8092 --seed 127.0.0.1:8090&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The cluster assigns node IDs automatically. When a node joins, EzRaft writes a blank log entry and uses its index as the new node&amp;rsquo;s ID. Other log entries may be committed between two joins, so IDs such as 0, 8, and 17 are expected. &lt;code&gt;/api/metrics&lt;/code&gt; now shows three voters with node 0 as the leader:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-json&#34; data-lang=&#34;json&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;0&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;current_term&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;state&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;Leader&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;current_leader&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;0&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;membership_config&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:{&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;membership&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:{&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;configs&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:[[&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;0&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;8&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;17&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;]],&lt;/span&gt; &lt;span style=&#34;color:#f6f8fa;background-color:#82071e&#34;&gt;...&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}}}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h3 id=&#34;request-timestamps&#34;&gt;Request Timestamps&lt;/h3&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl -X POST 127.0.0.1:8090/time&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Run the command three times. Each request returns an &lt;a href=&#34;https://www.rfc-editor.org/rfc/rfc3339&#34; title=&#34;RFC 3339: Date and Time on the Internet&#34;&gt;RFC 3339&lt;/a&gt; timestamp as a JSON string:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;&amp;#34;2026-08-11T14:43:16.966734Z&amp;#34;&#xA;&amp;#34;2026-08-11T14:43:16.973216Z&amp;#34;&#xA;&amp;#34;2026-08-11T14:43:16.979556Z&amp;#34;&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The six-to-seven-millisecond gaps come from starting a new &lt;code&gt;curl&lt;/code&gt; process for each request. The timestamp allocation itself stays entirely in the leader&amp;rsquo;s memory.&lt;/p&gt;&#xA;&lt;p&gt;A follower rejects &lt;code&gt;/time&lt;/code&gt; and names the current leader in its response:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;$ curl -s -X POST 127.0.0.1:8091/time&#xA;has to forward request to: Some(0), Some(BasicNode { addr: &amp;#34;127.0.0.1:8090&amp;#34; })   [HTTP 503]&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;stop-the-leader&#34;&gt;Stop the Leader&lt;/h3&gt;&#xA;&lt;p&gt;Press Ctrl-C in the terminal running node 0, then keep sending &lt;code&gt;next_timestamp&lt;/code&gt; requests to the two remaining nodes:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;14:43:28.998  curl 8090  -&amp;gt;  &amp;#34;2026-08-11T14:43:28.998630Z&amp;#34;                        [200]&#xA;14:43:29      kill node 0&#xA;14:43:29.3    curl 8091  -&amp;gt;  has to forward request to: Some(0), ...:8090          [503]&#xA;   ...        (for about five seconds, both 8091 and 8092 return the same 503)&#xA;14:43:34.8    curl 8091  -&amp;gt;  has to forward request to: Some(17), ...:8092         [503]&#xA;14:43:34.8    curl 8092  -&amp;gt;  no reserved timestamp is currently available          [503]&#xA;14:43:35.169  curl 8092  -&amp;gt;  &amp;#34;2026-08-11T14:43:35.169756Z&amp;#34;                        [200]&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The timeline shows two stages of recovery. At first the followers still point clients at node 0. Node 17 is then elected leader, but its first &lt;code&gt;Reserve&lt;/code&gt; entry has not committed, so its in-memory cache is still empty. As soon as that reservation commits, the service resumes.&lt;/p&gt;&#xA;&lt;p&gt;The timestamps stay monotonic. The first value from the new leader, &lt;code&gt;14:43:35.169756&lt;/code&gt;, is greater than the last value from the old leader, &lt;code&gt;14:43:28.998630&lt;/code&gt;. The unused interval between them is a harmless gap. &lt;code&gt;/api/metrics&lt;/code&gt; confirms that the cluster has advanced from term 1 to term 2:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-json&#34; data-lang=&#34;json&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;17&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;current_term&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;2&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;state&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;Leader&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;current_leader&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;17&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;8&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;current_term&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;2&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;state&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;Follower&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;current_leader&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;17&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;!-- No diagram: the timestamped output already shows the sequence before and after leader failover, so another illustration would be redundant. --&gt;&#xA;&lt;h2 id=&#34;what-raft-owns&#34;&gt;What Raft Owns&lt;/h2&gt;&#xA;&lt;p&gt;The consensus layer does not need to see every timestamp. It needs to see the ceiling, and it needs to say who may spend below that ceiling. Once those two facts are durable and current, &lt;code&gt;next_timestamp&lt;/code&gt; is a local increment.&lt;/p&gt;&#xA;&lt;p&gt;That split is the whole service. EzRaft handles replication, election, and failover. The application code persists the reservation boundary, tags each cached range with a term, and advances a cursor.&lt;/p&gt;&#xA;&lt;h2 id=&#34;links&#34;&gt;Links&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Example: &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs&#34; title=&#34;ezraft timestamp_oracle example&#34;&gt;timestamp_oracle.rs&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;EzRaft: &lt;a href=&#34;https://github.com/drmingdrmer/ezraft&#34; title=&#34;ezraft&#34;&gt;github.com/drmingdrmer/ezraft&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;crates.io: &lt;a href=&#34;https://crates.io/crates/ezraft&#34; title=&#34;ezraft on crates.io&#34;&gt;crates.io/crates/ezraft&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Documentation: &lt;a href=&#34;https://docs.rs/ezraft&#34; title=&#34;ezraft docs&#34;&gt;docs.rs/ezraft&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;OpenRaft: &lt;a href=&#34;https://github.com/databendlabs/openraft&#34; title=&#34;OpenRaft&#34;&gt;github.com/databendlabs/openraft&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Previous article: &lt;a href=&#34;https://blog.openacid.com/algo/ezraft/&#34; title=&#34;EzRaft: Build a Distributed KV Store in 100 Lines&#34;&gt;EzRaft: Build a Distributed KV Store in 100 Lines&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Reserve / Interval / TimeState / impl EzApp : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L66-L112&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L66-L112&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;EzRaft::linearizable : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/src/raft.rs#L343-L365&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/src/raft.rs#L343-L365&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;TimeService::leader_term : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L207-L210&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L207-L210&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;TimeService::next_timestamp : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L162-L168&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L162-L168&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;TimeService::refill : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L188-L205&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L188-L205&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;TimeService::run_reserver : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L170-L186&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L170-L186&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Reserved::take : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L124-L133&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs#L124-L133&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ezraft on crates.io : &lt;a href=&#34;https://crates.io/crates/ezraft&#34;&gt;https://crates.io/crates/ezraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;TiDB: Timestamp Oracle (TSO) : &lt;a href=&#34;https://docs.pingcap.com/tidb/stable/glossary/#timestamp-oracle-tso&#34;&gt;https://docs.pingcap.com/tidb/stable/glossary/#timestamp-oracle-tso&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ezraft docs : &lt;a href=&#34;https://docs.rs/ezraft&#34;&gt;https://docs.rs/ezraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;EzApp : &lt;a href=&#34;https://docs.rs/ezraft/latest/ezraft/app/trait.EzApp.html&#34;&gt;https://docs.rs/ezraft/latest/ezraft/app/trait.EzApp.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Raft::ensure_linearizable : &lt;a href=&#34;https://docs.rs/openraft/0.10.0-alpha.33/openraft/raft/struct.Raft.html#method.ensure_linearizable&#34;&gt;https://docs.rs/openraft/0.10.0-alpha.33/openraft/raft/struct.Raft.html#method.ensure_linearizable&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;OpenRaft ReadPolicy : &lt;a href=&#34;https://docs.rs/openraft/0.10.0-alpha.33/openraft/raft/enum.ReadPolicy.html&#34;&gt;https://docs.rs/openraft/0.10.0-alpha.33/openraft/raft/enum.ReadPolicy.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;EzRaft: Build a Distributed KV Store in 100 Lines : &lt;a href=&#34;https://blog.openacid.com/algo/ezraft/&#34;&gt;https://blog.openacid.com/algo/ezraft/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Linearizable Transactions in Distributed Systems : &lt;a href=&#34;https://blog.openacid.com/algo/linearizable/&#34;&gt;https://blog.openacid.com/algo/linearizable/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;How OpenRaft Optimizes ReadIndex : &lt;a href=&#34;https://blog.openacid.com/algo/openraft-read/&#34;&gt;https://blog.openacid.com/algo/openraft-read/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Quorum Reads and Writes with a Minority : &lt;a href=&#34;https://blog.openacid.com/algo/quorum/&#34;&gt;https://blog.openacid.com/algo/quorum/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Raft : &lt;a href=&#34;https://raft.github.io/&#34;&gt;https://raft.github.io/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Large-scale Incremental Processing Using Distributed Transactions and Notifications : &lt;a href=&#34;https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Peng.pdf&#34;&gt;https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Peng.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ezraft : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft&#34;&gt;https://github.com/drmingdrmer/ezraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ezraft timestamp_oracle example : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/examples/timestamp_oracle.rs&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;OpenRaft : &lt;a href=&#34;https://github.com/databendlabs/openraft&#34;&gt;https://github.com/databendlabs/openraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;RFC 3339: Date and Time on the Internet : &lt;a href=&#34;https://www.rfc-editor.org/rfc/rfc3339&#34;&gt;https://www.rfc-editor.org/rfc/rfc3339&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Rust : &lt;a href=&#34;https://www.rust-lang.org/&#34;&gt;https://www.rust-lang.org/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Multiversion concurrency control : &lt;a href=&#34;https://en.wikipedia.org/wiki/Multiversion_concurrency_control&#34;&gt;https://en.wikipedia.org/wiki/Multiversion_concurrency_control&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="algo"/>
    <category term="raft"/>
    <category term="consensus"/>
    <category term="distributed"/>
    <summary type="html">`timestamp_oracle` is a 200-line timestamp service on EzRaft. A background task reserves ranges through Raft; `next_timestamp` allocates from memory. Leader leases and term-scoped caches keep the sequence strictly increasing across failovers, with no Raft write on the request path.</summary>
  </entry>
  <entry>
    <title type="html">EzRaft: Build a Distributed KV Store in 100 Lines</title>
    <link href="https://blog.openacid.com/algo/ezraft/" rel="alternate" type="text/html" title="EzRaft: Build a Distributed KV Store in 100 Lines"/>
    <published>2026-08-01T00:00:00&#43;00:00</published>
    <updated>2026-08-01T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/algo/ezraft</id>
    <content type="html" xml:base="https://blog.openacid.com/algo/ezraft/">&lt;p&gt;&lt;img src=&#34;/post-res/ezraft/f45129e0aa8db968-ezraft-banner.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;h1 id=&#34;ezraft-build-a-distributed-kv-store-in-100-lines&#34;&gt;EzRaft: Build a Distributed KV Store in 100 Lines&lt;/h1&gt;&#xA;&lt;p&gt;For the past few years, I have spent most of my time working on &lt;a href=&#34;https://github.com/databendlabs/openraft&#34; title=&#34;openraft&#34;&gt;OpenRaft&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;OpenRaft is an open-source project that runs in production at companies around the world. To support such a wide range of applications, it has to be highly flexible. Log entries, log IDs, state machines, and storage are all expressed as &lt;a href=&#34;https://www.rust-lang.org/&#34; title=&#34;Rust&#34;&gt;Rust&lt;/a&gt; traits, so users can adapt them to their own needs.&lt;/p&gt;&#xA;&lt;p&gt;Today, OpenRaft&amp;rsquo;s API and performance are both fairly mature. The trade-off is that its model of the &lt;a href=&#34;https://raft.github.io/&#34; title=&#34;Raft&#34;&gt;Raft&lt;/a&gt; algorithm remains quite abstract.&lt;/p&gt;&#xA;&lt;h2 id=&#34;the-openraft-learning-curve&#34;&gt;The OpenRaft Learning Curve&lt;/h2&gt;&#xA;&lt;p&gt;That abstraction creates a steep learning curve. There are three main parts:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Understanding the Raft consensus algorithm:&#xA;Before you can build a distributed store on OpenRaft, you need to understand what Raft guarantees—and what it does not.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Understanding Rust:&#xA;OpenRaft is written in Rust. Reading the code and building an application requires some familiarity with Rust&amp;rsquo;s abstractions and generics.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Understanding OpenRaft&amp;rsquo;s own abstractions:&#xA;Abstraction makes software easier to use once you understand the model. It gives experienced users convenience and flexibility. But when you are learning the system from the bottom up, the abstraction itself becomes another thing you must first understand.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;This is a common problem with abstractions: before the whole system makes sense, you must understand how all of its abstract pieces fit together. Most people learn more naturally in the other direction—from concrete examples to general ideas.&lt;/p&gt;&#xA;&lt;p&gt;Give someone an abstraction and ask them to imagine a concrete example, and the idea can be hard to grasp. Give them a concrete example first, and the underlying pattern is much easier to see. OpenRaft&amp;rsquo;s abstractions make it a powerful, stable tool, but not necessarily an easy place to start learning.&lt;/p&gt;&#xA;&lt;h2 id=&#34;why-i-built-ezraft&#34;&gt;Why I Built EzRaft&lt;/h2&gt;&#xA;&lt;p&gt;As the maintainer of OpenRaft, I regularly hear from developers who are new to distributed systems, Rust, or OpenRaft itself. They want to build something of their own—perhaps a distributed storage system—but the amount they need to learn first can be overwhelming.&lt;/p&gt;&#xA;&lt;p&gt;I began to wonder: could I build a layer that lets people focus on their business logic first, without requiring them to master Raft, Rust, and OpenRaft up front?&lt;/p&gt;&#xA;&lt;p&gt;That question led to EzRaft.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&#34;https://github.com/drmingdrmer/ezraft&#34; title=&#34;ezraft&#34;&gt;EzRaft&lt;/a&gt; is a streamlined distributed storage library built on top of OpenRaft. It hides the details you do not need at the beginning, so you can concentrate on the application rather than the machinery underneath it.&lt;/p&gt;&#xA;&lt;p&gt;Here is a rough comparison:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;OpenRaft&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;EzRaft&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Traits to implement&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;7+ (&lt;code&gt;RaftLogStorage&lt;/code&gt;, &lt;code&gt;RaftStateMachine&lt;/code&gt;, &amp;hellip;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;2 (&lt;code&gt;EzApp&lt;/code&gt;, &lt;code&gt;EzStorage&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Methods to implement&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;21+&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;4&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Types to define&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;12 generic parameters&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;2 (&lt;code&gt;Request&lt;/code&gt;, &lt;code&gt;Response&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Network code&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;About 100 lines, written by the user&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Built in, 0 lines&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/ezraft/db3f610ca0318cb2-ezraft-layers.png&#34; alt=&#34;The layers of EzRaft and OpenRaft: users implement EzApp, EzRaft provides storage, networking, and service APIs, and OpenRaft handles consensus&#34;&gt;&lt;/p&gt;&#xA;&lt;h2 id=&#34;the-four-things-users-define&#34;&gt;The Four Things Users Define&lt;/h2&gt;&#xA;&lt;p&gt;EzRaft asks you to define just four things:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The shape of a request: &lt;code&gt;Request&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The shape of a response: &lt;code&gt;Response&lt;/code&gt;. Requests and responses represent your application&amp;rsquo;s operations, so only you can define them.&lt;/li&gt;&#xA;&lt;li&gt;How a request changes the state machine: &lt;code&gt;apply()&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;How to read from the state machine: &lt;code&gt;read()&lt;/code&gt;. You can think of the state machine as a key-value store and use a key to retrieve one part of it.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Together, these four pieces define the boundary of your application. EzRaft handles everything below that boundary: storage, network communication, service APIs, and more. When building directly on OpenRaft, you must provide these pieces yourself.&lt;/p&gt;&#xA;&lt;p&gt;With EzRaft, you can build the business logic first and explore the lower-level details later, when you actually need them.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;trait&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;EzApp: &lt;span style=&#34;color:#1f2328&#34;&gt;Serialize&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;+&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;DeserializeOwned&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;type&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Request&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;type&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Response&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;apply&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;req: &lt;span style=&#34;color:#1f2328&#34;&gt;Self&lt;/span&gt;::Request&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Self&lt;/span&gt;::Response&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;read&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;key: &lt;span style=&#34;color:#cf222e&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;str&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#6639ba&#34;&gt;Option&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;serde_json::Value&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;That is the complete application-facing API. Define those four pieces, add a few lines of boilerplate to connect them, and let EzRaft handle the rest.&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;EzApp&lt;/code&gt; value holds the application&amp;rsquo;s entire state. EzRaft persists and restores that state through serialization and deserialization, which means a snapshot is simply the serialized form of the whole application.&lt;/p&gt;&#xA;&lt;p&gt;This deliberate simplification makes EzRaft a good fit for applications whose state fits in memory. We will return to that constraint later.&lt;/p&gt;&#xA;&lt;h2 id=&#34;build-a-distributed-kv-store-in-three-steps&#34;&gt;Build a Distributed KV Store in Three Steps&lt;/h2&gt;&#xA;&lt;p&gt;Building our KV store takes three steps:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Implement &lt;code&gt;EzApp&lt;/code&gt; to define the business model.&lt;/li&gt;&#xA;&lt;li&gt;Assemble a service from the components EzRaft provides.&lt;/li&gt;&#xA;&lt;li&gt;Start three nodes and try the cluster for yourself.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;h2 id=&#34;step-one-define-the-application&#34;&gt;Step One: Define the Application&lt;/h2&gt;&#xA;&lt;p&gt;Let us start with the application itself. The service supports two write operations: set a key and delete a key.&lt;/p&gt;&#xA;&lt;p&gt;The complete application state lives in a KV map, backed here by a &lt;a href=&#34;https://doc.rust-lang.org/std/collections/struct.BTreeMap.html&#34; title=&#34;BTreeMap&#34;&gt;B-tree map&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;apply&lt;/code&gt; takes a set or delete request and applies it to the map.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;read&lt;/code&gt; looks up a value by key and returns it.&lt;/p&gt;&#xA;&lt;p&gt;That is all the application needs to do. EzRaft takes care of everything else.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;enum&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Request&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;Set&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;key: &lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;value: &lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;Delete&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;key: &lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;KvApp&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;data: &lt;span style=&#34;color:#1f2328&#34;&gt;BTreeMap&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;impl&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;EzApp&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;for&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;KvApp&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;type&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Request&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;Request&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;type&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Response&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Option&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;apply&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;req: &lt;span style=&#34;color:#1f2328&#34;&gt;Request&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#6639ba&#34;&gt;Option&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;match&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;req&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;Request::Set&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;key&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;value&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;data&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;insert&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;key&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;value&lt;span style=&#34;color:#1f2328&#34;&gt;),&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;Request::Delete&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;key&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;data&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;remove&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;key&lt;span style=&#34;color:#1f2328&#34;&gt;),&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;read&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;key: &lt;span style=&#34;color:#cf222e&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;str&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#6639ba&#34;&gt;Option&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;serde_json::Value&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;data&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;get&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;key&lt;span style=&#34;color:#1f2328&#34;&gt;).&lt;/span&gt;cloned&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;map&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;serde_json::Value::&lt;span style=&#34;color:#6639ba&#34;&gt;String&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Both &lt;code&gt;insert&lt;/code&gt; and &lt;code&gt;remove&lt;/code&gt; return the key&amp;rsquo;s previous value, so &lt;code&gt;apply&lt;/code&gt; returns it as well. Each write tells the caller exactly what it replaced or removed, without an extra read.&lt;/p&gt;&#xA;&lt;p&gt;To keep the example readable, I omitted the derive attributes. A real implementation needs them. &lt;code&gt;Request&lt;/code&gt; travels over the network and is written to the Raft log, so it must support serialization through &lt;a href=&#34;https://serde.rs/&#34; title=&#34;serde&#34;&gt;serde&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;OpenRaft also prints requests in its diagnostic logs, which requires &lt;code&gt;Debug&lt;/code&gt; and &lt;code&gt;Display&lt;/code&gt;. EzRaft may need to retain a copy while forwarding a request to the leader, so &lt;code&gt;Request&lt;/code&gt; also needs &lt;code&gt;Clone&lt;/code&gt;. Finally, &lt;code&gt;KvApp&lt;/code&gt; is the state itself, and serde turns that state into snapshots.&lt;/p&gt;&#xA;&lt;p&gt;The repository includes a complete, runnable version in &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/kvstore.rs&#34; title=&#34;ezraft kvstore example&#34;&gt;examples/kvstore.rs&lt;/a&gt;, with all derives and imports in place. You can run it directly with &lt;code&gt;cargo run&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;It differs from the code in this article in two small ways. First, it uses clap to parse &lt;code&gt;--addr&lt;/code&gt; and &lt;code&gt;--seed&lt;/code&gt; instead of positional arguments. Second, it returns a &lt;code&gt;Response&lt;/code&gt; struct rather than &lt;code&gt;Option&amp;lt;String&amp;gt;&lt;/code&gt;, so the HTTP response is &lt;code&gt;{&amp;quot;value&amp;quot;:&amp;quot;world&amp;quot;}&lt;/code&gt; instead of &lt;code&gt;&amp;quot;world&amp;quot;&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-two-assemble-a-working-service&#34;&gt;Step Two: Assemble a Working Service&lt;/h2&gt;&#xA;&lt;p&gt;Now we can turn the business logic into a working server. The server accepts two arguments at startup:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;This node&amp;rsquo;s address. Other EzRaft nodes use it for cluster communication, and clients use it to send requests.&lt;/li&gt;&#xA;&lt;li&gt;A seed address. A new node with uninitialized storage contacts the seed to join an existing cluster.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The first node does not need a seed. It creates a single-node cluster and can begin serving requests immediately.&lt;/p&gt;&#xA;&lt;p&gt;The second and third nodes use the first node as their seed. Each asks to join the existing cluster, and together they form a three-node cluster.&lt;/p&gt;&#xA;&lt;p&gt;EzRaft and OpenRaft handle the entire process: admitting each node, assigning its ID, and synchronizing its data.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/ezraft/1b91902c8e63f4b5-ezraft-cluster-formation.png&#34; alt=&#34;How three EzRaft nodes form a cluster: the first creates it, while two more join through the seed, catch up, and become voters&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;The server code is short: read the command-line arguments, then initialize storage.&lt;/p&gt;&#xA;&lt;p&gt;This example uses EzRaft&amp;rsquo;s built-in &lt;code&gt;FileStorage&lt;/code&gt;. It demonstrates every responsibility of a storage implementation, with clarity taking priority over performance.&lt;/p&gt;&#xA;&lt;p&gt;A production system should replace it with a proper storage backend. &lt;code&gt;FileStorage&lt;/code&gt; does not call fsync, so a power failure could lose a write that the cluster has already acknowledged to the client. That can &lt;a href=&#34;https://blog.openacid.com/algo/raft-io-order-complete-cn/&#34; title=&#34;I/O Ordering in Raft&#34;&gt;break Raft&amp;rsquo;s correctness guarantees&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Fortunately, &lt;code&gt;EzStorage&lt;/code&gt; has only three methods. The &lt;code&gt;FileStorage&lt;/code&gt; source provides a complete template for implementing them.&lt;/p&gt;&#xA;&lt;p&gt;After initializing storage, create the &lt;code&gt;KvApp&lt;/code&gt; instance and pass it, the storage backend, and a configuration to EzRaft. The server is then ready to run.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;#[tokio::main]&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;async&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;main&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#1f2328&#34;&gt;io&lt;/span&gt;::&lt;span style=&#34;color:#6639ba&#34;&gt;Result&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Usage: kvstore [addr] [seed-addr]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;addr&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;std::env::args&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;nth&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;).&lt;/span&gt;unwrap_or&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;127.0.0.1:8080&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;to_string&lt;span style=&#34;color:#1f2328&#34;&gt;());&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;seed&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;std::env::args&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;nth&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;2&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;storage&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;FileStorage::new&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;format!&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;./data/&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;addr&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;replace&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;:&amp;#39;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;-&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;))).&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;app&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;config&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;KvApp::default&lt;span style=&#34;color:#1f2328&#34;&gt;(),&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;EzConfig::default&lt;span style=&#34;color:#1f2328&#34;&gt;());&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;raft&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;match&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;seed&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Some&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;seed&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;EzRaft::join&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;addr&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;seed&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;app&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;storage&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;config&lt;span style=&#34;color:#1f2328&#34;&gt;).&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;None&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;EzRaft::create&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;addr&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;app&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;storage&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;config&lt;span style=&#34;color:#1f2328&#34;&gt;).&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;};&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;raft&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;serve&lt;span style=&#34;color:#1f2328&#34;&gt;().&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;await&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;create&lt;/code&gt; and &lt;code&gt;join&lt;/code&gt; are deliberately separate methods rather than a single method with an &lt;code&gt;Option&amp;lt;seed&amp;gt;&lt;/code&gt; parameter. Creating a cluster and joining an existing one are fundamentally different operations.&lt;/p&gt;&#xA;&lt;p&gt;Suppose a node is meant to join a cluster, but its seed is accidentally omitted. If it calls &lt;code&gt;create&lt;/code&gt;, it forms a second, independent single-node cluster—and the two clusters will never merge. Requiring the caller to choose explicitly is safer than letting an empty configuration value make that decision.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-three-start-a-three-node-cluster&#34;&gt;Step Three: Start a Three-Node Cluster&lt;/h2&gt;&#xA;&lt;p&gt;Open three terminals and start one node in each:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo run -- 127.0.0.1:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo run -- 127.0.0.1:8081 127.0.0.1:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo run -- 127.0.0.1:8082 127.0.0.1:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Open a fourth terminal for client requests.&lt;/p&gt;&#xA;&lt;p&gt;First, write a value. A write returns the key&amp;rsquo;s previous value. Because &lt;code&gt;hello&lt;/code&gt; does not exist yet, the response is &lt;code&gt;null&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl -X POST 127.0.0.1:8080/api/write &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -H &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;Content-Type: application/json&amp;#39;&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -d &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;{&amp;#34;Set&amp;#34;: {&amp;#34;key&amp;#34;: &amp;#34;hello&amp;#34;, &amp;#34;value&amp;#34;: &amp;#34;world&amp;#34;}}&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# null&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Now read the value from another node. The request goes to 8082, even though no client has written to that node directly:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;127.0.0.1:8082/api/read?key=hello&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# &amp;#34;world&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Every node holds a complete copy of the application state. A read can therefore go straight to local memory, without contacting the leader or running a round of consensus.&lt;/p&gt;&#xA;&lt;p&gt;This local read is fast, but it can be slightly stale. The node may not yet have received a write that the cluster just acknowledged. If you need a strictly &lt;a href=&#34;https://blog.openacid.com/algo/linearizable/&#34; title=&#34;Linearizable Transactions in Distributed Systems&#34;&gt;linearizable&lt;/a&gt; read, call &lt;code&gt;linearizable()&lt;/code&gt; first. It &lt;a href=&#34;https://blog.openacid.com/algo/openraft-read/&#34; title=&#34;How OpenRaft Optimizes ReadIndex&#34;&gt;confirms the leader&amp;rsquo;s identity&lt;/a&gt; and waits for the local state to catch up; the following &lt;code&gt;read()&lt;/code&gt; will then see every earlier acknowledged write.&lt;/p&gt;&#xA;&lt;h2 id=&#34;send-writes-to-any-node&#34;&gt;Send Writes to Any Node&lt;/h2&gt;&#xA;&lt;p&gt;In Raft, only the leader can coordinate a client write, but EzRaft hides that restriction from the client. Every node exposes the same write endpoint. A follower forwards the request to the leader, waits for it to commit, and then returns the result to the caller.&lt;/p&gt;&#xA;&lt;p&gt;That is why the following request succeeds when sent to 8081. The response, &lt;code&gt;&amp;quot;world&amp;quot;&lt;/code&gt;, is the old value replaced by the write:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl -X POST 127.0.0.1:8081/api/write &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -H &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;Content-Type: application/json&amp;#39;&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -d &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;{&amp;#34;Set&amp;#34;: {&amp;#34;key&amp;#34;: &amp;#34;hello&amp;#34;, &amp;#34;value&amp;#34;: &amp;#34;again&amp;#34;}}&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# &amp;#34;world&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The client never needs to discover or track the leader; it can connect to any node. A delete works the same way and returns the value it removed:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl -X POST 127.0.0.1:8080/api/write &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -H &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;Content-Type: application/json&amp;#39;&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -d &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;{&amp;#34;Delete&amp;#34;: {&amp;#34;key&amp;#34;: &amp;#34;hello&amp;#34;}}&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# &amp;#34;again&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;After the deletion, the key no longer exists, so another read returns HTTP 404:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;127.0.0.1:8080/api/read?key=hello&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# no value for key &amp;#34;hello&amp;#34;     [HTTP 404]&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;test-a-node-failure&#34;&gt;Test a Node Failure&lt;/h2&gt;&#xA;&lt;p&gt;So far, a single HashMap on one machine could provide the same behavior. The reason to run three nodes is fault tolerance: the service keeps working when one node fails.&lt;/p&gt;&#xA;&lt;p&gt;Write a key, then stop the leader—the first node, running on 8080:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl -X POST 127.0.0.1:8080/api/write &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -H &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;Content-Type: application/json&amp;#39;&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -d &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;{&amp;#34;Set&amp;#34;: {&amp;#34;key&amp;#34;: &amp;#34;k&amp;#34;, &amp;#34;value&amp;#34;: &amp;#34;before-failover&amp;#34;}}&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# null&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# Press Ctrl-C in the first terminal, or kill the process.&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The two remaining nodes still form a &lt;a href=&#34;https://blog.openacid.com/algo/quorum/&#34; title=&#34;Quorum Reads and Writes with a Minority&#34;&gt;majority&lt;/a&gt;, so they elect a new leader within a few heartbeats. The data written before the failure remains available, and the cluster continues to accept writes:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;127.0.0.1:8081/api/read?key=k&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# &amp;#34;before-failover&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl -X POST 127.0.0.1:8081/api/write &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -H &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;Content-Type: application/json&amp;#39;&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;\&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    -d &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;{&amp;#34;Set&amp;#34;: {&amp;#34;key&amp;#34;: &amp;#34;k&amp;#34;, &amp;#34;value&amp;#34;: &amp;#34;after-failover&amp;#34;}}&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# &amp;#34;before-failover&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl 127.0.0.1:8081/api/metrics &lt;span style=&#34;color:#1f2328&#34;&gt;|&lt;/span&gt; jq -c &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;{id, state, current_leader}&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# {&amp;#34;id&amp;#34;:2,&amp;#34;state&amp;#34;:&amp;#34;Follower&amp;#34;,&amp;#34;current_leader&amp;#34;:6}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Node 6 is now the leader. The cluster completed the failover without any manual intervention.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/ezraft/85aed0f352257dfd-ezraft-failover.png&#34; alt=&#34;Failover in a three-node cluster: after the original leader fails, the other two nodes form a majority and elect a new leader&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Now restart the failed node with exactly the same command:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo run -- 127.0.0.1:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Notice that the command does not include a seed. The node ID is already stored in &lt;code&gt;./data/127.0.0.1-8080/&lt;/code&gt;, so the node loads it during startup instead of joining again.&lt;/p&gt;&#xA;&lt;p&gt;The original seed is no longer needed, even if it has since left the cluster. Once the restarted node is running, it automatically catches up on the write it missed:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;127.0.0.1:8080/api/read?key=k&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;# &amp;#34;after-failover&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;A three-node cluster tolerates one failure; a five-node cluster tolerates two. Even-numbered clusters provide no additional fault tolerance: four nodes, like three, can still tolerate only one failure.&lt;/p&gt;&#xA;&lt;h2 id=&#34;cluster-state&#34;&gt;Cluster State&lt;/h2&gt;&#xA;&lt;p&gt;EzRaft includes a default endpoint for inspecting the current state of a node and its cluster:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#39;127.0.0.1:8080/api/metrics&amp;#39;&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;|&lt;/span&gt; jq&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The response comes directly from OpenRaft and exposes its full set of metrics:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-json&#34; data-lang=&#34;json&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;running_state&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;Ok&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#cf222e&#34;&gt;null&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;0&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;current_term&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;vote&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;term&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;voted_for&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;0&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;committed&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#cf222e&#34;&gt;true&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;last_log_index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;10&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;local_committed&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;10&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;cluster_committed&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;10&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;committed&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;10&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;last_applied&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;10&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;snapshot&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#cf222e&#34;&gt;null&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;purged&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#cf222e&#34;&gt;null&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;state&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;Leader&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;current_leader&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;0&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;millis_since_quorum_ack&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;3&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;last_quorum_acked&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1785595777894002000&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;membership_config&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;log_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;9&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;membership&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;configs&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;[&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;[&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;0&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;2&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;6&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;]&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;],&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;nodes&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;0&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;addr&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;127.0.0.1:8080&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;2&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;addr&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;127.0.0.1:8081&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;6&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;addr&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;127.0.0.1:8082&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;committed_membership_config&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;log_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;9&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;membership&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;configs&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;[&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;[&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;0&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;2&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;6&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;]&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;],&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;nodes&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;0&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;addr&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;127.0.0.1:8080&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;2&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;addr&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;127.0.0.1:8081&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;6&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;addr&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;127.0.0.1:8082&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      &lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;heartbeat&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;0&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1785595777894002750&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;2&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1785595777894002209&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;6&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1785595777894002125&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;replication&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;0&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;10&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;2&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;10&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;},&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;6&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;leader_id&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;:&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;10&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The three node IDs are 0, 2, and 6—not 0, 1, and 2. That is because the cluster assigns node IDs instead of reading them from configuration.&lt;/p&gt;&#xA;&lt;p&gt;When a node joins, the leader first writes a blank log entry, then uses that entry&amp;rsquo;s index as the new node&amp;rsquo;s ID. Log indexes are unique, so the resulting node IDs are unique as well. They are not consecutive because other entries are written between two joins: one adds the new node as a learner, and a membership change promotes it to a voter. The gaps between IDs are a visible trace of those steps.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/ezraft/57254dc9bd5dea86-ezraft-node-id-log.png&#34; alt=&#34;How node IDs 0, 2, and 6 relate to Raft log indexes written by join, add learner, and membership change operations&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;A new node first joins as a learner. It receives log entries, but it does not vote and does not count toward the cluster&amp;rsquo;s majority. Once it has caught up, EzRaft automatically promotes it to a voter.&lt;/p&gt;&#xA;&lt;p&gt;From the user&amp;rsquo;s perspective, joining the node is enough. EzRaft handles synchronization and promotion, so the node begins contributing to fault tolerance automatically.&lt;/p&gt;&#xA;&lt;h2 id=&#34;when-to-use-ezraft&#34;&gt;When to Use EzRaft&lt;/h2&gt;&#xA;&lt;p&gt;EzRaft is designed for applications whose state fits in memory but must never be lost or allowed to diverge across nodes. Configuration services, service registries, discovery systems, metadata stores, distributed locks, and task schedulers all fit this pattern.&lt;/p&gt;&#xA;&lt;p&gt;The data is usually small, but it is critical: losing it or allowing replicas to disagree can disrupt every system built on top of it. &lt;a href=&#34;https://zookeeper.apache.org/&#34; title=&#34;Apache ZooKeeper&#34;&gt;ZooKeeper&lt;/a&gt; and &lt;a href=&#34;https://etcd.io/&#34; title=&#34;etcd&#34;&gt;etcd&lt;/a&gt; serve this same class of workload.&lt;/p&gt;&#xA;&lt;p&gt;This focus follows directly from EzRaft&amp;rsquo;s snapshot model. A snapshot is the serialized form of the complete &lt;code&gt;EzApp&lt;/code&gt;, and installing one simply deserializes that state back into the application.&lt;/p&gt;&#xA;&lt;p&gt;As a result, snapshot support shrinks from &amp;ldquo;implement an entire snapshot mechanism&amp;rdquo; to &amp;ldquo;add one derive annotation to a struct.&amp;rdquo; Applications whose state fits in memory get the full benefit of that simplification.&lt;/p&gt;&#xA;&lt;p&gt;EzRaft also works well for read-heavy workloads with relatively few writes. Every node holds a complete copy of the state, and reads go straight to local memory without contacting the leader or running consensus. Clients can read from any node, so adding nodes increases read throughput.&lt;/p&gt;&#xA;&lt;p&gt;EzRaft is also a practical way to see Raft working end to end. The roughly one hundred lines above already exercise leader election, log replication, membership changes, snapshots, and failover.&lt;/p&gt;&#xA;&lt;p&gt;That makes the example easy to experiment with. Stop a node and watch the cluster elect a new leader. Follow the log index in the metrics. Change the heartbeat interval in &lt;code&gt;EzConfig&lt;/code&gt; and see how it affects election time. Once these ideas are concrete, moving down a layer to &lt;a href=&#34;https://github.com/databendlabs/openraft&#34; title=&#34;openraft&#34;&gt;OpenRaft&lt;/a&gt; is much easier than starting with a set of trait definitions.&lt;/p&gt;&#xA;&lt;p&gt;To turn this example into a production system, you need to replace two pieces. The first is &lt;code&gt;FileStorage&lt;/code&gt;, which should become a production-grade storage backend.&lt;/p&gt;&#xA;&lt;p&gt;The second is the service API. The built-in HTTP endpoints are designed for the example and provide neither authentication nor encryption. Use &lt;code&gt;EzServer&lt;/code&gt; as a reference, define routes that fit your application, and call &lt;code&gt;EzRaft::write&lt;/code&gt; and &lt;code&gt;EzRaft::read&lt;/code&gt; underneath. The rest of the stack can remain unchanged.&lt;/p&gt;&#xA;&lt;h2 id=&#34;links&#34;&gt;Links&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Code: &lt;a href=&#34;https://github.com/drmingdrmer/ezraft&#34; title=&#34;ezraft&#34;&gt;github.com/drmingdrmer/ezraft&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;crates.io: &lt;a href=&#34;https://crates.io/crates/ezraft&#34; title=&#34;ezraft on crates.io&#34;&gt;crates.io/crates/ezraft&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Documentation: &lt;a href=&#34;https://docs.rs/ezraft&#34; title=&#34;ezraft docs&#34;&gt;docs.rs/ezraft&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;OpenRaft: &lt;a href=&#34;https://github.com/databendlabs/openraft&#34; title=&#34;openraft&#34;&gt;github.com/databendlabs/openraft&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ezraft on crates.io : &lt;a href=&#34;https://crates.io/crates/ezraft&#34;&gt;https://crates.io/crates/ezraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ezraft docs : &lt;a href=&#34;https://docs.rs/ezraft&#34;&gt;https://docs.rs/ezraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;etcd : &lt;a href=&#34;https://etcd.io/&#34;&gt;https://etcd.io/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Linearizable Transactions in Distributed Systems : &lt;a href=&#34;https://blog.openacid.com/algo/linearizable/&#34;&gt;https://blog.openacid.com/algo/linearizable/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;How OpenRaft Optimizes ReadIndex : &lt;a href=&#34;https://blog.openacid.com/algo/openraft-read/&#34;&gt;https://blog.openacid.com/algo/openraft-read/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Quorum Reads and Writes with a Minority : &lt;a href=&#34;https://blog.openacid.com/algo/quorum/&#34;&gt;https://blog.openacid.com/algo/quorum/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;I/O Ordering in Raft : &lt;a href=&#34;https://blog.openacid.com/algo/raft-io-order-complete-cn/&#34;&gt;https://blog.openacid.com/algo/raft-io-order-complete-cn/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Raft : &lt;a href=&#34;https://raft.github.io/&#34;&gt;https://raft.github.io/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ezraft : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft&#34;&gt;https://github.com/drmingdrmer/ezraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;ezraft kvstore example : &lt;a href=&#34;https://github.com/drmingdrmer/ezraft/blob/main/examples/kvstore.rs&#34;&gt;https://github.com/drmingdrmer/ezraft/blob/main/examples/kvstore.rs&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;openraft : &lt;a href=&#34;https://github.com/databendlabs/openraft&#34;&gt;https://github.com/databendlabs/openraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;BTreeMap : &lt;a href=&#34;https://doc.rust-lang.org/std/collections/struct.BTreeMap.html&#34;&gt;https://doc.rust-lang.org/std/collections/struct.BTreeMap.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Rust : &lt;a href=&#34;https://www.rust-lang.org/&#34;&gt;https://www.rust-lang.org/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;serde : &lt;a href=&#34;https://serde.rs/&#34;&gt;https://serde.rs/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Apache ZooKeeper : &lt;a href=&#34;https://zookeeper.apache.org/&#34;&gt;https://zookeeper.apache.org/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="algo"/>
    <category term="raft"/>
    <category term="consensus"/>
    <category term="distributed"/>
    <summary type="html">EzRaft builds on OpenRaft and hides the storage, networking, and service layers. Define just four things—Request, Response, apply, and read—and you have a fault-tolerant, three-node distributed KV store.</summary>
  </entry>
  <entry>
    <title type="html">Do Not Read: Raf Is a Useless and Valueless Failed Experiment</title>
    <link href="https://blog.openacid.com/algo/raf-without-term/" rel="alternate" type="text/html" title="Do Not Read: Raf Is a Useless and Valueless Failed Experiment"/>
    <published>2026-05-11T00:00:00&#43;00:00</published>
    <updated>2026-05-11T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/algo/raf-without-term</id>
    <content type="html" xml:base="https://blog.openacid.com/algo/raf-without-term/">&lt;p&gt;&lt;img src=&#34;/post-res/raf-without-term/a9a9ca7fa5a1b329-raf-banner-small.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Summary: &lt;code&gt;raf: Raft without [T]erm&lt;/code&gt; is an experimental Raft variant. It does not persist &lt;code&gt;currentTerm&lt;/code&gt; as a separate piece of state. Instead, a candidate reserves a log index when it starts an election, and that index becomes the leader term. This does not remove Raft&amp;rsquo;s logical time model. It only changes where the term is derived from in storage.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Declaration: This approach is just the same as saving terms in the separate first extra slot in the terms array. So it actually still stores the term and has no value at all. Please do not read this as a useful design; it is only a failed personal experiment.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Note: The idea in this article came from Zhang Yanpo. The code was implemented by Zhang Yanpo by hand. This article was drafted and refined with Codex.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Repository: &lt;a href=&#34;https://github.com/drmingdrmer/raf/tree/v0.1.1&#34; title=&#34;raf&#34;&gt;raf&lt;/a&gt; (&lt;code&gt;v0.1.1&lt;/code&gt;).&lt;/p&gt;&#xA;&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;&#xA;&lt;p&gt;I have seen a few interesting proposals that try to remove the term from Raft. The idea is appealing: if a consensus protocol can maintain one less piece of persistent state, perhaps both the model and the implementation become simpler. But Raft&amp;rsquo;s term is not just a counter. It represents logical time, lets nodes distinguish old leaders from new ones, and participates in Raft&amp;rsquo;s commit safety rule. So the real question is not whether we can simply delete the term. The useful question is whether we can express it in a different way.&lt;/p&gt;&#xA;&lt;p&gt;The name &lt;code&gt;raf&lt;/code&gt; comes from &lt;code&gt;Raft without [T]erm&lt;/code&gt;. Here, &amp;ldquo;without term&amp;rdquo; does not mean the protocol has no term at all. It means &lt;code&gt;currentTerm&lt;/code&gt; is no longer persisted as an independent field. The project turns this idea into a small but serious implementation: avoid storing &lt;code&gt;currentTerm&lt;/code&gt; separately, while still giving every part of Raft that needs a term a reliable source of logical time.&lt;/p&gt;&#xA;&lt;p&gt;This article explains that core idea. The term still exists as a concept. Logs are still compared by &lt;code&gt;(term, index)&lt;/code&gt;. What changes is the source of the term: it no longer comes from a separately incremented persistent counter; it comes from a log index reserved by an election. The goal of this implementation is not to prove that it is a drop-in replacement for standard Raft in every engineering detail. The goal is to see whether this storage representation preserves the most important safety intuition behind Raft.&lt;/p&gt;&#xA;&lt;p&gt;We will walk through the storage model, election, replication, commit, and the three-node example in the repository. I assume the reader is already familiar with the basic Raft flow: leader election, AppendEntries, quorum commit, and log ids in the form &lt;code&gt;(term, index)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;why-term-cannot-disappear&#34;&gt;Why Term Cannot Disappear&lt;/h2&gt;&#xA;&lt;p&gt;In a consensus algorithm, the log records events that have been chosen or are being proposed. The term tells us which logical time those events belong to.&lt;/p&gt;&#xA;&lt;p&gt;Standard Raft uses the term for several jobs:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Leader election advances the term before choosing a leader. The term gives leaders an ordering.&lt;/li&gt;&#xA;&lt;li&gt;Because of that, log freshness is compared by term first, then by index.&lt;/li&gt;&#xA;&lt;li&gt;A leader may directly commit only entries from its own term.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;This role is similar to the ballot number in Paxos. It lets the system decide which history is newer and which candidate is eligible to become leader, even when a node does not know every other node&amp;rsquo;s complete log.&lt;/p&gt;&#xA;&lt;p&gt;In short: &lt;strong&gt;a log index is a local event position; a term is the logical time used to compare histories across nodes.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;raf&lt;/code&gt; keeps the concept of term, but removes its separate storage.&lt;/p&gt;&#xA;&lt;h2 id=&#34;core-idea&#34;&gt;Core Idea&lt;/h2&gt;&#xA;&lt;p&gt;Standard Raft usually persists state shaped roughly like this:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;StandardRaftStorage&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;current_term: &lt;span style=&#34;color:#1f2328&#34;&gt;Term&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;voted_for: &lt;span style=&#34;color:#6639ba&#34;&gt;Option&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;NodeId&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;log: &lt;span style=&#34;color:#6639ba&#34;&gt;Vec&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;LogEntry&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;LogEntry&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;term: &lt;span style=&#34;color:#1f2328&#34;&gt;Term&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;cmd: &lt;span style=&#34;color:#1f2328&#34;&gt;Cmd&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;raf&lt;/code&gt; represents the persistent state as two &lt;code&gt;Vec&lt;/code&gt;s aligned by log index:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;RafStorage&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;terms: &lt;span style=&#34;color:#6639ba&#34;&gt;Vec&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;Term&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;cmds: &lt;span style=&#34;color:#6639ba&#34;&gt;Vec&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;Cmd&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Here:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;terms[i]&lt;/code&gt; is the leader term of the log entry at index &lt;code&gt;i&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;cmds[i]&lt;/code&gt; is the application command of the log entry at index &lt;code&gt;i&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;log_id(i)&lt;/code&gt; is still &lt;code&gt;(terms[i], i)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The following is one possible storage state. &lt;code&gt;ø&lt;/code&gt; means empty command. &lt;code&gt;cmds&lt;/code&gt; only reaches index &lt;code&gt;7&lt;/code&gt;, so indexes &lt;code&gt;8&lt;/code&gt; and &lt;code&gt;9&lt;/code&gt; are not complete log entries yet.&lt;/p&gt;&#xA;&lt;!-- [ASCII source](assets/storage-layout.txt) --&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raf-without-term/10e4df1ad2741f72-storage-layout.png&#34; alt=&#34;Storage layout&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Index by index, this state means:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Index &lt;code&gt;0&lt;/code&gt;: the fixed default entry. &lt;code&gt;terms[0] = 0&lt;/code&gt;, and &lt;code&gt;cmds[0]&lt;/code&gt; is an empty command.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;1&lt;/code&gt;: a successful election for term &lt;code&gt;1&lt;/code&gt;. The leader reserved index &lt;code&gt;1&lt;/code&gt; when it was elected and wrote its first empty command there.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;2&lt;/code&gt;: a successful election for term &lt;code&gt;2&lt;/code&gt;. The new leader reserved index &lt;code&gt;2&lt;/code&gt;; its first log entry is also an empty command.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;3&lt;/code&gt;: a user log entry &lt;code&gt;C3&lt;/code&gt; written by the leader of term &lt;code&gt;2&lt;/code&gt;, so &lt;code&gt;terms[3] = 2&lt;/code&gt; and &lt;code&gt;cmds[3] = C3&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;4&lt;/code&gt;: this position used to be an election attempt for term &lt;code&gt;4&lt;/code&gt;, but it did not produce an established leader. Later, when the leader of term &lt;code&gt;6&lt;/code&gt; was established, this position was filled as an empty command owned by term &lt;code&gt;6&lt;/code&gt;, so now &lt;code&gt;terms[4] = 6&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;5&lt;/code&gt;: this position used to be another failed election attempt for term &lt;code&gt;5&lt;/code&gt;. It was also later filled as an empty command owned by term &lt;code&gt;6&lt;/code&gt;, so now &lt;code&gt;terms[5] = 6&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;6&lt;/code&gt;: a successful election for term &lt;code&gt;6&lt;/code&gt;. The leader reserved index &lt;code&gt;6&lt;/code&gt; when it was elected, and wrote its first empty command there.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;7&lt;/code&gt;: a user log entry &lt;code&gt;C7&lt;/code&gt; written by the leader of term &lt;code&gt;6&lt;/code&gt;, so &lt;code&gt;terms[7] = 6&lt;/code&gt; and &lt;code&gt;cmds[7] = C7&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;8&lt;/code&gt;: a new election attempt for term &lt;code&gt;8&lt;/code&gt;. So far we have only seen &lt;code&gt;terms[8] = 8&lt;/code&gt;; there is no command for this index yet, so it is not a complete log entry.&lt;/li&gt;&#xA;&lt;li&gt;Index &lt;code&gt;9&lt;/code&gt;: another election attempt for term &lt;code&gt;9&lt;/code&gt;. Like index &lt;code&gt;8&lt;/code&gt;, it currently has only a term record and no command. From this state alone, we cannot tell whether it will eventually become a leader.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;em&gt;Standard Raft persists a separate &lt;code&gt;current_term&lt;/code&gt; and an array of &lt;code&gt;(term, command)&lt;/code&gt; log entries. &lt;code&gt;raf&lt;/code&gt; is similar, but splits the term and command at each index into two aligned &lt;code&gt;Vec&lt;/code&gt;s.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;h2 id=&#34;storage-model&#34;&gt;Storage Model&lt;/h2&gt;&#xA;&lt;p&gt;Index &lt;code&gt;0&lt;/code&gt; is a fixed default entry. This keeps the types simple and avoids using &lt;code&gt;Option&lt;/code&gt; for the initial position:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;terms[0] = 0&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cmds[0]  = empty&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;When both &lt;code&gt;Vec&lt;/code&gt;s have a value at the same index, that index is a complete log entry. Otherwise, the index has a term but no command. That state represents an election in progress: the term has been observed, but no log entry has been written at that position yet.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;log[index] = (terms[index], cmds[index])&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;log_id     = (terms[index], index)&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;em&gt;&lt;code&gt;terms&lt;/code&gt; and &lt;code&gt;cmds&lt;/code&gt; share the same log index. During election, &lt;code&gt;terms&lt;/code&gt; may be ahead of &lt;code&gt;cmds&lt;/code&gt;.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;During an election, &lt;code&gt;terms&lt;/code&gt; can temporarily be longer than &lt;code&gt;cmds&lt;/code&gt;. A candidate first reserves an index as its term. Multiple failed elections may reserve multiple indexes. Only after some candidate becomes an established leader are the positions with missing commands filled with empty commands.&lt;/p&gt;&#xA;&lt;p&gt;So this implementation maintains a few basic facts:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;cmds&lt;/code&gt; should never be longer than &lt;code&gt;terms&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;When an election first reserves a term slot, &lt;code&gt;terms[term] = term&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;When an established leader fills positions that still miss commands, those positions are rewritten to the current &lt;code&gt;leader.term&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Therefore &lt;code&gt;terms[i] &amp;lt;= i&lt;/code&gt; is not a protocol invariant. A backfilled empty entry may have &lt;code&gt;terms[i] &amp;gt; i&lt;/code&gt;. That is not a problem by itself; standard Raft terms can also be greater than log indexes. The important property is different: every complete log entry must carry the term of an established leader, except for the fixed default entry at index &lt;code&gt;0&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;why-split-terms-and-cmds&#34;&gt;Why Split &lt;code&gt;terms&lt;/code&gt; and &lt;code&gt;cmds&lt;/code&gt;&lt;/h2&gt;&#xA;&lt;p&gt;This implementation separates leader terms and application commands into two &lt;code&gt;Vec&lt;/code&gt;s. Raft&amp;rsquo;s protocol semantics do not require this layout. It is mainly a storage design choice that creates room for cleaner optimization.&lt;/p&gt;&#xA;&lt;p&gt;For example, a leader may write many consecutive log entries during one term, and all of those entries have the same term. A storage engine could compress long runs of identical terms into a compact representation, while storing commands according to application needs. Once the two streams are separated, term compression, command persistence, and payload encoding can evolve independently.&lt;/p&gt;&#xA;&lt;p&gt;This is the experimental value of the project. It expresses the relationship between &amp;ldquo;logical time&amp;rdquo; and &amp;ldquo;log position&amp;rdquo; in Raft more directly, then asks whether that expression can simplify persistent state.&lt;/p&gt;&#xA;&lt;h2 id=&#34;starting-an-election&#34;&gt;Starting an Election&lt;/h2&gt;&#xA;&lt;p&gt;When a candidate starts an election, it uses the next index of the &lt;code&gt;terms&lt;/code&gt; array as the new term:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;len&lt;span style=&#34;color:#1f2328&#34;&gt;();&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;push&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;term&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;In this code:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The candidate declares that it wants to use &lt;code&gt;term&lt;/code&gt; as its leader term.&lt;/li&gt;&#xA;&lt;li&gt;The local persistent state records that this index has been reserved by an election.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The candidate then sends &lt;code&gt;RequestVote&lt;/code&gt;. This part is the same as standard Raft. The request carries two important pieces of information:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;term&lt;/code&gt;: the leader term the candidate wants to use.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;last_log_id&lt;/code&gt;: the &lt;code&gt;(term, index)&lt;/code&gt; of the candidate&amp;rsquo;s last complete log entry.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;code&gt;last_log_id&lt;/code&gt; is computed from the last index in &lt;code&gt;cmds&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;last_log_index&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;cmds&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;len&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;-&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;last_log_id&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;[&lt;/span&gt;last_log_index&lt;span style=&#34;color:#1f2328&#34;&gt;],&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;last_log_index&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The meaning of &lt;code&gt;last_log_id&lt;/code&gt; is the same as in standard Raft: the voter uses it to check whether the candidate&amp;rsquo;s log is at least as up to date as its own. Notice the separation here. The new election term comes from &lt;code&gt;terms.len()&lt;/code&gt;, while &lt;code&gt;last_log_id&lt;/code&gt; comes from &lt;code&gt;cmds.len() - 1&lt;/code&gt;. These can point to different indexes.&lt;/p&gt;&#xA;&lt;p&gt;In the following example, the complete log currently reaches only index &lt;code&gt;3&lt;/code&gt;, so &lt;code&gt;last_log_id = (2, 3)&lt;/code&gt;. When the candidate starts a new election, it uses &lt;code&gt;terms.len() = 4&lt;/code&gt; as the new term and first writes index &lt;code&gt;4&lt;/code&gt; into &lt;code&gt;terms&lt;/code&gt;. At this point, &lt;code&gt;cmds&lt;/code&gt; still reaches only index &lt;code&gt;3&lt;/code&gt;, because this candidate has not become an established leader yet.&lt;/p&gt;&#xA;&lt;!-- [ASCII source](assets/leader-election-term4.txt) --&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raf-without-term/289a45a69ad1dd8d-leader-election-term4.png&#34; alt=&#34;Leader election term=4&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;If the election for term &lt;code&gt;4&lt;/code&gt; does not reach a quorum, it leaves only an observed term index in &lt;code&gt;terms&lt;/code&gt;; it does not create a new command. The next election will use &lt;code&gt;terms.len()&lt;/code&gt; again, which is now term &lt;code&gt;5&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;!-- [ASCII source](assets/leader-election-term5.txt) --&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raf-without-term/3863b4a533d8cdf5-leader-election-term5.png&#34; alt=&#34;Leader election retry term=5&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;At this point &lt;code&gt;last_log_id&lt;/code&gt; is still &lt;code&gt;(2, 3)&lt;/code&gt;, because &lt;code&gt;cmds&lt;/code&gt; has not moved beyond index &lt;code&gt;3&lt;/code&gt;. What changed is the candidate term: it advanced from &lt;code&gt;4&lt;/code&gt; to &lt;code&gt;5&lt;/code&gt;. Only after an election succeeds and a leader is established does the system rewrite the positions with missing commands to this leader&amp;rsquo;s term and fill &lt;code&gt;cmds&lt;/code&gt; with empty commands.&lt;/p&gt;&#xA;&lt;h2 id=&#34;how-a-voter-handles-requestvote&#34;&gt;How a Voter Handles RequestVote&lt;/h2&gt;&#xA;&lt;p&gt;When a voter receives &lt;code&gt;RequestVote&lt;/code&gt;, it checks three things:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Whether the requested term is greater than the last term observed locally.&lt;/li&gt;&#xA;&lt;li&gt;Whether the requested term slot does not already exist locally.&lt;/li&gt;&#xA;&lt;li&gt;Whether the candidate&amp;rsquo;s log, represented by &lt;code&gt;last_log_id&lt;/code&gt;, is fresh enough. It must not be older than the voter&amp;rsquo;s own log.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Aside from the term check, the rest of the logic is standard Raft. The subtle difference is that the term is no longer stored independently, so term freshness is expressed through &lt;code&gt;terms&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The core condition looks like this:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;local_last_log_index&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;cmds&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;len&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;-&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;local_last_log_id&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;[&lt;/span&gt;local_last_log_index&lt;span style=&#34;color:#1f2328&#34;&gt;],&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;local_last_log_index&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;local_last_term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;[&lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;len&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;-&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;];&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;can_vote&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;req&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;local_last_term&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;req&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;len&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;req&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;last_log_id&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;local_last_log_id&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The next diagram sends the same &lt;code&gt;RequestVote { term: 5, last_log_id: (2, 3) }&lt;/code&gt; to voters with three different local states. The candidate&amp;rsquo;s own state is at the top. The three branches below it show how each voter makes its decision.&lt;/p&gt;&#xA;&lt;!-- [ASCII source](assets/request-vote.txt) --&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raf-without-term/d310c7c93bee8cbb-request-vote.png&#34; alt=&#34;RequestVote scenarios&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;The three outcomes are:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;granted&lt;/code&gt;: the voter&amp;rsquo;s &lt;code&gt;terms&lt;/code&gt; reaches only index &lt;code&gt;3&lt;/code&gt;, and &lt;code&gt;cmds&lt;/code&gt; also reaches only index &lt;code&gt;3&lt;/code&gt;. Therefore &lt;code&gt;req.term = 5&lt;/code&gt; is greater than the last locally observed term, names a term slot that has not appeared locally, and &lt;code&gt;req.last_log_id = (2, 3)&lt;/code&gt; is not behind the voter. The vote can be granted.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;rejected: term=7&lt;/code&gt;: the voter has already observed a later term &lt;code&gt;7&lt;/code&gt;. Since &lt;code&gt;req.term = 5&lt;/code&gt; is not greater than the last locally observed term, the candidate&amp;rsquo;s requested term is stale from this voter&amp;rsquo;s perspective, so the vote is rejected.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;rejected: last log id = (4,4)&lt;/code&gt;: the voter&amp;rsquo;s last complete log entry is &lt;code&gt;(4, 4)&lt;/code&gt;, which is newer than the candidate&amp;rsquo;s &lt;code&gt;(2, 3)&lt;/code&gt;. Even if the requested term could be recorded, the log freshness check still fails, so the vote is rejected.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;If the request is valid, the voter records the term in local &lt;code&gt;terms&lt;/code&gt;. If local &lt;code&gt;terms&lt;/code&gt; is shorter than &lt;code&gt;req.term&lt;/code&gt;, it fills the missing positions with default indexes until local state contains index &lt;code&gt;req.term&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;can_vote&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;while&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;len&lt;span style=&#34;color:#1f2328&#34;&gt;()&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;req&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;index&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;len&lt;span style=&#34;color:#1f2328&#34;&gt;();&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;push&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;index&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;These default items have term values equal to their own indexes. They mean the node has observed the corresponding term indexes. They do not mean those indexes are complete log entries, because the matching &lt;code&gt;cmds&lt;/code&gt; may not exist yet. If a later leader fills those positions as complete empty entries, it rewrites their terms to its own &lt;code&gt;leader.term&lt;/code&gt;. On the final iteration, &lt;code&gt;index == req.term&lt;/code&gt;, so the voter has observed and accepted that term. Later, it will not accept an older term or a term index that already exists locally.&lt;/p&gt;&#xA;&lt;p&gt;This replaces the role of standard Raft&amp;rsquo;s persisted &lt;code&gt;currentTerm&lt;/code&gt;, but it is not fully equivalent to standard Raft&amp;rsquo;s &lt;code&gt;votedFor&lt;/code&gt;. The current implementation does not persist &amp;ldquo;which candidate this term was granted to.&amp;rdquo; As a result, RequestVote retry and restart behavior are more conservative. We will return to this trade-off in &amp;ldquo;Current Boundaries.&amp;rdquo;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;The candidate chooses &lt;code&gt;terms.len()&lt;/code&gt; as its term. Other nodes record that term in their local &lt;code&gt;terms&lt;/code&gt; when they grant the vote.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;After the candidate receives granted replies from a quorum, it becomes an established leader. It first rewrites the local &lt;code&gt;cmds.len()..terms.len()&lt;/code&gt; range to its own &lt;code&gt;leader.term&lt;/code&gt;, then appends empty commands so that &lt;code&gt;cmds.len()&lt;/code&gt; catches up with &lt;code&gt;terms.len()&lt;/code&gt;. The index reserved by the leader&amp;rsquo;s election becomes this leader&amp;rsquo;s first complete log entry.&lt;/p&gt;&#xA;&lt;h2 id=&#34;establishing-leader-state&#34;&gt;Establishing Leader State&lt;/h2&gt;&#xA;&lt;p&gt;After a candidate becomes an established leader, it keeps the core state of this leadership in memory. You can think of it as the following structure:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;LeaderState&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;term: &lt;span style=&#34;color:#1f2328&#34;&gt;Term&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;granted_nodes: &lt;span style=&#34;color:#6639ba&#34;&gt;Vec&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;NodeId&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;replications: &lt;span style=&#34;color:#1f2328&#34;&gt;BTreeMap&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;NodeId&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;ReplicationState&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;ReplicationState&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;matched: &lt;span style=&#34;color:#1f2328&#34;&gt;LogIndex&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;end: &lt;span style=&#34;color:#1f2328&#34;&gt;LogIndex&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;inflight: &lt;span style=&#34;color:#cf222e&#34;&gt;bool&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The important fields are:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;term&lt;/code&gt;: the log index reserved when this leader was elected. All later log entries produced by this leader write this term into the corresponding positions of the &lt;code&gt;terms&lt;/code&gt; array.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;granted_nodes&lt;/code&gt;: the node ids that granted this leadership. This proves the leader was chosen by a quorum.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;replications&lt;/code&gt;: the replication progress of each node from the leader&amp;rsquo;s point of view. &lt;code&gt;matched&lt;/code&gt; is the largest index known to match on that node; &lt;code&gt;end&lt;/code&gt; is the upper bound used for further probing or replication; &lt;code&gt;inflight&lt;/code&gt; prevents sending multiple Append requests to the same node at the same time.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The leader also has its own replication state.&#xA;This makes commit calculation uniform:&#xA;look at which nodes have &lt;code&gt;matched&lt;/code&gt; covering an index, then check whether those nodes form a quorum.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;After the leader is established, every position that already exists in local &lt;code&gt;terms&lt;/code&gt; but is still missing from &lt;code&gt;cmds&lt;/code&gt; is taken over by the current leader: the term at that position is rewritten to &lt;code&gt;leader.term&lt;/code&gt;, and the command is filled with an empty command. After that, every local index on the leader has a corresponding command, and new application writes can start at the next index.&lt;/p&gt;&#xA;&lt;!-- [ASCII source](assets/establish-leader.txt) --&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raf-without-term/3494a2b8ffc3e936-establish-leader.png&#34; alt=&#34;Establish leader&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;In this example, index &lt;code&gt;4&lt;/code&gt; used to be a term slot left behind by a failed election, and term &lt;code&gt;5&lt;/code&gt; is the index reserved by the current leader. When the candidate for term &lt;code&gt;5&lt;/code&gt; becomes an established leader, indexes &lt;code&gt;4&lt;/code&gt; and &lt;code&gt;5&lt;/code&gt; are both rewritten as term &lt;code&gt;5&lt;/code&gt; empty log entries. The &lt;code&gt;ø&lt;/code&gt; at index &lt;code&gt;5&lt;/code&gt; is the entry reserved by this leader&amp;rsquo;s election; the &lt;code&gt;ø&lt;/code&gt; at index &lt;code&gt;4&lt;/code&gt; is the backfilled entry that keeps the log prefix contiguous.&lt;/p&gt;&#xA;&lt;h2 id=&#34;appending-a-log-entry&#34;&gt;Appending a Log Entry&lt;/h2&gt;&#xA;&lt;p&gt;After a node becomes leader, each new application write appends a log entry. The term does not change. It is still the term chosen when the leader was elected:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;terms&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;push&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;leader&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;term&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cmds&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;push&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;user_cmd&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;So within one leader term, all later log entries have the same &lt;code&gt;terms[i]&lt;/code&gt;. This matches standard Raft behavior. Only the source of the term is different.&lt;/p&gt;&#xA;&lt;h2 id=&#34;log-replication&#34;&gt;Log Replication&lt;/h2&gt;&#xA;&lt;p&gt;The leader sends Append requests to the other nodes. Conceptually, each request first names a previous log id that is already known to match, then carries a contiguous segment of log entries after that position:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Append&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;term: &lt;span style=&#34;color:#1f2328&#34;&gt;Term&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;prev_log_id: &lt;span style=&#34;color:#1f2328&#34;&gt;LogId&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;terms: &lt;span style=&#34;color:#6639ba&#34;&gt;Vec&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;Term&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;cmds: &lt;span style=&#34;color:#6639ba&#34;&gt;Vec&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;Cmd&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Here &lt;code&gt;prev_log_id&lt;/code&gt; is the matching point. &lt;code&gt;terms&lt;/code&gt; and &lt;code&gt;cmds&lt;/code&gt; are the real entries after that point. They must have the same length, and their first item corresponds to &lt;code&gt;prev_log_id.index + 1&lt;/code&gt;. The follower first checks whether its local log has the same &lt;code&gt;LogId&lt;/code&gt; at &lt;code&gt;prev_log_id.index&lt;/code&gt;. Only if that previous position matches does it accept the entries that follow.&lt;/p&gt;&#xA;&lt;p&gt;This design is close to standard Raft&amp;rsquo;s AppendEntries. Standard Raft carries &lt;code&gt;prevLogIndex&lt;/code&gt; and &lt;code&gt;prevLogTerm&lt;/code&gt; separately; &lt;code&gt;raf&lt;/code&gt; combines them into a single &lt;code&gt;prev_log_id&lt;/code&gt;. That is clearer than using the first entry in the request as the matching point, because the request&amp;rsquo;s &lt;code&gt;terms&lt;/code&gt; and &lt;code&gt;cmds&lt;/code&gt; represent only the real entries to replicate.&lt;/p&gt;&#xA;&lt;p&gt;The following diagram shows one Append request applied to several follower states. The request has &lt;code&gt;term = 5&lt;/code&gt;, &lt;code&gt;prev_log_id = (2, 3)&lt;/code&gt;, and carries the consecutive log entries for indexes &lt;code&gt;4..=5&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;!-- [ASCII source](assets/append-replication.txt) --&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raf-without-term/a0e8c4e1f7671c25-append-replication.png&#34; alt=&#34;Append replication scenarios&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;The three outcomes are:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;accepted&lt;/code&gt;: the follower matches the leader at &lt;code&gt;prev_log_id = (2, 3)&lt;/code&gt;, so it can accept this Append. &lt;code&gt;{...}&lt;/code&gt; marks the &lt;code&gt;terms&lt;/code&gt; range overwritten by this request, as well as the command range appended because it was missing locally. Here the term at index &lt;code&gt;4&lt;/code&gt; is updated from an old value to &lt;code&gt;5&lt;/code&gt;, and command &lt;code&gt;c5&lt;/code&gt; is appended at index &lt;code&gt;5&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;conflict at prev_log_id&lt;/code&gt;: &lt;code&gt;*&lt;/code&gt; marks the conflict position. The follower&amp;rsquo;s term at index &lt;code&gt;3&lt;/code&gt; is &lt;code&gt;3&lt;/code&gt;, while the request&amp;rsquo;s &lt;code&gt;prev_log_id&lt;/code&gt; is &lt;code&gt;(2, 3)&lt;/code&gt;. Since the previous log id does not match, the follower returns a conflict index immediately. The leader must try again with an earlier &lt;code&gt;prev_log_id&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;rejected: follower has newer term&lt;/code&gt;: the follower has already observed term &lt;code&gt;6&lt;/code&gt; at index &lt;code&gt;6&lt;/code&gt;, while the Append request is from term &lt;code&gt;5&lt;/code&gt;. This request comes from a stale leader, so the follower rejects it without modifying the log.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The handling logic is:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If the request term is older than the last observed local term, reject it.&lt;/li&gt;&#xA;&lt;li&gt;Check the follower&amp;rsquo;s local log at &lt;code&gt;prev_log_id&lt;/code&gt;. If it does not match, return the conflict index.&lt;/li&gt;&#xA;&lt;li&gt;If &lt;code&gt;prev_log_id&lt;/code&gt; matches, process the real entries starting at &lt;code&gt;prev_log_id.index + 1&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If later local commands diverge from the leader, truncate the local commands.&lt;/li&gt;&#xA;&lt;li&gt;Overwrite the local &lt;code&gt;terms&lt;/code&gt; range covered by this request.&lt;/li&gt;&#xA;&lt;li&gt;Append only the commands that are missing locally.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;em&gt;Append first finds the common prefix with &lt;code&gt;prev_log_id&lt;/code&gt;, then truncates the follower&amp;rsquo;s conflicting suffix, and finally copies the leader&amp;rsquo;s entries that the follower is missing.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;This is still Raft&amp;rsquo;s core replication model: the leader finds a shared log prefix, then replaces the follower&amp;rsquo;s divergent suffix with its own.&lt;/p&gt;&#xA;&lt;h2 id=&#34;advancing-commit&#34;&gt;Advancing Commit&lt;/h2&gt;&#xA;&lt;p&gt;Replication to a quorum does not mean every historical entry can be committed immediately. Standard Raft has an important rule: a leader may only directly commit log entries from its own current term. Entries from older terms become committed only as a consequence of committing an entry from the current term.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;raf&lt;/code&gt; keeps this rule. Although an established leader rewrites earlier gap slots to its own term, the leader still only directly commits matched indexes that are not smaller than its election index. Empty entries backfilled before &lt;code&gt;leader.term&lt;/code&gt; are committed only indirectly, together with the leader&amp;rsquo;s election index or a later entry.&lt;/p&gt;&#xA;&lt;p&gt;Intuitively:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;quorum_has_matched&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;index&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;index&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;leader&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;term&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;commit&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;index&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The reason is the same as in standard Raft: once an index is committed, every future valid leader must contain it and must not overwrite it.&lt;/p&gt;&#xA;&lt;p&gt;The following state shows why &amp;ldquo;replicated to a quorum&amp;rdquo; is not enough. The leader of term &lt;code&gt;6&lt;/code&gt; has replicated the old log entries at indexes &lt;code&gt;4&lt;/code&gt; and &lt;code&gt;5&lt;/code&gt; to quorum &lt;code&gt;A+B&lt;/code&gt;, but that quorum has not yet matched the leader&amp;rsquo;s own term index &lt;code&gt;6&lt;/code&gt;. Therefore indexes &lt;code&gt;4&lt;/code&gt; and &lt;code&gt;5&lt;/code&gt; still cannot be committed:&lt;/p&gt;&#xA;&lt;!-- [ASCII source](assets/not-committed.txt) --&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/raf-without-term/d98978c7f1b7fab2-not-committed.png&#34; alt=&#34;Not committed yet&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;If a new leader for term &lt;code&gt;7&lt;/code&gt; appears later, and its &lt;code&gt;last_log_id=(5,6)&lt;/code&gt; is newer, it can overwrite those uncommitted log entries. In the diagram, &lt;code&gt;{x}&lt;/code&gt; marks the range replaced by the new leader.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;A leader only directly commits an index that is both covered by a quorum and inside the current leader&amp;rsquo;s term range.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;h2 id=&#34;example&#34;&gt;Example&lt;/h2&gt;&#xA;&lt;p&gt;The repository includes a three-node in-process example that demonstrates the basic flow described in this article. It creates three &lt;code&gt;Raf&lt;/code&gt; nodes, connects them with &lt;code&gt;InProcessNetwork&lt;/code&gt;, explicitly triggers an election on node 1, and then writes a few log entries through the leader. Metrics show the role, term, commit index, and replication progress.&lt;/p&gt;&#xA;&lt;p&gt;The example source is here:&lt;/p&gt;&#xA;&lt;p&gt;https://github.com/drmingdrmer/raf/blob/v0.1.1/examples/three_node.rs&lt;/p&gt;&#xA;&lt;p&gt;Run it from the repository root:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo run --example three_node&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This example is not a production deployment template. It is a minimal demonstration for observing the core protocol state transitions. It logs to stderr and does not include an election timer, heartbeat, snapshots, or membership changes.&lt;/p&gt;&#xA;&lt;h2 id=&#34;current-boundaries&#34;&gt;Current Boundaries&lt;/h2&gt;&#xA;&lt;p&gt;The current implementation is intentionally small. It leaves out several features that a complete production system would usually need:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Automatic election triggering.&lt;/li&gt;&#xA;&lt;li&gt;Snapshots and log compaction.&lt;/li&gt;&#xA;&lt;li&gt;Membership changes.&lt;/li&gt;&#xA;&lt;li&gt;Heartbeats.&lt;/li&gt;&#xA;&lt;li&gt;RequestVote retry logic.&lt;/li&gt;&#xA;&lt;li&gt;Persistence semantics for application payloads.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Automatic election triggering corresponds to the election timer in standard Raft. A node periodically checks whether it has gone too long without seeing a valid leader. If it times out, it starts a new election. This can be implemented by an external timer that calls &lt;code&gt;Raf::elect()&lt;/code&gt;. It does not need to live inside the core &lt;code&gt;raf&lt;/code&gt; state machine, so the current implementation leaves it out.&lt;/p&gt;&#xA;&lt;p&gt;RequestVote retry has a subtler boundary. Suppose a target node successfully handles a &lt;code&gt;RequestVote&lt;/code&gt;, but the reply is lost in the network. If the candidate retries the same request, the target node has already recorded this term in &lt;code&gt;terms[req.term]&lt;/code&gt;. Under the current rule, it rejects the retry because that term index already exists.&lt;/p&gt;&#xA;&lt;p&gt;One optional fix is to add an in-memory &lt;code&gt;voted_for&lt;/code&gt; field that records which candidate owns a term. Then a retry from the same candidate for the same term can be recognized and granted again. This field does not necessarily need to be persisted. If a node restarts and loses &lt;code&gt;voted_for&lt;/code&gt;, it can conservatively reject every &lt;code&gt;RequestVote&lt;/code&gt; that uses a term already present locally. That creates a small availability issue, but only after restart; it does not change the persisted relationship between logs and terms.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;If a RequestVote reply is lost, the retry sees an already existing term. An optional in-memory &lt;code&gt;voted_for&lt;/code&gt; field can improve availability in that case.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;These features can all be added around the core model. This article focuses on the central question: if the term comes from the log index, can Raft election, replication, and commit still be expressed in the familiar way?&lt;/p&gt;&#xA;&lt;h2 id=&#34;summary&#34;&gt;Summary&lt;/h2&gt;&#xA;&lt;p&gt;&lt;code&gt;raf&lt;/code&gt; is not &amp;ldquo;Raft without any term.&amp;rdquo; It still has terms, and it still compares logs by &lt;code&gt;(term, index)&lt;/code&gt;. What it removes is the independently persisted &lt;code&gt;currentTerm&lt;/code&gt;; the leader term is bound to the log index reserved by an election.&lt;/p&gt;&#xA;&lt;p&gt;This change turns the storage state into two &lt;code&gt;Vec&lt;/code&gt;s aligned by index:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;RafStorage&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;terms: &lt;span style=&#34;color:#6639ba&#34;&gt;Vec&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;Term&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;cmds: &lt;span style=&#34;color:#6639ba&#34;&gt;Vec&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;Cmd&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;During election, a candidate chooses &lt;code&gt;terms.len()&lt;/code&gt; as its term. After it becomes leader, both the missing-command gap slots and later log entries use this leader term. Replication and commitment still follow the basic Raft rules.&lt;/p&gt;&#xA;&lt;p&gt;That is the core of this experimental implementation: keep Raft&amp;rsquo;s logical time model, but change where that logical time comes from in persistent state.&lt;/p&gt;&#xA;&lt;p&gt;Repository: &lt;a href=&#34;https://github.com/drmingdrmer/raf/tree/v0.1.1&#34; title=&#34;raf&#34;&gt;raf&lt;/a&gt; (&lt;code&gt;v0.1.1&lt;/code&gt;).&lt;/p&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;raf : &lt;a href=&#34;https://github.com/drmingdrmer/raf/tree/v0.1.1&#34;&gt;https://github.com/drmingdrmer/raf/tree/v0.1.1&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="algo"/>
    <category term="distributed"/>
    <category term="raft"/>
    <category term="consensus"/>
    <category term="storage"/>
    <summary type="html">raf is an experimental Raft variant that does not persist currentTerm as a separate field. Instead, each election reserves a log index, and that index becomes the leader term.</summary>
  </entry>
  <entry>
    <title type="html">Histogram Done Right: 2KB Memory, 0.2% Error</title>
    <link href="https://blog.openacid.com/algo/histogram/" rel="alternate" type="text/html" title="Histogram Done Right: 2KB Memory, 0.2% Error"/>
    <published>2026-04-02T00:00:00&#43;00:00</published>
    <updated>2026-04-02T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/algo/histogram</id>
    <content type="html" xml:base="https://blog.openacid.com/algo/histogram/">&lt;p&gt;&lt;img src=&#34;/post-res/histogram/7b72af58792aed59-histagram-banner.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;h2 id=&#34;the-problem-tracking-request-latency-without-slowing-things-down&#34;&gt;The Problem: Tracking Request Latency Without Slowing Things Down&lt;/h2&gt;&#xA;&lt;p&gt;When we were building &lt;a href=&#34;https://github.com/databendlabs/databend&#34;&gt;Databend&lt;/a&gt; and &lt;a href=&#34;https://github.com/databendlabs/openraft&#34;&gt;OpenRaft&lt;/a&gt;,&#xA;we ran into a familiar need: we wanted to see how request latency was distributed across the system, in real time, without burning CPU or memory to do it.&#xA;This article explains the design behind &lt;a href=&#34;https://github.com/drmingdrmer/base2histogram&#34;&gt;base2histogram&lt;/a&gt;, the library we built to solve it.&lt;/p&gt;&#xA;&lt;p&gt;Consider the life of a single Raft log entry. It passes through several stages, and each one has its own latency profile:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Received → written to storage&lt;/li&gt;&#xA;&lt;li&gt;Persisted to local disk&lt;/li&gt;&#xA;&lt;li&gt;Replicated to remote nodes&lt;/li&gt;&#xA;&lt;li&gt;Acknowledged by a majority quorum&lt;/li&gt;&#xA;&lt;li&gt;Committed → applied to the state machine&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;A &lt;a href=&#34;https://en.wikipedia.org/wiki/Histogram&#34;&gt;histogram&lt;/a&gt; is the natural tool here — plot latency on the x-axis, request count on the y-axis, and you get an immediate picture of where time is being spent.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/d0883e49611bab5c-001-article-latency.png&#34; alt=&#34;001-latency histogram&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;This kind of visibility is what lets you find bottlenecks and fix the right thing.&lt;/p&gt;&#xA;&lt;p&gt;But there&amp;rsquo;s a catch: collecting metrics can&amp;rsquo;t get in the way of doing actual work. So the histogram needs to be:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;O(1) to record&lt;/strong&gt; — no sorting, no rebalancing, nothing that can stall a hot path&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Tiny in memory&lt;/strong&gt; — the system may run hundreds or thousands of these at once&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Queryable for &lt;a href=&#34;https://en.wikipedia.org/wiki/Percentile&#34;&gt;percentiles&lt;/a&gt;&lt;/strong&gt; — P50, P95, P99&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s walk through how we designed one that hits all three.&lt;/p&gt;&#xA;&lt;h2 id=&#34;recording-getting-samples-into-buckets&#34;&gt;Recording: Getting Samples Into Buckets&lt;/h2&gt;&#xA;&lt;h3 id=&#34;why-log-scale-buckets&#34;&gt;Why Log-Scale Buckets&lt;/h3&gt;&#xA;&lt;p&gt;Most requests cluster around some typical latency, with a few outliers on both ends. This is a &lt;a href=&#34;https://en.wikipedia.org/wiki/Log-normal_distribution&#34;&gt;log-normal distribution&lt;/a&gt; — take the log of the latency values, and the shape becomes a classic &lt;a href=&#34;https://en.wikipedia.org/wiki/Normal_distribution&#34;&gt;bell curve&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The signature look: a peak at lower values, then a gradual &lt;a href=&#34;https://en.wikipedia.org/wiki/Long_tail&#34;&gt;long tail&lt;/a&gt; stretching to the right.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/1e4a9be7908c5de8-002-lognormal-distribution.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;To build a histogram, we divide the x-axis into buckets and count how many samples land in each one.&lt;/p&gt;&#xA;&lt;p&gt;The key question is how to size those buckets.&#xA;Equal-width buckets work great for a normal distribution, but latency is log-normal — the data only looks uniform on a logarithmic scale.&#xA;So the buckets need to grow on a &lt;strong&gt;&lt;a href=&#34;https://en.wikipedia.org/wiki/Logarithmic_scale&#34;&gt;log scale&lt;/a&gt;&lt;/strong&gt;, not a &lt;strong&gt;linear&lt;/strong&gt; one.&lt;/p&gt;&#xA;&lt;p&gt;The simplest version of this: each bucket is twice as wide as the one before it.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;[0,1), [1,2), [2,4), [4,8), [8,16), ...&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;Why powers of 2? Because multiplying by 2 is free on a CPU, and mapping a value to its bucket takes a single &lt;a href=&#34;https://en.wikipedia.org/wiki/Find_first_set#CLZ&#34;&gt;leading zero count&lt;/a&gt; instruction.&lt;/p&gt;&#xA;&lt;p&gt;Simulate a log-normal workload, plot the bucket counts with the bucket index on the x-axis (effectively a log transform), and the result is a clean bell curve:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/0587ffc4d78c5e01-003-log2-bucketing.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Storage-wise, this is great — 65 buckets cover the entire u64 range.&#xA;Resolution-wise, not so much. The last bucket spans half of all possible values. Everything that lands there is a blur.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/823fba54ce3daa43-004-log2-coarse.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;h3 id=&#34;a-tempting-fix-we-passed-on&#34;&gt;A Tempting Fix We Passed On&lt;/h3&gt;&#xA;&lt;p&gt;An obvious improvement: use a smaller growth factor, like 1.1× instead of 2×.&#xA;More buckets, finer resolution:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/4431414f6473152b-005-1.1x-buckets.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;The problem is cost. Finding the right bucket for a value &lt;code&gt;l&lt;/code&gt; means solving for the smallest &lt;code&gt;x&lt;/code&gt; where &lt;code&gt;1 + 1.1 + 1.1^2 + ... + 1.1^x &amp;gt;= l&lt;/code&gt; — and that requires floating-point logarithms. That&amp;rsquo;s real overhead on a hot path.&lt;/p&gt;&#xA;&lt;p&gt;We wanted to stay in the world of integers and bit operations.&lt;/p&gt;&#xA;&lt;h3 id=&#34;the-trick-float-like-encoding&#34;&gt;The Trick: Float-Like Encoding&lt;/h3&gt;&#xA;&lt;p&gt;Here&amp;rsquo;s the idea that makes everything work. We keep the roughly exponential bucket sizes, but we encode each bucket using a fixed number of bits — a parameter we call WIDTH.&lt;/p&gt;&#xA;&lt;p&gt;Think of a bucket&amp;rsquo;s lower bound as a tiny &lt;a href=&#34;https://en.wikipedia.org/wiki/Floating-point_arithmetic&#34;&gt;floating-point number&lt;/a&gt;.&#xA;The &lt;a href=&#34;https://en.wikipedia.org/wiki/Bit_numbering#Most_significant_bit&#34;&gt;MSB&lt;/a&gt; position gives you the exponent (which group of buckets you&amp;rsquo;re in),&#xA;and the next few bits give you the offset within that group.&lt;/p&gt;&#xA;&lt;p&gt;With WIDTH=3 (the default), a bucket boundary looks like this in binary:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;00..00 1 xx 00..00&#xA;       |&#xA;       MSB&#xA;&amp;lt;- significant&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The position of the leading &lt;code&gt;1&lt;/code&gt; picks the group. The two bits that follow pick the bucket within the group.&lt;/p&gt;&#xA;&lt;p&gt;Here&amp;rsquo;s what the first few groups look like — each bucket is fully described by just 3 bits:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;WIDTH = 3:&#xA;&#xA;range     bucket index        bucket size&#xA;[0, 1)     0  0b0 ..... 000    1&#xA;[1, 2)     1  0b0 ..... 001    1&#xA;[2, 3)     2  0b0 ..... 010    1&#xA;[3, 4)     3  0b0 ..... 011    1&#xA;&#xA;[4, 5)     4  0b0 ..... 100    1&#xA;[5, 6)     5  0b0 ..... 101    1&#xA;[6, 7)     6  0b0 ..... 110    1&#xA;[7, 8)     7  0b0 ..... 111    1&#xA;&#xA;[8, 10)    8  0b0 .... 1000    2&#xA;[10, 12)   9  0b0 .... 1010    2&#xA;[12, 14)  10  0b0 .... 1100    2&#xA;[14, 16)  11  0b0 .... 1110    2&#xA;&#xA;[16, 20)  12  0b0 ... 10000    4&#xA;[20, 24)  13  0b0 ... 10100    4&#xA;[24, 28)  14  0b0 ... 11000    4&#xA;[28, 32)  15  0b0 ... 11100    4&#xA;&#xA;[32, 40)  16  0b0 .. 100000    8&#xA;[40, 48)  17  0b0 .. 101000    8&#xA;[48, 56)  18  0b0 .. 110000    8&#xA;[56, 64)  19  0b0 .. 111000    8&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The pattern is clean:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each group contains &lt;code&gt;2^(WIDTH-1) = 4&lt;/code&gt; buckets&lt;/li&gt;&#xA;&lt;li&gt;The 2 bits after the MSB select the bucket within the group&lt;/li&gt;&#xA;&lt;li&gt;It&amp;rsquo;s a 3-bit float: 1 implicit leading bit + 2 fractional bits&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/077aac24f7b55056-006-bit-decomposition.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Bucket sizes grow roughly logarithmically,&#xA;and computing the bucket index is just a matter of extracting the top WIDTH bits — a handful of integer and bit ops. Recording a sample is &lt;strong&gt;O(1)&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Walk-through with latency = 42:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;value = 42 (binary: 0b101010)&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  MSB position: 5&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  group: 5 - 2 = 3&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  2 bits after MSB: 01 (from 1[01]010)&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  offset in group: 1&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  Bucket index: 4 + (3 × 4) + 1 = 17&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/126a7a9d50366663-007-bucket-layout.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;h3 id=&#34;tuning-width-the-precisionmemory-knob&#34;&gt;Tuning WIDTH: The Precision–Memory Knob&lt;/h3&gt;&#xA;&lt;p&gt;WIDTH sets how many buckets each group gets (&lt;code&gt;2^(WIDTH-1)&lt;/code&gt;).&#xA;Groups top out at 64 (they still double in size, covering the full u64 range).&lt;/p&gt;&#xA;&lt;p&gt;Here&amp;rsquo;s how the trade-off plays out:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;WIDTH&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Buckets&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Mem/slot&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Buckets per group&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;65&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;520 B&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;2&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;128&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1.0 KB&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;2&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;3&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;252&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;2.0 KB&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;4 (default)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;4&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;496&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;3.9 KB&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;8&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;5&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;976&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;7.6 KB&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;16&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;6&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1920&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;15.0 KB&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;32&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;At the default WIDTH=3, one histogram costs 2 KB and records every sample in O(1).&lt;/p&gt;&#xA;&lt;p&gt;That&amp;rsquo;s the write side sorted. Now for the read side.&lt;/p&gt;&#xA;&lt;h2 id=&#34;percentile-estimation-getting-answers-out&#34;&gt;Percentile Estimation: Getting Answers Out&lt;/h2&gt;&#xA;&lt;p&gt;Once we&amp;rsquo;ve collected the counts, we want percentiles: at what latency have 50% of requests finished (P50)? 90% (P90)? 99% (P99)?&lt;/p&gt;&#xA;&lt;h3 id=&#34;locating-the-right-bucket&#34;&gt;Locating the Right Bucket&lt;/h3&gt;&#xA;&lt;p&gt;The basic idea is simple.&#xA;For P50: count the total samples, take 50% to get a target rank &lt;code&gt;p&lt;/code&gt;, then walk through the buckets from the start, accumulating counts until you pass &lt;code&gt;p&lt;/code&gt;. That&amp;rsquo;s your bucket.&lt;/p&gt;&#xA;&lt;p&gt;But a bucket spans a range, not a point. We still need to estimate where inside the bucket the percentile actually falls.&lt;/p&gt;&#xA;&lt;p&gt;Here are a few ways to do that, from rough to precise.&#xA;All error numbers below come from a log-normal distribution (API latency scenario), WIDTH=3, 1,000,000 samples.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Midpoint&lt;/strong&gt;: just return &lt;code&gt;(min + max) / 2&lt;/code&gt;.&#xA;Many histogram libraries do this (e.g., &lt;a href=&#34;https://github.com/iopsystems/histogram&#34;&gt;iopsystems/histogram&lt;/a&gt;).&#xA;It&amp;rsquo;s a blind guess — it ignores everything about how samples are distributed within the bucket.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;P50&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;P95&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;P99&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;midpoint&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;5.018%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;7.732%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;4.861%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;strong&gt;Uniform interpolation&lt;/strong&gt;: assume samples are spread evenly across the bucket (a flat rectangle), then interpolate linearly: &lt;code&gt;estimate = min + (max - min) × rank / count&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Better than midpoint — at least it uses where in the bucket the target rank falls. But &amp;ldquo;evenly spread&amp;rdquo; is a rough assumption. Log-normal data is skewed even within a single bucket.&lt;/p&gt;&#xA;&lt;h3 id=&#34;trapezoid-interpolation-our-approach&#34;&gt;Trapezoid Interpolation (Our Approach)&lt;/h3&gt;&#xA;&lt;p&gt;Uniform interpolation treats the density inside a bucket as flat. In reality, it&amp;rsquo;s sloped — denser on the side closer to the peak of the distribution.&lt;/p&gt;&#xA;&lt;p&gt;If we know which way the density tilts, we can swap the rectangle for a trapezoid and land much closer to the true value.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/851f5482f43e8c4c-008-trapezoid.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Each bucket stores only a count — we don&amp;rsquo;t want to add extra fields. So where does the slope information come from? The neighbors.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;The densities of the left and right buckets tell us how the density slopes through the current bucket.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Here&amp;rsquo;s the recipe.&#xA;Compute the average density of the left bucket: &lt;code&gt;d0 = c0/(x1-x0)&lt;/code&gt;, and treat it as the density at that bucket&amp;rsquo;s midpoint &lt;code&gt;m0&lt;/code&gt;.&#xA;Do the same for the right bucket: &lt;code&gt;d2 = c2/(x3-x2)&lt;/code&gt; at midpoint &lt;code&gt;m2&lt;/code&gt;.&#xA;Assume density varies linearly from &lt;code&gt;m0&lt;/code&gt; to &lt;code&gt;m2&lt;/code&gt; — over this short range, that&amp;rsquo;s a reasonable approximation.&#xA;This gives us the slope &lt;code&gt;k&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Inside the target bucket, the density now forms a trapezoid: a sloped line with slope &lt;code&gt;k&lt;/code&gt;, pinned so that the density at the bucket&amp;rsquo;s midpoint &lt;code&gt;(x1+x2)/2&lt;/code&gt; equals the bucket&amp;rsquo;s own average density &lt;code&gt;d1 = c1/(x2-x1)&lt;/code&gt; (the midpoint of a linear function always equals its average).&lt;/p&gt;&#xA;&lt;p&gt;To find the percentile, we solve for the x-position where the trapezoid&amp;rsquo;s area from &lt;code&gt;x1&lt;/code&gt; equals the target rank.&lt;/p&gt;&#xA;&lt;p&gt;Same distribution, same buckets — here&amp;rsquo;s how it stacks up:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;P50&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;P95&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;P99&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;midpoint&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;5.018%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;7.732%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;4.861%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;trapezoid&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0.000%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0.080%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0.086%&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Two orders of magnitude better, with zero additional storage.&lt;/p&gt;&#xA;&lt;p&gt;The three-bucket layout:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/histogram/bd85430d86e35843-009-slope-estimation.png&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Variable&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;x0, x1, x2, x3&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Boundaries of the three adjacent buckets&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;w0, w1, w2&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Bucket widths: &lt;code&gt;w0 = x1-x0&lt;/code&gt;, &lt;code&gt;w1 = x2-x1&lt;/code&gt;, &lt;code&gt;w2 = x3-x2&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;c0, c1, c2&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Sample counts in each bucket&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;rank&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;How many samples into the target bucket the percentile falls&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;d0 = c0 / w0       -- left bucket density&#xA;d1 = c1 / w1       -- target bucket density&#xA;d2 = c2 / w2       -- right bucket density&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Midpoints of the left and right buckets: &lt;code&gt;m0 = (x0+x1)/2&lt;/code&gt;, &lt;code&gt;m2 = (x2+x3)/2&lt;/code&gt;.&#xA;Slope:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;k = (d2 - d0) / (m2 - m0)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Then solve for the x-position where the trapezoid&amp;rsquo;s cumulative area from &lt;code&gt;x1&lt;/code&gt; equals the rank.&lt;/p&gt;&#xA;&lt;p&gt;The whole thing runs on three counts and their bucket boundaries. Nothing else stored, nothing else needed.&lt;/p&gt;&#xA;&lt;h2 id=&#34;benchmarks-seven-distributions-six-width-settings&#34;&gt;Benchmarks: Seven Distributions, Six WIDTH Settings&lt;/h2&gt;&#xA;&lt;p&gt;We tested across 7 representative distributions with 1,000,000 samples each, using trapezoid interpolation.&lt;/p&gt;&#xA;&lt;p&gt;The rows to watch are &lt;strong&gt;LN-API&lt;/strong&gt; and &lt;strong&gt;LN-DB&lt;/strong&gt; at &lt;strong&gt;W=3&lt;/strong&gt; — these are the real-world latency cases, running on the default 2 KB configuration:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|                   W=1      W=2      W=3      W=4      W=5      W=6&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| ------------------------------------------------------------------&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Uniform P50    0.108%   0.028%   0.012%   0.018%   0.019%   0.002%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P95    2.317%   1.988%   1.035%   0.475%   0.005%   0.005%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P99    4.290%   4.129%   3.706%   1.486%   0.298%   0.162%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| &#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| LN-API  P50    2.281%   0.182%   0.000%   0.000%   0.000%   0.000%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P95   20.256%   3.963%   0.080%   0.040%   0.040%   0.000%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P99   11.951%   3.594%   0.086%   0.000%   0.029%   0.000%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| &#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Bimodal P50    1.381%   0.394%   0.394%   0.197%   0.197%   0.197%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P95    3.918%   0.172%   0.012%   0.028%   0.038%   0.008%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P99    1.521%   1.344%   0.543%   0.078%   0.016%   0.014%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| &#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Expon   P50    1.012%   0.000%   0.145%   0.145%   0.145%   0.000%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P95   10.989%   0.200%   0.000%   0.000%   0.033%   0.033%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P99   18.665%   4.574%   0.824%   0.022%   0.022%   0.022%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| &#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| LN-DB   P50    2.018%   0.034%   0.000%   0.000%   0.000%   0.034%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P95    2.027%   0.368%   0.039%   0.006%   0.019%   0.026%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P99    3.764%   1.066%   0.187%   0.007%   0.003%   0.062%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| &#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Sequent P50    0.095%   0.000%   0.000%   0.000%   0.000%   0.000%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P95    2.271%   1.967%   1.011%   0.496%   0.000%   0.000%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P99    4.272%   4.118%   3.696%   1.521%   0.305%   0.169%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| &#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Pareto  P50   10.127%   1.899%   0.633%   0.633%   0.633%   0.000%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P95    9.239%   0.272%   0.000%   0.136%   0.000%   0.000%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;|         P99    3.517%   0.879%   0.231%   0.093%   0.046%   0.046%&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| &#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| ------------------------------------------------------------------&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Buckets            65      128      252      496      976     1920&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Mem/slot        520 B   1.0 KB   2.0 KB   3.9 KB   7.6 KB  15.0 KB&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;| Mem total      1.0 KB   2.0 KB   3.9 KB   7.8 KB  15.2 KB  30.0 KB&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;What each distribution models:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Uniform (&lt;a href=&#34;https://en.wikipedia.org/wiki/Continuous_uniform_distribution&#34;&gt;uniform distribution&lt;/a&gt;)&lt;/strong&gt;: synthetic benchmarks&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;LN-API (&lt;a href=&#34;https://en.wikipedia.org/wiki/Log-normal_distribution&#34;&gt;log-normal&lt;/a&gt; σ=0.5)&lt;/strong&gt;: API and microservice latency&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Bimodal (&lt;a href=&#34;https://en.wikipedia.org/wiki/Multimodal_distribution&#34;&gt;bimodal distribution&lt;/a&gt;)&lt;/strong&gt;: cache hit/miss — 90% fast path ~500μs, 10% slow path ~50ms&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Expon (&lt;a href=&#34;https://en.wikipedia.org/wiki/Exponential_distribution&#34;&gt;exponential distribution&lt;/a&gt;)&lt;/strong&gt;: network and I/O waits&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;LN-DB (log-normal σ=1.0)&lt;/strong&gt;: database query latency, with a wider tail&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Sequent (sequential)&lt;/strong&gt;: adversarial worst case&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Pareto (&lt;a href=&#34;https://en.wikipedia.org/wiki/Pareto_distribution&#34;&gt;Pareto distribution&lt;/a&gt; α=1.5)&lt;/strong&gt;: heavy-tailed workloads like request sizes&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;For the latency distributions we care about most — LN-API and LN-DB — WIDTH=3 delivers sub-0.2% error on 2 KB of memory.&lt;/p&gt;&#xA;&lt;h2 id=&#34;summary&#34;&gt;Summary&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;2 KB memory&lt;/strong&gt; (WIDTH=3, 252 buckets of u64), P50/P95/P99 error under 0.2% for log-normal latency&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;O(1) recording&lt;/strong&gt;, O(buckets) querying&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Trapezoid interpolation&lt;/strong&gt; is what makes it work — over 10× more accurate than midpoint, with zero extra storage&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;WIDTH is tunable&lt;/strong&gt;: 520 B for bare-minimum tracking, up to 15 KB for maximum precision&lt;/li&gt;&#xA;&lt;li&gt;Code: &lt;a href=&#34;https://github.com/drmingdrmer/base2histogram&#34;&gt;base2histogram&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Databend : &lt;a href=&#34;https://github.com/databendlabs/databend&#34;&gt;https://github.com/databendlabs/databend&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;MSB : &lt;a href=&#34;https://en.wikipedia.org/wiki/Bit_numbering#Most_significant_bit&#34;&gt;https://en.wikipedia.org/wiki/Bit_numbering#Most_significant_bit&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;OpenRaft : &lt;a href=&#34;https://github.com/databendlabs/openraft&#34;&gt;https://github.com/databendlabs/openraft&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Pareto distribution : &lt;a href=&#34;https://en.wikipedia.org/wiki/Pareto_distribution&#34;&gt;https://en.wikipedia.org/wiki/Pareto_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;base2histogram : &lt;a href=&#34;https://github.com/drmingdrmer/base2histogram&#34;&gt;https://github.com/drmingdrmer/base2histogram&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;bimodal distribution : &lt;a href=&#34;https://en.wikipedia.org/wiki/Multimodal_distribution&#34;&gt;https://en.wikipedia.org/wiki/Multimodal_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;exponential distribution : &lt;a href=&#34;https://en.wikipedia.org/wiki/Exponential_distribution&#34;&gt;https://en.wikipedia.org/wiki/Exponential_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;floating-point number : &lt;a href=&#34;https://en.wikipedia.org/wiki/Floating-point_arithmetic&#34;&gt;https://en.wikipedia.org/wiki/Floating-point_arithmetic&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;histogram : &lt;a href=&#34;https://en.wikipedia.org/wiki/Histogram&#34;&gt;https://en.wikipedia.org/wiki/Histogram&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;iopsystems/histogram : &lt;a href=&#34;https://github.com/iopsystems/histogram&#34;&gt;https://github.com/iopsystems/histogram&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;leading zero counting : &lt;a href=&#34;https://en.wikipedia.org/wiki/Find_first_set#CLZ&#34;&gt;https://en.wikipedia.org/wiki/Find_first_set#CLZ&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;log scale : &lt;a href=&#34;https://en.wikipedia.org/wiki/Logarithmic_scale&#34;&gt;https://en.wikipedia.org/wiki/Logarithmic_scale&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;log-normal distribution : &lt;a href=&#34;https://en.wikipedia.org/wiki/Log-normal_distribution&#34;&gt;https://en.wikipedia.org/wiki/Log-normal_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;long tail : &lt;a href=&#34;https://en.wikipedia.org/wiki/Long_tail&#34;&gt;https://en.wikipedia.org/wiki/Long_tail&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;normal distribution : &lt;a href=&#34;https://en.wikipedia.org/wiki/Normal_distribution&#34;&gt;https://en.wikipedia.org/wiki/Normal_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;percentile : &lt;a href=&#34;https://en.wikipedia.org/wiki/Percentile&#34;&gt;https://en.wikipedia.org/wiki/Percentile&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;uniform distribution : &lt;a href=&#34;https://en.wikipedia.org/wiki/Continuous_uniform_distribution&#34;&gt;https://en.wikipedia.org/wiki/Continuous_uniform_distribution&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="algo"/>
    <category term="histogram"/>
    <category term="latency"/>
    <category term="percentile"/>
    <category term="metrics"/>
    <category term="performance"/>
    <summary type="html">A lightweight histogram that tracks latency distributions in 2KB of memory with sub-0.2% error. Uses a float-like encoding for O(1) bucket indexing and trapezoid interpolation for accurate percentile estimation — no floating-point math needed.</summary>
  </entry>
  <entry>
    <title type="html">xp 的 AI 工作流</title>
    <link href="https://blog.openacid.com/life/xp-vibe-coding/" rel="alternate" type="text/html" title="xp 的 AI 工作流"/>
    <published>2026-01-04T00:00:00&#43;00:00</published>
    <updated>2026-01-04T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/life/xp-vibe-coding</id>
    <content type="html" xml:base="https://blog.openacid.com/life/xp-vibe-coding/">&lt;p&gt;目前我的日常开发基本只需要&amp;quot;动嘴&amp;quot;就能完成。这篇文章分享我在用的工具组合。&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;偏见声明&lt;/strong&gt;：我更希望把精力放在问题分析上，所以市面上流行的工具并没有一一尝试。只要满足需要就不会替换，除非发现明显的效率瓶颈。&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h2 id=&#34;工作流概览&#34;&gt;工作流概览&lt;/h2&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;分析与决策&lt;/strong&gt; — 在 IDE 中浏览代码，理解项目，决定要做什么&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;描述需求&lt;/strong&gt; — 用语音输入把想法描述出来&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;代码生成&lt;/strong&gt; — 交给 AI 命令行工具生成代码&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;审核与提交&lt;/strong&gt; — 用 tig 逐行 review，增量提交到 Git&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;注意上面的几个步骤一般是穿插进行的, 一个这样的修改看做一个 session 的话, 一般我会同时做 2, 3 个工作流 session, 这是因为大模型输出代码比我 review 的速度慢一些, 我自己经常会有 IO 等待, 所以同时开 2,3 个 session 可以把我的 CPU 占满. 但是切换任务也会让自己脑袋里的 context 频繁切换, 降低效率, 所以一般我开 1 个需要我仔细思考的困难任务(例如增加新 feature), 和 1 或 2 个不太需要深度思考的简单任务(例如代码重构), 就不会产生大量颅内 context 切换. 而且忙起来似乎有助于多巴胺分泌.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/xp-vibe-coding/ef3c8085ef56b7b8-workflow.webp&#34; alt=&#34;workflow&#34;&gt;&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h2 id=&#34;第一步分析与决策--rust-rover&#34;&gt;第一步：分析与决策 — Rust Rover&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&#34;https://www.jetbrains.com/rust/&#34;&gt;Rust Rover&lt;/a&gt; 是 JetBrains 开发的 Rust IDE。JetBrains 的 IDE 系列（IntelliJ IDEA、PyCharm、WebStorm 等）在 AI 时代之前一直是开发者的首选，以代码跳转、重构、调试等功能著称。&lt;/p&gt;&#xA;&lt;p&gt;虽然 Rust Rover 在 AI 功能上相对保守，但传统的代码跳转、全局搜索等功能依然扎实，足以快速建立对项目的整体认知。&lt;/p&gt;&#xA;&lt;p&gt;在这个阶段，我主要根据要做的事情回顾代码结构，确定具体的实现思路；或者在没有具体任务时浏览项目，寻找需要改进的地方。&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/xp-vibe-coding/5499429f7c2ff0a1-rust-rover.webp&#34; alt=&#34;Rust Rover&#34;&gt;&lt;/p&gt;&#xA;&lt;h2 id=&#34;第二步描述需求--语音输入&#34;&gt;第二步：描述需求 — 语音输入&lt;/h2&gt;&#xA;&lt;p&gt;口述比打字快，语音输入法是重要补充。&lt;/p&gt;&#xA;&lt;p&gt;语音输入的错误可被大模型理解能力覆盖——只要大致表达清晰，就能完成任务。因此我对语音输入的准确率要求不高，即使口述转文字时有些错误也没关系。只要描述足够详细、提供足够的冗余信息，大模型就能准确理解需求。&lt;/p&gt;&#xA;&lt;p&gt;以下是几种我尝试过、都能满足需要的语音输入法：&lt;/p&gt;&#xA;&lt;h3 id=&#34;闪电说本地模型&#34;&gt;&lt;a href=&#34;https://shandianshuo.cn/&#34;&gt;闪电说&lt;/a&gt;（本地模型）&lt;/h3&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/xp-vibe-coding/3676e2058bc85027-shandianshuo.webp&#34; alt=&#34;闪电说&#34;&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;识别准确率中等&lt;/li&gt;&#xA;&lt;li&gt;离线可用，响应快&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h3 id=&#34;豆包桌面版&#34;&gt;&lt;a href=&#34;https://www.doubao.com/&#34;&gt;豆包桌面版&lt;/a&gt;&lt;/h3&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/xp-vibe-coding/53677e8f2cbbc865-doubao.webp&#34; alt=&#34;豆包&#34;&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;在线识别更精准&lt;/li&gt;&#xA;&lt;li&gt;网络延迟所以稍慢&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h3 id=&#34;智谱语音输入法&#34;&gt;&lt;a href=&#34;https://autoglm.zhipuai.cn/autotyper/&#34;&gt;智谱语音输入法&lt;/a&gt;&lt;/h3&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/xp-vibe-coding/264208e2cb986c5f-zhipu.webp&#34; alt=&#34;智谱语音输入法&#34;&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;AI 矫正能力强，准确率高&lt;/li&gt;&#xA;&lt;li&gt;网络延迟所以稍慢&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr&gt;&#xA;&lt;h2 id=&#34;第三步代码生成--claude-code--glm-47&#34;&gt;第三步：代码生成 — Claude Code + GLM 4.7&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&#34;https://github.com/anthropics/claude-code&#34;&gt;Claude Code&lt;/a&gt; 是我的主力工具，CLI 框架成熟，即便不用内置模型，作为命令行交互载体也很高效。&lt;/p&gt;&#xA;&lt;p&gt;日常开发中，我直接口述需求给 Claude Code，全程不需要键盘打字。&lt;/p&gt;&#xA;&lt;p&gt;为了简化开发流程（也为了好玩），我买了一个只有三个键的小键盘：一个键触发语音输入，一个键是删除，一个键是回车。目前正在尝试只用这三个键完成日常开发。&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/xp-vibe-coding/ba4b118935fc9e66-3-key.webp&#34; alt=&#34;三键小键盘&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;模型用的是 &lt;a href=&#34;https://open.bigmodel.cn/&#34;&gt;GLM 4.7&lt;/a&gt;（&lt;a href=&#34;https://www.zhipuai.cn/&#34;&gt;智谱 AI&lt;/a&gt; 发布）。需求描述清晰的话，完成&amp;quot;文字到代码&amp;quot;的转化没问题。与顶尖模型相比，处理抽象需求时有差距，但性价比高，国内网络环境下响应快。&lt;/p&gt;&#xA;&lt;p&gt;就算说的乱 78 糟, 大模型也知道我说的 &lt;code&gt;IOarrow&lt;/code&gt; 是 &lt;code&gt;io::Error&lt;/code&gt;:&#xA;&lt;img src=&#34;/post-res/xp-vibe-coding/4757c5dfa48f6dc4-claude.webp&#34; alt=&#34;Claude Code&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;我已完全摒弃交互式辅助模式，改用&amp;quot;全量需求描述 + AI 独立实现&amp;quot;。&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h2 id=&#34;第四步审核与提交--tig&#34;&gt;第四步：审核与提交 — tig&lt;/h2&gt;&#xA;&lt;p&gt;我发现最高效的工作方式不是等大模型写完 review, 而是：在大模型生成代码的过程中就开始 review 产生的变更，逐段将变更加入版本控制。确认一个变更正确且符合要求后，立即加入 Git stage(git add)。这样后续修改时, review 工作区变化可以忽略已确认(git add)的部分，效率很高。&lt;/p&gt;&#xA;&lt;p&gt;因此这个阶段最需要的工具，是能够高效的逐段将代码加入 Git 的交互式工具。&lt;code&gt;git add -p&lt;/code&gt; 是最基本的选择，我使用的是 tig。&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&#34;https://github.com/jonas/tig&#34;&gt;tig&lt;/a&gt; 是基于 ncurses 的 Git 文本界面工具。&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h3 id=&#34;为什么用-tig-而非-git-add--p&#34;&gt;为什么用 tig 而非 &lt;code&gt;git add -p&lt;/code&gt;&lt;/h3&gt;&#xA;&lt;p&gt;&lt;code&gt;git add -p&lt;/code&gt; 只能显示固定的 2-3 行上下文，有时不够理解修改(我的 context 太小 🤔)。&lt;/p&gt;&#xA;&lt;p&gt;tig 通过 &lt;code&gt;tig status&lt;/code&gt; 后进入交互界面, 可以：&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;逐行查看差异，像 Vim 一样导航(&lt;code&gt;j/k&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;快捷键&lt;code&gt;[&lt;/code&gt; 和 &lt;code&gt;]&lt;/code&gt; 随时调整 diff 上下文大小&lt;/li&gt;&#xA;&lt;li&gt;逐行(&lt;code&gt;1&lt;/code&gt;)或逐块(&lt;code&gt;u&lt;/code&gt;)将修改加入 Git stage(stage 部分也叫 cached 或 index)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;R&lt;/code&gt;(shift-r) 刷新页面, 显示最新修改&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;一般我习惯在 tmux 里左右分两屏, 右边 pua 大模型干活, 左边逐行确认修改. 因为有时大模型修改的滚动太快了, 看不清它到底做了什么. &lt;code&gt;tig&lt;/code&gt; 的逐行交互式 review 容许我异步查看每一行修改, 一旦看到有问题的修改, 就可以及时切到右边叫停努力的大模型, 重新调整:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&#34;/post-res/xp-vibe-coding/05dc7b8862daff46-tig.webp&#34; alt=&#34;tig&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;可以看到 tig 的上下文可以用&lt;code&gt;[&lt;/code&gt;和&lt;code&gt;]&lt;/code&gt;任意宽度展开&lt;/p&gt;&#xA;&lt;p&gt;Vim 的 Git fugitive 插件也可以提供类似功能但需要先启动 Vim，而且操作稍繁琐。&lt;/p&gt;&#xA;&lt;h3 id=&#34;增量-review&#34;&gt;增量 Review&lt;/h3&gt;&#xA;&lt;p&gt;确认 OK 的部分立即加入 Git，后续再次让大模型做出修改后只 review not-staged 部分, 不再重复审查上一步已经加入 stage, 确认 OK 的代码。AI 增量修改时只需关注变化的部分。&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h2 id=&#34;vibe-coding-的核心理念&#34;&gt;Vibe Coding 的核心理念&lt;/h2&gt;&#xA;&lt;p&gt;AI 能极大提升效率——前提是: 你了解这个领域。否则无法判断对错，错误会吃掉所有效率提升。用 AI 开发的前提是我比 AI 更了解这个项目。&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h2 id=&#34;工具清单&#34;&gt;工具清单&lt;/h2&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;工具&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;用途&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://www.jetbrains.com/rust/&#34;&gt;Rust Rover&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;代码分析&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://shandianshuo.cn/&#34;&gt;闪电说&lt;/a&gt; / &lt;a href=&#34;https://www.doubao.com/&#34;&gt;豆包&lt;/a&gt; / &lt;a href=&#34;https://autoglm.zhipuai.cn/autotyper/&#34;&gt;智谱语音输入法&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;语音输入&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/anthropics/claude-code&#34;&gt;Claude Code&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;AI 编程&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://open.bigmodel.cn/&#34;&gt;GLM 4.7&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;后端模型&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/jonas/tig&#34;&gt;tig&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;代码审核，增量提交&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;hr&gt;&#xA;&lt;p&gt;&lt;em&gt;这篇文章是口述完成的。&lt;/em&gt;&lt;/p&gt;&#xA;&lt;!-- 链接定义 --&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Claude Code : &lt;a href=&#34;https://github.com/anthropics/claude-code&#34;&gt;https://github.com/anthropics/claude-code&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;GLM 4.7 : &lt;a href=&#34;https://open.bigmodel.cn/&#34;&gt;https://open.bigmodel.cn/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;Rust Rover : &lt;a href=&#34;https://www.jetbrains.com/rust/&#34;&gt;https://www.jetbrains.com/rust/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;tig : &lt;a href=&#34;https://github.com/jonas/tig&#34;&gt;https://github.com/jonas/tig&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;智谱 AI : &lt;a href=&#34;https://www.zhipuai.cn/&#34;&gt;https://www.zhipuai.cn/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;智谱语音输入法 : &lt;a href=&#34;https://autoglm.zhipuai.cn/autotyper/&#34;&gt;https://autoglm.zhipuai.cn/autotyper/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;豆包 : &lt;a href=&#34;https://www.doubao.com/&#34;&gt;https://www.doubao.com/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;闪电说 : &lt;a href=&#34;https://shandianshuo.cn/&#34;&gt;https://shandianshuo.cn/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="life"/>
    <category term="vide-coding"/>
    <category term="ai"/>
    <category term="voice-input"/>
    <summary type="html">xp 的 AI 开发工作流</summary>
  </entry>
  <entry>
    <title type="html">Raft Node Rejoin Bug</title>
    <link href="https://blog.openacid.com/algo/raft-rejoin-bug/" rel="alternate" type="text/html" title="Raft Node Rejoin Bug"/>
    <published>2025-11-20T00:00:00&#43;00:00</published>
    <updated>2025-11-20T00:00:00&#43;00:00</updated>
    <id>https://blog.openacid.com/algo/raft-rejoin-bug</id>
    <content type="html" xml:base="https://blog.openacid.com/algo/raft-rejoin-bug/">&lt;p&gt;&lt;img src=&#34;/post-res/raft-rejoin-bug/1d34a13b1c0631e8-raft-rejoin-bug-banner.webp&#34; alt=&#34;&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;In Raft cluster operations, there&amp;rsquo;s an easily overlooked bug: when a node is removed and then re-added to the cluster within the same term, delayed AppendEntries responses from the old membership configuration can corrupt the leader&amp;rsquo;s replication progress tracking for that node, causing the leader to enter an infinite retry loop.&lt;/p&gt;&#xA;&lt;p&gt;The root cause is the lack of a &lt;strong&gt;replication session isolation mechanism&lt;/strong&gt;. When the same node joins the cluster at different times, these should be treated as different replication sessions. However, without explicit session identifiers, the leader cannot distinguish which session a response belongs to. The result is that delayed responses from old sessions incorrectly update the progress records of new sessions.&lt;/p&gt;&#xA;&lt;p&gt;While this creates operational challenges—continuous resource consumption and nodes unable to catch up with the cluster—the good news is that Raft&amp;rsquo;s commit protocol ensures data safety remains intact.&lt;/p&gt;&#xA;&lt;p&gt;Analyzed Raft libs:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Implementation&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th style=&#34;text-align: right&#34;&gt;Stars&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Language&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Status&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Analysis&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Apache Ratis&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;1,418&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Java&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;✓ PROTECTED&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/apache-ratis.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;NuRaft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;1,140&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;C++&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;✓ PROTECTED&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/nuraft.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;OpenRaft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;1,700&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Rust&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;✓ PROTECTED&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/openraft.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;RabbitMQ Ra&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;908&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Erlang&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;✓ PROTECTED&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/rabbitmq-ra.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;braft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;4,174&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;C++&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;✓ PROTECTED&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/braft.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;canonical/raft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;954&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;C&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;✓ PROTECTED&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/canonical-raft.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;sofa-jraft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;3,762&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Java&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;✓ PROTECTED&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/sofa-jraft-analysis.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;LogCabin&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;1,945&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;C++&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/logcabin.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;PySyncObj&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;738&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;Python&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/pysyncobj.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;dragonboat&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;5,262&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;Go&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/dragonboat.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;etcd-io/raft&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;943&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;Go&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/etcd-raft.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;hashicorp/raft&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;8,826&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;Go&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/hashicorp-raft-analysis.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;raft-java&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;1,234&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;Java&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/raft-java.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;raft-rs (TiKV)&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;3,224&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;Rust&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/raft-rs.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;redisraft&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;841&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;C&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/redisraft.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;willemt/raft&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;&lt;strong&gt;1,160&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;C&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;strong&gt;✗ VULNERABLE&lt;/strong&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/willemt-raft.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;eliben/raft&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td style=&#34;text-align: right&#34;&gt;1,232&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Go&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;N/A&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug/blob/main/analysis/eliben-raft.md&#34;&gt;Report&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;This article uses raft-rs, the Raft implementation used by TiKV, as a case study to analyze this bug&amp;rsquo;s trigger conditions, impact, and potential solutions.&lt;/p&gt;&#xA;&lt;p&gt;Complete analysis and survey of other Raft implementations can be found in the &lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug&#34;&gt;Raft Rejoin Bug Survey&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h2 id=&#34;raft-log-replication-basics&#34;&gt;Raft Log Replication Basics&lt;/h2&gt;&#xA;&lt;p&gt;In Raft, the leader replicates log entries to followers through AppendEntries RPC calls, while maintaining a replication state machine for each follower to track replication progress.&lt;/p&gt;&#xA;&lt;h3 id=&#34;appendentries-request-response-flow&#34;&gt;AppendEntries Request-Response Flow&lt;/h3&gt;&#xA;&lt;p&gt;Here&amp;rsquo;s how it works: The leader sends AppendEntries requests with the current &lt;code&gt;term&lt;/code&gt;, the &lt;code&gt;prev_log_index&lt;/code&gt; and &lt;code&gt;prev_log_term&lt;/code&gt; pointing to the position just before the new entries, the &lt;code&gt;entries[]&lt;/code&gt; array to replicate, and the leader&amp;rsquo;s &lt;code&gt;leader_commit&lt;/code&gt; index. The follower responds with its own &lt;code&gt;term&lt;/code&gt;, the highest log &lt;code&gt;index&lt;/code&gt; it replicated, and whether the operation succeeded.&lt;/p&gt;&#xA;&lt;h3 id=&#34;progress-tracking&#34;&gt;Progress Tracking&lt;/h3&gt;&#xA;&lt;p&gt;The leader relies on these responses to track each follower&amp;rsquo;s replication status. It uses &lt;code&gt;matched&lt;/code&gt; to record the highest log index confirmed to be replicated on that follower, and &lt;code&gt;next_idx&lt;/code&gt; to mark where to send next. When a successful response comes back with &lt;code&gt;index=N&lt;/code&gt;, the leader updates &lt;code&gt;matched=N&lt;/code&gt; and calculates &lt;code&gt;next_idx=N+1&lt;/code&gt; for the next round.&lt;/p&gt;&#xA;&lt;p&gt;This tracking mechanism has an implicit assumption: responses correspond to the current replication session.&lt;/p&gt;&#xA;&lt;p&gt;If this assumption isn&amp;rsquo;t handled properly, when a node rejoins the cluster, the leader can get stuck in an infinite retry loop. It keeps sending AppendEntries requests, the node keeps rejecting them, and the cycle repeats endlessly while that node never manages to catch up with the cluster.&lt;/p&gt;&#xA;&lt;h2 id=&#34;raft-rs-progress-tracking&#34;&gt;raft-rs Progress Tracking&lt;/h2&gt;&#xA;&lt;p&gt;raft-rs tracks replication progress using a Progress structure for each follower node:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// From raft-rs/src/tracker/progress.rs&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Progress&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;matched: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;      &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Highest log index known to be replicated&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;next_idx: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;     &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Next log index to send&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;state: &lt;span style=&#34;color:#1f2328&#34;&gt;ProgressState&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// ... other fields&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The &lt;code&gt;matched&lt;/code&gt; field records the highest log index successfully replicated to this follower. Whenever the leader receives a successful AppendEntries response, it updates this field:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// From raft-rs/src/tracker/progress.rs&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;maybe_update&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;n: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#cf222e&#34;&gt;bool&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;need_update&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;matched&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;n&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;  &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Only check monotonicity&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;need_update&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;matched&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;n&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;  &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Accept the update!&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;resume&lt;span style=&#34;color:#1f2328&#34;&gt;();&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;need_update&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Notice the update logic is quite simple: as long as the new index is higher than the current &lt;code&gt;matched&lt;/code&gt;, it accepts the update. When a node gets removed from the cluster, its Progress record is deleted. When it rejoins, a brand new Progress record is created with &lt;code&gt;matched = 0&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;bug-reproduction-sequence&#34;&gt;Bug Reproduction Sequence&lt;/h2&gt;&#xA;&lt;p&gt;Let&amp;rsquo;s walk through a concrete timeline to see how this bug unfolds. Pay special attention to the fact that all events happen within a single term (term=5)—this is key to understanding why term-based validation fails.&lt;/p&gt;&#xA;&lt;h3 id=&#34;event-timeline&#34;&gt;Event Timeline&lt;/h3&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;| Time | Event                                         | Progress State&#xA;|------|-----------------------------------------------|----------------&#xA;| T1   | log=1, members={a,b,c}                        | C: matched=0&#xA;|      | Leader sends AppendEntries(index=1) to C      |&#xA;|      | (Network delay causes slow delivery)          |&#xA;|      |                                               |&#xA;| T2   | log=5, members={a,b}                          | C: [deleted]&#xA;|      | Node C removed from cluster                   |&#xA;|      | Progress[C] deleted from leader&amp;#39;s tracker     |&#xA;|      |                                               |&#xA;| T3   | log=100, members={a,b,c}                      | C: matched=0 (new)&#xA;|      | Node C rejoins the cluster                    |&#xA;|      | New Progress[C] created with matched=0        |&#xA;|      |                                               |&#xA;| T4   | Delayed response arrives from T1:             |&#xA;|      | {from: C, index: 1, success: true}            |&#xA;|      | Leader finds Progress[C] (the new one!)       |&#xA;|      | maybe_update(1) called: 0 &amp;lt; 1, so update!     | C: matched=1 ❌&#xA;|      |                                               |&#xA;| T5   | Leader calculates next_idx = matched + 1 = 2  |&#xA;|      | Sends AppendEntries(prev_index=1)             |&#xA;|      | Node C rejects (doesn&amp;#39;t have index 1!)        |&#xA;|      | Leader can&amp;#39;t decrement (matched == rejected)  |&#xA;|      | Infinite loop begins...                       |&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;response-handling-at-t4&#34;&gt;Response Handling at T4&lt;/h3&gt;&#xA;&lt;p&gt;At time T4, that response sent at T1 and delayed in the network finally arrives. Here&amp;rsquo;s how the leader handles it:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// From raft-rs/src/raft.rs&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;handle_append_response&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;m: &lt;span style=&#34;color:#cf222e&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;Message&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Find the progress record&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;pr&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;match&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;prs&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;get_mut&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;m&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;from&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;Some&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;pr&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;pr&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;None&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;debug!&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;logger&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;no progress available for {}&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;m&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;from&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;return&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;};&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Update progress if the index is higher&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;!&lt;/span&gt;pr&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;maybe_update&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;m&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;index&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;return&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// ...&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Here&amp;rsquo;s where things go wrong: The leader does find a Progress record for node C, but it&amp;rsquo;s the new one created at T3. Since the message&amp;rsquo;s term matches the current term, it passes the term check in the &lt;a href=&#34;https://github.com/tikv/raft-rs/blob/master/src/raft.rs#L1346-L1478&#34;&gt;&lt;code&gt;step()&lt;/code&gt; function&lt;/a&gt;, and the leader updates progress with this stale index value.&lt;/p&gt;&#xA;&lt;h2 id=&#34;root-cause-analysis&#34;&gt;Root Cause Analysis&lt;/h2&gt;&#xA;&lt;p&gt;The root of this bug is that &lt;strong&gt;request-response messages lack replication session identification&lt;/strong&gt;. When node C gets removed at T2 and rejoins at T3, these should be two distinct replication sessions—but the leader has no way to distinguish between responses from requests sent at T1 versus responses from requests sent after T3.&lt;/p&gt;&#xA;&lt;p&gt;Look at raft-rs&amp;rsquo;s Message structure:&lt;/p&gt;&#xA;&lt;p&gt;File: &lt;a href=&#34;https://github.com/tikv/raft-rs/blob/master/proto/proto/eraftpb.proto#L71-L98&#34;&gt;&lt;code&gt;proto/proto/eraftpb.proto:71-98&lt;/code&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-protobuf&#34; data-lang=&#34;protobuf&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;message&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Message&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    MessageType msg_type &lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;1&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#cf222e&#34;&gt;uint64&lt;/span&gt; &lt;span style=&#34;color:#cf222e&#34;&gt;to&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;2&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#cf222e&#34;&gt;uint64&lt;/span&gt; from &lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;3&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#cf222e&#34;&gt;uint64&lt;/span&gt; term &lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;4&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;        &lt;span style=&#34;color:#57606a&#34;&gt;// Only term, no session identifier!&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#cf222e&#34;&gt;uint64&lt;/span&gt; log_term &lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;5&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#cf222e&#34;&gt;uint64&lt;/span&gt; index &lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;6&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#57606a&#34;&gt;// ...&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The Message only has a &lt;code&gt;from&lt;/code&gt; field identifying the sending node, but the same node ID joining the cluster at different times should be treated as different replication sessions. The leader needs to distinguish: is this response from node C&amp;rsquo;s first session or its second session? But the current Message structure provides no way to tell.&lt;/p&gt;&#xA;&lt;h2 id=&#34;impact-analysis&#34;&gt;Impact Analysis&lt;/h2&gt;&#xA;&lt;h3 id=&#34;infinite-retry-loop&#34;&gt;Infinite Retry Loop&lt;/h3&gt;&#xA;&lt;p&gt;Once the leader incorrectly sets &lt;code&gt;matched=1&lt;/code&gt;, trouble begins. Here&amp;rsquo;s what happens:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// From raft-rs/src/tracker/progress.rs&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;maybe_decr_to&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;rejected: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;match_hint: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;..&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;-&amp;gt; &lt;span style=&#34;color:#cf222e&#34;&gt;bool&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;state&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;==&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;ProgressState::Replicate&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Can&amp;#39;t decrement if rejected &amp;lt;= matched&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;rejected&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;matched&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;||&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;rejected&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;==&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;matched&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;request_snapshot&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;==&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;INVALID_INDEX&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;            &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;return&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;false&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;  &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Ignore the rejection!&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// ...&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The leader sends AppendEntries with &lt;code&gt;prev_log_index=1&lt;/code&gt;, but node C&amp;rsquo;s log is empty—it doesn&amp;rsquo;t have index 1. Node C rejects the request. The leader wants to decrement &lt;code&gt;next_idx&lt;/code&gt; to retry an earlier position, but here&amp;rsquo;s the problem: because &lt;code&gt;rejected (1) == matched (1)&lt;/code&gt;, the decrement logic returns false and refuses to decrement. So the leader just sends the same request again, node C rejects it again, and this cycle continues forever.&lt;/p&gt;&#xA;&lt;h3 id=&#34;operational-impact&#34;&gt;Operational Impact&lt;/h3&gt;&#xA;&lt;p&gt;This bug creates a series of operational problems. First, there&amp;rsquo;s resource exhaustion: the continuous AppendEntries-rejection cycle keeps consuming CPU and network bandwidth.&lt;/p&gt;&#xA;&lt;h2 id=&#34;why-data-remains-safe&#34;&gt;Why Data Remains Safe&lt;/h2&gt;&#xA;&lt;p&gt;Despite all the operational chaos, there&amp;rsquo;s good news: data integrity remains intact. Raft&amp;rsquo;s safety properties ensure that even with corrupted progress tracking, the cluster won&amp;rsquo;t lose any committed data.&lt;/p&gt;&#xA;&lt;p&gt;The reason is that commit index calculation still works correctly. Even if the leader mistakenly thinks node C has &lt;code&gt;matched=1&lt;/code&gt;, it calculates the commit index based on the actual majority. For example, node A has matched=100, node B has matched=100, and node C has matched=1 (which is wrong, but doesn&amp;rsquo;t matter). The majority looks at A and B with matched=100, so the commit index is correctly calculated as 100. Combined with Raft&amp;rsquo;s overlapping majorities property, any newly elected leader will necessarily have all committed entries, keeping data safe.&lt;/p&gt;&#xA;&lt;h2 id=&#34;solutions&#34;&gt;Solutions&lt;/h2&gt;&#xA;&lt;h3 id=&#34;solution-1-add-membership-version-recommended&#34;&gt;Solution 1: Add Membership Version (Recommended)&lt;/h3&gt;&#xA;&lt;p&gt;The most straightforward fix is to add a membership configuration version to messages:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-protobuf&#34; data-lang=&#34;protobuf&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;message&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Message&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#57606a&#34;&gt;// ... existing fields&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#cf222e&#34;&gt;uint64&lt;/span&gt; membership_log_id &lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#0550ae&#34;&gt;17&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;  &lt;span style=&#34;color:#57606a&#34;&gt;// New field&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Then validate it when processing responses:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;fn&lt;/span&gt; &lt;span style=&#34;color:#6639ba&#34;&gt;handle_append_response&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;mut&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;m: &lt;span style=&#34;color:#cf222e&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;Message&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;let&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;pr&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;prs&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;get_mut&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;m&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;from&lt;span style=&#34;color:#1f2328&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;?&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Check membership version&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;if&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;m&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;membership_log_id&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#0550ae&#34;&gt;!=&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#6a737d&#34;&gt;self&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;current_membership_log_id&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#6639ba&#34;&gt;debug!&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;&lt;span style=&#34;color:#0a3069&#34;&gt;&amp;#34;stale message from different membership&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;        &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;return&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;;&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;pr&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;maybe_update&lt;span style=&#34;color:#1f2328&#34;&gt;(&lt;/span&gt;m&lt;span style=&#34;color:#1f2328&#34;&gt;.&lt;/span&gt;index&lt;span style=&#34;color:#1f2328&#34;&gt;);&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This directly fixes the root cause—the leader can now tell which membership configuration a message comes from.&lt;/p&gt;&#xA;&lt;h3 id=&#34;solution-2-generation-counters&#34;&gt;Solution 2: Generation Counters&lt;/h3&gt;&#xA;&lt;p&gt;Another approach is to add a generation counter to Progress that increments each time a node rejoins:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#f7f7f7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;struct&lt;/span&gt; &lt;span style=&#34;color:#1f2328&#34;&gt;Progress&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;{&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;matched: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;next_idx: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#cf222e&#34;&gt;pub&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt; &lt;/span&gt;generation: &lt;span style=&#34;color:#cf222e&#34;&gt;u64&lt;/span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;,&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;  &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// Incremented on each rejoin&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#fff&#34;&gt;    &lt;/span&gt;&lt;span style=&#34;color:#57606a&#34;&gt;// ...&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#1f2328&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#fff&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Include the generation in messages and validate it when responses arrive. This is lighter weight than solution 1, but you need to carefully manage the generation lifecycle.&lt;/p&gt;&#xA;&lt;h2 id=&#34;summary&#34;&gt;Summary&lt;/h2&gt;&#xA;&lt;p&gt;This bug shows us that when membership changes happen within the same term, relying on term-based validation alone isn&amp;rsquo;t enough to ensure message freshness. Without explicit session isolation, delayed responses from old membership configurations can corrupt progress tracking.&lt;/p&gt;&#xA;&lt;p&gt;Fortunately, because Raft&amp;rsquo;s commit index calculation and overlapping quorum mechanisms provide strong guarantees, this bug doesn&amp;rsquo;t compromise data safety. The main impact is operational—the symptoms look like data corruption, which can send operations teams down the rabbit hole investigating a data loss problem that doesn&amp;rsquo;t actually exist.&lt;/p&gt;&#xA;&lt;p&gt;For production Raft implementations, it&amp;rsquo;s recommended to introduce explicit session management mechanisms. This can be achieved through membership versioning or generation counters. The most recommended approach is to add a membership_log_id field to messages, which lets the leader clearly distinguish which membership configuration a response comes from.&lt;/p&gt;&#xA;&lt;p&gt;Complete analysis and survey of other Raft implementations can be found in the &lt;a href=&#34;https://github.com/drmingdrmer/raft-rejoin-bug&#34;&gt;Raft Rejoin Bug Survey&lt;/a&gt;&lt;/p&gt;&#xA;</content>
    <author><name>Zhang Yanpo (drdr.xp)</name></author>
    <category term="algo"/>
    <category term="distributed"/>
    <category term="分布式"/>
    <category term="raft"/>
    <category term="en"/>
    <summary type="html">Analyzes a replication session isolation bug in Raft implementations. When a node rejoins the cluster within the same term, delayed AppendEntries responses can corrupt progress tracking, causing infinite retry loops. While data safety remains intact, it creates operational issues like resource exhaustion. Uses raft-rs as a case study to examine trigger conditions and solutions.</summary>
  </entry>
</feed>
