come kill my cluster!

jul 18, 2026 · updated jul 30, 2026 · by anush sonone

try to kill my cluster! :)

the live cluster

each card is one machine. it shows that machine's role, its term, and how far it has committed.

four of the seven have to stay alive, which makes the leader the interesting kill. machines heal themselves after about ten seconds, and you get one kill every two seconds.

one thing to watch for in the readout: writes and reads per second are the leader's, while vm cpu and vm mem are the whole host.

what this is

kill-my-cluster is a raft key-value store i wrote from scratch in go. there's no consensus library under it, so the log, the elections, the replication, and the read path are all mine. seven of those nodes run in docker on a single oracle vm with 4 ocpu.

it's a demo and a learning project, not a production database. i wouldn't put anything you care about in it.

the problem it mocks

put your data on one machine and you've got two problems. the machine can die and take the data with it. worse, it can stay up and quietly serve wrong answers — and from the outside, those two look the same.

the fix: keep the data on several machines, and make them agree on the order of every change before any of them acts on it. that agreement is consensus, and raft is one way to reach it.

you're probably already depending on it. etcd runs raft, and every kubernetes cluster keeps its entire state there. consul and cockroachdb run it too.

this page is that shape at small scale, where the failure is a button instead of a paragraph.

how it works

start with the leader. one node holds that job, and every change goes to it first. the leader writes the change into its log, then ships it to the other six. once four of the seven have it, the change is committed and applied to the map each node keeps in memory.

four is the whole trick. any two majorities have to share at least one node, so two conflicting decisions can never both commit.

one more word you'll need: term. it's the election round number, and it goes up by one each time the cluster picks a new leader.

kill a follower and not much happens. kill the leader and there's a short pause while the rest elect a replacement.

how the nodes talk

there are only three messages in the whole protocol, and the one below does almost all the work. here's a single write making the round trip.

the other two messages are requestvote, which is the next diagram, and installsnapshot.

installsnapshot is for a follower that's been gone long enough that the entries it needs were already compacted away. the leader gives up on catching it up entry by entry and ships the whole map instead.

that's what happens to every machine you kill here: ten seconds away at this write rate is further behind than the leader bothers to remember.

what an election looks like

this is the part i got wrong first, and the part i'm most pleased with now.

the pre-vote step in the middle isn't in the original paper — it's from the author's thesis, and skipping it cost me six days.

here's what goes wrong without it. a node that's cut off keeps timing out, and under the naive rule every timeout raises its term. when it comes back, it's holding a term nobody else has seen, so the real leader stands down out of politeness and the cluster elects again.

mine reached term 36,000 doing that on a loop, burning 87% of the cpu to commit nothing at all. asking first costs one round trip and makes the whole failure mode impossible.

how the pieces fit together

the cluster itself is only part of it, so read the diagram below in two halves.

the top half is everything sitting behind the panel above: how a kill you press actually reaches a machine. the bottom half is what that kill does once it lands, which is where raft stops being plumbing and starts being the whole point.

the filled dot is the leader, and the crossed ones are machines you killed.

the third case is the interesting one. raft would rather refuse your write than take it on three machines and lose it, so a cluster with no majority stops instead of guessing. the machines come back on their own about ten seconds later, and the filled dot reappears.

a note on the plumbing: node_exporter runs alongside prometheus and reports the host cpu and memory you see in the readout, and everything except the tunnel is a docker container on the same box. there's no separate demo hostname, so this page only ever sees what the control plane chooses to share.

how i built it

i started at the bottom, with a write-ahead log and snapshots on disk. nothing above that layer means much if a crash halfway through a write can corrupt the record, so it felt like the honest place to begin.

the raft core came next, then a key-value map on top of it with per-client ids, so a retried write applies once instead of twice.

all of that worked, and then i made my first mistake: i sent reads through the raft log the same way as writes.

that's technically correct. it's also thousands of proposals a second squeezed through one leader and six disks. on my laptop it election-stormed until throughput fell apart.

the fix was readindex, from section 6.4 of the paper. the leader confirms it still leads with a single round of heartbeats, waits for its own commits to land, then answers straight from memory. reads stopped touching disk, and the whole thing went quiet again.

so i shipped it. seven containers, a control plane behind the kill buttons, load generators, prometheus, grafana, and a cloudflare tunnel. i ran a load ladder to find where the box actually gave out instead of picking a number that sounded impressive, and put the demo online.

then in july it broke for six days straight, which is the part i'd rather not have learned the hard way.

the raft log was never being compacted, so it grew to 21.6 million entries. every restart replayed all of them, pushing the node past 10gb of memory until the kernel killed it — which triggered another election, which restarted another node.

term passed 36,000. the host got so busy that ssh barely answered, and getting back in meant racing a loop against the boot window.

the fixes came from reading what actually happened rather than turning knobs:

• prevote, so a node that got cut off can't disrupt a healthy leader just by raising the term.
• checkquorum, so a leader that's lost the majority steps down instead of pretending.
• backoff before a node runs again, and one replicator per follower.
• automatic compaction, plus cpu caps on every container.
• load generators that ease off when the cluster is struggling, instead of leaning harder.

the third mistake was smaller and, honestly, my favorite. the table that remembers client ids swept itself clean on every write, so once the load generator ran 96 workers it was scanning the whole thing constantly, on all seven nodes. writes fell from about 950 a second to 11.

raft itself stayed perfectly healthy the entire time — term flat, quorum fine — which is exactly why it took me so long to find. it looked nothing like a consensus problem because it wasn't one.

three mistakes, two of which took the live cluster down. they taught me more than the parts that worked on the first try.

what it does today

as i write this the cluster is holding about 1,160 writes and 1,500 reads a second, with the host near 73% busy across its 4 ocpu. the readout above is the live version of those numbers, so it won't match exactly, and it dips while people are killing things.

the load generator aims at 1,400 writes and 2,000 reads, but it backs off when errors or latency climb. those are ceilings it moves toward, not rates it holds regardless.

one thing that looks alarming but isn't: followers routinely trail the leader by hundreds of commit indexes under load. that's ordinary catch-up, not drift.

the ladder below is where this box saturates, measured 2026-07-21 before the cpu caps existed. every step held term at 1 with no errors, so what ran out was cpu rather than stability.

c5 was the peak. c6 asked for more and got less, which is what a saturated machine looks like from outside. today's rates sit below that on purpose, so kills still have headroom.

what i'd still fix

• node memory drifts from about 470mb to 726mb over an hour of load, and i haven't watched it long enough to prove it flattens out.
• the security hardening checklist for the public endpoints is written but not finished.
• reads are cheap now, but they still all go to the leader. serving them from followers would spread the work out, and i haven't built that.
• 4 ocpu is the ceiling, and no amount of tuning changes that. forgetting it is how i got the first election storm.

learn more

the code's at kill-my-cluster, laid out roughly like this:

• internal/storage is the write-ahead log and the snapshots.
• internal/raft is where elections, replication, readindex, and batching live.
• internal/kv is the replicated map and the exactly-once client ids.
• internal/controlplane handles kill, heal, and the snapshot stream.
• deploy/compose brings the whole stack up locally.

if raft's new to you, this is the order i'd go in:

1. kill the leader in the panel above and watch term and commit index move.
2. read the secret lives of data, which animates the whole algorithm in a few minutes. it's the friendliest starting point i know of.
3. watch diego ongaro's talk below. he wrote the paper, and hearing him explain it is easier than reading it cold.
4. then read the paper itself, sections 5 and 6. section 6.4 is the read path i described above, and raft.github.io lists other implementations if you want to compare.
5. clone the repo, bring up compose, and read internal/raft/node.go with the paper open beside it.

the nodes talk over grpc with protocol buffers. core concepts and the go quickstart are enough to follow the code.