come kill my cluster!

jul 18, 2026 · updated aug 1, 2026 · by anush sonone

try to kill my cluster! :)

each card above is one machine, showing its role, its term, and how far it has committed. four of the seven have to stay alive for the cluster to keep working, which is what makes the leader the interesting one to kill. machines heal themselves after about ten seconds, and you get one kill every two seconds.

in the readout at the top, writes and reads per second are the leader's alone, while vm cpu and vm mem cover the whole host.

what this is

kill-my-cluster is a raft key-value store i wrote from scratch in go. there's no consensus library under it, so the log, the elections, the replication, and the read path are all mine. seven of those nodes run in docker on a single oracle vm with 4 ocpu.

the problem it mocks

put your data on one machine and you've got two problems. the machine can die and take the data with it, or it can stay up and quietly serve wrong answers, and from the outside those two failures look identical.

the fix is to keep the data on several machines and make them agree on the order of every change before any of them acts on it. that agreement is what consensus means, and raft is one way to reach it. you're probably already depending on it somewhere, since etcd runs raft and every kubernetes cluster keeps its entire state there, and consul and cockroachdb run it too.

how it works

one node is the leader, and every change goes to it first. the leader writes the change into its log and ships it to the other six, and once four of the seven have it, the change is committed and applied to the map that each node keeps in memory.

that number four is the whole trick, because any two majorities of seven have to share at least one node, which is why two conflicting decisions can never both commit. the other word worth knowing is term, the election round number, which goes up by one each time the cluster picks a new leader.

kill a follower and not much happens. kill the leader and there's a short pause while the rest elect a replacement.

how the nodes talk

there are only three messages in the whole protocol, and the one below does almost all the work. here's a single write making the round trip.

the other two messages are requestvote, which is the next diagram, and installsnapshot. installsnapshot is for a follower that's been gone long enough that the entries it needs were already compacted away, so the leader stops trying to catch it up entry by entry and ships the whole map instead. that's what happens to every machine you kill on this page, since ten seconds away at this write rate leaves it further behind than the leader bothers to remember.

what an election looks like

this is the part i got wrong first, and the part i'm most pleased with now.

the pre-vote step in the middle isn't in the original paper. it comes from the author's thesis, and skipping it cost me six days.

without it, a node that gets cut off keeps timing out, and under the naive rule every timeout raises its term. when that node comes back it arrives holding a term nobody else has seen, so the real leader stands down out of politeness and the cluster has to elect again. asking first costs one round trip and makes that failure mode impossible, which is why the version running above does it. i left it out at first and paid for it in july, which is the story further down.

how the pieces fit together

the cluster itself is only part of the system, so the diagram below is worth reading in two halves. the top half is everything sitting behind the panel above, and it shows how a kill you press actually reaches a machine. the bottom half is what that kill does once it lands, which is where raft stops being plumbing and starts being the whole point.

the filled dot is the leader and the crossed ones are machines you killed. the third case is the interesting one, because raft would rather refuse your write than accept it on three machines and lose it, so a cluster without a majority stops instead of guessing. the machines come back on their own about ten seconds later and the filled dot reappears.

everything except the tunnel is a docker container on the same box, and node_exporter runs alongside prometheus to report the host cpu and memory you see in the readout. there's no separate demo hostname, so this page only ever sees what the control plane chooses to share.

how i built it

i started at the bottom, with a write-ahead log and snapshots on disk, because nothing above that layer means much if a crash halfway through a write can corrupt the record. the raft core came next, and then a key-value map on top of it with per-client ids, so that a retried write applies once instead of twice.

all of that worked, and then i made my first mistake: i sent reads through the raft log the same way as writes. that's technically correct, but it also means thousands of proposals a second squeezed through one leader and six disks, and on my laptop it election-stormed until throughput fell apart.

the fix was readindex, from section 6.4 of the paper, where the leader confirms it still leads with a single round of heartbeats, waits for its own commits to land, and then answers straight from memory. reads stopped touching disk and the whole thing went quiet again.

with reads sorted out, i shipped it, which meant seven containers, a control plane behind the kill buttons, load generators, prometheus, grafana, and a cloudflare tunnel. i ran a load ladder to find where the box actually gave out instead of picking a number that sounded impressive, and then put the demo online.

then in july it broke for six days straight, which is the part i'd rather not have learned the hard way. the raft log was never being compacted, so it grew to 21.6 million entries, and every restart replayed all of them, pushing the node past 10gb of memory until the kernel killed it. that triggered another election, which restarted another node, and the cycle fed itself from there.

this was the missing pre-vote step playing out for real: term passed 36,000, and the cluster burned 87% of the cpu to commit nothing at all. the host got so busy that ssh barely answered, and getting back in meant racing a loop against the boot window.

the fixes came from reading what actually happened rather than turning knobs:

• prevote, so a node that got cut off can't disrupt a healthy leader just by raising the term.
• checkquorum, so a leader that's lost the majority steps down instead of pretending.
• backoff before a node runs again, and one replicator per follower.
• automatic compaction, plus cpu caps on every container.
• load generators that ease off when the cluster is struggling, instead of leaning harder.

the third mistake came later and was smaller, though it's honestly my favorite. the table that remembers client ids swept itself clean on every write, so once the load generator ran 96 workers it was scanning the whole thing constantly, on all seven nodes, and writes fell from about 950 a second to 11. raft itself stayed perfectly healthy throughout, with term flat and quorum fine, which is exactly why it took me so long to find. it looked nothing like a consensus problem because it wasn't one.

the fourth mistake was the worst, and it arrived the day after i thought the cluster was finally safe. on july 28 a visitor killed three nodes, they healed on schedule, and the cluster then served zero writes and zero reads for forty-eight hours. every container stayed up and every metric stayed green the whole time, so nothing i had built to catch trouble caught anything, and it recovered only because another visitor happened to kill the wedged leader two days later.

the reason is the part i keep coming back to. checkquorum asks whether the followers still answer, and they did, so pre-vote and leader stickiness went on faithfully protecting a leader that had stopped committing anything at all. every mechanism i added after the election storm made that zombie more stable, because all of them measured whether the cluster was alive and none of them measured whether it was making progress. the fix was a watchdog that deposes a leader holding uncommitted entries when the commit index stops moving for thirty seconds, and the cluster now keeps a durable log of those state changes so its worst day is reconstructable afterwards.

that's four mistakes in all, three of which took the live cluster down, and they taught me more than the parts that worked on the first try.

what it does today

as i write this the cluster is holding about 1,110 writes and 1,460 reads a second, with the host near 78% busy across its 4 ocpu. the readout above is the live version of those numbers, so it won't match exactly, and it dips while people are killing things.

the load generator aims at 1,400 writes and 2,000 reads but backs off when errors or latency climb, so those are ceilings it moves toward rather than rates it holds regardless. followers routinely trail the leader by hundreds of commit indexes under load, which looks alarming but is ordinary catch-up rather than drift.

the ladder below is where this box saturates, measured 2026-07-21 before the cpu caps existed. every step held term at 1 with no errors, so what ran out was cpu rather than stability.

c5 was the peak, and c6 asked for more and got less, which is what a saturated machine looks like from the outside. today's rates sit below that on purpose, so that kills still have headroom.

what i'd still fix

• every healed node comes back far enough behind that the leader has to ship it the whole map. retaining enough entries to avoid that costs 20 to 50% of throughput on a node capped this small, so there's no setting that dodges both, and for now i keep the small default and pay for the snapshot on heal.
• the security hardening checklist for the public endpoints is most of the way through. the grafana credentials and the incident record are done, and the tunnel path restrictions and the rate limit on kills are not.
• reads are cheap now, but they still all go to the leader. serving them from followers would spread the work out, and i haven't built that.
• 4 ocpu is the ceiling, and no amount of tuning changes that. forgetting it is how i got the first election storm.

learn more

the code's at kill-my-cluster, laid out roughly like this:

• internal/storage is the write-ahead log and the snapshots.
• internal/raft is where elections, replication, readindex, and batching live.
• internal/kv is the replicated map and the exactly-once client ids.
• internal/controlplane handles kill, heal, and the snapshot stream.
• deploy/compose brings the whole stack up locally.

if raft's new to you, this is the order i'd go in:

1. kill the leader in the panel above and watch term and commit index move.
2. read the secret lives of data, which animates the whole algorithm in a few minutes. it's the friendliest starting point i know of.
3. watch diego ongaro's talk below. he wrote the paper, and hearing him explain it is easier than reading it cold.
4. then read the paper itself, sections 5 and 6. section 6.4 is the read path i described above, and raft.github.io lists other implementations if you want to compare.
5. clone the repo, bring up compose, and read internal/raft/node.go with the paper open beside it.

the nodes talk over grpc with protocol buffers. core concepts and the go quickstart are enough to follow the code.