The Pain Nobody Posts on LinkedIn
In a single week, we brought down the production database twice. It wasn't because we lacked backups—we had daily, tested backups. The problem was that when the primary node went down, nobody knew with confidence how to bring the database up on the other node without corrupting anything, preferably automatically.
PostgreSQL is single-primary
With streaming replication, one node writes and streams data to read-only replicas. Unidirectional replication is not a limitation—it is the safeguard against corruption. Failover elects a new primary; the old one must resynchronize before returning (the "switchback dance").
Two nodes aren't enough: split-brain
With two nodes, a partition becomes a coin toss: without a majority, no tool can safely decide which one is the primary. The solution is arithmetic: a third vote (witness). In our case, it's a cheap droplet that only casts a vote.
Patroni: a platform solution
It relies on a DCS (etcd/Consul/ZooKeeper) and performs automatic fencing: whoever loses quorum demotes itself. It costs Python (more RAM), wraps PostgreSQL, and adds etcd (+HAProxy in Docker). But etcd isn't ZooKeeper—it's lightweight Go software (~50 MB). It's worth it when you're going to standardize HA across multiple services.
repmgr: the bolt-on
Written in C (lightweight), it uses a witness + repmgrd, with no external DCS, alongside your PostgreSQL instance. repmgr standby switchover and repmgr node rejoin --force-rewind provide an elegant switchback. Fencing is more your responsibility, but with the witness, repmgrd requires a majority before promoting a node. In Docker, you need an image with the repmgr extension—even so, it is far less invasive than Patroni.
DNS routing changes the game
The app and database are per-node: each app instance talks to its local database, and DNS decides which one serves traffic. So, no HAProxy: failover means promoting the local database and changing DNS. Patroni's heaviest component becomes dead weight.
What about losing data?
Almost none. Content (articles, images, authors) lives in MinIO, which is replicated. During an outage, the only writes are visit counters; with slot-based replication, there are 0 bytes of lag. The real danger was never lag—it was a poorly executed switchover. That's an operational risk, addressed with a good tool + a witness.
The decision
repmgr + the droplet as a witness: it runs with the existing PostgreSQL setup, without etcd or changes to the CI/CD pipeline, while providing supported automatic failover + switchover/rejoin and reusing the DNS change. Patroni can wait until HA becomes a platform requirement.
Closing thoughts
HA means choosing the tool you can maintain and trust at 3 a.m. For a small team, it is almost always the simplest tool that solves the right problem—split-brain. A cheap third vote is worth more than a thousand lines of YAML you cannot debug under pressure.