Servers die. Databases should not.
Every stack has a database at the bottom of it, and a database on a single server is a countdown. This page explains how the Node Platform keeps databases available through hardware failure: what replication and automatic failover mean in plain English, and what your apps actually experience when a machine dies. It is part of our Under the hood series, expanding on How the platform is built.
The problem, plainly
Most of what runs on a platform can be restarted without drama. Kubernetes does exactly that: if an app's container dies, it is restarted or rescheduled, and the interruption is brief. Databases are the hard case, because they hold state. You cannot simply start a fresh copy somewhere else, because the fresh copy would be empty. Whatever machine holds the data is, by default, a single point of failure, and single points of failure do not fail at convenient times. They fail mid-afternoon, mid-invoice-run, mid-checkout.
The answer is to make sure no single machine ever holds the only copy.
Primary and replicas, in plain English
A highly available database is a small cluster rather than a server. One member, the primary, takes the writes. The others, the replicas, receive a continuous stream of every change the primary makes, a mechanism PostgreSQL calls streaming replication, and apply it to their own copies. At any moment there are several machines holding the data, one of them leading.
Replication answers the first half of the availability question: when the primary dies, the data is not lost, because up-to-date copies already exist elsewhere. The second half is harder: who takes over, and who decides?
Leader election: who decides, when no one is in charge
Promoting a replica by hand works, if an engineer is awake, notices quickly, and picks the right replica. Automating it safely is the interesting part, because the cluster must agree on one answer to "who is the primary now?". Two nodes each believing they lead, a condition known as split-brain, is worse than an outage, because both accept writes and the histories diverge.
For our PostgreSQL clusters this job belongs to Patroni, an open source high-availability framework. Each database node runs a Patroni agent, and the agents coordinate through a consensus store: a small, strongly consistent system whose job is to hold one agreed version of the truth. The primary must continually renew its claim to leadership there. If it dies, or is cut off, its claim lapses, the healthy replicas hold an election, the most up-to-date one is promoted, and the remaining replicas follow the new primary. No human in the loop, and no possibility of two winners, because the consensus store will only ever grant leadership once.
Our MySQL workloads take a related approach with Percona XtraDB Cluster, where the members operate as a coordinated cluster rather than a lone server. In both cases the principle is the same: the machines agree amongst themselves, quickly, so that a hardware failure becomes a handover rather than an outage.
One stable address
Failover would still be disruptive if every application had to know which server currently leads. So they do not. As described in How the platform is built, HAProxy fronts our database clusters: an application talks to one stable address, and HAProxy routes to whichever member is primary right now. When the cluster fails over, scales or gets patched underneath, the address the application knows never changes. A typical application experiences a failover as a brief pause and a reconnect, not as an incident.
Small blast radius, by design
One more design choice matters as much as the clustering. Each tenant on the platform gets its own database instances, not a slice of one shared multi-tenant database. Tenant isolation is structural on this platform, namespaces, network policy, storage and identity are all per-tenant, and databases follow the same rule.
The consequence is that failures stay small. A runaway query, a corrupted table or an app bug in one tenant's database is that tenant's problem to fix with us, not a platform-wide event. There is no single enormous database whose bad day is everyone's bad day.
What this means for you
When a server dies mid-afternoon, the sequence is: the cluster notices within moments, elects a new primary, applications reconnect through the same address, and engineers are alerted to deal with the dead machine at leisure. Your part in this is usually nothing at all.
Failover is one layer of a larger answer. It protects you from hardware; it cannot protect you from a mistaken deletion, because replication copies mistakes as faithfully as everything else. That is what backups and rehearsed restores are for, and the layer below both, the storage itself, has its own protections covered in how we prevent silent data corruption. We do not publish availability percentages we have not evidenced; a custom SLA is available for larger rollouts.
Frequently asked questions
Is my data sitting on a single server?
No. The platform's databases are highly available clusters, replicated across nodes, so every change is copied to more than one machine as it happens. A single failure never takes a database offline, because the data already exists elsewhere and a replica can take over.
Is failover automatic, or does someone have to notice first?
Automatic. Our PostgreSQL clusters run Patroni, which continuously watches the primary and, if it stops responding, holds an election and promotes a replica without waiting for a human. Engineers are alerted and investigate the failed node, but your apps are not waiting on us to wake up.
What do applications experience during a failover?
Applications talk to one stable address, fronted by HAProxy, rather than to a specific server. When the cluster fails over behind that address, a typical application sees a brief pause and reconnects to the new primary. The address it knows never changes.
Do all customers share one big database?
No. Each tenant gets its own database instances rather than a shared multi-tenant database. That keeps the blast radius small: a database problem lives inside one tenant rather than rippling across everyone on the platform.
Does replication mean I do not need backups?
No, and be wary of anyone who implies otherwise. Replication protects against hardware failure by copying every change immediately, which means it also copies mistakes immediately. Deletions and corruption replicate just as faithfully as good data. Point-in-time backups cover that, and the platform runs both.