The failures you hear about are not the dangerous ones.
A dead disk announces itself. The dangerous failure is the quiet one: a block that degrades in place, a write that lands in the wrong spot, and a storage system that keeps serving the damaged result without a murmur. This page explains how the Node Platform's storage detects and repairs silent corruption. It is part of our Under the hood series, expanding on the storage section of How the platform is built.
Silent corruption, in plain English
Storage is physics, and physics is not perfect. Magnetic domains weaken, flash cells leak charge, cables and controllers occasionally corrupt or misplace data in transit. Individually these events are rare; across many terabytes and many years they are a certainty. The industry name for the slow version is bit rot.
The uncomfortable part is the "silent". A conventional filesystem mostly trusts the disk: ask for a block, get a block, pass it along. If the block has quietly changed since it was written, nothing complains. The corruption simply sits there until a person opens the file, and a file nobody opens for two years can be broken for two years before anyone finds out, at which point the backups of it may be broken too.
Checksums, end to end
The fix is for the filesystem to stop trusting and start verifying. ZFS, the filesystem our storage is built on, stores a checksum, a small fingerprint of the data, with every block it writes, kept separately from the block itself. On every read, the data is fingerprinted again and compared. If they do not match, the data has changed, and ZFS knows before you do. The OpenZFS project calls end-to-end checksums a key differentiator over other RAID implementations and filesystems, and it is the property this whole page rests on: corruption on this storage is detectable by design, not by luck.
Detection alone would only turn silent corruption into loud corruption. The useful half is repair. Because the storage holds redundant copies of data across disks, a block that fails its checksum is re-fetched from a good copy, served correctly, and the damaged copy is rewritten. This is what self-healing means: the bad read is fixed in flight, and the first sign you see of it is a counter in an engineer's dashboard rather than a broken spreadsheet.
Copy-on-write, and why snapshots are cheap
ZFS never overwrites live data in place. Every change is written to fresh space first, and only then does the filesystem point to the new version: copy-on-write. Two things follow. A power cut mid-write cannot half-overwrite your old data, because the old data was never touched. And snapshots, frozen pictures of a volume at a moment in time, cost almost nothing, because a snapshot is simply a promise not to recycle the old blocks. Snapshotting is therefore something the platform does routinely rather than sparingly, which gives every volume a set of recent points to roll back to.
Scrubs: checking the files nobody is reading
Checksums verify data as it is read, which leaves a gap: data nobody reads is never verified. Scrubbing closes it. A scrub walks the pool deliberately, verifying every block against its checksum and repairing what it finds from redundancy, catching what OpenZFS describes as latent media degradation before it matters. It is the difference between finding rot when a scheduled process checks, and finding it when your accountant opens the 2024 year-end file.
RAID is not a backup
A rule worth engraving. Redundancy, RAID or otherwise, answers one question: can the system keep running when a drive dies? It says nothing about the other ways data is lost. Delete a file and redundancy diligently preserves the deletion. Encrypt a volume with ransomware and every disk holds a perfect copy of the damage. Checksums and scrubs guard the integrity of what is stored; they do not decide whether what is stored is what you meant. That is the job of backups, and restores that have actually been rehearsed, which are a separate line of defence on this platform, not a substitute for this one.
How the Node Platform runs it
Persistent data on the platform lives on TrueNAS, built on the ZFS filesystem, which as How the platform is built puts it gives us checksummed integrity, cheap snapshots and efficient replication for free. The hardware underneath is our own, in a UK datacentre, so there is no further provider beneath us whose storage practices we would have to take on faith.
An app asks for a volume of a given size and performance tier, fast tiers on SSD, standard tiers on spinning disk, and the platform provisions it from TrueNAS; the checksumming, snapshots and self-healing described above come with every volume, on every tier, rather than as a premium feature. The same efficient replication feeds the wider protection story: storage snapshots are the first line of defence, ahead of database failover and the nightly encrypted backups and off-site disaster-recovery copy described in how your data is backed up.
The point of all of it is a boring one, and boring is the goal: the file you wrote in 2024 and open in 2027 is the file you wrote, and if physics disagreed at any point in between, the storage noticed first.
Frequently asked questions
What is silent data corruption, or bit rot?
It is data changing on disk without anything reporting an error: a charge leaks, a sector degrades, a controller misdirects a write. Most filesystems trust whatever the disk returns, so the damage surfaces only when someone finally opens the file, possibly years later. ZFS, the filesystem under our storage, checksums every block so corruption is detected the moment the data is read.
How would you know a file from 2024 has not quietly rotted?
Two ways. Every read is verified against its checksum, so a corrupted block can never be served silently. And because files nobody reads would otherwise never be checked, scheduled scrubs walk the pool and verify data proactively, repairing latent degradation from redundant copies before anyone is waiting on that file.
Is RAID not enough to protect our data?
RAID keeps storage available when a drive fails, which is necessary but narrow. Plain RAID cannot tell which copy is correct when two disks disagree, and no form of redundancy protects you from deletion or ransomware, because it faithfully preserves those too. ZFS adds checksums so the correct copy is knowable, and backups cover what redundancy never can.
What are snapshots, and are they backups?
A snapshot is a frozen, read-only picture of a volume at a moment in time. Because ZFS copies on write, snapshots are cheap to take and keep. They are excellent for quick rollback, but they live on the same system as the data, so they are not backups. Separate nightly encrypted backups and an off-site copy cover that.
Whose hardware is this actually on?
Ours. The storage layer runs on TrueNAS, on servers we own in a UK datacentre, described openly in How the platform is built. We chose the stack, we can inspect every layer of it, and your data does not sit behind another provider's abstraction.