Data Scrubbing & Self-Healing in ZFS

The Problem: Silent Data Corruption (Bit Rot)

Traditional file systems trust underlying storage controllers. However, disk drives, SAS/SATA cables, firmware bugs, and controller memory bit flips can cause data to alter silently on disk without returning an I/O error to the OS.

When a traditional filesystem reads this corrupted block, it delivers garbage data to the application with no warning.

How ZFS Detects & Self-Heals Corruption

 [Application Read Request]
             │
             ▼
      [Read Data Block]
             │
             ▼
   [Compute Checksum] <─── Compare with Parent Pointer's Checksum
             │
     ┌───────┴───────┐
     │ Match?        │
    YES              NO (Corruption Detected!)
     │               │
     ▼               ▼
[Deliver Data]  [Fetch Good Copy from Mirror / Parity VDEV]
                     │
                     ├─► [Deliver Correct Data to Application]
                     │
                     └─► [Automatically Overwrite Corrupted Block on Disk]

1. Merkle Tree Checksums

In ZFS, every block pointer in the metadata hierarchy contains the cryptographic checksum of its child block (Fletcher4, SHA-256, or Blake3). A block cannot be quietly swapped or altered without failing verification.

2. On-the-Fly Healing

If an error is detected during any regular application read:

1. ZFS intercepts the checksum failure.
2. ZFS fetches the correct block from a mirror or calculates it from RAID Levels & ZFS RAID-Z parity.
3. ZFS returns valid data to the user without raising an application error.
4. ZFS writes the repaired block back to the failing physical device in the background.

3. Periodic Background Scrubbing

• zpool scrub <pool>: Traverses every active allocated block across the entire pool, verifying every checksum.
• Regular weekly or monthly scrubs ensure latent corruption on seldom-read files is identified and healed before additional disk failures occur.