Design the failure paths

Claim: a design names how each part fails and how each failure is handled, and its plan tests those paths.

What it asks

  • In the Consequences, list the ways the design can fail: a part down, a network partition, a slow dependency.
  • For each, say how it is detected and handled, and whether data can be lost.
  • In the plan, test the error paths, not only the main path.

Why

  • In five distributed systems, 92% of 48 catastrophic failures came from incorrect handling of non-fatal errors, and 58% of them could have been caught by simple tests (Yuan 2014).
  • Of 136 failures under network partitions, 80% were catastrophic and 90% silent (Alquraan 2018).
  • Of the 88 taken from issue trackers, 47% needed a redesigned mechanism, and design flaws took 2.5 times longer to fix (Alquraan 2018).

Limits

  • The studies cover distributed data systems; other kinds of software were not studied the same way.
  • They describe failures; no trial shows that writing the failure paths down prevents them.