Protocol: how the pool of sources was built

The searches, screening and scoring of the system-design review are set out here so that they can be rerun. The rules that every review shares are in the evidence method. All searches of this review ran on 2026-10-29. Its terms of architecture are defined in architecture-descriptions; module-level design is in architecture-design.

1. Why grey literature

Garousi 2019 Table 4 lists seven questions for grey literature, where a single yes suffices (the rule). For system design every answer is yes.

# Question Answer for this topic
1 Complex, not solvable from formal literature alone? yes: the main forms (primers, guidelines, postmortems) come from practice
2 Lack of volume or quality of evidence? yes: no controlled study found for estimation or for most principles
3 Context important? yes: load, failure model and team decide the design
4 Corroborate research with practice? yes: failure studies against postmortem practice
5 Challenge practice with research, or the reverse? yes: primers state CAP as a binary choice (position.md)
6 Would a synthesis serve both communities? yes
7 Large practitioner volume? yes: 376k stars on one primer (state-of-the-art.md)

2. Questions

Id Question
Q1 Which principles of computer system design are stated, and which are tested?
Q2 What do consistency and availability trade-offs say, as theory and as practice?
Q3 What do failure studies say designs get wrong?
Q4 How are capacity and load estimated before building?
Q5 How are interfaces between services designed, and what do usability studies show?
Q6 How is a design checked before it is built?
Q7 What do systems engineering standards ask of design?
Q8 What do interview books and primers teach, and how does it compare with research?

3. Existing reviews (Garousi Guideline 2)

  • Searched first, by relevance searches on Semantic Scholar (four returned listings) and by web searches, on these topics (R1 to R6).
  • R1: multivocal or systematic review of site reliability engineering, chaos engineering.
  • R2: systematic review of consistency models; of REST and microservice API design rules.
  • R3: systematic review of cloud outages, incidents, postmortems.
  • R4: systematic mapping of formal methods, TLA+, in distributed design.
  • R5: systematic review of capacity planning (junk results, see section 10).
  • R6: systematic review of large language models and architecture design.
  • Found: 7 reviews; all read at least to the abstract, and Bogli 2024 and Costa 2025 beyond it. They are placed in review.md strand 0.
  • Boundary: a review of architecture design with models (Schmid 2025, arXiv 2505.16697) was located and left to architecture-design.

6. Stopping rule

  • Fixed before searching: effort bounded at N = 6 results per relevance search and 30 per Q/A string, with saturation at k = 5 for web searches.
  • Formal: effort bounded for the relevance searches; evidence exhaustion for leads (each checked).
  • Practice: evidence exhaustion; the named producers were all visited.
  • Q/A sites: see 5.1.

7. Selection

  • Include: a source on principles, trade-offs, failures, estimation, interfaces, checking, standards or teaching of designing a running system; English; readable on the search date.
  • Exclude:
    • module decomposition and styles (architecture-design);
    • specification languages in general and contracts (algorithm-design);
    • hardware computer architecture (the benchmarks ArchEval and QuArch);
    • vendor pages selling a product with no design content;
    • sources whose abstract could not be read and whose title does not settle relevance ("located only").
  • Grey sources must also pass section 8.

8. Quality of grey sources (Garousi Guideline 11, Table 7)

  • For the 20 criteria, the tier scale and the threshold of 10 of 20, see the evidence method.
  • Tiers as applied here: books, magazines, theses, standards and vendor guidance pages are tier 1. Presentations, project documentation, Q/A and wiki pages are tier 2. Blogs are tier 3.
  • Impact is judged from stars, citations, views or votes.
  • None of the grey sources fell below the threshold, so none is only an example of practice.
  • A company page is its owner's claim about itself; the text says so.

9. Extraction and synthesis

  • Extraction and synthesis follow the evidence method. Each number carries its location, and "(secondary)" when it came from an abstract or summary.
  • The synthesis runs by question, Q1 to Q8; each claim traces to a row of sources.md.

10. Limits

  • The limits shared by every review are in the evidence method.
  • One reviewer, no second rater; the scores are one person's.
  • Snowballing was one level, and forward only one page of 100 records.
  • Seven formal sources are known by title and venue only after the second pass; sources.md says which.
  • Q4 (capacity estimation) returned junk (604 hits for "capacity planning" and software, mostly hardware and IT). A better string was not found in the time, so absence of a study is not proof.
  • Semantic Scholar answered some queries with errors on the day; those queries were repeated or replaced, and counts are from the successful calls.