Protocol: how the pool of sources was built

This file states how another reviewer can repeat the search, screening and scoring of the design-decisions review. It follows Garousi 2019, a multivocal literature review (MLR): formal and grey literature in one pool. All searches of the review ran on 2026-10-28. The architecture terms it uses (viewpoint, architectural knowledge) are defined in architecture-descriptions.

1. Why grey literature

Garousi 2019 Table 4: one "yes" suggests including grey literature (the rule). All seven answers here are yes.

# Question Answer for this topic
1 Complex, not solvable from formal literature alone? yes: the main forms (ADR, design doc, RFC) come from practice
2 Lack of volume or quality of evidence? yes: two controlled experiments on documents in 20 years of work (Tofan 2014; Nogueira 2026)
3 Context important? yes: company size and process decide which form fits
4 Corroborate research with practice? yes
5 Challenge practice with research, or the reverse? yes: practice runs ahead of the evidence (position.md)
6 Would a synthesis serve both communities? yes
7 Large practitioner volume? yes: 875 repositories tagged adr (state-of-the-art.md)

2. Questions

Id Question
Q1 What does a decision record hold?
Q2 Does recorded rationale help later decisions?
Q3 How is a proposal structured?
Q4 How is a proposal judged, and why is it declined?
Q5 Do people adopt and keep ADRs?
Q6 Why is rationale not captured?
Q7 Which rationales drive decisions?

3. Existing reviews (Garousi Guideline 2)

  • Searched first, by three web searches (R1 to R3) and by the lead the commissioner named.
  • R1: "systematic mapping study architectural decisions" or "past and future of software architectural decisions".
  • R2: "architecture decision records systematic literature review OR mapping study".
  • R3: "multivocal literature review architecture decision documentation" (formal side: strings F2, F6).
  • Found: 9 reviews and maps; 2 read in full, 3 by abstract or page summary, 4 by title only. They are placed in review.md strand 0.

6. Stopping rule

  • Stated before searching: effort bounded, N = 10 results per string, with saturation, k = 5.
  • Saturation means: stop a string after 5 consecutive results that add no new concept. G3, G5 and G7 stopped this way, since they returned only known sources or off-topic pages.
  • Formal: evidence exhaustion for F2, F3, F6 (small sets); effort bounded (top 15) for the rest.
  • Q/A sites: see the subsection Q/A sites; the search stopped after string DD18.

7. Selection

  • Include: a source on the form, content, use or adoption of documents that propose or record software design decisions; English; readable on the search date.
  • Exclude:
    • tools or LLM generators with no bearing on document form (adjacent only, Gupta 2026);
    • vendor posts selling a documentation product;
    • duplicates and non-software design review;
    • sources whose abstract could not be read and whose title does not settle relevance (kept as "located only").
  • Grey sources must also pass section 8.

8. Quality of grey sources (Garousi Guideline 11, Table 7)

  • 20 criteria, each 1, 0.5 or 0: authority of the producer (4), methodology (6), objectivity (4), date (1), position with respect to related sources (1), novelty (2), impact (1), outlet type (1).
  • Outlet tier: books, magazines, theses, government and standards documents, vendor guidance pages = tier 1 (1); presentations, project documentation, wiki pages = tier 2 (0.5); blogs = tier 3 (0).
  • Impact is judged from stars, citations or the size of the user base, as the source allows.
  • Threshold: 10 of 20. Below it a source stays only as an example of practice, never as support for a claim; the text that cites it says so.
  • Scores, with subtotals, are in sources.md.

9. Extraction and synthesis

  • Extracted per source: kind, year, the claim, the evidence offered, any number (marked "(secondary)" if not read in the primary).
  • Synthesis is narrative, by question Q1 to Q7, and every claim traces to a row of sources.md.

10. Limits

  • A second rater re-scored 7 of this topic's grey sources blind (method): all 7 decisions agree. The second scores were equal or higher, so the scores of exactly 10 (Microsoft 2026 ADR, Rust 2026b) sit near the threshold on one side only.
  • Search engines personalise and change; counts will differ on a repeat.
  • Semantic Scholar and arXiv counts are not phrase-exact; the OpenAlex counts, taken a day after the search, are.
  • Forward snowballing was one page; backward snowballing covered two works.
  • Many formal sources were read by abstract only; sources.md says which.
  • This search did not find a study that measures reviewers reading proposals; that is not proof none exists.
  • Q/A threads of design-decisions: one scorer, no second rater.