Protocol: how the pool of sources was built

This file records the search for the review of architecture descriptions. A reader can repeat the search from it. The review is a multivocal literature review (MLR, Garousi 2019).

Grey literature, which the review includes, is defined in the design-decisions frame. All searches of the review were run on 2026-10-28 by one reviewer.

1. Why grey literature

Garousi 2019 Table 4: one "yes" suggests including grey literature (the rule). All seven answers are yes for this topic.

# Question Answer for this topic
1 Is the subject complex and not solvable by the formal literature alone? Yes: the templates in use (arc42, C4, ARCHITECTURE.md) come from practice.
2 Is volume or quality of evidence lacking? Yes: one controlled comparison of formats (Ernst 2023).
3 Is context important? Yes: readers, system size and team differ.
4 Is the goal to corroborate research with practice? Yes: reader surveys against vendor surveys.
5 Is the goal to challenge practice with research? Yes: templates are taught without trials.
6 Would a synthesis serve both communities? Yes.
7 Do many practitioner sources exist? Yes: templates, blogs, project guides.

2. Questions

  • Q0: which reviews of this field exist, and what do they cover?
  • Q1: how is a description divided (views, sections)?
  • Q2: what does a reader see first?
  • Q3: how many levels of abstraction?
  • Q4: one document, or one per task?
  • Q5: how does a reader find information in it?
  • Q6: how is it kept true?
  • Q7: what is its effect on understanding?
  • Q8: what do projects have?
  • Q9: what do machine readers need?

3. Existing reviews (Garousi Guideline 2)

Searched before anything else; each is placed in strand 0 of the review.

  • Strings F6, F4, W1, W2 and O1 below, plus the reviews that appeared as citing papers.
  • 19 secondary studies were kept; none has as its subject the organisation of a description for its reader.
  • None was read in full: ten through an abstract or a search summary, nine by title only. The review marks each.

6. Stopping rule

Stated before searching.

  • Formal: effort bounded. The first 25 results of the review string (F6) and the first 8 to 14 of every other string, sorted by citations; snowballing stops after one pass that adds no new work on the questions.
  • Grey: effort bounded at N = 10 results per query, with saturation at k = 3. A query stops being read after 3 consecutive results that add no new concept; vendor blogs repeat each other.
  • Q/A sites: effort bounded at N = 30 results by votes per string and site, with saturation at k = 10; stated in the subsection Q/A sites, where the search stopped after string AD15.
  • Named producers (arc42, C4, the SEI books) are read to exhaustion, since the pool is small.

7. Selection

Include if all hold:

  • The subject is how a description of a system's architecture is organised, entered or read, or a review of that.
  • The source states who wrote it and when, or can be dated.
  • It is in English.
  • Its content was read in full, or the review says which part was read.

Exclude if one holds:

  • E1: notation and architecture description languages (review system-diagrams).
  • E2: decision records (review design-decisions).
  • E3: enterprise architecture documentation, another object.
  • E4: marketing with no architecture content.
  • E5: a figure no one could trace to a source.
  • E6: the source could not be read (paywall or bot protection); it is listed as found, not read.
  • E7: documentation in general, with no architecture content.
  • E8: tools that generate or recover documentation, with no evaluation by readers.
  • E9: reviews whose subject is another architecture topic (evolution, evaluation, analysis).

Formal sources carry the evidence grade of the standard (measured, supported with caveat, convention). Grey sources are scored in section 8.

8. Quality of grey sources (Garousi Guideline 11, Table 7)

Twenty criteria, each scored 1, 0.5 or 0, in eight groups:

Group Criteria Max
A authority of the producer reputable publisher; author tied to a reputable organisation; author's other work; expertise 4
M methodology clear aim; stated method; authoritative references; limits stated; specific question; particular case 6
O objectivity balanced; objective; no vested interest; conclusions follow the data 4
D date clearly stated date 1
P position related sources linked or discussed 1
N novelty adds something; strengthens or refutes a position 2
I impact citations, backlinks, adoption signals, aggregated 1
T outlet type tier 1 = 1, tier 2 = 0.5, tier 3 = 0 1
  • Threshold: 10 of 20 to include.
  • A source below it is kept only as an example of practice, never as support for a claim; sources says which, and the text that names it says so.
  • A part of a source that was not seen scores 0 on its criteria.

9. Extraction and synthesis

  • Extracted per source: kind, what it covers, numbers (read in the primary or marked secondary), stated limits, grade.
  • Synthesis is narrative by question, traceable through the "used in" column of sources.
  • A vendor or owner page is read as its owner's claim about itself.

10. Limits

  • A second rater re-scored 4 of this topic's grey sources blind (method): 3 of 4 decisions agree; the arc42 site (Starke 2026) scores 9.5 and 12 and stays an example of practice.
  • Search engines rank and personalise results and change over time; the dates above are the only anchor.
  • The formal engine matched loosely and the counts are not those of a curated database.
  • The Helsinki thesis and the Gothenburg essay were read through their abstracts only (bot protection blocked the full text).
  • Some reviews were read through abstracts or search summaries and are marked (secondary).
  • Recorded talks were not watched.
  • Q/A threads of architecture-descriptions: one scorer, no second rater.