Sources: the pool, its flow and its quality
The pool, read on 2026-11-02 and extended on 2026-11-03 by forward citation: 42 sources, 34 formal and 8 grey.
Shape: sources
- Searches are in protocol; terms in frame.
- Below 10 of 20 a grey source would be an example of practice only.
- After the second rating and the rereading (protocol section 12), Hashimoto 2026 is below at 8.5.
- It was already used as an example.
Flow
| Stage | Count |
|---|---|
| Strings run (formal F1 to F4, grey G1 to G5) | 9 |
| Forward citation: three seed works, 4,103 citing works listed, titles screened by regex (protocol section 11) | 5 added |
| Formal works recalled, then verified by DOI or arXiv id | 29 |
| Formal works found by forward citation of three seeds (2026-11-03), abstract read | 5 |
| Grey sources opened | 9; 8 kept, 1 dropped (HTTP 403) |
| Read in full or in the sections cited | 15: 7 formal, 8 grey |
| Abstract read | 22, all formal |
| Located only (metadata checked) | 5, all formal |
| In the review | 42 |
Formal sources
How found: R recalled and verified; E reading list of the Empirical Standards; C cited in a read source; F forward citation of a seed (2026-11-03); W web search. Read: full, sections, abstract, located.
| Key | Found | Read | Grade | Used in |
|---|---|---|---|---|
| Runeson 2009 | R, W | full | convention, from existing checklists and the authors' experience | baseline, review A |
| Runeson 2012 | E | located | convention | review A |
| Yin 2018 | E | located | convention | frame, review A |
| Flyvbjerg 2006b | R | abstract | argument | review A, against |
| Kitchenham 1995 | E | abstract | convention | review A |
| Kitchenham 2002 | R | abstract | convention, from medical guidelines | review A |
| Stol 2018 | R | abstract | convention, taxonomy | review A |
| Ralph 2020 | R | full, two standards | convention, community consensus | baseline, review A |
| Baltes 2022 | E | full | measured, descriptive, a review of sampling | review A, against |
| Lethbridge 2005 | E | located | convention, taxonomy | frame |
| Garousi 2019 | R | sections | convention, method | protocol |
| Rainer 2019 | R | abstract | supported with caveat, a review | review E |
| Kalliamvakou 2014 | E | full | measured, descriptive | review B |
| Kalliamvakou 2016 | C | located | measured, descriptive | review B |
| Bird 2009 | R | abstract | measured, descriptive | review B |
| Bird 2009b | R | abstract | measured | review B |
| Hindle 2008 | R | abstract | measured, descriptive | review B |
| Kagdi 2007 | R | abstract | supported with caveat, a survey | review B |
| Collier 1996 | W | abstract | convention | review C |
| Birk 2002 | W | abstract | convention | review C |
| Dingsøyr 2005 | R | located | supported with caveat, a review | review C |
| Washburn 2016 | R | abstract | measured, descriptive | review C |
| Gagnier 2013 | R | abstract | convention, consensus of 27 people | review A |
| Tate 2016 | R | abstract | convention, consensus | review A |
| Becker 2025 | R | full, sections | measured, a randomised trial | review D |
| Korn 2026 | F | abstract | measured, descriptive; the guideline itself is convention | review A |
| Lenberg 2023 | F | abstract | convention, an argument and a literature survey | review A |
| Pinna 2026 | F | abstract | measured, descriptive | review D |
| Rainer 2024 | F | abstract | measured, descriptive | review A |
| Song 2026 | F | abstract | measured, descriptive, from usage data | review D |
| Peng 2023 | R | abstract | measured, a controlled experiment | review D |
| Ziegler 2024 | R | abstract | supported with caveat, owner's survey | review D |
| Li 2025b | R | sections | measured, descriptive, a dataset | review D |
| Watanabe 2025 | R | abstract | measured, descriptive | review D |
Grey sources
Quality is Garousi Table 7, out of 20: authority 4, methodology 6, objectivity 4, date 1, position 1, novelty 2, impact 1, outlet 1. Tier 1 books, 2 calls and lists, 3 blogs.
| Key | Kind | Found | Score | Subtotals | Used in |
|---|---|---|---|---|---|
| ICSE 2003 | tier 2, conference call | G3 | 11 | 3, 3, 2, 1, 0, 1, 0.5, 0.5 | baseline, review A |
| USENIX 2010 | tier 2, conference call | G3 | 10 | 3, 2.5, 2, 1, 0, 0.5, 0.5, 0.5 | review A |
| Beyer 2016 | tier 1, company book | R | 15 | 4, 4.5, 1.5, 1, 0.5, 1.5, 1, 1 | review C |
| Luu 2026 | tier 2, community list | R | 12 | 3, 3, 2.5, 0.5, 0.5, 1, 1, 0.5 | review C |
| Carlini 2026 | tier 3, company post | G2 | 15 | 4, 5, 1.5, 1, 0.5, 2, 1, 0 | review D |
| Lin 2026 | tier 3, company post | G2 | 12.5 | 3.5, 4, 1.5, 1, 0, 1.5, 1, 0 | review D |
| Hashimoto 2026 | tier 3, personal blog | R | 8.5 | 1, 3, 1.5, 1, 0, 1, 1, 0 | review D, as an example |
| Git 2026 | tier 2, tool documentation | R | 10.5 | 3, 2, 2.5, 1, 0, 0.5, 1, 0.5 | review B |
- Changed on rereading (2026-11-03), each where the second rater differed by more than 3 points. The reason is the page text, not the rater's score:
- ICSE 2003, 12.5 to 11: objectivity 3.5 to 2. A call reports no data, so conclusions cannot follow it (0), and its organisers have an interest in submissions (half).
- USENIX 2010, 11 to 10: objectivity 3 to 2, for the same reasons. It stays at the threshold.
- Hashimoto 2026, 10.5 to 8.5: authority 3 to 1. The page names no organisation. It names Ghostty, the author's own product, as other work (half), and states experience (half). The earlier 3 used knowledge of the author from outside the page.
- Git 2026, 12 to 10.5: authority 4 to 3, objectivity 3.5 to 2.5, date 0.5 to 1. The page has no named author (other work 0). A reference page has no conclusions (0), and the project documents its own tool (half). The release history gives dates for the version shown (1).
- A company post about its own work is its owner's claim about itself: Carlini 2026 and Lin 2026 carry a vested interest, scored as such.
- Hashimoto 2026 reports no measurement; its numbers are self-reported.
Not scored
None.