Sources: the pool, its flow and its quality
The pool of this first pass, read on 2026-11-02: 33 formal works and 5 grey sources. On 2026-11-03 two formal works were added by forward citation, so the pool is 35 formal and 5 grey. Searches are in protocol; terms in frame.
Shape: sources
A source marked located was found and its record checked, but its text was not opened. A claim that rests on it is marked as such. Below 10 of 20, a grey source is an example of practice only.
Flow
| Stage | Count |
|---|---|
| Formal strings run | 14 (S1 to S7, S11 to S14 and the variants S2b, S3b, S3c); S14 screened, nothing included |
| Grey strings run | 4 web strings and 4 direct visits |
| Titles screened | up to 10 per string; the exact number was not logged |
| Forward citations, 2026-11-03 | 3 works screened: 378 titles and 6 abstracts; 2 added (protocol, section 11) |
| Reached through a string or a direct visit | 30 of the 38; several also named by the brief |
| Reached only from the brief's leads or the reviewer's recall, then checked against a bibliographic record | 8 of the 38 (Simon 1969, Webster 2002, Garousi 2019, Kitchenham 2009, Polya 1945, Kellogg 1988, Flyvbjerg 2006, Booth 2016) |
| Read in the text or the sections cited | 5 formal, 4 grey |
| Read at abstract level | 13 formal: 11 from the first pass, and Hohn 2002 and Mettler 2014 |
| Read only through secondary summaries | 1 formal (Hembree 1992) |
| Located, text not opened | 16 formal, 1 grey |
| Grey scored | 4 by the first reviewer; after the second rating and the rereads, two at 10 or more and two below |
Formal sources
Quality is the evidence grade, with what the grade rests on. A number from an abstract or a summary is marked (secondary).
| Key | Kind | Found via | Read | Grade | Used in |
|---|---|---|---|---|---|
| Simon 1969 | formal | lead (brief) | located | convention, book not opened | frame |
| Hevner 2004 | formal | lead (brief); S1 | located; seen through Peffers 2006 | convention, guidelines not opened | frame, review |
| Peffers 2006 | formal | lead (brief); S1; S2 | full text | convention: a synthesis of seven process models, demonstrated on two cases | frame, review, position, baseline |
| Peffers 2007 | formal | lead (brief); S2 | located; journal version of Peffers 2006 | convention, as Peffers 2006 | frame, state of the art |
| Gregor 2013 | formal | lead (brief); S1 | located | convention, not opened | frame |
| Wieringa 2014 | formal | lead (brief); S1 | located | convention, book not opened | frame |
| Webster 2002 | formal | lead (brief) | located | convention, not opened | frame, position |
| Garousi 2019 | formal | lead (brief) | method, as the family cites it | convention: guidelines | protocol |
| Kitchenham 2009 | formal | lead (brief) | located; not opened | review of reviews, not graded further | review strand 0 |
| Lund 2016 | formal | S3; S3c | first line of the abstract | convention: an argument; its call is that no new study begins without a systematic review | review Q1, position |
| Chalmers 2009 | formal | S3 | located; abstract not retrieved | review of evidence, not opened | review Q1 |
| Macleod 2014 | formal | S3 | located; abstract not retrieved | review of evidence, not opened | review Q1 |
| Glasziou 2014 | formal | S3 | abstract | measured, descriptive: reports of studies (secondary) | review Q1 |
| Boehm 2001 | formal | S2b | full text (three pages) | expert report, not a measurement; cites studies | review Q1, against |
| Menzies 2017 | formal | S12 | abstract page | measured: 171 projects (secondary) | against |
| Fairley 2005 | formal | S3b | abstract | expert report | review Q1 |
| Hofmann 2001 | formal | S11 | abstract | expert report; the 75 percent is Jones's, quoted (secondary) | review Q1 |
| Eveleens 2010 | formal | S11 | abstract | measured: 5,457 forecasts of 1,211 projects (secondary) | against |
| Nawaz 2022 | formal | S3b | abstract | review of 23 studies, global software development only (secondary) | review strand 0, Q1 |
| Dybå 2008 | formal | S11 | located; abstract not retrieved | review of evidence, not opened | review strand 0, Q1 |
| Serrador 2015 | formal | S11 | summaries of the abstract | measured, correlational: 1,002 projects (secondary) | review Q1 |
| Polya 1945 | formal | lead (brief) | located | convention, book not opened | frame, review Q1 |
| Schoenfeld 1985 | formal | lead (brief); S4 | located | measured, by case study; book not opened | review Q1 |
| Hembree 1992 | formal | S4 | located; findings from two secondary summaries | measured: meta-analysis of 487 reports (secondary) | review Q1, against |
| Kellogg 1988 | formal | lead (brief) | summaries of the abstract | measured: experiments, judged quality (secondary) | review Q2 |
| Noble 2009 | formal | S5 | text, through a summary | convention: a practice guide | review Q3, baseline |
| Sandve 2013 | formal | S5 | rule titles | convention: a practice guide | review Q3 |
| Wilson 2017 | formal | S5 | sections 3a, 4a and 5f, and the overview, in the text | convention: a practice guide; over 11,000 people since 2010, in the text | review Q3, baseline |
| Parnin 2010 | formal | S13 | located; abstract not retrieved | measured, by study of programmers; not opened | review Q3 |
| Altmann 2002 | formal | S13 | abstract | measured by simulation against latency and error data | review Q3 |
| Prechelt 2001b | formal | S6 | abstract | measured: a controlled experiment (secondary) | review Q1, against |
| Hayes 1997 | formal | S6 | located; not opened | measured, descriptive; report not opened | review Q1 |
| Flyvbjerg 2006 | formal | lead (brief) | located; not opened | convention, not opened | position |
| Hohn 2002 | formal | forward from Hembree 1992 | abstract | measured: two controlled experiments, 223 elementary students (secondary) | review strand C |
| Mettler 2014 | formal | forward from Peffers 2006 | abstract | convention: a review of experiments in design science and a proposed evaluation framework, not tested (secondary) | review strands 0 and A, position |
Grey sources
Scores follow Garousi 2019, Table 7: authority A (of 4), methodology M (6), objectivity O (4), date D (1), position P (1), novelty N (2), impact I (1), outlet tier T (1). The first reviewer scored all four. A second rater then scored them blind, and each disagreement of more than 3 points was resolved by rereading the whole text; see protocol, section 12.
| Key | Kind | Found via | Read | Score of 20 | Used in |
|---|---|---|---|---|---|
| Schwaber 2020 | grey tier 2 | direct | the whole guide, read on 2026-11-03 (first pass: through a summary) | final 10.0; first reviewer 12; second rater 8: A 2.5, M 2, O 1.5, D 1, P 0, N 1.5, I 1, T 0.5. At the threshold | review Q1, state of the art |
| Singer 2019 | grey tier 1 | direct | the web edition's contents page, read on 2026-11-03 (first pass: chapter list and definitions) | final 9; first reviewer 12.5; second rater 0: A 2, M 4, O 0, D 0, P 0, N 2, I 0, T 1. Below the threshold: an example of practice only | review Q1 (example) |
| Chapman 2000 | grey tier 1 | S8 | the first 25,000 characters, the introduction and the contents, read on 2026-11-03 (first pass: sections 1.1 and 1.4, the contents) | final 14.5; first reviewer 15.5; second rater 11.5: A 3, M 5, O 1.5, D 1, P 0, N 2, I 1, T 1 | review Q1, baseline |
| Ubl 2020 | grey tier 3 | direct | the whole post, read on 2026-11-03 (first pass: first section) | final 9.5; first reviewer 9; second rater 3: A 1, M 4, O 1, D 1, P 0, N 1.5, I 1, T 0. Below the threshold: an example of practice only | review Q1 (example) |
| Booth 2016 | grey tier 1 | lead (brief) | located; not opened, not scored | not scored: a practitioner book | frame |
Keys
Garousi 2019, Polya 1945 and Ubl 2020 are keys the family already holds, with the same work. Prechelt 2001b is a different work from the family's other Prechelt entry of 2001, so it takes a letter.