Sources
Every source the two passes on reports included or set aside, each with its grade or its score out of 20 (scores). The searches behind them are in protocol.
Flow
| Stage | Formal | Grey |
|---|---|---|
| Listed by the strings (323 in all) | 150 (E 47 + F 103) | 173 (G) |
| Included | 11 | 7 |
| Read in full (or the part opened) | 11 (Nesi 2012, chapter 1; and, on 2026-10-29, Gardner 2013, Mullins 2002, Kiley 2004, Paltridge 2002, Bunton 2005, Holbrook 2007, Bourke 2013, Lorch 1996, Pinelli 1984, Pinelli 1997) | 6 (NISO 2010, Pinelli 1982, TSE 2017, EOST 2026, INSPE 2021, Sheffield 2017) |
| Read as abstract or record page | 0 | 0 |
| Metadata verified, nothing else read | 0 | 1 (ISO 1986) |
- The 11 formal include Gardner 2013, Nesi 2012, Mullins 2002, Kiley 2004, Paltridge 2002, Bunton 2005, Holbrook 2007, Bourke 2013, Lorch 1996, Pinelli 1984 and Pinelli 1997.
- The six that had metadata only (Kiley 2004, Paltridge 2002, Bunton 2005, Holbrook 2007, Bourke 2013, Pinelli 1997) were read in full on 2026-10-29 (full text), like Gardner 2013, Mullins 2002 and Lorch 1996. Kiley 2004 prints 2004 in the journal and 2005 in the DOI and copyright; the key stays.
- The 7 grey are NISO 2010, ISO 1986, TSE 2017, EOST 2026, INSPE 2021, Sheffield 2017 and Pinelli 1982.
By string
| String | Listed | Read | Added |
|---|---|---|---|
| E1 BAWE genres | 10 | 2 | 2 (Gardner 2013, Nesi 2012) |
| E2 thesis examiners | 10 | 1 | 2 (Mullins 2002, Kiley 2004) |
| E3 executive summary effect | 9 | 0 | 0 |
| E4 internship report research | 9 | 0 | 0 |
| E5 conclusion first or last | 9 | 0 | 0 |
| F1 Paltridge | 20 | 0 | 1 |
| F2 Bunton | 9 | 0 | 1 |
| F3 examiner reports | 9 | 0 | 2 (Holbrook 2007, Bourke 2013) |
| F4 headings | 9 | 1 | 1 (Lorch 1996) |
| F5 écrits universitaires | 9 | 0 | 0 |
| F6 engineering design report | 9 | 0 | 0 |
| F7 Paltridge structures | 9 | 0 | 0 |
| F8 how examiners read | 9 | 0 | 0 |
| F9 thesis length | 20 | 0 | 0 |
| G1 NISO Z39.18 | 10 | 1 | 1 |
| G2 ISO 7144 | 10 | 0 | 1 (record) |
| G3 rapport de stage grids | 10 | 2 | 2 (TSE 2017, EOST 2026) |
| G4 mémoire Grenoble | 10 | 1 | 1 (INSPE 2021) |
| G5 plain English | 10 | 0 | 0 |
| G6 CTI | 30 | 1 (a course sheet, left out) | 0 |
| G7 Ensimag | 18 | 0 | 0 |
| G8 project report grids | 9 | 0 | 0 |
| G9 readers of reports | 9 | 2 | 2 (Sheffield 2017, Pinelli 1984) |
| G10 Pinelli components | 20 | 1 | 2 (Pinelli 1982, Pinelli 1997) |
| G11 host company | 18 | 0 | 0 |
| G12 mémoire jury | 19 | 0 | 0 |
| Total | 323 | 12 | 18 |
Pages read: 12 in all; Gardner 2013 is counted once.
Excluded
| Reason | Examples | Count |
|---|---|---|
| EC1 blog or relay | blogs on how an examiner reads a thesis; consultancy and editing pages; a VinUni opinion piece on word count | not counted one by one |
| EC3 not retrievable (403 or bot check) | ScienceDirect pages of Paltridge 2002 and Bunton 2005; the Newcastle examiner-report PDF and project page; a Geneseo textbook chapter on industry readers; a DOAJ page on internship reports; a HAL chapter by Delcambre and colleagues (bot check, not bypassed) | 7 |
| EC3 not found | the UGA faculty of economics note on the mémoire (the link served a site page); the ESGT final-project referential (download failed); a CTI text fixing a report format (none found) | 3 |
| Paywalled | ISO 7144 full text | 1 |
| EC4 duplicate | the Pinelli report found by three strings | not counted |
Pool
For a formal source the last column of quality holds the evidence grade. For a grey source it holds the score out of 20 and its eight subtotals, whose letters are defined in the evidence README.
Of 7 grey sources, 2 score below 10: TSE 2017 (9.5) and EOST 2026 (9). Both sources are examples of practice only; no claim leans on them. Another source, Sheffield 2017, scores exactly 10 and is opinion. A further source, ISO 1986, is scored on its catalogue record; its text was not read.
| Key | Kind | Found via | Quality | Used in |
|---|---|---|---|---|
| Gardner 2013 | formal | E1 | measured, descriptive (full text) | review, state-of-the-art |
| Nesi 2012 | formal | E1 | measured, descriptive (chapter 1 excerpt read) | review, state-of-the-art, baseline |
| Mullins 2002 | formal | E2 | supported with caveat (full text) | review, position |
| Kiley 2004 | formal | E2 | supported with caveat (full text; prints 2004, but the copyright year and the DOI say 2005) | review |
| Paltridge 2002 | formal | F1 | supported with caveat (full text) | review, against |
| Bunton 2005 | formal | F2 | supported with caveat (full text) | review |
| Holbrook 2007 | formal | F3 | supported with caveat (full text) | review, academic-report shape |
| Bourke 2013 | formal | F3 | supported with caveat (full text) | review |
| Lorch 1996 | formal | F4 | measured (full text) | review, state-of-the-art, against |
| Pinelli 1984 | formal | G9 | measured, descriptive (full text, 2026-10-29; stated preferences of 838 respondents) | review, against |
| Pinelli 1997 | formal | G10 | measured, descriptive (full text); a 1996 replication of the 1981 survey, usable and citable | listed, not used |
| NISO 2010 | grey tier 1 | G1 | 16.5: A4 M4 O3 D1 P1 N1.5 I1 T1 | review, baseline |
| ISO 1986 | grey tier 1 | G2 | 11 on the catalogue record: A4 M0.5 O3 D1 P0 N0.5 I1 T1 | review |
| Pinelli 1982 | grey tier 1 | G10 | 17.5: A3.5 M5.5 O3.5 D1 P1 N1.5 I0.5 T1 | review, state-of-the-art, against |
| TSE 2017 | grey tier 2 (example of practice only) | G3 | 9.5: A3 M1.5 O2 D1 P0 N1 I0.5 T0.5 | review, academic-report shape |
| EOST 2026 | grey tier 2 (example of practice only) | G3 | 9: A3 M1.5 O2.5 D0 P0 N1 I0.5 T0.5 | review, academic-report shape |
| INSPE 2021 | grey tier 2 | G4 | 12: A3.5 M2.5 O2.5 D1 P0.5 N1 I0.5 T0.5 | review, academic-report shape |
| Sheffield 2017 | grey tier 2 | G9 | 10: A3 M2 O2 D1 P0 N1 I0.5 T0.5 | review, against |
Second pass (2026-10-29): sources added
Rows to append to sources.md. Scores for grey sources use the same eight subtotals: authority A (4), methodology M (6), objectivity O (4), date D (1), position P (1), novelty N (2), impact I (1), outlet T (1).
Flow rows
| Stage | Formal | Grey |
|---|---|---|
| Listed by the strings (371 in all) | 266 (E 112 + F 154) | 105 (G) |
| Citing papers screened by title (snowball) | 304 | 0 |
| Included, new to the pool | 22 | 5 |
| Pool sources upgraded (abstract first; all three then read in full on 2026-10-29) | 3 (Holbrook 2007, Bourke 2013, Mullins 2002) | 0 |
| Read in full or the relevant part | 11 (Golding 2014, Bodnar 2018, Brown 2023, Reddy 2010 and, on 2026-10-29, Holbrook 2008, Carter 2008, Williams 2018, Sharmini 2015, Suchan 1989, Kieras 1978, Kieras 1980; all full text) | 5 (Army 2020 section, UT 2024, ULB 2025, UGA 2026, BTS 2010) |
| Read as abstract (Crossref, OpenAlex or Semantic Scholar copy of the publisher abstract) | 9 new (Hodgson 2020, Chetcuti 2022, Jansen 2013, Koetsenruijter 2022, Spyridakis 2000, Spyridakis 2002, Davey 1990, Bridge 1984, Silyn-Roberts 2002), plus 3 pool sources upgraded | 0 |
| Record or search summary only | 0 | 0 |
| Metadata only, cited through Golding 2014 | 1 (Johnston 1997) | 0 |
- Of 5 grey, 1 scores below 10: BTS 2010 (9). It is an example of practice only, and the low end of the report-weight range (17%) rests on it, so the range without it is 60% to 75%.
- New formal sources, 22 in all: Golding 2014, Hodgson 2020, Chetcuti 2022, Sharmini 2015, Williams 2018, Bodnar 2018, Brown 2023, Reddy 2010, Jansen 2010, Jansen 2013, Koetsenruijter 2022.
- And: Suchan 1989, Spyridakis 2000, Spyridakis 2002, Davey 1990, Bridge 1984, Silyn-Roberts 2002, Kieras 1978, Kieras 1980, Carter 2008, Holbrook 2008, Johnston 1997.
- New grey: Army 2020, UT 2024, ULB 2025, UGA 2026, BTS 2010.
- Jansen 2010 is also held in the shared bibliography under the same key; the review there read it beyond the abstract. It was read in full on 2026-10-29; the paper prints in 2011.
By string
| String | Listed | Read | Added |
|---|---|---|---|
| E6 examiners systematic review | 9 | 1 | 1 (Golding 2014) |
| E7 executive summary experiment | 10 | 1 | 1 (Koetsenruijter 2022) |
| E8 synthèse en tête (FR) | 28 | 0 | 0 |
| E9 text structure reviews | 9 | 0 | 0 |
| E10 rubric reviews | 9 | 2 | 2 (Reddy 2010, Brown 2023) |
| E11 thesis examination reviews 2015-2025 | 18 | 0 | 1 (Sharmini 2015) |
| E12 reading of technical reports | 10 | 0 | 1 (Silyn-Roberts 2002) |
| E13 inverted pyramid reviews | 10 | 0 | 0 |
| E14 placement assessment reviews | 9 | 0 | 0 |
| F10 Mullins full text | 10 | 0 | 0 |
| F11 Carter 2008 | 19 | 0 | 1 (Carter 2008) |
| F12 Holbrook 2007 | 20 | 0 | 0 (abstract read through OpenAlex) |
| F13 Bourke 2013 | 9 | 0 | 0 (abstract read through OpenAlex) |
| F14 capstone rubric reliability | 10 | 1 | 1 (Bodnar 2018) |
| F15 Jansen explanations first | 10 | 0 | 2 (Jansen 2010, Jansen 2013) |
| F16 summary before or after | 9 | 0 | 0 |
| F17 main idea first or last | 19 | 0 | 2 (Davey 1990, Kieras 1980) |
| F18 deductive or inductive | 20 | 0 | 0 |
| F19 Kieras 1978 | 9 | 0 | 1 (Kieras 1978) |
| F20 Britt and Larson | 10 | 0 | 0 |
| F21 Johnston 1997 | 9 | 0 | 0 (Johnston 1997 by metadata) |
| Snowball (forward from Mullins 2002, Jansen 2010, Britt 2003, Kieras 1978) | 304 titles | 0 | 7 (Hodgson 2020, Chetcuti 2022, Williams 2018, Suchan 1989, Spyridakis 2000, Spyridakis 2002, Bridge 1984) |
| Snowball (backward from Golding 2014) | 3 | 0 | 2 (Holbrook 2008, Johnston 1997) |
| G13 internship rubric | 10 | 1 | 1 (UT 2024) |
| G14 grille rapport de stage | 10 | 2 | 2 (ULB 2025, BTS 2010) |
| G15 critères implicites (FR) | 18 | 0 | 0 |
| G16 UFR IM2AG | 9 | 1 | 1 (UGA 2026) |
| G17 mémoire jury (FR) | 9 | 0 | 0 |
| G18 executive summary readability | 10 | 0 | 0 |
| G19 BLUF AR 25-50 | 9 | 0 | 0 |
| G20 AR 25-50 text | 10 | 1 | 1 (Army 2020) |
| G21 Pyramid Principle | 10 | 0 | 0 |
| G22, a book by Lovitts | 10 | 0 | 0 |
Carter 2008 appears under F11 because the string found the record; Holbrook 2008 and Johnston 1997 come from Golding 2014's reference list. Carter 2008 and Holbrook 2008 were read in full on 2026-10-29.
Excluded rows
| Reason | Examples | Count |
|---|---|---|
| EC1 blog or relay | a doctoral blog on how an examiner reads a thesis (read order: abstract, references); a consultancy blog on the Pyramid Principle; vendor blogs on BLUF and executive summaries; a Wikipedia entry | not counted one by one |
| EC2 commercial page | AI-writing and slide-vendor pages on executive summaries | not counted |
| EC3 403 or access denied | the Newcastle examiner-report PDF and project page (again); the Sage page of Jansen 2010; the DTIC technical reports of Kieras (ADA073773); the Tandfonline PDF of the 2024 rubrics paper; the Université de Genève sociology grid (access-denied notice); the Utrecht repository record of Jansen 2010 | 6 (counting the Newcastle PDF and project page once) |
| EC3 bot check, not bypassed | the Simon Fraser repository record of a thesis on summary timing (Anubis page) | 1 |
| EC3 login wall | a UniSA course copy of Mullins 2002 (redirects to a VPN-required page; no sign-in tried) | 1 |
| EC3 not found or not a document | Strasbourg M1 consignes (404 at 2 addresses); a Wageningen rubric page (404); a Springer PDF of Kieras 1980 (paywall page); a Calhoun copy of Suchan 1989 (served as an app shell, no text) | 4 |
| EC3 scanned, not readable | NPS Naval Writing chapter 9 (image-only PDF; no OCR run) | 1 |
| Off topic after reading the record | three studies by Li, by Salmerón and by Kuechler (published in 2015, 2009 and 2006): manual structure for Western and Chinese users, graphical overviews in hypertext, goals in process narratives; and meta-analyses of text structure instruction (instruction, not order) | 4 |
| Search summary only, not in pool | module guides of Nanterre (equal thirds), IUT Caen (report 15 and oral 15), Strasbourg (report half, oral half), Leiden, Wageningen, HKU; claims from them are not used | not counted |
| Paywalled or a book | a book by Lovitts (published 2007); Johnston 1997 (only through Golding 2014) | 2 |
Pool rows
For a formal source the quality column holds the evidence grade and what was read. For a grey source it holds the score out of 20 with the eight subtotals.
| Key | Kind | Found via | Quality | Used in |
|---|---|---|---|---|
| Golding 2014 | formal | E6 | supported with caveat (qualitative systematic review, 30 articles; full text, open access; the chapter 1 or 2 claim is printed on p. 568 with no primary study cited) | review-pass2, requirements-pass2 |
| Hodgson 2020 | formal | snowball (Mullins 2002) | convention, untested (abstract only) | review-pass2 |
| Chetcuti 2022 | formal | snowball (Mullins 2002) | supported with caveat (abstract read) | review-pass2 |
| Sharmini 2015 | formal | E11 | supported with caveat (full text, 2026-10-29) | review-pass2 |
| Williams 2018 | formal | snowball (Mullins 2002) | measured (full text; prints 2019, online 2018) | review-pass2 |
| Bodnar 2018 | formal | F14 | measured (read in full, open access) | review-pass2 |
| Brown 2023 | formal | E10 | measured (read, open access) | review-pass2 |
| Reddy 2010 | formal | E10 | supported with caveat (review; full text) | review-pass2 |
| Jansen 2010 | formal | F15 | measured, ratings (full text; prints in 2011) | review-pass2, requirements-pass2 |
| Jansen 2013 | formal | F15 | measured, ratings (abstract read) | review-pass2 |
| Koetsenruijter 2022 | formal | E7 | measured, weak: nonrandomized pilot (abstract read, Crossref copy) | review-pass2, requirements-pass2 |
| Suchan 1989 | formal | snowball (Kieras 1978) | measured (full text; bundled styles, recall only) | review-pass2, requirements-pass2 |
| Spyridakis 2000 | formal | snowball (Kieras 1978) | measured (abstract read) | review-pass2, requirements-pass2 |
| Spyridakis 2002 | formal | snowball (Kieras 1978) | measured (abstract read) | review-pass2, requirements-pass2 |
| Davey 1990 | formal | F17 | measured, children (abstract read) | review-pass2 |
| Bridge 1984 | formal | snowball (Kieras 1978) | measured (abstract read) | review-pass2 |
| Silyn-Roberts 2002 | formal | E12 | supported with caveat: stated preference, N = 744 (abstract read) | review-pass2, requirements-pass2 |
| Kieras 1978 | formal | F19 | supported with caveat (full text) | review-pass2 |
| Kieras 1980 | formal | F17 | supported with caveat (full text) | review-pass2 |
| Carter 2008 | formal | F11 | supported with caveat (full text; informal panels of 23 Auckland faculty, notes by the author) | review-pass2 |
| Holbrook 2008 | formal | backward (Golding 2014) | supported with caveat (full text; 804 theses, 2121 reports) | review-pass2 |
| Johnston 1997 | formal | backward (Golding 2014) | metadata only; claims secondary through Golding 2014 | review-pass2 |
| Mullins 2002 | formal (pool) | first pass | upgraded: abstract read through OpenAlex, then full text read on 2026-10-29 | review-pass2 |
| Kiley 2004 | formal (pool) | first pass | upgraded: full text read on 2026-10-29; prints 2004 in the journal, 2005 in the DOI and copyright | review-pass2 |
| Holbrook 2007 | formal (pool) | first pass | upgraded: abstract read (501 candidates, 1310 reports), then full text read on 2026-10-29 | review-pass2 |
| Bourke 2013 | formal (pool) | first pass | upgraded: abstract read (353 and 74 examiners), then full text read on 2026-10-29; not an editorial | review-pass2 |
| Army 2020 | grey tier 1 | G20 | 14: A4 M3.5 O2 D1 P0.5 N1 I1 T1 | review-pass2, requirements-pass2 |
| UT 2024 | grey tier 2 | G13 | 12: A3.5 M3 O2.5 D1 P0 N1 I0.5 T0.5 | review-pass2, requirements-pass2 |
| ULB 2025 | grey tier 2 | G14 | 10.5: A3.5 M2 O2.5 D1 P0 N0.5 I0.5 T0.5 | review-pass2 |
| UGA 2026 | grey tier 2 | G16 | 10: A3.5 M2 O2.5 D0.5 P0 N0.5 I0.5 T0.5 (fiche last updated 2024-09-06, catalogue generated 2026-10-29; the D subtotal 0.5 is to be rechecked now the date is known; Master STAPS, not the M1 AI course) | review-pass2 |
| BTS 2010 | grey tier 2 (example of practice only) | G14 | 9: A3 M2 O2.5 D0 P0 N0.5 I0.5 T0.5 (BTS is not master level) | review-pass2 |
Score arithmetic, checked; no subtotal exceeds its maximum:
| Source | A + M + O + D + P + N + I + T | Total |
|---|---|---|
| Army | 4 + 3.5 + 2 + 1 + 0.5 + 1 + 1 + 1 | 14 |
| UT | 3.5 + 3 + 2.5 + 1 + 0 + 1 + 0.5 + 0.5 | 12 |
| ULB | 3.5 + 2 + 2.5 + 1 + 0 + 0.5 + 0.5 + 0.5 | 10.5 |
| UGA | 3.5 + 2 + 2.5 + 0.5 + 0 + 0.5 + 0.5 + 0.5 | 10 |
| BTS | 3 + 2 + 2.5 + 0 + 0 + 0.5 + 0.5 + 0.5 | 9 |