Protocol: how the pool of sources was built

Every search, count and rule of the instructional-text review, so it can be repeated; all searches ran on 2026-10-29. Shared rules: evidence README.

1. Why grey literature

The seven questions of Garousi 2019, Table 4, for a study text: each answer is yes.

# Question Answer for this topic
1 Complex, not solvable from formal literature alone? yes: textbook apparatus, lesson templates and course rubrics come from publishers and projects
2 Lack of volume or quality of evidence? yes: no experiment was found that removes one aid from a real textbook
3 Context important? yes: school, college and workshop lessons carry different apparatus
4 Corroborate research with practice? yes
5 Challenge practice with research, or the reverse? yes: refutation text and guided notes have meta-analyses and no place in the templates read (position.md)
6 Would a synthesis serve both communities? yes
7 Large practitioner volume? yes: 4,980 Carpentries workshops since 2012 (Carpentries 2025)

2. Questions

The sub-questions Q1 to Q7 are in frame.

3. Existing reviews (Garousi Guideline 2)

  • Searched in the first batch of strings (F8, meta-analyses of objectives) and by strings aimed at reviews: F14 refined to titles with "meta-analysis", F32b (glosses), F35 (refutation).
  • Found and placed in review.md strand 0: 5 reviews without pooling, 11 meta-analyses, 1 systematic review; 2 further meta-analyses located by title only.
  • No systematic review of textbook pedagogical aids as a set, of study guides or of crib sheets was found.

6. Stopping rule

  • Stated before the first search: effort bounded, the top 20 per formal string; saturation after 5 strings in a row that add no new included source. Grey web: the first result page per string. Q/A: the shared method.
  • Formal: saturation was reached at F36 to F40, five strings that added no new included source; the search stopped there.
  • Grey web: G3 and G4 added nothing; producer visits ended when each practice form named in the commission had been read.
  • Q/A: as above, ended after IT13.

7. Selection

  • Include: a source on the parts of an explanatory study text (aids, objectives and alignment, summaries, glosses, refutation, notes and guides, explanations in solutions), their use or their effect. The source is in English and readable on the day at least by abstract or, for a book, by its catalogue record.
  • Exclude:
    • effects the core already holds, linked instead (frame.md, last table);
    • questions and their format, cards, and the order of exercises and access to their solutions, passed to the sibling reviews (handoff.md);
    • language course books, glossaries of course manuals and other word collisions;
    • sources whose abstract could not be read and whose title does not settle relevance (kept as located only).
  • Leads named in the commission and their status:
    • Verified and cited: Hartley 1976, Gurung 2003, Weiten 1996 (located), Marek 1999, Biggs 1996, Cohen 1987, Mager 1962, Reder 1980 (located; read in full after the audit), Guzzetti 1993. Also cited: Tippett 2010, Schroeder 2022, Konrad 2009, Dickson 2005b, Wilson 2019, the Carpentries template, the Quality Matters rubric;
    • Dickson and Miller 2005 is the core's key Dickson 2005, linked; its DOI in the core bibliography resolves to another item (handoff.md);
    • Hartley's Designing Instructional Text: found as the 1978 book (ERIC record); a 2004 version was not located;
    • Wiggins and McTighe 2005: catalogue record found, book not opened;
    • the taxonomy book by Anderson and Krathwohl (2001): not opened; Krathwohl 2002 cited for the revised taxonomy (title and venue);
    • the lead Gurung and Daniel (2005): a chapter of that title is in Crossref (2006, pp. 59-74) with no authors, and authorship was not verified, so it is not cited;
    • an OpenStax style guide: not found; the preface of one book is used.

8. Quality of grey sources (Garousi Guideline 11, Table 7)

  • Outlet tier: books (and a book's front matter) = tier 1 (1); project documentation, company and university pages, annual reports and Q/A threads = tier 2 (0.5); blogs = tier 3 (0).
  • Scores, with subtotals, are in sources.md.

9. Extraction and synthesis

  • Extracted per source: kind, year, design, sample, the claim, any number (marked "(secondary)" if read in an abstract), and the sub-question it answers.

10. Limits

  • One reviewer; no second rater on this topic.
  • Most formal sources were read by abstract only; publisher pages of SAGE, Springer, Elsevier and Taylor and Francis refused access without a login, and none was used.
  • One Semantic Scholar string was refused three times (F4); Semantic Scholar counts are not phrase-exact.
  • The ERIC relevance order and the web engine's ranking are not documented and may change.
  • Archived copies exist for six practice pages (the bibliography); OpenStax, OSCQR and Teaching Tech Together had none on the day.
  • This search did not find an experiment that removes one aid from a real textbook; that is not proof none exists.