Protocol: how the pool of sources was built

Every search, count and rule of the exercise-sets review, so it can be repeated; every search ran on 2026-10-29. The rules the four reviews share are in the evidence README.

1. Why grey literature

Garousi 2019, Table 4, answered for exercise sets: seven yes.

# Question Answer for this topic
1 Complex, not solvable from formal literature alone? yes: the exercise set as published (textbook sections, answer keys, online homework) comes from practice
2 Lack of volume or quality of evidence? yes: the EEF review found 12 classroom studies of interleaving (Perry 2021); none tests a whole set design
3 Context important? yes: subject, age and whether homework is graded change what works
4 Corroborate research with practice? yes
5 Challenge practice with research, or the reverse? yes: 9.7% of textbook problems are interleaved against classroom trials favouring interleaving (Rohrer 2020b)
6 Would a synthesis serve both communities? yes: teachers ask for the evidence on Q/A sites and do not find it (MathEd 2021b, example only)
7 Large practitioner volume? yes: online homework systems, problem sites and solution sites with millions of users (state-of-the-art.md)

2. Questions

Id Question
Q1 Blocked against interleaved or mixed review, at the level of a sheet or homework, in classrooms
Q2 From worked example to problem: fading schedules, completion problems, example-problem pairs, Parsons problems
Q3 Difficulty order and variability across a set
Q4 Hints, answer keys and full solutions: when shown; help seeking and its abuse; copying
Q5 How much practice; homework effects
Q6 Practice and adoption: textbooks, problem books, past exams, online homework, mastery exercises, programming exercise sites

3. Existing reviews (Garousi Guideline 2)

  • Searched first, by strings F1 (interleaving reviews), F3 (homework meta-analyses) and the review hits of F2, F10, F15 and F31.
  • Found: 13 reviews and meta-analyses, read by abstract except Fan 2017 (title and venue only); and the EEF review, read first-hand in its summary (Perry 2021, grey). They are placed in review.md strand 0.
  • No review of the exercise set as a whole was found.

6. Stopping rule

  • Stated before searching (and written first in this file): effort bounded, the 20 most relevant ERIC records and 20 most cited Semantic Scholar records per string. Saturation at k = 5 strings in a row that add no new included source.
  • Applied: saturation at k = 5 was not reached; the longest run without a new source was three strings (F25 to F27). The search stopped on the effort bound, once every field name of the brief and every sub-question had been run in at least one string.
  • Evidence exhaustion: F1, F2, F6, F7, F9, F12, F13, F17, F25, F30, F31, F34, F35 returned 20 or fewer ERIC records, all screened.
  • Grey web: first result page per string. Q/A: as above.

7. Selection

  • Include: a source on how practice items are ordered, supported, solved, checked or assigned as a set, measured on learners or counted in practice. The source is in English and readable on the day at least by abstract (a title states the result only for Harms 2016).
  • Exclude:
    • motor and sport training (variability of practice in its original field), clinical and language-drill studies, matrix completion and other word collisions;
    • feedback content and timing, question format and folded answers to single questions (passed to practice-questions);
    • worked-solution writing as a genre (passed to instructional-text);
    • effects already held by the core, linked instead;
    • sources whose abstract could not be read and whose title does not settle relevance.

8. Quality of grey sources (Garousi Guideline 11, Table 7)

  • Outlet tier: government and charity reports, standards and textbooks = tier 1 (1); project and company pages, handouts, company reports and Q/A threads = tier 2 (0.5); blogs = tier 3 (0).
  • Scores are in sources.md.

9. Extraction and synthesis

  • Extracted per source: kind, year, sample, design, the outcome and its delay, any number (marked "(secondary)" when read in an abstract), and the question it answers.

10. Limits

  • One reviewer, no second rater for this topic yet.
  • Search engines personalise and change; Semantic Scholar counts are not phrase-exact; OpenAlex counts were taken afterwards by script, title and abstract, phrase search.
  • Publisher pages of ACM, APS, Elsevier and PNAS refused automated reading; several computing studies are known by title and abstract only.
  • Most formal sources were read by abstract; sources.md says which.
  • This search did not find a study of a whole exercise set built from all the measured parts; that is not proof none exists.