Against: null results, controversies and limits

This file gathers what cuts against the rules on exercise sets: null classroom results, studies that disagree, practice that contradicts the evidence, and the limits of the evidence.

Interleaving at the sheet level does not always win

  • A full year of interleaved problem sets in 62 Nigerian classrooms raised short-term retention by 0.29 SD but not the average score on a cumulative test (van der Haar 2023, secondary).
    • Gains at the bottom of the distribution appeared offset by losses at the top (same source).
  • In a university Calculus I course with online homework, mixed and blocked sets gave no difference on the final exam (Gorgievski 2024, secondary).
  • Distributed homework sets in business statistics showed only some evidence of a gain, possibly an instructor or Hawthorne effect (Crissinger 2015, secondary).
  • Spaced retrieval practice in nine introductory STEM courses gave significant gains on the criterial test in two (Bego 2024, secondary).
  • Inside courses the spacing effect in mathematics is smaller than in isolation: g = 0.24 against 0.43 (Murray 2025, secondary).
  • The EEF review found interleaving tested almost only in mathematics, and calls the classroom evidence base small and thin (Perry 2021).
  • Most interleaving evidence comes from laboratory studies of undergraduates (Firth 2021, secondary); Rohrer 2012 called it limited and asked for ecologically valid studies (abstract).
  • Interleaving helps categories that are similar; categories dissimilar within and between may be better blocked (Carvalho 2014, linked in reading-order). Yet Rohrer 2014 found a large effect with dissimilar problem types (secondary). Which types to mix is not settled.

Learners and teachers prefer the blocked sheet

  • Students judged the least spaced and least interleaved schedules most effective, and found spaced, interleaved ones harder and less enjoyable (Hartwig 2022, secondary).
  • Massed practice left Year 7 students overconfident about their test scores (Emeny 2021, secondary).
  • Textbooks keep blocked sets: 9.7% of 13,505 problems interleaved in six texts (Rohrer 2020b).
  • So the shape the evidence favours feels worse to the learner and is rare in print: a rule must expect resistance, not demand.

Example-problem order and fading: mixed results

  • With identical items the example-first advantage disappeared at test (van Gog 2011b, 16 per group, secondary).
  • Example-problem did not beat problem-example for technical students; studying examples only beat both (van Harsel 2019, secondary).
  • Order had no effect on inquiry skills for 156 middle-school students (Ganaiem 2025, secondary).
  • Faded examples were no better than provided examples for declarative concepts and took longer (Zamary 2018, secondary).
  • In four college homework sets, problem solving beat fading with self-explanation prompts, and fading alone did no better (Miller-Cotto 2019, secondary).
  • Help that grew more similar to the problem (fade in) beat help that grew less similar (fade out), the reverse of the usual fading direction (Jennings 2020, secondary).
  • Free to choose, learners rarely studied an example before a problem (Foster 2018, secondary); a set that leaves the order to them loses the pairs.

Parsons problems

  • Their effectiveness, as a question type and as a learning tool, is called uncertain for lack of replicated studies (Du 2020).
  • Distractor lines lowered learning efficiency for young novices (Harms 2016, title).
  • The efficiency gain disappeared when the expected solution was unusual (Haynes 2021).

Difficulty and variability

  • Variability helped experienced learners and hurt novices (Likourezos 2019, secondary), so "vary the problems" is not a rule for every reader.
  • Higher success rates led to more practice and larger gains in one adaptive program (Jansen 2013b, secondary), which sits uneasily with "desirable difficulties": a difficulty that makes the learner stop is not desirable.

Hints and solutions

  • Teaching students to seek help well changed their help use but not their learning (Aleven 2016).
  • Students prone to gaming were not brought level by delaying hints (Vanacore 2024, secondary).
  • Generated hints failed quality checks on 32% of problems before a correction step (Pardos 2024, secondary).
  • Only one study found compared a course with and without the solutions manual, and it compared two cohorts, not randomised groups (Rystrom 2005).
  • Practitioners split: publish solutions for review and fairness (MathEd 2014b; MathEd 2014c), or withhold them so that students try (MathEd 2016b; Academia 2021a). No trial decides between them.

Homework and how much of it

  • Every homework design in the 1987 to 2003 syntheses had flaws (Cooper 2006, secondary).
  • The homework effect pooled over studies is small, d = 0.229 (Bas 2017, secondary).
  • Completing more homework correlated with worse exams for low-aptitude physics students (Kontur 2015); a comment found the reverse in two larger courses (Rieger 2016, secondary). The disagreement is about populations, not settled.
  • The share of students who did not benefit from homework rose from 14% (2008) to 55% (2017), and students who reported copying did not benefit (Glass 2020, secondary). A remote physics course found no widespread copying (Chen 2022, secondary).
  • Online homework did not beat paper homework over 18 studies (Bishop 2024, secondary); the ASSISTments gain (Roschelle 2016) bundles hints, feedback and teacher reports.
  • Open access to a model during practice raised practice grades and lowered later exam grades (Bastani 2025, secondary).

The Saxon case

  • Saxon Math is built on daily cumulative mixed practice (Rohrer 2020b). Yet the WWC found mixed effects in its primary courses, kindergarten to pre-algebra, and no discernible effect for Algebra I (WWC 2017; WWC 2016).
  • Its first-grade advantage over two curricula (Agodini 2010) disappeared against one of them by second grade (Agodini 2013, secondary).
  • A whole curriculum differs in much more than mixed review, so this neither confirms nor refutes interleaving.

Limits of this review

  • Most formal sources were read by abstract; numbers from abstracts carry "(secondary)". Full texts were read for Rohrer 2015, Rohrer 2020b, Roschelle 2016 and Hou 2025.
  • Publisher pages of ACM, APS, Elsevier and PNAS refused access; several computing papers are known by title and abstract only.
  • Grey sources mostly describe adoption; the owners' pages report their own success (ASSISTments 2026, example only).
  • Q/A threads are opinion; one answer carries unpublished data (MathEd 2016a).