Against: null results, critiques and failed assumptions

This file gathers the evidence against the rules on the apparatus of a study text: null and negative results, critiques of the strongest studies, and practice that a study contradicts.

Aids that learners value and that do not pay

  • Reported use and felt helpfulness of textbook aids did not relate to exam scores, and the rated helpfulness of key terms related to worse scores (Gurung 2003, secondary).
  • Over a semester the use of aids stayed unrelated to scores; a poor first exam led only to a small rise in summary use (Gurung 2004, secondary).
  • The link from aids to grades, where found, ran through how much of the book was read (Landrum 2012, secondary).
  • Readers said they rarely study boxed text, yet skimmed boxes and plain text alike when measured (Benbassat 2008, secondary).
  • All of these are correlational or single studies in introductory psychology; none removes an aid and measures the loss.

Objectives and alignment

  • Objectives lower learning of what they do not name (Klauer 1984, in the core); a text with many objectives narrows study.
  • Students said the stated learning outcomes did little for their approach; teaching and assessment did the work (Hailikari 2022, secondary).
  • Imposed top-down for audit, alignment can lose its educational use, and staff come to vilify learning outcomes (Loughlin 2021, an argument).
  • Teachers' alignment to standards had "surprisingly weak" links to their contribution to test gains (Polikoff 2014, secondary).
  • The one controlled-looking test of a design rubric found no significant difference (Lynch 2020, two against two courses, secondary).
  • The claim of large effects for instructional alignment comes from its proponent's own synthesis (Cohen 1987); its numbers were not read here.

Summaries and glosses

  • Glossaries and in-text glosses were the least effective gloss forms for word learning (Yanagisawa 2020, second language, secondary).
  • Marginal glosses in first-language college reading may be of limited use (Stewart 1991, secondary).
  • The summary studies are small and old: three groups (Hartley 1979), 40 students (Cook 1981), 40 students (van der Meij 2012), all read by abstract (secondary).

Refutation text

  • Most studies test one topic, often right after reading, with a pretest that may prime the answer (Zengilowski 2021).
  • A refutation text alone is not enough and needs discussion (Guzzetti 2000), so a printed misconception box may do less than the classroom studies show.
  • Effect sizes differ widely across pools: g = 0.41 between subjects (Schroeder 2022, read in full) against 1.18 for conceptual change texts (Armagan 2017, secondary); the reasons were not read.
  • Pictures added to make a refutation text engaging hurt it (Jin 2024, secondary).
  • One answer to the pretest worry: a four-group design found no inflation from pretesting (Jin 2025b, 113 participants recruited online, secondary).

Guided notes, study guides, crib sheets

  • The guided-notes pools mix designs and outcomes: effects ranged from -2.21 to 10.39 in one pool, and on exam scores the mean was negative, d = -0.75, from special-education samples only (Larwin 2012, read).
  • The college pool is heterogeneous, Q(26) = 158.99, and 13 of its 27 effects are under 0.5 (Larwin 2013, read).
  • Against the instructor's full notes, the gain of guided notes falls to d = 0.27 (Larwin 2013).
  • A provided concept-list study guide lowered exam scores against students making their own (Cushen 2019, secondary).
  • Instructor-made guides did no better than student-made ones (Horner 2007, secondary).
  • Completing three quarters of a study guide did no better than completing a quarter (Dickson 2005b, secondary).
  • Authorised crib cards did not raise scores (Dickson 2005, core); making one did not teach, and expecting to use one bred dependency (Dickson 2008; Funk 2011, secondary).

Explained solutions

  • Instructional explanations added to worked examples gave minimal benefit and did no better than self-explaining (Wittwer 2010, secondary).
  • Withholding stepwise explanations gave more conceptual learning in three experiments (Richey 2013, secondary).
  • Explanations, and prompts to self-explain, did not help, and prompts hurt with modular examples (Gerjets 2006, secondary).

Limits of the practice evidence

  • Adoption pages are their owners' claims: OpenStax 2026c and Quality Matters 2026a score under 10 of 20 and give undated figures.
  • The OpenStax preface reports a review process, not a test of its features (OpenStax 2026b).
  • The rubrics state standards, not outcomes; SUNY 2026 (example only) gives no count of users.
  • Open textbooks matched commercial ones on learning (Clinton 2019, g = 0.01), so the shared apparatus cannot be credited or blamed by that comparison.
  • The revision-guide publisher's page refused access, and no outcome data for the genre was found.

What the Q/A threads add against

  • A lecturer reports that handing out notes rarely improves exam results, on anecdote (Academia 2017c).
  • Providing notes leads some students to stop attending or writing (MathEd 2014h), which matches the dependency found for crib sheets.
  • Textbooks bloat with an example for every problem type (MathEd 2014l, example only).

Failed or missing replications

  • This search found no replication of the summary-position studies after 1981.
  • No replication of the concept-list result (Cushen 2019) was found.
  • Reder 1980, the classic comparison of texts and summaries, read in full: its summary advantage closed when the text had three times the study time (Experiment 3). After 6 to 12 months it was not significant (Experiment 7).