Against: null results, controversies and limits

This file gathers the evidence against the rules on questions in study material: null results, disputed findings, and practice that a study contradicts. Terms are defined in frame.

Whether recall formats beat multiple choice

The lab says yes: cued recall g = 0.61 against recognition 0.29 (Rowland 2014).
Classrooms say no: recall 0.541 against recognition 0.607 when feedback follows (Yang 2021).
Short answer was little or no better than multiple choice in three of four experiments, and better only when initial recall succeeded (Smith 2013c); quiz formats tied in one course (Lawson 2021).
Yang 2021 reports format as a significant moderator in its results, Q(8) = 61.333, and in its summary says format does not significantly modulate the effect. The two statements were read in the same full text.
A rule "prefer short answer" is therefore not supported in classrooms; a rule "use competitive options" has lab support only (Little 2012).

Costs of multiple choice

Reading lures plants false answers that persist a week (Roediger 2005; Fazio 2010); feedback reduces it (Butler 2008).
The gain still outweighs the cost in the studies read (Marsh 2007), but none of them used flawed or test-bank items.
Showing the stem before the options helped in three of four experiments, one in a classroom (van den Broek 2023), and not on a children's quiz site of 13,060 users (Brom 2026).

Whether more feedback is better

High-information feedback has the largest pooled effect (Wisniewski 2020), but heterogeneity is high, I2 = 83.4%.
Immediate explanation feedback on demanding questions lowered effort and learning in one study of 59 students (Roelle 2016).
A simpler correct-answer feedback beat more complex versions in a 1985 study by Kulhavy and colleagues, as Hattie 2007 reports it (read).
Explanation feedback helped only new inference questions, not repeated ones (Butler 2013).
Answer-until-correct equalled standard feedback (Butler 2007).

Whether feedback should be delayed

Single experiments favour delay (Butler 2007; Metcalfe 2009, for children).
Classroom studies favoured immediate feedback (Kulik 1988, secondary), and the latest pool finds no average effect, g = 0.03 (Kandemir 2026, secondary).
Learners prefer immediate feedback: 95.33% chose it (Lefevre 2016).
A timing rule is not supported; the moderators (level, domain, time limits) are confounded by the authors' own account (Kandemir 2026).

Whether higher-order questions always help

Fact quizzes did not raise higher-order scores (Agarwal 2019), so a rule "facts first" lacks support; the reverse rule rests on few classroom studies (Jensen 2014: two sections of one course).
Testing effects shrink with element interactivity (core: van Gog 2015, with the reply Karpicke 2015); this review did not find a study of question level with complex material that settles it.

Weaknesses of the item-writing evidence

Guidelines are textbook consensus: nearly half of 43 rules had no study (Haladyna 1989).
Flaws had little effect on reliability (Downing 2005) and did not disadvantage borderline students (Tarrant 2008); effects on difficulty are inconsistent across flaws (Moore 2026).
The three-option result measures tests, not learning (Rodriguez 2005).
"None of the above" harmed only as a replacement for a good distractor (Pachai 2015).
Most item-writing studies come from medical and nursing exams (Downing 2005; Tarrant 2006; Masters 2001).

Whether peeking matters

The evidence is old and read by abstract: programmed instruction (Anderson 1971) and a 1977 review (Kulhavy 1977).
Covert answering makes retrieval invisible: a reader may peek without knowing it, and covert retrieval is weaker than overt (Yu 2025).
No modern test of folded answers in a study text was found.

Limits of the evidence on generated questions

The positive comparisons come from authors working for the vendor of the system (Van Campenhout 2021; Johnson 2024), or measure item statistics only (Isley 2026).
The negative ones are small samples: 24 doctors (Law 2025), 161 students (Laupichler 2023), 120 students (Chauhan 2025).
Lu 2021b compares generated practice with no practice, so it shows a testing effect, not a parity with written questions.

Limits of the practice evidence

Owner's pages report their own reach: OpenStax 2026c (example only, scored in the instructional-text review) and Moodle 2026 (undated).
The NBME guide was not opened; only its landing page was read (NBME 2026).
Butler 2018 was read through its university's news release (WashU 2018), which states its recommendations without data.
Q/A threads are opinion with no data, and several answerers cite the same two works (Haladyna 2002; Rodriguez 2005).

Failed or missing replications

This search found no replication of the higher-order transfer result of Jensen 2014 outside biology.
It found no replication of Anderson 1971 on peeking in a modern medium.
Kang 2007 and McDaniel 2007 on short answer against multiple choice were located by title only; their results are not claimed.