State of the art, per sub-question

This file names, for each sub-question, who holds the record and how it was measured. Two records are kept apart: most adopted (what study material and quiz tools do) and best evidence (what a study measured). Terms are defined in frame; every source is in review.

Best evidence

Sub-question Best evidence Measured how Measure
Q1 does a question transfer to new questions? Pan 2018 meta-analysis, 192 effects, N = 10,382 d = 0.40 overall; 0.32 to application and inference questions
Q1 does a higher-order question help lower-order learning? Jensen 2014; McDaniel 2013; Agarwal 2019 one course over a semester; middle-school classes; lab and classes yes for application and high-level questions; fact questions did not raise higher-order scores
Q2 recall or recognition? Yang 2021 (classroom); Rowland 2014 (lab) meta-analyses classroom: recall 0.541, recognition 0.607 with feedback; lab: cued recall 0.61, recognition 0.29
Q2 written or in the head? Yu 2025 meta-analysis, 18 studies, 2,560 participants (secondary) covert g = 0.23; overt better by g = 0.17
Q2 can a multiple-choice item induce retrieval? Little 2012; Little 2014; van den Broek 2023 lab experiments competitive options teach the facts behind the wrong options; stem shown before options helps vocabulary
Q3 how many options? Rodriguez 2005 meta-analysis, 80 years three, for psychometric quality, not measured on learning
Q3 what do flaws cost? Downing 2005; Tarrant 2008; Moore 2026 item analysis of real exams 0 to 15 points harder; high achievers penalised; confusing flaws add 14% to 29% response time
Q4 what should feedback say? Wisniewski 2020; Mertens 2022; Butler 2013 meta-analysis, 435 studies; network meta-analysis, 77 studies; two experiments more information, larger effect (high-information d = 0.99); explanation helps new inference questions
Q4 when should it come? Kandemir 2026 meta-analysis, 51 studies, 160 effects (secondary) no average difference, g = 0.03
Q5 does peeking cost? Anderson 1971; Kulhavy 1977 programmed instruction experiment; review learners who could peek learned less (secondary)
Q7 are generated items as good? Isley 2026; Laupichler 2023; Law 2025 item response theory in 91 classes; item analysis comparable after refinement; lower discrimination or easier without it

Most adopted

Sub-question Most adopted Source
Q1 level lower-order items: 93% of 9,713 biology exam items at Bloom levels 1 and 2 Momsen 2010
Q2 format four-option multiple choice at the end of a chapter Spielman 2020
Q3 writing consensus guidelines (31 to 43 rules) and a medical-school guide Haladyna 1989; Haladyna 2002; Frey 2005; NBME 2026 (example only)
Q4 feedback the quiz tool's setting: deferred, immediate, or multiple tries Moodle 2025
Q5 answers a solution folded under each challenge; answers away from the questions Carpentries 2026p; Spielman 2020
Q7 generation generated practice placed beside textbook text at scale Johnson 2024 (vendor's authors)

Adoption counts, descriptive, read 2026-10-29

Count Value Caveat
quiz questions on registered Moodle sites 12,654,431,762 on 144,984 sites undated owner's page; registered sites only (Moodle 2026)
students reached by OpenStax books 49 million all-time; 5.4 million a year owner's page, undated, example only (OpenStax 2026c)
generated questions answered in one vendor's textbooks over 800,000 questions, 5.2 million sessions, over 400,000 students vendor's authors, abstract (Johnson 2024)
learners who chose immediate over delayed feedback 95.33% of 599 one programme (Lefevre 2016)
flawed items in real exams 46.2% of 2,770; 47.3% of 10 papers; 36% to 65% of four exams nursing and medicine (Tarrant 2006; Tarrant 2008; Downing 2005)
guideline breaches in test banks sold with textbooks 75% of questions; 2,233 breaches in 2,913 questions auditing; nursing (Hansen 1997; Masters 2001)

Upper bounds (achieved)

  • Testing transfers to new application and inference questions at a literature level (Pan 2018).
  • A semester of high-level questions raised low-level scores too, in one course (Jensen 2014).
  • Multiple choice with competitive options matched cued recall and added learning about the wrong options (Little 2012).
  • Explanation feedback beat correct-answer feedback on new inference questions (Butler 2013).
  • Refined generated items reached expert items' psychometric quality in a field study (Isley 2026).

Lower bounds (not yet shown)

  • This search found no experiment comparing answers folded beside the question, at the chapter end and at the back of the book.
  • It found no study of learning, as opposed to item quality, from generated questions against written ones.
  • It found no study of learning as a function of the number of options; Rodriguez 2005 measures tests.
  • It found no study testing an item-writing guideline for its effect on learning, except distractor competitiveness (Little 2014) and lures (Roediger 2005).
  • Q/A sites (searched 2026-10-29) hold no data on any sub-question.