Review

This file reads the literature on questions written into study material, by strand: existing reviews first, then one strand per sub-question, then practice and Q/A sites. Terms are defined in frame. "(secondary)" marks a number read in an abstract or a summary, not in the paper; grey scores are in sources.

0. Existing reviews

Review Kind, size What it holds for this topic
Pan 2018 (full text) meta-analysis, 192 effect sizes, 122 experiments, N = 10,382 testing transfers, d = 0.40 [0.31, 0.50]; to application and inference questions d = 0.32 [0.085, 0.56], k = 41 (Table 2)
Rowland 2014, Yang 2021 (core; full text read for moderators) lab and classroom meta-analyses format and feedback moderators, strand 2 and 4
Wisniewski 2020 (full text) meta-analysis, 435 studies, 994 effects, about 61,000 subjects feedback d = 0.48 [0.44, 0.51] after outliers; I2 = 83.4%
Van der Kleij 2015; Mertens 2022; Bangert-Drowns 1991 feedback on items: 40 studies, 70 effects; network, 77 studies, 163 effects; 40 reports, 58 effects content of item feedback, strand 4 (abstracts)
Kulik 1988; Kandemir 2026 timing: 53 studies; 51 studies, 160 effects strand 4 (abstracts)
Hattie 2007 (full text); Shute 2008; Kulhavy 1977 narrative reviews of feedback levels of feedback; guidelines; presearch availability
Andre 1979 review of question level while reading higher-level questions help reproductive and productive knowledge (abstract)
Haladyna 1989; Haladyna 2002; Frey 2005; Rodriguez 2005; Gierl 2017 item-writing rules from 46, 27 and 20 textbooks; number of options, meta-analysis; distractors, review strand 3
Kurdi 2020; Gorgun 2024; Alhazmi 2024 AQG, 93 papers 2015-2019; evaluation methods; distractor generation strand 6
  • How this review differs: each review above takes one part (level, format, item, feedback, generation); this one reads them as parts of one artefact, a question in a study text.
  • It keeps learning outcomes apart from assessment quality: item-writing reviews measure tests, testing-effect reviews measure learning, and only Butler 2018 joins the two (read through a news release, strand 3).

1. Q1: kinds and levels of question

  • Agarwal 2019: middle-school and college students; higher-order and mixed quizzes raised delayed higher-order test scores, fact quizzes did not (abstract).
  • Jensen 2014: two sections of one biology course, a semester of low-level or high-level quizzes and exams; the high-level section scored higher on both low-level and high-level final items (abstract).
  • McDaniel 2013 (middle school): application quiz questions raised exam scores on application and on definition items. Collins 2018 (college): application quizzes raised application items and matched factual quizzes on definitions (abstracts).
  • Butler 2010 and Rohrer 2010: repeated testing improved new inferential questions a week later, and transfer map questions a day later in children (abstracts).
  • Pan 2018 moderators: transfer grows with response congruency (d = 0.28 without, 0.58 with), elaborated retrieval practice and initial test accuracy (full text).
  • Self-explanation and error-finding questions are core results (Bisra 2018; Alemdag 2025); no study here compared them with application questions inside one text.
  • Field record of what questions ask: 93% of 9,713 items in 77 introductory biology courses were at Bloom levels 1 and 2 (Momsen 2010, full text). Also: introductory biology and physics sequences mostly assess lower-order skills (Momsen 2013, abstract).
  • Dávila and Talanquer (Davila 2009): end-of-chapter items in general chemistry textbooks are mostly at application and analysis level, but narrow: algorithmic problems and predictions (abstract).

2. Q2: format

Source Setting Result
Rowland 2014 (core; not checked in the paper, APA, no subscription) lab meta-analysis initial cued recall g = 0.61 [0.52, 0.69]; free recall 0.29; recognition 0.29; with high exposure, recognition 0.36, cued 0.72, free 0.82
Yang 2021 (core; the published values not checked, APA, no subscription) 222 classroom studies format moderated, Q(8) = 61.333: short answer 0.638, multiple choice 0.567, free recall 0.238; recall 0.541 against recognition 0.607 with feedback. The author manuscript (223 studies) gives Q(8) = 60.678, 0.629, 0.569, 0.238, and 0.534 against 0.610
Smith 2013c 4 experiments, N = 372, texts short-answer and hybrid questions little or no better than multiple choice in three experiments; better when initial recall succeeded (abstract)
Lawson 2021 introductory psychology course multiple-choice and fill-in-the-blank quizzes raised exam scores equally (abstract)
Little 2012; Little 2014 lab items with competitive options improved recall of the tested fact and of facts behind the wrong options; cued recall did not (abstracts)
Little 2019 lab learners spontaneously retrieve facts about wrong options, and that predicts later retention (abstract)
Roediger 2005; Marsh 2007; Fazio 2010 lab lures read on a multiple-choice test come back as answers later; the effect persists a week; the gain still outweighs it (abstracts)
Butler 2008 lab feedback, immediate or delayed, raised correct answers and cut lure intrusions (abstract)
van den Broek 2023 4 experiments, N = 45 to 79 showing the question 4 s before the options raised vocabulary retention in three of four (abstract)
Brom 2026 children's quiz site, N = 13,060 stepwise options did not change short-term learning; learner-paced delay raised completion slightly (abstract)
Smith 2013b; Yu 2025; Rivers 2025 lab; meta-analysis, 18 studies, 2,560 participants; 2 experiments answering in one's head equalled writing in Smith 2013b; pooled, covert g = 0.23, overt better by g = 0.17; overt better for definitions (abstracts)

3. Q3: how an item is written

  • Haladyna 1989: a taxonomy of 43 rules drawn from 46 textbooks; 96 studies reviewed; for nearly half the rules no research was found (abstract).
  • Haladyna 2002: 31 guidelines for classroom assessment, from 27 textbooks and 27 studies since 1990 (abstract); Frey 2005: 40 rules on which 20 classroom-assessment textbooks agree (abstract).
  • Rodriguez 2005: three options are optimal, by 80 years of research on psychometric quality; more items fit in the same time (abstract).
  • Downing 2005: four medical-school exams; 36% to 65% of items flawed; flawed items 0 to 15 points harder; little effect on reliability (abstract).
  • Tarrant 2006: 46.2% of 2,770 nursing items flawed, over 90% at low level, 36.2% from test banks; Tarrant 2008: 47.3% flawed, and high achievers, not borderline students, were penalised (abstracts).
  • Moore 2026: 1,230 items in three domains; confusion-inducing flaws raised response time by 14% to 29%; cue-giving flaws showed no effect (abstract).
  • Pachai 2015: "none of the above" made an item harder only as the key; replacing a good distractor with it lowered discrimination (abstract).
  • Distractors from misconceptions: Sadler 1998 ranks children's astronomy ideas on one scale from distractor-driven items (abstract). Gierl 2017 finds distractor writing studied far less than stems (abstract); Little 2014 is the one learning result on distractor quality (strand 2).
  • Butler 2018, as reported by its university (WashU 2018): three plausible options, items most students can answer, no "all" or "none of the above", no complex formats, then review of answers.

4. Q4: feedback content and timing

  • Content, meta-analyses: high-information feedback d = 0.99, corrective 0.46, reinforcement or punishment 0.24 (Wisniewski 2020, Table 3). On items, EF 0.49, KCR 0.32, KR 0.05, and EF more effective for higher-order outcomes (Van der Kleij 2015, secondary). EF most likely best for lower- and higher-order outcomes, KR worst (Mertens 2022, secondary).
  • Bangert-Drowns 1991: effect 0.28 over 40 effects, as Hattie 2007 lists it in Table 1 (read). Its own abstract gives 40 reports and 58 effects; feedback corrected errors when learners were led to attend to it (abstract).
  • Single studies: explanation feedback equalled correct-answer feedback on repeated questions and beat it on new inference questions (Butler 2013). Also: correct answer after an error raised final retention by 494% in word learning, after a correct answer it changed little (Pashler 2005).
  • Against more content: immediate explanation feedback on demanding adjunct questions cut time and accuracy on the next questions, then learning (Roelle 2016, N = 59). Also: the correct answer alone gave higher later performance than more complex feedback, in Kulhavy, White, Topp, Chan and Adams (1985), on reading passages with multiple-choice items, as Hattie 2007 reports it (read).
  • Timing: classroom studies favoured immediate feedback, lab studies of test content delayed (Kulik 1988, abstract); with a 1988 to 2024 pool, timing made no average difference, g = 0.03 [-0.08, 0.13] (Kandemir 2026, abstract).
  • Single studies of timing: delayed beat immediate on multiple-choice tests, and AUC equalled standard feedback (Butler 2007). Delayed beat immediate for children, not for adults once the lag to test was equated (Metcalfe 2009). Delayed elaborated feedback did not beat immediate (Candel 2021, N = 133). Yet 95.33% of 599 learners chose immediate feedback (Lefevre 2016).
  • Classroom moderator: corrective feedback after quizzes g = 0.537 against 0.374 without (Yang 2021, full text).

5. Q5: where answers sit, and peeking

  • Kulhavy 1977: feedback does not help retention when it is highly available before the learner responds (abstract).
  • Anderson 1971: learners given the correct response right after answering learned more than those given none or those who could peek before answering (abstract).
  • Covert answering is allowed by the evidence only if retrieval happens (Smith 2013b), and it is weaker for definitions (Rivers 2025): an answer visible on the same line would make retrieval optional (an inference).
  • van den Broek 2023 controls availability within an item: the stem before the options; Brom 2026 found no gain in the wild.
  • No study was found that compares answers folded beside the question, at the end of the chapter, and at the back of the book.

6. Q7: generated questions

Source Comparison Result
Zhang 2016 machine-generated, web-collected and textbook biology questions rated pedagogy similar; knowledge-base questions shallower (abstract)
Van Campenhout 2021 automatic and human-authored questions in one course equal engagement, difficulty and persistence, by the vendor's authors (abstract)
Johnson 2024 5.2 million sessions, 800,000 generated questions, 9,000 textbooks features that predict student ratings (abstract)
Lu 2021b 91 students, generated short-answer practice against a control practice group learned more; not compared with written questions (abstract)
Laupichler 2023 161 students, 25 human and 21 ChatGPT items equal difficulty; discrimination 0.36 against 0.24 (abstract)
Law 2025 24 doctors, 100 and 100 items AI items easier (0.78 against 0.69), similar discrimination, more inaccuracies, lower-order; 24.5 against 96 person-hours (abstract)
Chauhan 2025 120 students, crossover 9 ChatGPT items against 19 faculty items met ideal criteria; nonfunctional distractors 33.75% against 13.75% (abstract)
Kuusemets 2024 GPT-4 against an expert, vectors the suggested key often needs correction (abstract)
Isley 2026 91 classes, nearly 1,700 students refined AI items comparable to expert items by item response theory (abstract)
Elkins 2023; Scaria 2024 teacher ratings; expert ratings by Bloom level judged useful; automated evaluation not on par with human (abstracts)
  • Kurdi 2020 found little work on difficulty control, feedback generation and presentation, and asked for harmonised evaluation; Gorgun 2024 found no standard evaluation framework.
  • No study found here measured learning from generated questions against learning from written ones.

7. Practice (Q6)

Who Since Form Evidence of use
OpenStax, Psychology 2e (Spielman 2020) 2020 chapter 1: 15 four-option review questions, one with "all of the above", and 8 critical thinking questions, no answer on the page; chapters 2 and 3 hold 24 and 20 review questions OpenStax reach, owner's page, example only (OpenStax 2026c)
The Carpentries page of 2025-08-05; 2026-10-06 challenges with hint and solution blocks in accordions; instructors trained to write MCQs whose wrong answers each diagnose a misconception (Carpentries 2025p; Carpentries 2026p) lesson template; reach in the instructional-text review
Moodle quiz page of 2025-07-16 deferred feedback, immediate feedback, interactive with multiple tries, certainty-based marking (Moodle 2025) 12,654,431,762 quiz questions on 144,984 registered sites (Moodle 2026, undated)
NBME sixth edition item-writing guide for medical schools; text behind a form, not opened (NBME 2026, example only) decades of exam items, by its own account
Teaching centres 2013 stem a question, plausible distractors from common errors, options parallel in grammar and length, negation avoided or emphasised, no "all" or "none of the above" (Brame 2013) archived page
IES practice guide 2007 quizzing to re-expose content: strong evidence; deep explanatory questions: strong; prequestions: low (Pashler 2007) federal guide
  • Test banks sold with textbooks: 75% of auditing test-bank questions broke a guideline (Hansen 1997); 2,233 violations among 2,913 nursing test-bank questions, 47.3% at knowledge level (Masters 2001).

8. Q/A sites

Mathematics Educators, Computer Science Educators and Academia Stack Exchange, searched 2026-10-29: 15 threads, none under 10 of 20. Practitioner opinion; no thread carries data.

  • Q3 is what teachers ask most: where plausible wrong answers come from.
    • An item writer builds distractors from grade-level misconceptions, intermediate steps and calculation slips (MathEd 2020p); a teacher collects short answers one year and turns the common ones into options the next (MathEd 2017p).
    • Three or four options are argued from Haladyna 2002 and Rodriguez 2005, or from guessing odds (MathEd 2018p; Academia 2016q).
    • Wording: negation in a stem confuses (MathEd 2025p); extra context suits homework, not exams (Academia 2018p).
    • Writing good items is slow; teachers reuse banks and add 10 to 15 items a year (CSEd 2018p; Academia 2021p).
  • Q2: teachers prefer free response for partial credit and for seeing reasoning, and use multiple choice for grading time (Academia 2016p; Academia 2021p). Asked for research, answerers found little on learning (MathEd 2016p; MathEd 2014p).
  • Q4: explanation shown only after a correct choice invites gaming by retrying (CSEd 2024p). Q5: answers left out of books, or printed only for odd-numbered problems, are explained by grading and reuse of problems, not by learning (MathEd 2014q; MathEd 2017q).
  • Q1: Bloom's verbs are judged a poor fit for mathematics (MathEd 2016q).