Position

This file says what is known about questions in study material, what is not, where practice and evidence part, and what this repository adds. Each "not known" is a statement about this search, not about the world. Terms are defined in frame.

Against the existing reviews

  • The testing-effect meta-analyses (Rowland 2014, Yang 2021, Pan 2018) measure learning, and treat the question as a condition: format, feedback, congruency.
  • The item-writing reviews (Haladyna 2002, Rodriguez 2005, Gierl 2017) treat the question as a measurement: difficulty, discrimination, reliability.
  • The feedback meta-analyses (Wisniewski 2020, Van der Kleij 2015, Mertens 2022, Kandemir 2026) treat feedback apart from the question it follows.
  • This search found no review that reads the three together as one artefact written into a text; Butler 2018 asks the question for multiple choice, and was read only through its news release.

Known

  • Answering questions on a text beats rereading it, and transfers to new application and inference questions, d = 0.40 overall (Pan 2018; core for the base effect).
  • Higher-order questions raise higher-order scores, and in two classroom studies lower-order scores too; fact questions do not raise higher-order scores (Agarwal 2019; Jensen 2014; McDaniel 2013).
  • In classrooms, recognition (mostly multiple choice) teaches about as much as recall (mostly short answer) when feedback follows (Yang 2021); in the lab, recall formats teach more (Rowland 2014).
  • Multiple-choice items with competitive wrong options induce retrieval and teach the facts behind them (Little 2012; Little 2014).
  • Wrong options can be learned as facts, and feedback cuts that cost (Roediger 2005; Butler 2008).
  • Feedback with more information has larger effects (Wisniewski 2020), and an explanation helps new questions more than the bare answer (Butler 2013).
  • Feedback timing has no average effect across studies (Kandemir 2026, secondary).
  • Item-writing flaws are common in real exams and test banks (Tarrant 2006; Masters 2001), and some make items harder or slower (Downing 2005; Moore 2026).

Known with caveat

  • Peeking at the answer before responding cuts learning: programmed-instruction studies and one review, read by abstract (Anderson 1971; Kulhavy 1977).
  • Answering in one's head works, less well than writing, and least for definitions (Yu 2025; Rivers 2025).
  • Three options are enough, for test quality (Rodriguez 2005); their effect on learning is inferred, not measured.
  • Generated items can match expert items on psychometric quality after refinement (Isley 2026); without it, they are easier, less discriminating or wrong more often (Law 2025; Laupichler 2023; Chauhan 2025).

Not known

  • Where the answer should sit in a study text: folded beside the question, at the end of the part, or at the back. This search found no comparison.
  • Whether a learner opens a folded answer after trying, or before; the progressive-disclosure review finds readers rarely open a second layer, and Lefevre 2016 found learners choose immediate feedback when offered.
  • Whether self-explanation, error-finding and application questions differ when placed in the same text.
  • Whether generated questions teach as much as written ones; only item quality and engagement were compared.
  • Whether the number of options changes learning.
  • How many questions per part, and which mix of levels, is best for a study text.

Where practice runs ahead of the evidence

  • Guidelines: nearly half of 43 item-writing rules had no research behind them (Haladyna 1989); later lists rest on textbook consensus (Frey 2005; Haladyna 2002).
  • The Carpentries train instructors to write distractors that each diagnose a misconception (Carpentries 2025p); learning from such items is measured only for competitive options in the lab (Little 2014).
  • Solutions folded under each challenge are the template's default (Carpentries 2026p); the fold is untested as a guard against peeking.
  • Generated practice questions are placed beside textbook text at scale (Johnson 2024), with engagement data and no learning comparison against written questions.

Where evidence is ignored by practice

  • Course exams and test banks sit mostly at low cognitive levels (Momsen 2010: 93%; Tarrant 2006: over 90%). Also, 47.3% of test-bank items ask for knowledge only (Masters 2001), though higher-order questions teach more (Agarwal 2019; Jensen 2014).
  • Practice items still use "all of the above" (Spielman 2020, chapter 1, question 1) against every guideline read (Brame 2013; WashU 2018).
  • Test banks sold with textbooks break guidelines in most questions (Hansen 1997: 75%).
  • Answer keys printed for odd-numbered problems only are explained by teachers through grading and reuse, not learning (MathEd 2017q).

What this repository adds

  • It reads the question in a study text as one artefact with four parts: level, format, wording, and the feedback and where it sits.
  • It names a baseline and a measure for comparing shapes (baseline.md).
  • It offers candidate rules, each graded and traced to review.md (shapes-and-checks.md); the rules on answer placement are convention, untested.
  • It does not claim a new result or the state of the art.