Baseline: what the shapes are compared against
This file names the baseline that a proposed shape for questions in study material must beat, and says how the comparison would be measured. Terms are defined in frame.
The named baselines
| Id | Baseline | Why this one |
|---|---|---|
| B1, to beat | a self-test of short-answer questions on recall, one per key idea, each followed by its correct answer once the learner has answered | the standard condition of the research: cued recall as the initial test gives g = 0.61 in the lab (Rowland 2014, to be checked in the paper), and feedback lifts classroom quizzes from 0.374 to 0.537 (Yang 2021) |
| B0, the practice control | end-of-chapter review questions as a textbook prints them: four-option multiple choice, mostly recall, answers away from the questions or absent | most adopted: 15 to 24 four-option items per chapter in Spielman 2020 (chapters 1 to 3); 93% of course items at the two lowest levels (Momsen 2010) |
- The shapes must beat B1: beating B0 is cheap, since B0 often lacks feedback and higher-order items, and B1 already supplies both repairs the evidence names.
- B0 stays in every comparison as the condition learners actually meet.
- The baseline sources are formal (Rowland 2014; Yang 2021; Momsen 2010) or a published textbook (Spielman 2020, 15 of 20 in sources.md).
Candidate variant to test
- Questions at two levels per part: a recall question, then an application or inference question (Agarwal 2019; Pan 2018).
- Multiple-choice items, where used, with three competitive options built from known misconceptions (Little 2014; Rodriguez 2005; Sadler 1998), no "all" or "none of the above".
- Feedback that explains the key and each wrong option, not only names the key (Butler 2013; Wisniewski 2020).
- The answer folded beside the question, opened only after a written attempt; the fold is convention, untested.
How a comparison would be measured
| Reader task | Outcome | Instrument | Prior study to borrow from |
|---|---|---|---|
| retain what was asked | proportion correct on the practised questions, reworded | delayed test, two days to a week | Butler 2010; Smith 2013c |
| transfer to new questions | proportion correct on new application and inference questions | items written before the study, never practised | Butler 2013; Pan 2018 (its transfer categories) |
| learn from wrong options | proportion correct on questions whose answers were distractors; rate of lure intrusions | cued-recall final test | Little 2014; Roediger 2005 |
| peek or not | share of answers opened before a written attempt; time to open | logged folds in a digital text | Anderson 1971 (peek condition); Brom 2026 (learner-paced delay) |
| cost | time on task; time to write the questions | logs; author timesheets | Law 2025 (24.5 against 96 person-hours) |
- Design: three arms (B0, B1, variant) between subjects, or a crossover over two chapters with counterbalanced order; same text and same time on task.
- Delay of at least two days before the final test, as in the cited studies.
- Effect sizes with confidence intervals, as Hedges' g; separate novices from learners with prior knowledge.
- Learning outcomes are kept apart from item statistics (difficulty, discrimination); the latter cannot stand in for the former (position.md).
A checklist in the baseline's own terms
Per question, from B1:
- It asks for one key idea of the part, in the learner's own words.
- It can be answered without the text in view.
- Its correct answer is given only after the learner has answered.
Per question, from item-writing guidance (Haladyna 2002; Brame 2013), when the item is multiple choice:
- The stem states a full question.
- Every option is plausible and comes from a common error.
- No "all of the above", "none of the above" or combined options.
- Options are parallel in grammar and length.
- The stem avoids negation, or marks it.
A rule may be rejected deliberately; the rejection is written where a reviewer looks for the missing element.