Baseline
This file names the baseline a report template is compared against, and how the comparison would be measured.
The named baseline
The report components of ANSI/NISO Z39.18-2005 (R2010) are the control template. They begin with a title section, an abstract of about 200 words, contents and a summary; a separate executive summary is often prepared when a print report passes 50 pages. After the summary come an introduction without findings, methods, results and discussion, and conclusions written to be read alone. Recommendations follow the conclusions if the report proposes action, then references and appendices.
Why this one:
- It is the only standard read in full, and it is written for the readers of a report, not only for its author (NISO 2010).
- It already puts a summary at the front, so a new template must beat a summary-first control, not a strawman.
- Its components are the ones surveyed readers say they use (Pinelli 1982), so the control is not arbitrary.
- It states its own limit: no single format fits every purpose, which is the reason to compare.
- The competing candidate, ISO 7144, is paywalled and unread (ISO 1986), so it cannot serve as a control yet.
An example of practice, for French schools
The school grid itself, run as the score: for an internship report, the TSE 2017 grid (form 7, substance 13 of 20). It scores 9.5 of 20, below the line of 10, so it is an example of practice only and not a baseline; NISO Z39.18 stays the baseline. For a thesis, Mullins 2002 reports the criteria that experienced examiners name as themes, not a numbered list (full text).
What to measure
| Outcome | Definition | Instrument used before | Source |
|---|---|---|---|
| Time to find the answer | seconds from question to correct answer | timed questions (not verified: format known only as summarised in Hartley 2004; the 1996 abstract gives only background) | linked from the academic papers review (Hartley 1996) |
| Accuracy of the answer | share of questions correct | same | same |
| Screening decision | whether a reader would read on after the first page | a survey of components used to decide | Pinelli 1982 |
| Recall | free recall of topics, scored against the main ideas | free recall task | Lorch 1996 |
| Grade | mark on the school grid | the grid | TSE 2017 (example of practice only) |
| Grader agreement | difference between two graders on the same report | none found for reports | open |
| Information content of the summary | items present over items applicable | checklist | linked from the academic papers review |
How to compare
- Write the same project twice, once to the baseline and once to the new template, by someone other than the experimenter.
- Hold length constant or report it; length has no measured effect on report readers, so it must not vary by accident.
- Use a within-subject crossover with counterbalanced order.
- Test the four readers separately: a grader, a jury member, a host supervisor and a colleague; their needs differ (review.md 4).
- Recruit readers from inside and outside the topic.
- Preregister the plan.