Baseline: the guideline a case here must meet
Step 5 of the review. The review is finished when it names the baseline. Terms are in frame; the findings are in review.
Shape: baseline
The baseline
The reader's checklist of Runeson 2009 (Table 11, items 39 to 50), the published guideline for reporting case study research in software engineering.
- It has 12 items, each a question a reader asks of a report.
- It is read beside the experience-report criteria of ICSE 2003, since the author took part, and the Case Study and Repository Mining standards (Ralph 2020).
- The measures program is held to the essential list of the Repository Mining standard.
- Grade: convention. No study shows that meeting it makes a report more trusted.
- The maintainer's practice, written down for this review, adds one rule: a count taken by hand is marked as such, and a program takes every other.
The 12 items, and where a case meets each
| Item | The reader asks | Part of a case |
|---|---|---|
| 39 | Are the objective and questions clear and relevant? | Summary, Case |
| 40 | Are the case and its units of analysis well defined? | Case |
| 41 | Is the suitability of the case motivated? | Why this case |
| 42 | Is it linked to existing literature? | Genre and baseline |
| 43 | Are the data collection procedures sufficient? | Data and procedure |
| 44 | Is enough raw data presented? | Results |
| 45 | Are the analysis procedures repeatable and transparent? | Analysis, Rerun |
| 46 | Is a chain of evidence established? | Chain of evidence |
| 47 | Are threats to validity analysed with countermeasures? | Threats to validity |
| 48 | Is triangulation applied? | Triangulation |
| 49 | Are ethical issues addressed, including intentions? | Roles and ethics |
| 50 | Are conclusions and implications suitable for the audience? | Lessons |
How a case is held to it
- Each item is answered in the case, met or not met with a reason.
- A case with an item not met says so in its Threats to validity, and is still published.
- The measures program is checked by a test, as the maintainer's practice asks of anything a program can decide.
- Triangulation (item 48) is the weak point for a report from inside; the shape asks the author to say which sources were not independent.
What would count as beating it
- A second reader's marks on the 12 items, compared with the author's.
- A reader study of whether the checklist changes what an absent reader believes: not run.