Review
Step 2 of the review: what each strand holds, by question. Existing reviews come first. Terms are in frame; grades and reading depth per source are in sources. A number from an abstract or a summary is marked (secondary).
Shape: review
Strand 0: existing reviews
- Nawaz 2022 reviews 23 studies of rework, 2009 to 2020, in global software development only. It sorts causes into 6 categories; communication and requirements management are reported most (secondary).
- Dybå 2008 reviews agile studies; its abstract was not retrieved.
- Hembree 1992 is a meta-analysis of problem-solving research; two secondary summaries put it at 487 reports.
- Chalmers 2009, Macleod 2014 and Glasziou 2014 are a series on research waste.
- No review of design science process models turned up, and none of what a project trail must hold.
Strand A: the process from problem to result
- Peffers 2006 synthesises seven process models into six activities: problem identification and motivation, objectives of a solution, design and development, demonstration, evaluation, communication.
- The order is nominal. A project may start at any activity and move outward; Peffers 2006 lists four entry points.
- Objectives are inferred from the problem. They can be quantitative, in terms in which a solution would be better than current ones.
- That step needs "knowledge of the state of problems and current solutions". The comparison target sits there, with no review step named.
- Evaluation compares the objectives with observed results, then decides whether to iterate to design or to go on.
- Communication is the last activity, and the paper may follow its structure.
- Grade: convention. The model is a synthesis and a demonstration on two cases, not a test.
- Mettler 2014 reviews experiments in design science and proposes an evaluation framework: questions to ask and pitfalls to avoid (secondary).
- The abstract names no review step and no comparison target, so the framework sharpens evaluation without adding the missing step.
- Hevner 2004, Gregor 2013 and Wieringa 2014 were located, not opened.
Strand B: reviewing before building
- Lund 2016 makes the call: "no new studies should be done without a systematic review of existing evidence".
- Glasziou 2014 reports, in its abstract, that investigators of new trials rarely set their findings in the context of a systematic review.
- It reports that 40 to 89 percent of published trials were not replicable, because their interventions were poorly described (secondary).
- The evidence measures the neglect. No source found measures that doing the review changes the outcome of a project.
- Grade: convention, untested for effect; the neglect is measured, descriptive.
- Webster 2002 and Booth 2016 were located as guides to reviewing; not opened.
Strand C: heuristics
- Polya 1945 and Schoenfeld 1985 were located, not opened.
- Hembree 1992, in secondary summaries: checking work, drawing diagrams and using heuristics go with higher problem-solving scores.
- The same summaries say heuristics had a clear effect only where teachers were trained in them, and that students told to follow fixed steps scored lower than students using flexible approaches.
- Hohn 2002 reports two controlled experiments with 223 elementary students (secondary).
- A named five-phase strategy, taught over several lessons, did better than traditional instruction, short term and delayed.
- The abstract does not say whether students followed the phases as fixed steps, so the older warning about fixed steps is not tested here.
- Grade: measured, for students of mathematics, now in two controlled experiments and the summaries; transfer to a person's own project is not tested.
Strand D: frameworks and personal process
| Source | What it prescribes before building | Measured |
|---|---|---|
| Schwaber 2020 | a Product Goal, a Sprint Goal and a Definition of Done, as commitments; a sprint of one month or less | nothing in the guide |
| Singer 2019 | shaping an idea before a bet; a pitch; a six-week cycle and a two-week cool-down | nothing in the contents page read; below the threshold on the second rating, an example of practice only |
| Chapman 2000 | phase 1: business objectives, situation, success criteria, a project plan; the process loops back | nothing in the sections read |
| Ubl 2020 | a design doc before coding, to find design issues while changes are cheap; an example only, below the threshold | the owner's claim |
| Serrador 2015 | agile use against project success | correlational, 1,002 projects (secondary) |
| Prechelt 2001b | a taught personal process: plan, log, review | controlled experiment |
- Prechelt 2001b found positive effects, but smaller than proponents assume. The trained group estimated productivity better and made fewer trivial mistakes.
- CRISP-DM names success criteria in phase 1. It is the closest practice to a named baseline.
- Grade for the prescriptions: convention, untested.
Strand E: rework and its causes
- Boehm 2001, section TWO: current projects spend about 40 to 50 percent of effort on avoidable rework. Section ONE: a late fix is "often" 100 times dearer, and 5:1 for small noncritical systems.
- Section THREE: about 80 percent of avoidable rework comes from 20 percent of the defects. Two major sources are hastily specified requirements and nominal-case design.
- Fairley 2005 adds that rework is not all bad: iteration needs some, and too little may signal too little review (abstract).
- Hofmann 2001 quotes Jones: requirements engineering is deficient in more than 75 percent of enterprises (secondary).
- Grade: the shares are expert reports. The cause, requirements specified in haste, is supported with caveat by Boehm and by Nawaz 2022 for global development.
Strand F: the trail
- Noble 2009: one folder per project with data, results, doc and source; dated experiment folders; a dated notebook. The reader is "someone else", often the author later.
- Wilson 2017: an overview file (3a), a folder per project (4a), a change log serving as a lab notebook (5f). Its workshops reached over 11,000 people since 2010.
- Sandve 2013: ten rules, first "for every result, keep track of how it was produced".
- Altmann 2002 models memory for goals and the cost of interruption, fitted to a puzzle task. Parnin 2010 on programmers was located, not opened.
- Grade: convention, untested. These guides agree with each other. No study tests that a set of files lets a second person resume.
Answers to the four questions
| Question | Answer | Grade |
|---|---|---|
| Q1 round trips | the process models name objectives and evaluation; the cause of avoidable rework is hasty specification; the benefit of a written scope, baseline or decision list is not measured | convention, untested |
| Q2 topics | an outline raised judged writing quality in an experiment (secondary, Kellogg 1988); a plan written as a topic is not tested | not tested; outline measured for writing |
| Q3 trail | practice guides converge on a README, a folder per project and a dated log | convention, untested |
| Q4 boundary | the DSR process ends in communication; a document's shape is outside the process | convention |