The Leprechauns of Software Engineering is a deconstruction of different claims of ‘fact’ in software engineering that are shown to have dubious origins. The three main claims discussed are: the origins of the waterfall process, 10x programmer productivity, and the defect cost curve. If you are a fan of meta analysis this book is definitely something you should check out. Even if you aren’t into that sort of thing the ‘facts’ discussed are commonly being cited in other places that are accepted without much thought.
The 10x study was especially affecting, because while many of us have anecdotal experience that says that this is true; however, the research that supposedly shows it was so haphazardly done that it shouldn’t be trusted. For instance, some of the supporting studies had people debugging a piece of code who weren’t familiar with the programming language it was written in, but were included in the study anyway. Of course the people with greater familiarity with the programming language would be greatly superior at debugging. In other studies, the researchers measured lines of code per hour as a measure of skill, but more lines is not necessarily the mark of greater productivity. Some of the studies had different participants using different programming languages and tools; some measured whole projects. Altogether, it wasn’t information that feels like it shows the claim with the certainty it was normally made with.
The investigation of the defect cost curve was more about the definition of the defect, the definitions of the various phases, the numbers attached to the curve, and how somewhat reasonable claims got distorted into the commonly seen form. The evidence does not show a rigid cone with phases and multipliers. However, to me, it shows that the cone does exist, but it shows the best you can expect to do, not how well you will do.
Overall, the book is an impressively researched and cited effort that digs through years of history to try and uncover the truth. It calls into question some of the foundational ‘facts’ of software engineering that may seem true, but we haven’t bothered to do the rigorous research to prove it. I want to help in getting the word out there on these misconceptions, and hope you join me. The sooner we acknowledge the basic research isn’t there the sooner we can build that solid foundation on the truth.