Words on software and web development


Archive for the ‘Root cause analysis’ Category

Root Cause Analysis

Recent events with the large Toyota recall for problems with the software controlling inputs from the brake and accelerator (which are no longer mechanically linked to the systems they control) highlight one of my pet peeves in the approach a company takes toward software development. Whether Toyota is guilty of this or not, I have no idea, but it’s a serious issue.

There is a related topic, which is whether software development is treated as software or as electronics. I’d like to deal with that separately, because it’s equally important (and usually, equally tied to a corporate approach to product development).

Regardless of how the software in a system is designed or architected, it is ultimately coded, reviewed and debugged by coders. When coders are driven by management to meet management goals such as adding features, closing open issues, etc. it is easy for them to lose sight of one of the most essential parts of software development: root cause analysis.

Root cause analysis has a motto: “fixes are cheap, bugs are precious.” When actual reproducible problems are found, they need to be cherished and treated as precious opportunities to find defects in design or in implementation. But for middle managers, this might mean that the programmer will be spending a few extra hours investigating a problem that could be fixed with a simple workaround. Although there is often the justification of “we can come back to that later,” we all know that especially in these crunch economic times, later never comes.

Sometimes a major defect manifests early warning symptoms in a seemingly innocuous bug which is only an annoyance. If a programmer feels empowered to follow the natural instinct to fully understand the problem, he/she can put some additional effort into chasing it down. Of course, this can be done in excess because while it’s nice to pursue zero defects in software, we have to get products out or do whatever pays the bills. But having the right balance in any software development organization is definitely the responsibility of leaders and management.

Managers need to understand the importance of root cause analysis, and need to encourage the front-line coders to pursue leads, because who knows? The next annoying little glitch you find in your test program’s results could be the next recall of 6 million vehicles, and a lone vigilant coder could be the one who went out on a limb to find it.