Benchmarking Value

Abstract

We’ve spoken a few times about the need to benchmark value in technology management. This is an entirely different concept from benchmarking the nuts and bolts of a deployment such as a software application. The latter can be done to a large extent numerically — does the application do what its supposed to do, if not to what extent does it not fit the requirement? What are its metrics in terms of response speed? and so on. All these can be identified and measured. I have to say, however, that even in this area, practice is polarised. One European bank has a 6-month test cycle for new software which costs €150,000 each time an application enters the pipeline. New vendors must submit to this test routine even before their software is delivered into a user acceptance testing (UAT) phase. Any change made or fault identified is logged and any fixes that are required must be built into a new version and resubmitted into the 6-month test cycle. This gives you a very clear picture of the level of conservatism in financial services. Another company I came across, a software vendor, had no quality control at all for the first ten years of its existence. Bugs found by clients were reported, fixed by the programmer that wrote the original product, tested by the same programmer, documented by the same programmer and released to the client by the same programmer. As a result, because different clients found different bugs, essentially this company was writing code ‘on the fly’ and maintaining over twenty different variations of the same product since the programmer couldn’t fix the problem in someone else’s code.