Its a fact that bugs and faults in networking products is not a key issue for customers. Indeed vendors rely on customer testing and deployment to find bugs before declaring their products as fully tested or generally available.
I believe this created a process of moral hazard and false incentives.
2. The Fundamental Truths – Rule 1 – It Has To Work.
Vendors constantly repeat that they are driven by what customers ask for. Customers are constantly asking for new features, a faster box, a cheaper purchase price or new line cards but they don’t ask for less bugs.
Across the board, product managers tell me that customers don’t see bugs as a key issue. Its accepted industry practice that bugs are inevitable, expected and acceptable.
Why ? Why should customers expect to purchase a faulty product ?
Wrong Incentives & Moral Hazard
Vendors have little incentive to reduce bugs. Today, customers are content to waste time, resources and risk business outages while locating bugs, gathering data, reporting them to vendors, working with vendors to implement a fix then bugs will be a low priority inside the development process.
In fact, networking professionals expect bugs. Thats insane. In IT, you expect your product to be faulty and you live with it. In fact you plan for it, allocate budget for bugs, time for testing and implement change control in a costly attempt to miitgate risk.
Accpetance mans that bugs haven’t gone away. That’s why your CIO hates you and the network. Fauly products are purchased, customer spends money & resources to detect & resolve the faults. Its a good deal for vendors who get a free pass on the issue.
Lets consider just one aspect of buying faulty products – We spend vast amount of money on Bug Scrubs and Proof of Concepts that simply should not be necessary.
The method we use to find and report bugs is “technical support” which requires substantial sums so as to be able to report bugs to a vendor. In nearly all cases, tech support is needed to trace/locate/verify and accept the bug. Its highly unlikely that the average customer could locate a vendor bug with high degree of certainty and get the vendor to accept it without paid technical support.
Bug Scrubs and Proof of Concepts
Tech Support services have the wrong incentives and do not drive the vendor to improve their software quality.
Proof of Concept – why am I paying for professional services to test a vendor product for bugs as if I am expecting them to happen ?
Vendor Bug Scrub – vendor staff have access to all of the data since vendors do not publish all of the biugs. They do their best but they aren’t the ones with their butts on the line if they miss one. Wrong incentives again.
Self Scrub – the right incentive because you are at risk however you don’t have access to all the resources to perform this.
I’ve used these as small example of misplaced incentives that encourage vendors to focus on product development instead of products that work reliably.
What if vendors paid a bounty on bugs in their products ? In the security industry, companies like Bug Crowd are managing teams of freelance professionals to locate bugs and pay handsomely when a vulnerability (aka “a bug”) is found.
Sure, there are some complex business and ethical issues here but the incentives are the right ones.
First, customer without support contracts can get bugs recognised.
Second, engineers have motivation to locate bugs and work with the vendor to resovel them. Many times I have located potential bugs but never reported them because of the time and cost of doing so.
Third, public disclosure of discovered bugs is good for customers. Vendors keep the majority of product failures and attempt to make service profits from them. This drive fixes in a shorter timeframe instead of being determined by the size of customer spend.
A New Competitive Strategy
Competition can be achieved in different ways. Cisco has dominant marketshare in the network market
This article first appeared in an Issue of the Human Infrastructure Magazine at Packet Pushers. You can signup to receive the magazine by email and for free by subscribing here.