The Nether World of NoTest

Ever ventured unwittingly into the shadow world of NoTest? It’s the hazy utopia where the ivory tower types proclaim that if the design is good and manufacturing is good then why test.  Sometimes this even came up in a snarky way, back when I worked in an integrated contract design and manufacturing (CDM) environment, as in “You did my design and you’re going to manufacture it, so then you don’t need test . And oh by the way take the test line item out of the open book pricing I made you provide.”  Followed by a sneer, or at least I envision a sneer in my memory when I think back unfondly on those days.

Of course doing without test in the real world is, well, fantasy.

Don’t get me wrong I’m a huge fan of doing a solid design, employing six sigma techniques where cost effective, and conducting serious design validation testing long before production release.  I’m also a fan of spending the effort needed to productize a design–get it ready for production and get production ready for the design.  But despite best efforts and intentions, I am also a believer in both Murphy, he of that oft quoted Law, and also in statistics, which basically say that even with a robust design and a low DPM (defect per million) manufacturing yield loss there are still product that will fail, and its better to know about a failure sooner rather than later.

Of course the level of test absolutely should be commensurate with the product robustness, yield issues, and the total cost of field failures (which while high are not infinite) including impact on brand, returns, customer satisfaction.  And while in reality NoTest is an extreme that is rarely ventured, many companies do neglect developing and implementing a production test plan that is as robust as their design methods and production practices. Often times this is due to lack of understanding of how test works; sometimes its part of the “beat up your contract manufacturer,” and all too often it’s just part of the productization chasm that widens with time as design teams and their manufacturing counterparts drift further and further part. (for more on this see our productization blog on the chasm analogy,

The right solution is really to put the same effort into making sure your test plan is robust as you do for design, marketing or any other function.  Start with requirements, look at tradeoffs, come up with a draft plan, scour it for cost effectiveness and get buy in from the cross functional team.  Make sure the risk / reward equation is properly balanced, and, as my Swedish friends would say, make sure the amount of test is lagom–neither too much nor too little.  And avoid the sirens as they lure you toward the shadowy land of NoTest.



  • Don

    Saving money is also the consumer’s response. When an item fails, the victim or former customer tells the world about the shoddy products made by that company. He tells everyone he knows for the next 10 to 20 years! These days, he talks in great detail on a blog, on a user group or other special group, then Google or Yahoo or Bing or any other search engine can find it as long as it the group or blog survives.

    I had a Motorola TV that died in 2001 after 4 months. Two authorized repair shops and 13 months after it failed, I was told by the second shop that even though they had it in their shop for 8 months, they were not authorized to repair it since it was opened when they received it (from the first repair shop when they could not repair it!).

    I do not know how many sales my story has cost Motorola, but that is the choice they made when their quality control failed their customer, and it was then compounded by their Customer Dis-service department.

    I know of 3 Motorola TV sales that were not made to friends that chose another brand. I know that I have not owned ANY Motorola product of any sort since then.

    See? I just told the story again! It does not matter if they fixed the problem later, they will never regain a previously satisfied customer, ME. And I know that many people have repeated my story to others over the years.

    Once you provide a flawed product to the public, it is public record, and all you get is what you earned, a bad reputation for product quality.

    TEST those products!

  • Vince Hileman

    You are absolutely correct. The big risk in “no test” comes with design changes that are “paper quals”. In complex computer systems, small changes in firmware can cause big problems in production and test compatibility. You may be interested in reading the chapter on Pass Fail testing in Forrest Brefogle’s book “Better Six Sigma Solutions” He discusses a factorial method that makes for very efficient and effective testing.

  • Chuck Post author

    Vince, good point re paper quals. The other area that I have found is risky is emulating the analog real world that the product sees in actual use–for products that have inputs or outputs other than digital. E.g. video, audio, sensors, etc.

Leave a Comment