Are Bad Tests Better Than No Tests? BLEEEEP NO!

By June 13, 2014 Uncategorized No Comments

My pal Justin Searls posted an interesting Tweet last week asking for feedback on a slide from the RubyNation conference stating “‘Bad tests’ are 100% more awesome than no tests.” Of course, being Twitter, there was absolutely no context around that slide. Perhaps the presenter was in the midst of a sarcastic, ironic stint on why you need to take care in writing your tests. Being a tester, I also wanted to know more about what was meant by “bad tests” since one person’s bad may mean totally different things than another’s.

I thought the straight reading of the slide was good fodder for some thoughts around bad tests. I’ve some strong opinions on the topic.[1]

Also, setting some important context, I’m not focusing on automated tests. A test is a test, regardless of whether it’s automated, manual, exploratory, or whatever. Some in the testing domain descend into pedantry debating “testing versus checking” or “sapient versus non-sapient”

What’s a Bad Test?

Good tests give the organization useful, valuable information about the state of the system. Good tests increase the “trust surplus” others in the organization have about the system and about the delivery team. Good tests minimize the cost of building and maintaining the test over time.

Everyone’s vernacular is different, but I’m the one writing this blogpost, so my definition of “bad test” wins.

To me, a bad test is one that:

  • Provides no useful information about the system
  • Provides false information about the system
  • Provides a false sense of security about the system
  • Increases cost of building and maintaining the test suite over time
  • Decreases trust in the system and the delivery team

Why Are Bad Tests Worse Than No Tests

With the above definitions in mind, it’s not too hard to see why bad tests might be worse than no tests at all. Bad tests make everyone dig the quality hole deeper and increase everyone’s frustration and stress levels.

False Sense of Security

Worse yet, organizations may be led into a horribly misplaced sense of safety when in reality the system is in a very bad state. Bad tests may be missing huge swaths of functionality, despite impressive but useless metrics. (“But we have 95% code coverage!” Yes, but they were lousy tests that did no effective checks or provided incorrect information.)

Absolution of Responsibility

A particularly insidious effect of bad tests is driving a cultural sense absolving the team of responsibility. Instead of constantly asking “What other information do we need to search for?” teams end up falling back on excuses like “We had 3,342 test cases with a 94.6% pass rate. Someone else missed something.”

Loss of Trust

Worst of all is the downward spiral of trust that bad tests inevitably create. Intermittent failures lead to frustration with other delivery team members. Bad tests miss critical bugs that your users end up finding–and ask your support folks how they feel when they’re getting hammered by angry customers.

Don’t Settle for Bad Tests

Delivering high-value software isn’t an easy thing. Avoid getting sucked into bad tests that mislead your team, organization, and customers into a never-ending spiral of doom. Work hard to have your entire team involved in creating high-value, high-quality tests regardless of whether they’re automated, manual checklists, exploratory sessions, or other tests.

Bad tests aren’t better than no tests. They’re potentially far, far worse.

[1] I know, you’re shocked.

The following two tabs change content below.
Jim has been in various corners of the IT world since joining the US Air Force in 1982. He’s spent time in LAN/WAN and server management roles in addition to many years helping teams and customers deliver great systems. Jim has lead successful delivery teams at software consulting firms and product companies alike. He’s worked with organizations ranging from start ups to Fortune 100 companies to improve their delivery processes and ship better value to their customers. Jim’s worked in many different environments but greatly prefers those adopting practices from Lean and Agile communities. When not at work you might find Jim in the kitchen with a glass of wine, playing Xbox, hiking with his family, or banished to the garage while trying to practice his guitar.