Are Your Tests Providing a False Sense of Security?

This article is part of our Testing from First Principles series. If you enjoyed this series, stay tuned for practical testing advice on my Cypress End-To-End Testing course on egghead.io

Tests as Checklists

I think of tests much in the way that Atul Gawande describes surgeons' checklists in The Checklist Manifesto.

Our jobs are complex, and we can miss incredibly basic things unless we have a checklist.

But a checklist is executed by a human. It forces us to consider the various complexities and edge cases one by one.

Automated tests are executed by a computer, without manual intervention, and without us being forced to review them.

Which leads to some questions to consider...

• When's the last time you looked at your tests?
• Do they test only the happy path? Or all edge cases?
• Did you mock out some objects to improve test speed? Or is everything tested end-to-end? Do you remember?

Why are you confident these tests will keep your application safe?

When we're confident the "checklist" is correct and complete, we can miss things that would have been blindingly obvious if we thought about the problem ourselves.

If your boss asks, "Is this code safe to ship?" and you reply, "The tests are passing," then you haven't answered the question. You haven't engaged with the process of doing so because it's much easier and requires less accountability to trust in the tests.

Tests can provide security when done right. But when we fail to consistently engage with our tests, that sense of security is undeserved. We have to know what the tests exercise, why, and what's missing.

• Is this test case an accurate representation of how users really interact with our website?
• Are we truly testing features end-to-end?
• Are we using real user data, or imaginary scenarios?

Your tests might be really killer. Or they might be a security blanket when what you need is a shield.

This article is part of our Testing from First Principles series. If you enjoyed this series, stay tuned for practical testing advice on my Cypress End-To-End Testing course on egghead.io