richrtesting

It isn’t always easy to identify the changes which affect us. Some changes require a sharp reminder of how things used to be. Take a look at a photo of yourself from several years ago and, if you are anything like me, you will recoil at the sight of a ridiculous hairstyle or some fashion faux pas. You may well ask yourself, “What was I thinking?”

I have become aware of a change in myself which I believe has taken several years to develop. I have become more uncertain.

I suspect that this is related to a realisation which I discussed in an earlier article – the realisation that I actually know very little. After all, if there is much that I don’t know, then how can I really be certain of anything?

Probably the biggest clue to this change has been in my use of language. I realised that some words or phrases feature regularly in my written and spoken communication. Here are some examples:

• Sometimes
• May
• Often
• Is likely to
• Tends to
• I suspect
• Probably

These words or phrases now occur more regularly than some others which I believe I used to use more often:

• Always
• Definitely
• Must
• Should

In my working life, I feel much more comfortable with my uncertain self. It occurred to me that uncertainty in testing was actually a very good thing, because certainty in testing may be something of a problem.

When we test software we are of course dealing with uncertainty. From the early stages of development, there are likely to be many people who have an interest in the product and they are uncertain about it. They do not know whether it will operate in the way it was intended to, whether it will facilitate the tasks or processes which it supports, whether it will be useful or whether it will even be used.

Testing can help address that uncertainty. Iain McCowatt explains this in his excellent series of articles about uncertainty. Iain also discusses how uncertainty is something we should embrace. In particular, I share the view that uncertainty drives us to test. Where we are uncertain we are likely to investigate further.

“doubt is not a pleasant condition but certainty is absurd” – Voltaire

I have observed that an unreasonable expectation is often placed on testers; the expectation that we can not only reduce uncertainty, but that we can eliminate it. In other words an expectation that we can provide certainty.

I understand why this expectation may occur. After all, businesses and organisations can spend vast amounts of money on development and testing of software. There is often great anticipation and excitement about the products which are developed. These products can make or break a business through the investment made and the return achieved. Decision makers can be under great stress.

Naturally then, there is pressure to answer questions about quality with absolute statements. The question, “Have you tested the product?” is likely to be asked.

The response, “We have tested some of it” may be unpopular, but it is really the only honest answer. It should of course be followed up with some more information (in many cases, much more) about what has been tested and what has not been tested.

The absolute response, “Yes, we have tested it” would be dishonest and misleading. As anyone who has read Gerald Weinberg’s ‘Perfect Software: and other illusions about testing’ will tell you, testing is infinite and therefore it is not possible to fully test any product.

I recall a situation where, after leading testing of a product, I was approached by an angry manager shortly after that product was released. The manager explained how a major fault had occurred in the live system and that the fault was causing confusion and delay. They demanded an explanation, because (the logic went) we had tested the product so we should have found the bug which caused the problem.

I explained that I would look into it but that I couldn’t guarantee we had tested this part of the system under the circumstances described, because we did not test everything. This caused an angry reaction, which surprised me as the manager had been present in meetings where I had explained what we had and had not tested, and where we thought problems existed.

As it turned out, we had tested this part of the system and found the bug. The bug had been fixed but an error had been made in the way the code was deployed and the bug had been reintroduced.

A desire for, or expectation of certainty can cause testers other problems:

• We can be expected to prove that ‘full coverage’ of a product has been achieved by mapping tests to requirements. This does not prove ‘full coverage’. At best it can prove that at least one test script has been written which provides for a degree of testing of each requirement.
• Associated to this is a belief that requirements documentation is a ‘source of truth’. In some testing teams and organisations, testing which does not directly address the requirements is frowned upon. This despite much evidence that requirements are often ineffective at accurately capturing intent and expectations.
• Perhaps the most significant problem with certainty is that its natural conclusion is likely to be a belief that there is only one solution to a problem, a guarantee of success, a standard which applies in all circumstances. I will save that subject for another day.

In confronting some of these expectations and beliefs, we may encounter some difficult conversations. The alternative – to accept that testers can provide certainty – will in my opinion lead to many more difficult conversations.

Adopting more cautious language may well help in your own conversations about testing. It may also help others adjust their expectations of testing and help us all to have more meaningful conversations in the future.