The Misinformation Game: three common pitfalls in test reporting

If, like me, you believe that one of the most important things for a tester to do is to provide information to people who make decisions then I have some observations on how this reporting can sometimes go wrong, and some suggestions on how to avoid your message being lost…

The Extrapolation Game

A common mistake when reporting is to take a pattern of behaviour and project an outcome based on the assumption that the pattern will continue. Let me give you an example which demonstrates why this is can be a flawed approach:

In 1977 (the year in which Elvis Presley died) there were around 170 Elvis impersonators.

By the year 2000 the number of Elvis impersonators worldwide had grown to 85,000.

At this rate of growth it is anticipated that by the year 2043, everybody in the world will be an Elvis impersonator.

We can laugh at the nonsensical assumptions built into this projection, and yet I can also recount a conversation with a Project Manager which went something like this:

PM: Can you please provide a graph showing the rate at which the outstanding defects will be fixed?

Me: Sorry, I don’t understand.

PM (patiently): I’ve promised the Steering Committee that I will let them have a view by tomorrow on when we can expect the defects to be fixed.

Me: Yes, I understand what you are asking for. I just don’t understand how you expect me to predict this.

PM (less patiently): Well it’s obvious isn’t it? Just look at how many fixes the developers have been providing each day and then look at how many defects we have. You can then tell me how many days it will take to fix the rest.

Me: But how will I know how many more bugs we might find? How can I understand the complexity of the work required for the developers? How can I know how much more testing might be required because of the problems we identify?

PM: (exasperated) Oh forget it. I’ll just give you a date when we need to finish and you tell me how many bugs they need to fix each day to meet it.

If you are asked to make predictions based on a pattern of behaviour observed up to this point in time, think very carefully about whether it is sensible to do so. The people who ask for this are likely to hold you to your prediction.

It is possible to use past behaviour as an indicator in some cases, perhaps in instances where significant data exists from multiple similar activities over an extended period of time. My advice is to carefully communicate your assumptions and only agree to this approach if you are comfortable with the decisions that are being made based on your prediction.

Good testers have many admirable qualities, but prophecy is not typically amongst them.

The Trepidation Game

Being responsible for reporting on testing is not always easy. There is a likelihood that you will have to deliver some disappointing news:

Perhaps you or your team were unable to complete the testing that was planned.
Perhaps you were able to test but you found some problems or bugs which will mean more work for the developers.
Perhaps you found something which means that a product launch will have to be delayed.

In each of these circumstances someone is likely to feel disappointed; the manager who wants to feel that his or her project is progressing according to an agreed schedule, the developers who wanted to start working on some new feature rather than fixing something they believed was working, the senior stakeholder who has to explain why the new website will not be available as promised.

This can be unnerving for some people. There are some organisations, particularly large organisations, where some of the people who take an interest in the progress and findings of testing have significant influence and power. There can be intimidating people with fearsome reputations and it is not uncommon for a shy, retiring tester to back away from delivering the bad news.

I have yet to encounter any decision maker who didn’t want to hear the bad news as soon as possible. If you are of the opinion that a product or application is not fit for launch, I can say with confidence that you would be better off making the case at the time when you come to that conclusion, rather than waiting until days or weeks later.

Follow the golden rule, “NO SURPRISES” and this will help you overcome the fear of speaking out.

If you use facts and present your opinion with reasoning, having already explored some possible solutions, it is highly likely that you will be treated with respect and your opinion will be sought in the future.

The Obfuscation Game

“88.2% of statistics are made up on the spot” – Vic Reeves

There are many good and honest people working in software development. There are also some unscrupulous people. There are some who will see the delivery of software on time as an objective which must be achieved, regardless of the implications.

Typically these people will not be around to deal with the consequences of delivering faulty software.

The poor people in the Operations Team, the people on the help desk and of course our friend the customer; these are the people who are left with the fallout.

I’m sure many testers have witnessed earnest discussions about the validity or categorisation of bugs during the latter stages of projects – “Is this really a bug? We can just close this one off can’t we?”

I’m sure many will also have seen colleagues who were once keen to ensure that comprehensive testing took place, make sudden U-turns and act as advocates for less testing – “I’m sure it will be fine. It isn’t like we changed much here.”

There may even be one or two who have produced a report only to see a sanitised version of that report make its way to more senior people.

My advice here is to build relationships with decision makers as early as possible and involve them in your testing activity. Ask them what sort of things concern them, promise to test these things and keep them informed on progress. If you distribute reports include them in the circulation. Try to catch them for a chat about your work. Invite them to come and talk to the testers about the kind of things they are seeing. This is the best way to circumvent anyone with a desire to hide your messages.

If you enjoyed the blog, please feel free to share and comment. Do you have any other common pitfalls in reporting? How should we avoid them?