Hitting the mark with test reporting

Have you ever encountered someone who doesn’t respond well to bad news? Have you ever worried about how to approach difficult conversations with colleagues or clients? Perhaps you anticipate an experience a little like this:

The good news is that I have yet to witness this level of anger from managers or stakeholders in software development work. The bad news is that sometimes colleagues might believe this is the kind of response they will get. And sometimes their worries lead them to make poor decisions in the way they choose to report information.

Sharing the news

There are many jobs in which an ability to report important news (whether good or bad) is crucial.

In medical care for example, our experience is often shaped by the manner in which doctors are able to communicate to us. If they get it right, even patients with serious conditions can be left feeling more at ease. If they get it wrong, people may feel confused and perhaps panicked. They may make the wrong decisions based on the information they have.

A financial advisor must notify clients of potential warning signs in financial markets, or perhaps new legislation which might affect people’s investments. If events occur which have a dramatic effect on a client’s future plans then the advisor’s first task is to inform them before recommending a course of action. If the information provided is not clear then the actions which follow might be made in error.

Jobs associated with software development often include a reporting element too. For anyone involved in testing this element is critical.

When your work requires you to give people news about something important to them, then the timing of that news and the detail you go into are also important. Effective communication is unlikely when reporting occurs too early or too late, or if too little or too much information is provided.

Imagine a doctor revealing that a patient has a serious illness several weeks after the test results are returned, or reporting that a blood test shows normal levels of cholesterol but failing to mention that high levels of liver enzymes are present, perhaps signaling liver disease.

Imagine a financial advisor calling a client each day to report small percentage adjustments in a pension investment which is not due for payment for another twenty years, or sending them a 200 page financial report for a company, without providing a brief summary of relevant points.

These examples seem ludicrous because we would expect people in these professions to keep patients or clients properly informed so that they can make the right decisions. Shouldn’t stakeholders or clients of testers reasonably expect the same?

Testers have a duty to keep their clients informed about what they discover and in order to do so they need a good understanding of who their stakeholders or clients are (the audience), what matters to those people, and how they like to be kept up to date (reporting mechanisms).

In deciding on the last of these, timing and depth of reporting are important factors:

A shortage of information means that reporting is likely to be inadequate for the people who need it to make decisions and take actions.
Too much information can be opaque and important messages can be obscured.
If that reporting is also too early then the information provided is sketchy.
Early reporting could be misleading if there are investigations ongoing which may help to better understand behaviour or result in a resolution to a problem. If that reporting is also short on information then the reporting is sketchy. If too much detail is also provided it could be confusing, particularly when there has not yet been meaningful analysis of what it might mean.
A measured report might still be ineffective if information arrives too late to affect decisions. If an excess of information is delivered late, it may be distracting; requiring time to absorb and filter whilst diverting attention from other important activities. Meanwhile, providing too little information too late is neglectful, leaving clients and stakeholders exposed and misinformed.

How do we know how much information is the right amount and when people might need it?

As with so many aspects of a tester’s work, the answer is subjective. There is no standard that will give you the right response here.

One simple phrase which often helps is ‘No surprises’. Withholding important information (perhaps due to a fear of delivering bad news) is a dangerous strategy. Keep in mind the patient waiting for the doctor’s diagnosis or the investor depending on their advisor’s financial summary. Your stakeholders will not thank you if you keep them in the dark.

Yet there are still judgement calls to make. Instinctively you may feel that your stakeholders need as much information as possible, as soon as possible, but are there circumstances in which caution should be exercised?

Perhaps you have found a problem but your team feel there is a good chance that it can be investigated and resolved quickly. You have talked to a developer who has a convincing hypothesis on the cause and how it can be resolved. You know you can rapidly deploy and revisit their fix to determine whether they have in fact resolved the problem, and to investigate whether they might have inadvertently caused some other problem.

Do you need to tell your stakeholders right away? You might feel that you can hold off for a while, that it is better to talk to them once you have more on the investigation, and potentially some good news about a fix.

What if the problem is potentially very serious to the organisation? Should you still wait?

Would you change your decision if you remembered that your product owner is due to update an executive committee meeting that afternoon? Shouldn’t they be made aware and have the opportunity to decide for themselves whether to alert others?

Perhaps you have some other information relating to testing. There are no problems to report; in fact you have observed a marked improvement in system performance since some changes were made.

Should you tell your stakeholders right away? The Operations Manager has concerns about performance after some previous problems in the live environment. You might feel that this good news is something they need to know.

What if there are further application changes to come which might affect performance? Would it be better to wait until further testing has been carried out?

Would you change your decision if you knew that some staff training was planned for later the same day. Perhaps the Operations Manager would prefer to hold off taking further changes in the training environment and would like the opportunity to carry out training whilst things were running well.

These are the kind of situations that might occur during the course of some projects, and the kind of questions that must be considered. Getting the right balance is not always easy but awareness of how your decisions might affect others could help you to hit the mark.