This page looks at questions you can ask whilst reporting, about the information you intend to provide. There is an associated blog post about filtering information as part of the ‘Assisting with inquiries’ series .
Here we address the broad question of WHAT information you might communicate. The answers to the questions are not necessarily intended to be included directly in the information you provide to your audience, but they should help decide what information is useful.
Below, I have briefly explained each of the questions in this branch of the mind-map and how you might answer those questions. Where possible, I have tried to provide some practical examples, based on my own experiences with reporting.
WHO?
Who is the customer (for the product)?
Understanding the person or people who will use a product, and their needs and expectations, will help guide your testing and also assist in identifying problems. When reporting on testing, this understanding also helps to frame the story and to explain why you perceive something as a problem. Referring to the customer and how they might experience using a product should help your audience to see that product (and the quality of that product) through the eyes of a real person.
Who was involved in testing?
The people you report information to may appreciate the opportunity to talk to other people involved in testing. They may need more information or some help in understanding the information they have been given. They may also be operating under a misapprehension about who was involved and may feel that someone else could provide a different, and interesting perspective.
WHAT?
What is the product?
Your audience may have detailed knowledge of the product you are testing, or they may have more limited knowledge. In some cases they may know very little about it. In some other cases they may know more than you do. Perhaps the product consists of a number of different elements which some of your audience understand but others do not. It is helpful to clarify what you mean when you refer to a product, particularly if making broad and wide reaching statements about its quality.
What might the customer desire or expect from the product?
The testing you do is likely to be influenced to a significant extent by the desires and expectations of the people who the product is intended for. It follows therefore that you should consider this in the information you provide. It might also make your reports more engaging for your audience. If, for example, a bug can be explained in terms of the way it affects a customer’s experience, then this may help people to understand how urgently it should be addressed.
What have you tested?
This is a significant element in the story you tell and (along with what you haven’t tested) provides your audience with some important context. If, for example, you report that your testing of a mobile application has not uncovered any significant problems, this information is misleading if you fail to also mention that so far you have only tested the installation of the app and none of its features. This may seem obvious, but in complex products which may not always be well understood by your audience, it is important to be clear on what has (and has not) been tested.
What have you been unable to test?
Although this is linked to the previous question, it is worth considering anything which you have been unable to test, perhaps due to factors outside your control. There may be elements of the product which are unavailable, or there may be bugs which are preventing you from reaching some other parts of the product. The implication here is that you intended to test something but for some reason could not.
What have you discovered?
You may report to some people on a frequent basis, in which case you could amend this question to ‘What have you discovered since you last reported?’ Your judgement in what to include will be shaped by your understanding of the needs of your audience. It is unlikely that they will want to hear about everything that you have discovered, so selecting the most useful information is important. Where there are many discoveries to report, it is also essential to give prominence to those that you believe are most significant.
What problems have been observed?
Sometimes you may be able to provide details of bugs which have been identified through your testing, but other times you may simply have a sense that something is not quite right. You may, for example, be concerned about the performance of a system, without having had the opportunity to carry out any benchmarking. Reporting this problem may trigger a decision to undertake a more thorough assessment of the system’s performance under different circumstances and loads.
What are the most significant problems affecting your testing?
You may encounter problems relating to the product itself which prevent you from carrying out some testing. You may also have more prosaic problems, for example the absence of a colleague whose help you needed. If there is someone who can help you to solve problems, or someone who needs to understand why you might not have provided some information which they are expecting, then it is sensible to include this information.
What are the most significant problems affecting the product?
This is a subjective assessment, but there may be some problems with the product which are clearly significant. It is worth emphasising these in your reporting and in some cases it might also be sensible to include some information on what steps are being taken to address these problems.
What might the customer consider a significant problem?
Although you are likely to have some means of categorising bugs according to how serious they are, these sort of categories do not always consider the customer. For example, an inconsistent use of colours in a website may be considered trivial by some definitions, but this may be considerably more serious to the (approximately) 8% of men and 4.5% of women who are colour blind.
WHEN?
When was the testing carried out?
Your audience may be interested in when the testing took place and how up to date the information is that you provide. There may be some other factors which they are aware of which make this particularly significant, perhaps relating to the code or the environment in which you were testing.
HOW?
How did you test (what was your approach)?
Some information on how you tested will help your audience understand what you are telling them, and might prompt them to suggest changes to the approach. If, for example, your testing was time-bound and some priorities were defined beforehand, then an awareness of what was deemed lower priority might result in a shift in priorities once the observations from that testing are reported.
How does the information you are providing relate to the product?
There are times when the tasks and activities relating to our testing, and the problems we uncover might be important to us, but not necessarily valuable to our audience. As a rule, most of the audience for information on testing are more likely to be interested in the product than in the day to day difficulties experienced by people involved in the testing.
How did you validate your assumptions and interpretations of the results of your testing?
It is often helpful to seek a second opinion on the results of testing, particularly if you have built up good working relationships with people who have a better understanding of the implications of those results. This may also help you to understand what information is valuable to your audience.
How did you recognise problems (what were your oracles)?
It is helpful to keep this question in mind whilst reporting on the problems you encountered (whether relating to the product or otherwise). As your audience absorb the information you provide they may well ask themselves the question ‘Why is this a problem?’ Understanding why you believe there might be a problem will help them make this connection.
How could your testing be improved?
You are likely to refine your approach to testing a particular product based on the behaviours you observe. You may, for example, decide that you have not considered in enough detail the different ways that a product might be used by different people, so you may decide to make use of personas during further testing. Keeping your audience apprised of changes will help to keep them engaged and may also spark some ideas from them about other ways you can improve your testing.
Quick links to the posts and pages in this series:
Assisting with inquiries – blog posts
- Introduction to the series
- Part 1 – your audience
- Part 2 – the mechanics
- Part 3 – filtering information
- Part 4 – how was it for you?
richrtesting 