Going back to where we once went wrong

When does a change become a new product?

There is a philosophical question which you might be familiar with, known as ‘the ship of Theseus’. If you haven’t heard of it before, then this video from the British comedy series ‘Only Fools and Horses’ might help explain it.

‘The ship of Theseus’ (and Trigger’s broom) present us with a debatable point; whether an object whose component parts have been replaced remains the same object. In software development we could well ask the same question about the products we work on and the code which contributes (in part at least) to those products.

Certainly a fascinating question to discuss, but perhaps less urgent for us to address than the question of what should happen when a product changes.

You may have heard a discussion that goes a little something like this:

“We’re going to make a few changes to the core application and we need to come up with a plan before the Executive Committee meeting.”

“What sort of changes are they?”

“I’m not too sure yet. There have been a few problems reported since we launched and so we need to fix some of those. The Committee met last week and based on the reports they are getting back from our customers they think we need to address these problems quickly.”

“I guess it makes sense to iron things out as soon as we can.”

“The thing is, whilst we’re doing that, I was wondering if maybe we could add a couple of features that didn’t make it for launch.”

“Ah ok, well I guess it depends on which features we’re talking about.”

“Nothing major. Just to improve a few things whilst we’re under the bonnet anyway.”

“Well, most of the team are working on the new website at the moment. The Product Owner was pretty upset about the delay to that work.”

“Hmm, yeah. I can understand that. The delays to the app launch probably blew any chance of the website being finished on time.”

“Yep, and diverting people back on to the app won’t go down too well.”

“Don’t worry about that – the Committee can work out the priorities. We just need a rough plan for now.”

“I’ll see what I can work out. Mary is back from leave next week and I hadn’t worked out what she might be doing so I can probably spare her for a few weeks.”

“Mary’s a good developer – sounds like a plan. What about testing?”

“Well if we’re only talking about relatively small changes, and we tested it pretty thoroughly for the launch, we probably just need to do a bit of Regression Testing.”

As the desire for more rapid development and delivery of software increases, so the frequency of change required increases too. And whenever we talk about change in software, you can be certain that the term ‘Regression Testing’ will not be far away.

So what do we mean by ‘Regression Testing’?

I’ve heard lots of different explanations of regression testing from testers and I have been left with the feeling that whilst it is a widely used term, its meaning is open to interpretation. The definitions I hear typically sound like one of the following:

“Checking that nothing which was working before has been broken”
“Retesting something to make sure it still works”
“Testing after a fix to make sure that fix didn’t introduce any other bugs”
“Rerunning tests after a change”

Let’s see how these measure up against the definition from the ISTQB glossary:

Putting aside the confusing punctuation, this is a rather strange definition as it suggests that we are only looking for problems in the unchanged areas of the product. Shouldn’t we consider those areas which have changed?

If you work with testers, try this exercise for yourself. Ask them what regression testing is and see if there is a common understanding.

Based on my conversations with colleagues throughout my career, and with people I have interviewed over a number of years, there is some common ground in what many testers understand by regression testing:

Some testing of a product has already happened
A change of some sort has been made to that product
Some further testing of the product takes place

Beyond that, there are different interpretations and opinions on the scope and objectives, based on what an individual has been taught or has experienced.

So, let us operate on the understanding that among testers there is not a consensus on what regression testing is. What would happen if we broadened this experiment out? How might a Project Manager, a Product Owner or a Developer describe regression testing? How might a Senior Manager or decision maker interpret the term? Would they believe that regression testing is focused only on the specific changes made, on other (unchanged) parts of the product, or that it is actually a wide ranging test of the product?

Why does any of this matter?

Divergent interpretations of terms raise the possibility of miscommunication, which can lead to confusion and in this instance, misplaced confidence.

If, our client believes that the purpose of regression testing is to repeat earlier wide ranging testing of a product, but our testing is concentrated on a narrow area of change then our client has a significantly different expectation. A poorly communicated message – “No problems found in our Regression Testing!” – could mean something quite different to our client.

Finding a problem in some part of a product which was previously working as intended clearly tells us that something has gone wrong. Failure to find a problem in that part of the product which has changed does not tell us that there are no problems with the product.

Changing the ‘R’ word

Conversations about testing following a change to a product could be more meaningful, and less open to interpretation if they were less concerned with regression and instead addressed another ‘R’ word – RISK.

Any change to a product presents a risk that problems or bugs are introduced (or reintroduced) to that product. If there is a risk, then testing can help us to understand it, but we might first make some assessment of how much risk a change presents:

how likely it is that there will be problems?
if there are problems, what does this mean for the product and of course, the people who will use it?

In answering these questions, we can use information resulting from earlier testing of the product to help us. This includes observations we made, and problems that were identified during that testing. If we encountered many problems in the part(s) of the product which are changing, this might tell us that further problems are likely. If what we learned through our testing tells us that critical ‘downstream’ actions or outputs could be profoundly affected by these changes, this might indicate that there is the potential for severe disruption to customers.

There are many other factors which might influence a discussion about our strategy for testing; time, cost, conflicting demands for skilled people, and so on, but the findings of earlier testing can inform a broader conversation about risk, and help decide where we might focus our efforts now.

Whilst testing, the tester may decide that those earlier tests and any problems or bugs identified can be useful again. If the change has been made in response to an earlier bug (in other words, a fix), repeating the original tests might be useful, but a different approach may also be desirable. It should not be assumed that repeating tests will find all the problems that could have been introduced by the change, or to put it another way, that the risk to the product has been fully explored.

In considering our response to change it may well be helpful to remember ‘the ship of Theseus’ and to pose another philosophical question:

‘If we change the code or components which contribute to a software product, do we have a new product?’

This isn’t an easy question to answer, but by considering it we might change the conversation about testing. If our change presents us with risk, then shouldn’t we consider that risk with the same rigour that we would for a new product?

I have a definition for ‘Regression Testing’ which you can find here.