richrtesting

Two swimmers are standing on a beach, looking out at the glistening water of a beautiful ocean bay. It is a warm day and the salty scent of the sea drifts in on a gentle breeze. The sea appears calm beyond the breaking waves. The beach is clean and there is no sign of litter, seaweed or debris in the water. The conditions seem perfect for them, yet the swimmers are apprehensive. They wait together, unsure as to whether to proceed. They are concerned that there is something else out there, lurking in the unseen depths.

“How long has it been?”, asks Stephen.

“The last sighting was two days ago”, Karen replies. “What do you think? Should we head in?”

“I’m just not sure. I mean it could be long gone by now. That’s if it was really here. But then….”

Stephen’s voice trails off. He knows there is no point in speculating. The truth is neither of them really know whether it is safe. If they venture into the water they are taking a risk.

It all feels faintly ridiculous. They know these waters, have swum in them for years; a familiar aspect of their lives. There has never been a problem before, but recently things have changed. There have been sightings of huge dark shapes in the water, passing beneath fishing boats and in some cases visible from cliff tops overlooking the sea. Rumours are spreading that a large shark is patrolling the area, searching for prey.

“How about we go in, but stay close to shore”, Karen muses. “That would be safe wouldn’t it?”

“Maybe… but I want to get further out, beyond the break. It’s no fun trying to swim with the waves crashing over you.”

They are quiet for a few moments, then Stephen has an idea.

“What we need is a boat”, he says brightly.

“A boat?” Karen asks. “What for?”

“To search the bay of course”, he replies. “We can ask one of the fishing crews to do a scout of the area. Make sure there are no sharks about.”

Karen considers this. It is true that there are plenty of fishing boats nearby and they know many of the people who work on them.

“I guess we could…”, she says “But they can only really see the surface. I mean the shark could be in the deeper water couldn’t it?”

“OK then”, replies Stephen, undeterred. “A net. That will do it.”

“A net? What sort of net?”

“A big one”, he says. “Something we can use to check the deeper water before we go in. One of the trawlers maybe. They could drag the net behind them and that way we could be certain there is nothing deeper down.”

Stephen is excited about his idea, but Karen is still not convinced.

“The thing is”, she says “the boat could only really make a pass of the bay. There’s a big old ocean out there and it’s not like sharks stay still. Even if they searched the bay, how do we know that the shark won’t come along afterwards?”

“It’s like you don’t want to go for a swim”, Stephen replies grumpily. He is feeling irritated that Karen is less than enthusiastic about his ideas.

“I do! Of course I do. I’m just feeling a little nervous that’s all. Look, you see if you can get one of the boats to come round and we’ll give it a try. If nothing else, the sound of the boat’s engine will probably be enough to scare the shark away…. that’s if there even is a shark. But lets not stray too far from shore. OK?”

“It’s a deal”, Stephen replies.

He heads off to contact one of the local fishermen, and within minutes is back with good news.

“We’re in luck!”, he says. “Jim is just heading out and said he will do a search for us.”

Sure enough, Karen sees a boat passing around the headland and beginning a systematic sweep of the bay. Starting close to shore, it moves from one side to another, gradually moving further out to sea. Eventually they see Jim on the deck, waving a green flag.

“That’s the signal”, declares Stephen. “Red flag means ‘do not proceed’. Green flag means we’re good to go. No sign of a shark! Let’s do this.”

Feeling more confident, Karen prepares herself and heads into the water. It is cool at first, but as she moves further out, she becomes accustomed to the change in temperature. The water is up to her waist now, and looking to her left she sees Stephen several metres away, diving head first into a breaking wave.

She prepares herself to follow suit, but stops short as Stephen emerges from the water and lets out a scream of pain. Feeling her blood run cold she shouts out to him.

“Stephen! What’s wrong? Are you OK?”

He doesn’t, or perhaps can’t, answer. Instead he gestures at the water. Looking down, Karen struggles to make sense of what is happening. She can’t see anything in the water and begins to make her way towards Stephen. Now he manages to speak.

“Don’t come any closer!”, he shouts. “Get out of the water!”

“Why? What’s happening?”, she answers, desperate to understand and to help her friend.

 “There’s a bloody Jellyfish!”, Stephen replies “I’ve been stung.”

---

When we make changes to our applications and systems, we introduce risk. Much like Stephen and Karen, we can take precautions intended to help us identify problems and minimise those risks, but even when we search for problems we can miss important things and if we do, we too can sometimes get a nasty surprise.

There is sometimes a misconception that ‘regression tests’ can provide a kind of foolproof safety net, which can somehow be depended on to identify any problem which might have been introduced.

For the swimmers in the story, the fishing boat and net seemed like a good solution to a potential problem, and perhaps would have been if their only potential problem was large sharks. Unfortunately, their solution was not so effective in identifying smaller (but potentially just as serious) problems.

So it can be with ‘regression packs’ or ‘regression suites’. If they only help us find our large sharks then we shouldn’t be surprised if we occasionally get stung. No matter how big the net, there is always a chance that something nasty can slip through.

Meanwhile, we must also be careful about the messages we send. Green flags can sometimes be misleading. We might take them as a sign that we can proceed based on the information we have, but this doesn’t mean that there is no risk in doing so.

As with the swimmers, there are no guarantees for those of us involved in developing and releasing software. Sometimes we can accept the risks and dive right in. At other times we might need to cast our nets wider and deeper to better understand those risks before we take the plunge.

---

This blog post was inspired by Elisabeth Hendrickson‘s fishing net analogy from the wonderful book ‘Explore It!‘