In my last blog, I asked whether the sense of unease which some testers feel about the future is justified. Some claim that ongoing changes in software development practices are so significant that they radically alter the core skills required for a tester, or dispense with the need for testers entirely. Should testers feel threatened, or optimistic about the opportunities the changing landscape presents?
If we consider (for the purpose of this exercise) testing as a service which is provided to clients, we can take a look at some of the factors which might affect organisations (and individuals) who offer this service. This is not on behalf of any company or organisation, but it might help to consider some of the factors from the perspective of a fictional testing services business.
Let us start with the threats. What factors, perceptions and attitudes might result in reduced demand for such services? In this diagram, I’ve attempted to capture the threats, and some the underlying factors which I believe contribute to them:
The ongoing commoditisation of testing
Commoditisation refers to the process by which goods or services become viewed as uniform, or at least barely distinguishable in terms of quality, resulting in competition based on price alone. Organisations who offer testing services, and who compete on this basis, can reduce their prices in a variety of ways, for example:
- reducing their own costs by using a low paid workforce
- minimising their training costs
- offering services at low margins, in the knowledge that the project may well be extended thus increasing revenue from the engagement
This process has been ongoing within testing for some time now. For many years, it has been identifiable through an outsourcing model which often uses offshore test centres. A standardised ‘factory’ approach is adopted, pumping out test cases which create an illusion of ‘coverage’, but don’t necessarily address the risks to the products being tested.
Ironically, this approach can often result in high costs for clients, as lengthy ‘test phases’ occur and useful, actionable information arrives late on, resulting in rework and delays.
Concerns about commoditisation might seem slightly outdated, as the flaws in factory methods have been exposed in recent years by changes in software development practices. Around the globe, there are clever and motivated people operating companies which make a great success of providing testing services which reject the commoditised model (some examples can be found at the end of this blog).
Yet, commoditisation still presents a threat. The factory model hasn’t gone away – it is still alive and well, with thousands of testers operating from low cost testing centres around the world. As recently as May of this year, an article about UBS discussed the need for vendors of outsourced testing to lower their costs further.
Perhaps, more importantly, damage has already been done by the industrialisation of testing. The very idea of what testing is has been shaped significantly in many organisations by the low-cost model which has dominated for so long. Testing is sometimes (or perhaps often) viewed as not only a low cost service, but also a low value service, where testers do little more than mechanically push buttons, blindly following test cases based on requirements documents. It isn’t surprising that those who hold this view might seek to reduce costs even further, and in some cases might believe that testers are no longer required. If testing is this easy, why pay for specialists?
For companies (and individuals) who seek to offer a more sophisticated and valuable service, these attitudes present a challenge. Tackling negative perceptions of testing requires patience and commitment. It shouldn’t surprise us if clients don’t want to talk about misperceptions in testing – as interesting as this might be to testers, the rest of the human race care rather less. Of course, businesses do care about their reputations, the quality of their products and their relationships with customers; they just might not appreciate how testing can help by providing information relating to these concerns, and the ways in which they might be put at risk.
Substitution Myths
“One of the reasons the introduction of automated technologies into complex work environments can fail or have surprising effects is an implicit belief on the part of designers that automation activities simply can be substituted for human activities without otherwise affecting the operation of the system.”
Let me state quite categorically that I am not labelling automation, or tools themselves, as a threat to testing. I believe there is a threat though, from the misapprehension that costs can be further reduced by replacing (with tools and scripts) the people who test.
This idea not only devalues the complex work of testing, it also devalues the skill and effort required to create and maintain the frameworks and scripts which enable automation. There is no testing tool that can replace testers, or that can operate effectively without people (who understand testing) to write and maintain code, interpret outcomes, investigate problems, and so forth.
Perceptions of exactly what testing is, and what testers do, have important implications here too. If it is believed that all testers do is perform binary checks, highlighting where behaviour deviates from some documented expectation, then it is easier to imagine that an automated script could perform this task in exactly the same way, and that the expensive people who do so could be replaced.
Similarly, the idea that it is simple to write and maintain the code used in automation, that such skills can be quickly attained, or to imagine that tools can operate autonomously (removing the ongoing need for skilled people), gives credence to the suggestion that automation is a cheap option.
These are myths, and some tool vendors like to promote these myths. Here are some ways in which they do so:
- Claim that tools can be used to automate any ‘manual test’
- Claim that some high percentage of testing (e.g. ‘up to 95% of test cases’) can be automated
- Propagate ‘Return On Investment’ calculations which make no distinction between what an automated script does and what a human being does
- In those same ‘Return On Investment’ calculations, fail to account for maintenance, monitoring, reviewing results and investigating problems – all tasks which require skilled human intervention
- Use statements which credit the tool rather than the person using the tool – ‘Our tool does x’, rather than ‘People can use our tool to assist them with x’
- Suggest that computers are actually more effective at testing than humans because computers don’t get bored
The myths which drive this threat also drive the kind of articles which I discussed in the previous blog. Demonstrating the benefits of using tools, and of automation, is one thing. Demonstrating the complexities of automation, the skills required to implement and support it, and the limits of what can be achieved is quite another. Those who are unwilling to acknowledge these complexities, skills and limits deserve to be challenged.
“To effectively exploit the capabilities that automation provides (versus merely increasing automation), the task work—and the interdependent teamwork it induces among players in a given situation— must be understood and coordinated as a whole.”
Threats from within
A common complaint from testers is that managers, and other colleagues, believe that testing is easy, or to put it another way, that anyone can test. In my experience, such a perception is unfortunately quite widely held. What is more, I believe this view of testing as a low skill activity underpins the two threats I have highlighted so far.
Unfortunately, some of the roots of this perception can be found in what has happened, and continues to happen under the banner of testing, and in some of the peripheral ventures and operations. Here are some factors which I believe contribute:
Low value training and certificates
What message do the bodies who claim to “promote the value of software testing as a profession” actually send through the certifications they offer and the training they facilitate through providers? Do three day courses, with rote learning of a syllabus (and no practical exercises whatsoever) really help promote the value of testing? For testers, isn’t it stretching credibility to complain that testing isn’t taken seriously whilst marketing yourself with a certificate which was attained after partaking in such an exercise and sitting a brief multiple choice test?
Dependency on standards and templates
Standards seem to me to be an excellent way of neutralising testers and of devaluing testing. If we aren’t required to think critically – if we instead follow rules and procedures – then we are more easily replaceable. Likewise, templates can become crutches which can prevent us from considering different ideas and being creative.
Characterising testing as simple binary checks
When we reduce the outcomes and observations which result from testing to a single word (‘Pass’ or ‘Fail’), we don’t allow for the nuanced responses which we (and customers) might have to the product. If a ‘Pass’ represents a product doing something we anticipated, then a ‘Fail’ captures a multitude of other outcomes; some potentially serious, some less so. Identifying the conditions under which these outcomes occur is not always simple, but this would not necessarily be clear to others if they are only presented with two possible results.
Characterising testing as requirement confirmation
Establishing a set of documented requirements as the sole basis for testing, might constrain the tester rather than encouraging them to use their instincts and judgement. Confirming that a requirement (which may or may not represent what the business intended, or what customers want) can be fulfilled, does not always tell us a great deal. Yet, these Crosby inspired definitions of testing persist, and fail to recognise the different ways in which a product can be investigated, some of which depend significantly on specialised testing skills.
Characterising testing as adhering to scripts
Quite early in my career I was taught to ‘write test cases which anyone could follow’; a set of repeatable steps with an expected outcome. I’m sure many others were taught the same thing. Of course, when I followed these steps, I noticed other things too, investigated them and learned from them. But a distinction was often made between the supposedly more challenging task of designing or preparing test scripts, and the supposedly simple task of executing them. The fiction of testing as an exercise in rigidly following a script or procedure overlooks the observation and analysis which occurs. Increasing awareness of exploratory testing techniques may have revised this view in many teams and organisations, but the practice of preparing and following scripts lives on in others and so do the misperceptions which accompany it.
Final thoughts
I have become increasingly aware as I have worked on this blog, that it could have been written at any time in the last ten years, and probably before that. It seems to me that the threats to testing are much the same as they have been for some time, and are driven by perceptions which have been forged over a long period.
I don’t see changing development practices as a threat to testing. In fact I see many opportunities for testers in the way in which teams are working, and in the information which organisations value. I will discuss these more next time.
Resources
Iain McCowatt has discussed the commoditisation of testing and its dangers. Some links to his valuable thoughts on this subject:
- Blog – http://exploringuncertainty.com/blog/archives/757
- Webinar – http://testhuddle.com/resource/the-commoditization-of-testing-with-iain-mccowatt/
Lee Hawkins also discussed it recently here:
This blog from Randy Wetmore of PQA explains some of the problems he has faced in selling testing services:
This article by Jeffrey M. Bradshaw, Robert R. Hoffman, Matthew Johnson, and David D. Woods discusses “The Seven Deadly Myths of Autonomous Systems”:
“How to Make Automated Systems Team Players” by Klaus Christoffersen and David D. Woods introduces the idea of the ‘Substitution Myth’:
Some websites for companies offering testing services:
- USA –Doran Jones
- Sweden – House of Test
- India – Test Insane
Finally, an important note:
This is an opinion piece, not intended as a detailed analysis of the market for testing services. The opinions here are based on observations, discussions with colleagues, and on other articles, blogs and in some cases marketing material produced by companies.
richrtesting 
4 thoughts on “Under threat?”