A solution to a non-existent problem

Recently, there was a post made to the Swedish community site “TestZonen” with some information about how the work with the new ISO/IEC/IEEE 29119 standard for software testing is progressing. That post sparked a rather intense series of comments from different members of the Swedish testing community.

My personal stance on the whole thing has in the past been that I “don’t care”. I don’t care for it and I don’t care much about it. If you’re not familiar with the standard, here’s what the organization’s website says the goal of it is:

The aim of ISO/IEC/IEEE 29119 Software Testing is to provide one definitive standard for software testing that defines vocabulary, processes, documentation, techniques and a process assessment model for software testing that can be used within any software development life cycle.

That’s a pretty tall order if you ask me. And words like “definitive” definitely set off some alarm bells in my head. I also think it’s a near impossible goal to reach, at least if one by “standard” mean anything near what I interpret the word to mean. So when I say I don’t care, I partly mean that I think it’s a waste of time and won’t help the craft or the industry, so why bother? After having thought a bit more about it though, I’ve come to realize that I do care a little bit. Why? Because not only do I not think it would help our craft, I also think it can actually hurt the software testing industry. Now, of course you can always opt-out and simply not use the standard in your organization, but I don’t think it’s as easy as that for the industry as a whole.

What follows below is a not-quite-but-pretty-close-to literal translation of what my comments to the article were. It’s a bit long and maybe a bit of a rant, sorry about that. Look, even an introvert can go off on rants sometimes. It started off with me quoting a line from the article where the author gave some examples about what he thought the might be the outcome of the release of this new standard.

One possibility is that it will show up as a requirement, in procurement negotiations or when selecting partners, that you need to be certified according to ISO/IEC/IEEE 29119.

Yes, that’s one of the risks I see with this standard. As if it’s not bad enough that some buyers of testing services today are already throwing in must-have requirements that they don’t know the value (or cost) of. Here comes even more crap that a big number of test professionals must spend time and money getting certified on, or risk becoming disqualified when competing for contracts.

Another risk is that this will be yet another silver bullet or best practice that well-meaning but dangerously uninformed stakeholders will decide to implement rather than taking the time to understand what software development is really all about. Especially since this silver bullet has ISO and IEEE written all over it. How much will the time saved picking this “best practice” over developing a context-appropriate approach end up costing their shareholders in the end…?

Ok, so isn’t it then a good thing that we get a standard so that buyers and stakeholders can feel more secure and know what they’re betting their money on? No, not in this case. Software development is much too context-dependent for that. Just because there’s a standard out there that somebody’s cooked up, doesn’t mean that people will all the sudden become more knowledgeable about software testing. Instead, this will become an opportunity for people to keep ignoring to learn anything about all the choices and judgement calls that go into creating good software. Just use this standard and quality will flow forth from your projects, that’s all you need to know. I don’t see how it would ever be possible to automatically equate using this (or any other) standard with delivering valuable testing and useful information. But unfortunately I believe that’s what a lot of uninformed people will think it means.

What I think is needed instead is a continued dialogue with our stakeholders. We need to gain their understanding for our job and have them realize that creating software is a task filled with creativity, checking and exploration, and that we perform this exercise guided by sound principles and past experiences, but very rarely by standards. And we can of course talk about these principles and explain and defend them as well as use them when we’re asked to estimate amounts of work, or report on progress, or give a status report. But to standardize the practical application of a principle is as unthinkable to me as it is to tell a boxer that he or she will win against any opponent as long as a certain combination of punches is used, regardless of what the opponent is doing. But, teach the same boxer a few principles like how it’s generally a good idea to keep the guard up and keep moving, and their chance to win increases.

The principles and heuristics we use have very little to do with the kind of rigid process maps I find on the standard organization’s website though. I sincerely hope those are old drafts that have been discarded a long time ago. Judging by that material though, I think the best case scenario we’re looking at is that we’ll end up with a new alternative to RUP or some other fairly large and prescriptive process. And since this process will have ISO and IEEE in its name, it will likely be welcomed by many industries and once again they will go back to focusing on being “compliant” rather than making sure to deliver something valuable to customers. So again we’ll be back in that rigid grind where documents are being drawn up just for the heck of it, just because the process says so. Again.

Like many other comments to the article said: What’s the use of this standard? Where’s the value? Who is it really benefiting? To me, it seems like a solution to a non-existent problem!

The article asked if the standard will mean new opportunities or if it will mean that our everyday work will be less open to new ideas. I believe it will mean neither to me. Because I don’t see the opportunities, and I also don’t see that it’s even possible to do good work without being open to new ideas and having the freedom to try them out and then look back on the result and evaluate those ideas. What I see instead is yet another meaningless abbreviation that testers will get hit over the head with time and time again and that we will have to spend time and energy arguing against following, if we want to do valuable testing in our projects and organizations.

The standard is scheduled for release soon. You can choose to ignore it or you can choose to fight it. But please don’t adopt it. You’ll only make life harder for your peers and you’re unlikely to gain anything of value.

10 Responses to A solution to a non-existent problem

  1. It seems to me that all these ISO and IEEE standards for software development and testing are modelled around developing standards for assembly line manufacturing. The problem with that is, software development is a fundamentally different endeavour.

    Assembly line manufacturing is intended to produce the same thing over and over again, while software development is unique in each iteration. Thus, these sorts of sweeping standards really don’t have much place in software development. If they really want to come up with some generally good practices (and I do think they exist), they would be further ahead to use a model associated with craftsmanship, instead of manufacturing.

    • I agree. And I don’t mean to imply that all ISO or IEEE standards are bad. Just that the type of problems they usually try to solve, don’t appear to be of the same kind that we usually have to deal with as testers. Like you say, it’s a very different endeavor compared to the manufacturing style engineering disciplines.

      I like how Johanna Rothman summarized it once in a talk she gave: “Software development is like… no other activity in the history of the world!”

      But I agree that it’s at least more like craftsmanship than manufacturing, which I think is a helpful mindset to start off with.

  2. Training community would benefit from this standard.

    By the way Stuart Reid, a former ISTQB president http://www.istqb.org/about-istqb/working-rules-and-management-structure.html is one of the leaders of ISO/IEC JTC1/SC7 WG26 http://www.softwaretestingstandard.org/aboutWG26.php.

    • Hi Yury,

      Thank you for your comment. I appreciate the dialogue.

      I agree, providers of testing courses and training will benefit from this standard. It will give them more stuff to create training around. What’s your opinion about this fact that we seem to agree on? Is that good or bad?

      In my mind, I don’t think that giving trainers more course material (or worse, a basis for new certification schemes) to sell is a good enough justification to create the standard (and I’m saying this as a training provider myself). Not unless that training can actually provide professional testers with something worthwhile. And since I very much doubt that this standard will even come close to reaching its goal of being a definitive standard for software testing, I don’t see much potential there. It is however, a very apparent risk in my mind, that this ends up being more worth as something to “sell” rather than something to “use”. Was that what you meant as well or can you make a more positive spin on it?

      And yes, I’m aware that Stuart Reid is involved in developing the standard. That doesn’t sway my opinion about this soon-to-be-existent standard one way or the other though.


  3. Pingback: Five Blogs – 5 August 2013 | 5blogs

  4. Hi Johan,

    For the sake of this discussion let’s define 2 schools of testing:
    1. “Context-driven School of Testing”
    2. “CMM/TMM/ISTQB-driven School of Testing”

    In this post I use “Context-driven School of Testing” and “CMM/TMM/ISTQB-driven School of Testing” just as labels to denote these schools.

    It seems to me that these are two different paradigms with different values and vocabularies. People who belong to these schools do not understand each other. For example neither you nor me have a good understanding of the value of ISTQB. The same way ISO/IEC/IEEE 29119 is a puzzle for us. We do not understand why it’s required.

    At the same time quite a few people and organizations are involved in development of ISO/IEC/IEEE 29119. Apparently they understand the value of this new standard.

    In order to understand ISTQB or ISO/IEC/IEEE 29119 we need to understand what needs are met by these organizations.
    Right now I intuitively understand that such needs exist with corresponding $$$ market demand but I can’t articulate specific details.

    Unfortunately there is a significant difference between “Context-driven School of Testing” and “CMM/TMM/ISTQB-driven School of Testing”.

    Such people as Cem Kaner, James Bach and many other were able to create a theory explaining “Context-driven School of Testing”.

    On the other hand representatives of “CMM/TMM/ISTQB-driven School of Testing” either do not have similar understanding of their School or are not willing to share their esoteric knowledge with people outside their School.

    If we want to understand both sides of this testing divide we need to start this discussion within Context-driven testing community.
    I believe that as a group we have enough experience and knowledge to figure this out.


    • Hi Yury,

      Thank you for your comment.

      I consider myself fairly knowledgeable in what you label the CMM/TMM/ISTQB-driven style of testing and I have an understanding of what it tries to accomplish or bring to the table. In my opinion, it’s about control. By following rigorous processes and standards, you gain (an illusion of) control over your testing and are able to “prove” that requirements are covered and show what testing has actually been done by pointing to detailed test cases.

      I don’t think the problem is a lack of understanding. I think the problem is that the factory approach doesn’t work if you want to do good testing. Software development is about learning, and the factory model doesn’t allow for learning to take place. Or at least not in a way that would influence or change “The Plan That Must Be Followed”. I’m being a bit facetious, I agree, but still, I think the problem is not one of not understanding.

      There is also a lot of money to be made from being able to standardize a way of doing something and then making everybody adhere to that standard. Expensive training programs, certification programs, accreditation schemes, mandated certifications for suppliers… the list could be made longer still, and it’s already a rather familiar list, wouldn’t you say?

      That said though, I do agree that we need to have discussions within the CDT community, always. We need to counter the power-grabbing tactics of those who are more worried about control than they are about good testing. It’s one of the reasons why I’m part of the International Society for Software Testing which I hope will be able to do just that.

      (Sorry about the late reply, I only noticed your last comment today when I was going back to check on my original post.)

  5. Pingback: Solving the real problem | Let's Talk Testing

  6. @halperinko - Kobi Halperin

    As this standard is not coming into an industry where no other standard was available before, I guess the value or lack of it should be evaluated based on comparison to the number of existing software testing standards it comes to replace:
    IEEE 829 Test Documentation
    IEEE 1008 Unit Testing
    BS 7925-1 Vocabulary of Terms in Software Testing
    BS 7925-2 Software Component Testing Standard

    It may ease the contracts which are now based on those, or worsen the terms.

    If any of you know of a short comparison list we can base this discussion on (so that we will not waste our time already on reading the whole thing to figure it out, or just run empty discussions based on our previous standing without even reading it) – that will be great.

    @halperinko – Kobi Halperin

  7. Kobi,

    I have read the 3 sections of ISO 29119 published so far. To a large extent, it simply swallows up and entrenches those previous standards, as it is the working group’s intent to have one standard replace all the others. In so doing, it seeks to further institutionalise all the pernicious qualities — the rigid, cumbersome, expensive, document-heavy practices — of the previous standards, which are completely antithetical to Agile practices. As Johan wrote, following these standards (or any other) provides an illusion of control, where control is the wrong thing to be concerned about.

    Testing (and software development in general) cannot be done effectively as a pre-defined template-driven activity. It is far too context-dependent for that. As many critics of the standard have pointed out, the emphasis on documentation too often results in goal displacement, where production of documents becomes the objective of testing. As I wrote almost 3 years ago here: http://www.agileconnection.com/article/tester-know-your-product documents are not the tester’s product. Our product is information about the quality of a system that allows our stakeholders to make informed decisions.

    A tester who is churning out fat documents is not testing. Frequently, he/she is not even thinking about testing. Not-testing, not-thinking testers can’t uncover useful information for stakeholders.

    This “new” standard is like the last gasp of a dying old guard. It’s an attempt to resurrect and entrench the ineffective and expensive practices of the bad old days.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.