10,000+ synonyms for Quality Assurance

I recently had a conversation with my team about what we should call the status between when work is passed “code review” but not yet “done”. The one thing I didn’t want to call it was “In QA”. One of the developers on my team had another idea that I decided to run with:

Let me explain.

Much like Richard Bradshaw says in this video on “QA” as a term, it doesn’t make much sense to me to name one stage of a feature’s development as if that was the only stage at which QA was done. Michael Bolton regularly insists that quality assurance isn’t even a thing testers do (or should do). (Although interestingly when I brought this up in Testers Chat, Michael was the one to ask whether the name we picked would even make a material difference.) The argument generally goes that testers don’t assure quality. We can do all sorts of other things to it—inform it, study it, encourage it, foster it—but we can’t guarantee or enforce it. We especially can’t do it after the code is written.

Maybe this one is better?

I’ve mentioned before that the terminology at my company is “QA” rather than “Testing”. Asking the difference between “QA” and “Testing” is another sure-fire way to spark debate but I don’t think a piece of development should ever be in a discrete “In Testing” phase either. Generally I’m not too concerned about calling it one or the other; I’m much more interested in what people are actually doing. I haven’t seen any of the dire warnings about using “quality assurance” come true where I am now, but I’m not going to risk encouraging it with an “In QA” phase.

Here’s a third attempt at a better name for QA:

The idea that the developer on my team had was this: if I was so set against calling it anything but “QA”, let’s just take synonyms for “quality” and “assurance” and come up with something that didn’t have all that baggage. He was joking, but I ran with it. I ran with it about 13 thousand times.

Here, come up with some of your own:

This is a little script that will randomly pick alternative terms for “quality assurance”. Very rarely it might actually suggest you stick with “quality assurance”. I do not vouch for any of these being good suggestions, but I think at this point I’m more interested in discussing the merits of “quirk investigation” vs “constitution corroboration” than I am hearing more complaints about “quality assurance” as a term.

The standalone link is here if you want to keep generating more ideas, and I even made a helpful Twitter robot that’ll tweet out a new idea every day. Hit me up on my Twitter or leave a comment if you want to make sure your favourite synonyms are included. Let the pedantry begin!

Testing is like a box of rocks

I was inspired today by Alan Page’s Test Automation Snowman. He makes all good points, but let’s be honest, the model is the same as the good ol’ test pyramid. The only difference is that he’s being explicit about tests at the top of the pyramid being slow and tests at the bottom being fast. Ok, so maybe the snowman thing is a joke, but it did make me think about what might make a better visualization. I quickly doodled something on a sticky note:

A sticky note with a lot of tiny circles in the bottom third, medium circles in the middle third, and a few large circles in the top third.

If the point we want to emphasize is that UI tests are slow (and therefore take a lot of time), we should include that in the visualization! The problem with the pyramid (and the snowman) is that the big tests take up the least amount of space; the small triangle at the top makes it look like having fewer UI tests also means you do less UI testing.

It doesn’t.

At least, not proportionately. If you had an equal number of UI and unit tests, it’s a safe bet that you’re going to spend more of your time working on the UI tests.

So instead, let’s say testing is like a box of rocks. Each rock is a test, and I have to choose how to allocate the space in that box to fit all the rocks that I want. A few big rocks are going to take up a lot more space than a bunch of tiny pebbles. Unless I have a good reason why that big boulder is a lot more interesting than the hundred little rocks I could put in its place, I’m going to go for the little rocks! If I have to add a new rock (to kill a new bug, say) I probably want to choose the smallest one that’ll still do the job.

You can still think about the different levels (unit vs API vs UI, for example) if you picture the little rocks at the bottom forming a foundation for bigger rocks on top. I don’t know if rocks work like that. Just be careful not to get this whole thing confused with that dumb life metaphor.

Ok, it might not be the best model, but I’ll stick with it for now. And like the Alan’s snowman, you’re free to ignore this one too.

Agile Testing book club: Everyone is a Tester

If you’ve even dipped your toe into the online testing community, there’s a good chance that you’ve heard Agile Testing by Janet Gregory and Lisa Crispin as a recommended read. A couple weeks ago I got my hands on a copy and thought it would be a useful exercise to record the highlights of what I learn along the way. There is a lot in here, and I can tell that what resonates will be different depending on where I am mentally at the time. What I highlight will no doubt be just one facet of each chapter, and a different one from what someone else might read.

So, my main highlight from Chapters 1 and 2:

everyone is a tester

Janet and Lisa immediately made an interesting distinction that I would never have thought of before: they don’t use “developer” to refer to the people writing the application code, because everybody on an agile team is contributing to the development. I really like emphasizing this. I’m currently in an environment where we have “developers” writing code and “QA” people doing tests, and even though we’re all working together in an agile way, I can see how those labels can create a divide where there should not be one.

Similarly surprising and refreshing was this:

Some folks who are new to agile perceive it as all about speed. The fact is, it’s all about quality—and if it’s not, we question whether it’s really an “agile” team. (page 16)

The first time I encountered Agile, it was positioned by managers as being all about speed. Project managers (as they were still called) positioned it as all about delivering something of value sooner than possible otherwise, which is still just emphasizing speed in a different way. If asked myself, I probably would have said it was about being agile (i.e., able to adapt to change) because that was the aspect of it that made it worth adopting compared to the environment we worked in before. Saying it’s all about quality? That was new to me, but it made sense immediately, and I love it. Delivering smaller bits sooner is what lets you adapt and change based on feedback, sure, but you do that so you end up with something that everyone is happier with. All of that is about quality.

So now, if everybody on the team should be counted as a developer, and everything about agile is about delivering quality, it makes perfect sense that main drive for everybody on the team should be delivering that quality. The next step is obvious: “Everyone on an agile team is a tester.” Everyone on the team is a developer and everyone on the team is a tester. That includes the customer, the business analysts, the product owners, everybody. Testing has to be everybody’s responsibility for agile-as-quality to work. Otherwise how do you judge the quality of what you’re making? (Yes, the customer might be the final judge of quality means to them, but they can’t be the only tester any more than a tester can be.)

Now, the trick is to take that understanding and help a team to internalize it.

Tests vs Checks and the complication of AI

There’s a lot to be made in testing literature of the differences between tests and checks. This seems to have been championed primarily by Michael Bolton and James Bach (with additional background information here), though it has not been without debate. I don’t have anything against the distinction such as it is, but I do think it’s interesting to look at what whether it’s really a robust one.

The definitions, just as a starting point, are given in their most recent iteration as:

Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

These come with a host of caveats and clarifications; do go back and read the full explanation if you haven’t already. The difference does not seem intuitive from the words themselves, which may be why there is such debate. Indeed, I’ve never seen anybody other than testing professionals make the distinction, so in normal usage I almost exclusively hear “test” used, and never “check”. Something I might call an automated test, others might call—and insist that it be called—an automated (or machine) check. This is just a consequence of working day-to-day with developers, not with testing nerds who might care about the difference.

Along those lines, I also find it interesting that this statement, still quoting from James Bach’s blog:

One common problem in our industry is that checking is confused with testing. Our purpose here is to reduce that confusion.

goes by with little explanation. There is a clear desire to differentiate what a human can do and what a computer can do. The analogy in the preamble to craftspeople being replaced by factory workers tries to illustrate the problem, but I’m not sure it really works. The factory model also has advantages and requires it own, different, set of skilled workers. I may just be lucky in that I haven’t ever worked in an environment where I was under pressure to blindly automate everything and dismiss the value humans bring to the process, so I’ve never needed the linguistic backing to argue against that. This affords some privilege to wonder whether this distinction has come about only because of a desire to differentiate between what a computer and a human can do, or because there actually is a fundamental difference.

Obviously, as far as what is possible today, there is no argument. But the more we see AI coming into use in testing, the more difficult this distinction will become. If I have an AI that knows how to use different kind of apps, and I can give it an app without giving it any specific instructions, what is it doing? Can it ever be testing, or is it by definition only checking? There are AI products being pushed by vendors now that can report differences between builds of an app, though for now these don’t seem to be much more than a glorified diff tool or monitor for new error messages.

Nonetheless, it’s easy to imagine more and more advanced AIs that can better and better mimic what a real end user (or millions of end users) might do and how they would react. Maybe it can recognize UI elements or simulate the kinds of swipe gestures people make on a phone. Think of the sort of thing I, as a human user, might do when exploring a new app: click everywhere, swipe different ways, move things around, try changing all the settings to see what happens, etc. It’s all that exploration, experimentation, and observation that’s under the “testing” definition above, with some mental model of what I expect from each kind of interaction. I don’t think there’s anything there that an AI fundamentally can’t do, but even then, there would be some kind of report coming out the other end about what the AI found that would have to be evaluated and acted upon by a human. Is the act of running the AI itself the test, and every thing else it does just checks? If you’re the type that wants to say that “testing” by its nature can’t be automated, then do you just move the definition of testing to mean interpreting and acting on the results?

This passage addresses something along those lines, and seems to answer “yes”:

This is exactly parallel with the long established convention of distinguishing between “programming” and “compiling.” Programming is what human programmers do. Compiling is what a particular tool does for the programmer, even though what a compiler does might appear to be, technically, exactly what programmers do. Come to think of it, no one speaks of automated programming or manual programming. There is programming, and there is lots of other stuff done by tools. Once a tool is created to do that stuff, it is never called programming again.

Unfortunately “compiling” and “programming” are both a distracting choice of words for me (few of the tools I use involve compiling and the actual programming is the least interesting and least important step in producing software). More charitably, perhaps as more and more coding becomes automated (and it is), “programming” as used here becomes the act of deciding how to use those tools to get to some end result. When thinking about how the application of AI might confuse “tests” vs “checks”, this passage stuck out because it reminded me of another idea I’ve heard which I can only paraphrase: “It’s only called machine learning (or AI) until it works, then it’s just software”. Unfortunately I do not recall who said that or if I am even remembering the point correctly.

More to the point, James also notes in the comments:

Testing involves creative interpretation and analysis, and checking does not

This too seems to be a position that, as AI becomes more advanced and encroaches on areas previously thought to be exclusive to human thought, will be difficult to hold on to. Again, I’m not making the argument that an AI can replace a good tester any time soon, but I do think that sufficiently advanced tools will continue to do more and more of what we previous thought was not possible. Maybe the bar will be so high that expert tester AIs are never in high enough demand to be developed, but could we one day get to the point where the main responsibility a human “tester” has is checking the recommendations of tester AIs?

I think more likely the addition of real AIs to testing just means less checking that things work, and more focus on testing whether they actually do the right thing. Until AIs can predict what customers or users want better than the users themselves, us humans should still have plenty to do, but that distinction is a different one than just “test” vs “check”.