Unit tests versus the unit tested

I recently read the great and oft-cited article about testing microservice architectures by Cindy Sridharan over on Medium. It’s broadly applicable beyond just “microservices”, so I highly recommend giving it a read. I was struck by this passage in particular:

The main thrust of my argument wasn’t that unit testing is completely obviated by end-to-end tests, but that being able to correctly identifying the “unit” under test might mean accepting that the unit test might resemble what’s traditionally perceived as “integration testing” involving communication over a network.
— Cindy Sridharan, “Testing Microservices, the Sane Way

Articulating what you’re actually trying to test is one of the most underrated skills in testing. It seems like a simple question to ask, but often a forgotten one. Asking “what information does this test give me that no other test does?” is great heuristic for determining whether a test, especially an automated one, is worth keeping. It’s also a convenient go-to when evaluating whether a tester knows what they’re doing, and when trying to understand what a test does.

What interests me about this passage is that Cindy is highlighting how “the thing being tested” gets conflated with the “unit” in “unit testing”.

When I used to train new testers at my company, I would present four levels of testing: unit, component, integration, and system. Invariably, people would ask what the difference between component and integration was. Unit tests were easy because that tended to be the first (and sometimes) only kind of testing incoming devs were familiar with. System tests were easily understood as the ones at the end with everything up and running. But when are you testing a component versus an integration? It’s not obvious, so it’s no surprise that three-tiered descriptions are much more common these days.

For us, testing a component was sometimes a single service. At other times it was testing a well defined system of a larger program (one that probably would have been a microservice if we built the system from scratch). Inputs and outputs were controlled directly, and no other services needed to be running for it to communicate with. We defined a “unit” as the smallest possible thing that could be tested (often a single function) and the “component” was running the actual program.

Is that different from an “integration” test of those units? Not really. Running the component is an integration of smaller units. But it was convenient for us to separate those tests from testing how separate services communicated with each other. You can ask the same question anywhere along the continuum from function to system. Is testing a class a unit test or a component test? Where do you draw the lines?

The confusion highlighted in the passage above, to me, is only because of different definitions of “unit”. If you want to call the thing, the service or behaviour, that you’re testing a “unit” instead of a “component”, go for it. If you want to call the communication between two services the “unit” (as this article does), great. This ambiguity should not be an obstacle to understanding the point: you should know what you’re testing and have a reason for testing it.

And, of course, that you should be testing the most important things. Which means, for example, don’t mock away the database if your service’s main responsibility is writing to the database. The problem isn’t that Cindy is saying “unit tests should be communicating over the network,” though you might read it that way if you’re dogmatic about the term “unit test”. She’s saying “communicating over the network is important, so I’m going to prioritize that communication as the subject (unit) of my tests.”

For all these terms it’s unlikely we’re ever going to setting on unambiguous definitions. Let’s just try to be clear about what we mean when we use them, and clear about what we’re testing with each test.

If you didn’t test it, it doesn’t work

Gary Bernhardt has a great talk online from 2012 called Boundaries, about how design patterns influence, for better and worse, the testing that can be done. In particular he advocates a “core” and “shell” model, having many independent functional cores and one orchestrating shell around them. The idea is that each functional core can be tested independently from the others more effectively by not having to worry about mocks and dependencies. The tests of each core can be greatly simplified, while focusing on what it’s actually supposed to do.

The concept is definitely solid, but I do take issue with one thing he says towards the end. I’ll acknowledge upfront that I’m drawing much more meaning from his choice of words than I think was intended, it just happened to strike a nerve. It happens at about the 29m30s mark, when he’s describing how small the tests for the functional cores became in his example program. As for the shell, the orchestrating code responsible for connecting the wires between these cores, he says:

The shell is actually untested. The only conditional in the shell is the case statement that we saw that was choosing between keys. There’s no other decision made in the entire shell. So I don’t even really feel the need to test it. I fire the program up, I hit ‘j’, as long as that worked, pretty much the whole thing works.

The obvious objection here is that just because the cores work independently doesn’t mean that the shell manipulates those cores correctly. Conditionals aren’t the only things that can have bugs in them. But that’s not my objection. It’s this:

Firing the program up and hitting ‘j’ to see if that works is testing it.

Are there other tests he could do? Sure. At a minimum, ‘j’ isn’t the only keystroke the program supports. So he’s made a risk assessment that the shell is simple enough that this is the only test he needs. It might be an informal assessment, even an unconscious one, but it’s still based on the foundation of unit tests, his knowledge of how the program works, and how it’s going to be used. There’s absolutely nothing wrong with that.

Has he automated his test? No. But is something only a test if it’s automated? Also no. Much like he made a risk assessment that he only needs one test, there’s also a judgement call (perhaps implicit) that this one test isn’t worth automating. There could be any number of reasons for making that call. That doesn’t mean the sequence of firing it up and trying it out is something outside of “testing it”.

I think there is a tendency these days to think that a test only counts if it’s automated, but we should know better than that. It’s been argued that the parts we automate shouldn’t even be called “tests”. Whether or not you agree with the terminology, it has to be acknowledged that testing encompasses a lot more than automating some checks.

I think this same sentiment is what motivates the #NoTesting hashtag on Twitter. If you have a traditional, waterfall style, gated, or heavily regimented picture of what software testing is, you’ll exclude from that umbrella all kinds of other activities that people naturally do to poke at their products. “We don’t want that stuffy bottleneck-creating deadline-breaking time-suck kind of testing around here”, they say, “so therefore we don’t want testing at all.” I bring this up not to put Gary into that camp—in the context of the talk this quibble of mine wouldn’t affect the point he was making—but to point out that people do take this idea too far.

Coincidentally, the same day I came across this talk, I also read The Honest Manual Writer Heuristic by Michael Bolton. It ends with a line that captures this idea that people do a lot more testing than they realize:

Did you notice that I’ve just described testing without using the word “testing”?

If you didn’t test it, it doesn’t work.

or, to make it perfectly clear what I’m getting at, let’s phrase it like this:

If you know it works, you must have tested it.

Debating the Modern Testing Principles

Last week I had the opportunity to moderate a discussion on the Modern Testing Principles being developed by Alan Page and Brent Jensen with a group of QA folks. I’m a relative late-comer to the AB Testing podcast, having first subscribed somewhere around Episode 60, but have been quite interested in this take on testing. Armed primarily with the 4 episodes starting from their initial introduction and some back-up from the article by the Ministry of Testing, we had a pretty interesting discussion.

Discussing the seven principles

After giving a bit of a preamble based on the mission statement—“Accelerate the Achievement of Shippable Quality”—we went through the principles one by one. For each one I asked (roughly) these questions:

  1. What do you think this statement means?
  2. How do you feel about this a core principle of testing?
  3. How well (or not) does this describe your approach to testing?
  4. Is this a principle you would adopt?

For the first four principles, there was a lot of agreement. We discussed building better products versus trying to assure the product’s quality, the importance of prioritization of tests and identifying bottlenecks, leaky safety nets, data-driven decisions, and the easy alignment with a whole-team Agile mindset. Then it started to get a bit more interesting.

Disagreement one: Judging Quality

The fifth principle started to get problematic for some people:

5. We believe that the customer is the only one capable to judge and evaluate the quality of our product.

There was a lot of debate here. Although a couple people were on board right away, the biggest question for most in the room was: who is the “customer”? Lots of people could fall into that category. Internally there are stakeholders in different parts of the business, product owners in our team, managers in our department, and team itself to some degree. We also have both our current end users and the people we want to attract into regular users. Some of you may have simpler environments with a clear cut individual client, but others could be even more complicated.

What we did agree on was that you have to use the product to be able to judge it. The people testing have to think like the customer and have a good idea of what their expectations are. Interestingly, when we changed “customer” to “anybody who uses the product”, everybody around the table could agree with the principle as a whole.

I suspect, though, that if we only say “anybody who uses the product is capable of judging and evaluating the quality of the product”, the statement loses its power. My feeling is that if this principle feels problematic in its original form, you may just not have a firm idea of who your customer really is. This just highlights for me how important it is to ask who’s opinion, at the end of the day, is the one that counts.

Disagreement two: The dedicated specialist

It’s likely unsurprising that a principle suggesting the elimination of the testing specialist would raise a few eyebrows in a group of testing specialists.

7. We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.

There was no disagreement with the first clause. Many people immediately connected it with the 4th principle, to “coach, lead, and nurture the team towards a more mature quality culture”. Surely endeavouring to “expand the testing abilities and know-how across the team” is a good way to achieve that. When the group initially discussed the 4th principle, we were all in agreement that we wanted to drive a culture of quality and a whole-team approach to testing.

I am still unsure whether the disagreement with eliminating the dedicated specialist was just a knee-jerk reaction or not. I tried to use an analogy of the tester-as-Mary-Poppins: She stays only as long as she is truly needed, and then takes to the wind again to find a new family in need. It didn’t seem to sell the point. We agreed that our teams should be able to function without us… temporarily. There was one assertion that QA was the most important part of the whole process and therefore could not be eliminated. Another one that the skills are different from other roles. And yet another that not everybody wants to be a dev. (Although, of course, the principle doesn’t end with “… so that they can become a developer.”)

Additional context from Alan and Brent helps here too. In some of the episodes after the principles were first introduced, they do talk about now not every tester needs to be a Capital-M Capital-T Modern Tester. I don’t believe the intent is to eventually eliminate the need for testing specialists full stop. It’s not even a given that the specialist would be eliminated on a particular team, just that the need for a specialist should be reduced. To me this principle is a corollary of reducing bottlenecks and building the testing know-how on the team, albeit phrased more provocatively.

Nonetheless, the closest we got to agreement on this was to say we could eventually eliminate the singular position of a testing specialist, but not eliminate the function.

Is that any different or just an easier pill to swallow?

Wrapping up

Both of these, the two biggest objections to the Modern Testing Principles, have a common theme. The 4th principle asserts that testers aren’t the judge of quality or even truly capable of evaluating it. The 7th pushes the idea that given the right expertise and know-how, a testing specialist may not even be needed. Both of these can feel like a threat. Both speak to a fear of losing agency. Alan and Brent also talked about this in the podcasts: one of the motivations for formulating these principles was to prepare people for how testing is changing so that we aren’t all caught unprepared. While I have doubts that there’s an apocalyptic testing singularity coming—something I plan to write on in another post—it does emphasize how important it is to be prepared for new ways of thinking in the industry.

To wrap up the discussion, we did a quick summary of the words and concepts that had come up as common themes in the principles. Then, to compare, I asked for testing concepts or buzzwords that had been conspicuously absent. Chief among the latter were automation, defect tracking, reporting, traceability, documentation, and not once did we talk about writing a test case. Highlighting what was not explicitly mentioned in the principles seemed to be a great way to highlight what makes this a different approach compared to a lot of our day-to-day experience. Though some of those “missing” elements may come out naturally as tools necessary to really embrace these principles, I felt it important to highlight that they were not the goal in and of themselves.

In the end, these differences and the disagreements were the most interesting part of the Modern Testing Principles. Alan described presenting the principles at Test Bash in much the same way—it’s not much fun if everybody just agrees with everything! Hopefully the discussions sparked some new ways of thinking, if only a little bit.

Testing on Dune (Part 2)

Previously I highlighted a passage in Frank Herbert’s Dune that mentioned testing and said it was the first of two. This is the second passage:

Arrakis teaches the attitude of the knife—chopping off what’s incomplete and saying: “Now it’s complete because it’s ended here.”

—from “Collected Sayings of Maud’Dib” by the Princess Irulan, in Frank Herbert’s Dune

Testing isn’t explicitly mentioned this time, but most testers who have been around the block a few times will probably recognize why this caught my eye. Often it feels like there are an infinite number of things to test, and the only way testing can be “done” is by putting down the knife.

Is there any testing that the “attitude of the knife” doesn’t apply to?

Of course, we like to attribute more agency to ourselves than that. Maybe we say our testing is complete when we’ve tested all the aspects described in our well-reasoned test plan. You know, that perfect test plan that takes into consideration all the stakeholders’ needs, the risks involved, the impact to the users, and timeliness required to arrive at just the right amount of testing for this particular context. At the end you can push back from your terminal and say, “Yes, I tested everything I set out to test. Everything about it has been validated and verified to the extent reasonable, and I’m not just saying that because we’ve run out of time.”

But then again, did you not come up with your test plan or strategy knowing how much time would be reasonable to spend on this? Did you know that the knife was going to come down on a certain day? Did you have a sense of the tolerance of your stakeholders, and balance that against the risks?

We all know that exhaustive testing is impossible. Most of us probably realize that exhaustive testing isn’t the goal anyway, and not just because we don’t have infinite time. The knife has to come down sometime, and our testing will be complete because it ended.*


  • Though… does anybody ever finish testing anything? Short of moving on to another product or company entirely, I don’t think I do. Maybe we don’t live on Arrakis.

Agile Testing book club: Courage

This is the third part on my series highlighting lessons I’m taking out of reading Agile Testing by Lisa Crispin and Janet Gregory. Other entries in the series can be found here.

Chapter 4 is largely about transitioning non-agile processes to an agile workflow. There’s a lot here that would have been useful to me a couple years ago, but these days my work is great about not imposing cumbersome processes. Nonetheless, there was one passage that stood out:

On Courage

Courage is especially important. Get up and go talk to people; ask how you can help. Reach out to team members and other teams with direct communication. Notice impediments and ask the team to help remove them.

— Agile Testing, Lisa Crispin & Janet Gregory, Chapter 4

Often I find this one of my biggest personal challenges. Good communication is one of the most important elements of working as a team, and that means talking to people directly. I consider myself an introvert, and though I’m happy stepping up to lead a conversation when needed, it can be very easy to find excuses to avoid it. Sometimes I don’t want to bother someone or intrude, sometimes I’d rather avoid hashing out a point of disagreement.

There’s two main ways I try to overcome this.

Don’t borrow the jack

Some time ago, my husband told me a story about a man that got a flat tire out on a country road. According to Oprah.com the story originated with Danny Thomas in the 1950s. Quite a few versions have been nearly copy-pasted around the blogosphere already. The story goes that the man needed a borrow a jack to put his spare tire on. As he walked up to the closest farmhouse, he started imaging increasingly bad scenarios about what might happen. The farmer will probably demand money, get upset at being interrupted at the late hour, or just generally be an asshole. By the time the man gets to the front door, he’s worked himself up so much that he knows to expect the worst. When the farmer answers the knock, the man just shouts “You can keep your damn jack!” and storms off.

The moral of the story is to be careful about imagining worst-case scenarios and getting angry about hypotheticals. It’s this kind of thinking that leads to avoiding communication out of wanting to avoid conflict, even if the conflict is imaginary.  Give the guy a chance. and embrace the benefit of the doubt.

“Don’t borrow the jack” has become a shorthand now between my husband and I to warn ourselves about imagining the worst. I try to remind myself of that when I start getting into the mindset that I’d rather avoid talking to someone directly. Overcoming that mindset when it does set in can take a lot of courage.

Like going to the gym

Summoning the courage to get up and talk to someone isn’t always about overcoming conflict avoidance. Lisa and Janet point out an example in Chapter 5 of where processes in place can make it even more difficult:

Defect tracking systems certainly don’t promote communication between programmers and testers. They can make it easy to avoid talking directly to each other.

— Agile Testing, Lisa Crispin and Janet Gregory, Chapter 5

Too true. Written communication like defect trackers, documentation systems, and code review platforms have their purpose, but they’re also the easiest excuse to avoid conversation. A comment that provides background, context, or a clarification is great. One that continues a back-and-forth debate isn’t; go talk to the person and then come back to comment so there’s a record of the outcome. Five minutes talking in person could easily resolve something that would take hours or days in a writing. It can just as easily highlight a bigger issue at play that needs to be worked out.

The point is, as much as I might want to avoid it at times, I almost always come out of a face-to-face conversation happier for having done it. It’s like going to the gym. I may not want to, but just getting up and going for an hour is easier than agonizing over it all day, and is always worth it in the end.

Agile testing does, I think, ask more of us introverts than a documentation-heavy waterfall style. It can take courage to get up and talk to someone. Just don’t borrow the jack, and it’ll be worth it.