Debating the Modern Testing Principles

Last week I had the opportunity to moderate a discussion on the Modern Testing Principles being developed by Alan Page and Brent Jensen with a group of QA folks. I’m a relative late-comer to the AB Testing podcast, having first subscribed somewhere around Episode 60, but have been quite interested in this take on testing. Armed primarily with the 4 episodes starting from their initial introduction and some back-up from the article by the Ministry of Testing, we had a pretty interesting discussion.

Discussing the seven principles

After giving a bit of a preamble based on the mission statement—“Accelerate the Achievement of Shippable Quality”—we went through the principles one by one. For each one I asked (roughly) these questions:

  1. What do you think this statement means?
  2. How do you feel about this a core principle of testing?
  3. How well (or not) does this describe your approach to testing?
  4. Is this a principle you would adopt?

For the first four principles, there was a lot of agreement. We discussed building better products versus trying to assure the product’s quality, the importance of prioritization of tests and identifying bottlenecks, leaky safety nets, data-driven decisions, and the easy alignment with a whole-team Agile mindset. Then it started to get a bit more interesting.

Disagreement one: Judging Quality

The fifth principle started to get problematic for some people:

5. We believe that the customer is the only one capable to judge and evaluate the quality of our product.

There was a lot of debate here. Although a couple people were on board right away, the biggest question for most in the room was: who is the “customer”? Lots of people could fall into that category. Internally there are stakeholders in different parts of the business, product owners in our team, managers in our department, and team itself to some degree. We also have both our current end users and the people we want to attract into regular users. Some of you may have simpler environments with a clear cut individual client, but others could be even more complicated.

What we did agree on was that you have to use the product to be able to judge it. The people testing have to think like the customer and have a good idea of what their expectations are. Interestingly, when we changed “customer” to “anybody who uses the product”, everybody around the table could agree with the principle as a whole.

I suspect, though, that if we only say “anybody who uses the product is capable of judging and evaluating the quality of the product”, the statement loses its power. My feeling is that if this principle feels problematic in its original form, you may just not have a firm idea of who your customer really is. This just highlights for me how important it is to ask who’s opinion, at the end of the day, is the one that counts.

Disagreement two: The dedicated specialist

It’s likely unsurprising that a principle suggesting the elimination of the testing specialist would raise a few eyebrows in a group of testing specialists.

7. We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.

There was no disagreement with the first clause. Many people immediately connected it with the 4th principle, to “coach, lead, and nurture the team towards a more mature quality culture”. Surely endeavouring to “expand the testing abilities and know-how across the team” is a good way to achieve that. When the group initially discussed the 4th principle, we were all in agreement that we wanted to drive a culture of quality and a whole-team approach to testing.

I am still unsure whether the disagreement with eliminating the dedicated specialist was just a knee-jerk reaction or not. I tried to use an analogy of the tester-as-Mary-Poppins: She stays only as long as she is truly needed, and then takes to the wind again to find a new family in need. It didn’t seem to sell the point. We agreed that our teams should be able to function without us… temporarily. There was one assertion that QA was the most important part of the whole process and therefore could not be eliminated. Another one that the skills are different from other roles. And yet another that not everybody wants to be a dev. (Although, of course, the principle doesn’t end with “… so that they can become a developer.”)

Additional context from Alan and Brent helps here too. In some of the episodes after the principles were first introduced, they do talk about now not every tester needs to be a Capital-M Capital-T Modern Tester. I don’t believe the intent is to eventually eliminate the need for testing specialists full stop. It’s not even a given that the specialist would be eliminated on a particular team, just that the need for a specialist should be reduced. To me this principle is a corollary of reducing bottlenecks and building the testing know-how on the team, albeit phrased more provocatively.

Nonetheless, the closest we got to agreement on this was to say we could eventually eliminate the singular position of a testing specialist, but not eliminate the function.

Is that any different or just an easier pill to swallow?

Wrapping up

Both of these, the two biggest objections to the Modern Testing Principles, have a common theme. The 4th principle asserts that testers aren’t the judge of quality or even truly capable of evaluating it. The 7th pushes the idea that given the right expertise and know-how, a testing specialist may not even be needed. Both of these can feel like a threat. Both speak to a fear of losing agency. Alan and Brent also talked about this in the podcasts: one of the motivations for formulating these principles was to prepare people for how testing is changing so that we aren’t all caught unprepared. While I have doubts that there’s an apocalyptic testing singularity coming—something I plan to write on in another post—it does emphasize how important it is to be prepared for new ways of thinking in the industry.

To wrap up the discussion, we did a quick summary of the words and concepts that had come up as common themes in the principles. Then, to compare, I asked for testing concepts or buzzwords that had been conspicuously absent. Chief among the latter were automation, defect tracking, reporting, traceability, documentation, and not once did we talk about writing a test case. Highlighting what was not explicitly mentioned in the principles seemed to be a great way to highlight what makes this a different approach compared to a lot of our day-to-day experience. Though some of those “missing” elements may come out naturally as tools necessary to really embrace these principles, I felt it important to highlight that they were not the goal in and of themselves.

In the end, these differences and the disagreements were the most interesting part of the Modern Testing Principles. Alan described presenting the principles at Test Bash in much the same way—it’s not much fun if everybody just agrees with everything! Hopefully the discussions sparked some new ways of thinking, if only a little bit.

What the Bug!? (An attempt at knowledge sharing, two ways)

When I was making the transition from waterfall style projects to agile teams in a previous company, one of the main things I struggled with was the loss of the testing team as we all became generic “software engineers”. We all still did the same testing tasks, but none of us were “testers” any more. There were a lot of positive effects from the change, but I kept feeling like without dedicated testers focused on improving our testing craft, we’d stagnate.

What I was missing, as I eventually realized, was a testing community. A recent episode of the AB Testing podcast talked all about building a community of testers, which brought all of this back to mind. A few of the ideas Alan and Brent talked about I had actually tried at the time. One in particular was the idea of highlighting major fails of the week in a newsletter, even offering prizes for the “winner”. Back in 2016 I heard a similar idea at a STARCanada talk, where the engineering group at AppFolio would send an email to everybody in the company for every bug they found describing what was learned, again emphasizing that finding these bugs was a positive thing, not a blame game. (Sorry I can’t find now who gave that talk; if it was you let me know!)

The reason the idea stuck with me at the time was primarily because I had started to notice that as our newly agile teams specialized in subsets of what was previously a monolithic product, we started to loose visibility on bugs that other teams ran into. Different teams were getting bit by the same issues. It didn’t help that the code base was also old enough that newer people would run into old bugs, spend potentially hours debugging it, and then hear “oh yeah, that’s a known problem.” (My response to that was to scream “It isn’t known if people don’t know it!” silently to myself.)

Here’s what I tried to do about it:

What The Bug!?
Honestly, the part of the idea I’m most proud of might be the name

I wanted teams to start talking more about the bugs they found, both so that others could learn from them and so that we could all tune our spidey senses to the sorts of issues that were actually important. There wasn’t much appetite for an email newsletter—people didn’t seem to read the newsletters the company already had—but we ended up trying two alternatives, one of which was pretty successful and one of which really wasn’t.

Building A knowledge base

The first idea was to solicit short and easily digestible details about every production bug that got logged into our bug tracker. Anybody who logged a bug would get an email asking them to answer three questions. The key was that the answers had to be short—think one tweet—and written at a level that any developer in the company would be able to understand. Bonus points for memorable descriptions. The questions were roughly:

  1. What was the bug?
  2. What was important about it?
  3. What one lesson can we learn from it?

The answers were linked back to the original ticket and tagged with a bunch of meta data like which team found it, the part of the system it was found in, what language the code was in, and any other keywords that would make it easy to find again. The idea was that if I was going to start writing something in a new language or working on a new part of the system, I could go look up the related tags and immediately find a list of easily digestible problems that I should stay alert for. I think it was an okay idea, but there were issues.

First problem: People were pretty bad at writing descriptions of bugs that were short, but also useful and interesting. It not only took a lot of creativity, but in order to do it well you also really had to examine what was important about the bug in the first place. The example I used as a bad answer to the second question was “This caused an error every Tuesday”; What caused what kind of error every WHY TUESDAY!? This was especially problematic for the third question, where often the answers that came back were “testing is important” or “we’ll test this next time”. True, but shallow. What I was really hoping for would have been “There are different kinds of Unicode characters, so always consider testing different character planes, byte lengths, and reserved code points”. To really make the knowledge base as useful as it could have been, it would have needed committed editors who would talk to the people involved and craft a really good summary with them.

Second problem: The response rate was pretty lousy. It might have been that targeting every production bug was just too much. Not everybody is going to see something interesting in every bug, and not everybody is as interested in learning from them. That was part of the culture that I wanted to change—I wanted everybody else to be as excited about these bugs as I was!—but it wasn’t going to happen over night.

Third problem: It might seem minor, but the timing of the email prompt coming the day after the bug was logged was often just too soon to have digested what the real lesson learned was. This turned up as a problem as I asked around about why people weren’t taking part. They just didn’t have the answers yet.

All of this created a chicken-and-egg problem. Until people saw the value of this project, saw interesting summaries, and got excited to contribute their own experiences, we wouldn’t get the content we needed to build that interest or excitement. And, in all honesty, though there was a conscious effort with this to make an accessible library of bugs compared to the technical JIRA tickets, we were still basically asking people to log a bug in one database every time they logged a bug in another database. We needed something more active and engaging.

Welcome to the What The Bug!? Show

At the time we already had a weekly all-hands meeting every Friday afternoon where anybody could contribute a segment to demo something, talk about something interesting, or anything else they wanted. I was doing short segments on quality and testing topics every few weeks to try to promote testing in general, but it was largely a one-man show. A fellow tester/developer that I was working with on the What The Bug!? knowledge base had the idea to take just a couple minutes each week to present our favourite bugs of the week.

Turning a passive library of bugs into a weekly road show was a big success. We were basically getting the benefit of a bug-of-the-week newsletter with the added bonus of an already established live audience. Again, the branding helped to sell it, and because the two of us both embraced the idea that any production bug could be turned into an 2 minute elevator pitch for something interesting to learn, we were able to make pretty fun presentations. We at least got laughs and had people thinking about testing if only for 5 minutes, but the real highlight for me was that afterwards I had people come up to me and say “you know, I found something last week that would make a good segment next time.”

This was possible in part because she and I had both spent that time soliciting user-friendly descriptions from people logging bugs, so we knew what had been going on across all the teams for the last couple weeks. We had a lot to choose from for the feature. It would have been harder to do without that knowledge base, being limited to only the bugs we knew from the teams we worked directly with. I suspect, though, that once the weekly What The Bug!? segments built up enough momentum that people took on presenting their own bugs, the need for the email prompts and the knowledge base would fade away. An archive of the featured bugs could still be a useful resource for new people coming on board, but it would no longer be the primary driver.

Where are things now

I left the company shortly after starting What The Bug!?, but I recently had a chance to check in with an old colleague and inquire about where these two manifestations of the idea ended up. Unsurprisingly in retrospect, the knowledge base has largely fizzled. The combination of asking people to volunteer extra writing work and the low ROI of a poorly written archive doomed it pretty early on. On the flip side, bugs are still a regular topic at those weekly meetings, although I’m sorry to report that they dropped the What The Bug!? branding. If anybody else wants to try something similar, feel free to use the name, all I ask for is a hat tip. (A quick google of the phrase only turns up some etymologists and an edible insect company, but if someone else had the same idea in the context of software, do let me know and I’ll pass the hat tip on.)