Seven things I learned at CAST

My CAST debrief from last week ended up being mostly about personal reflection, but I also wanted to call out the some of the things I picked up more specific to testing. After going through my notes I picked out what I thought where the important themes or interesting insights. There are many other tidbits, interactions, and ideas from those notes that I may write about later, these are just the top seven that I think will have an impact right away.

Each section here is paraphrased from the notes I took in each speaker’s session.

1. “Because we don’t like uncertainty we pretend we don’t have it”
Liz Keogh presented a keynote on using Cynefin as a way of embracing uncertainty and chaos in how we work. We are hard wired to see patterns where they don’t exist. “We treat complexity as if it’s predictable.” People experience the highest stress when shocks are unpredictable, so let’s at least prepare for them. To be safe to fail you have to be able to (1) know if it works, (2) amplify it if it does work, (3) know of it fails, (4) dampen it if it does fail, and (5) have a realistic reason for thinking there will be a positive impact in the first place. It’s not about avoiding failure, it’s about being prepared for it. The linked article above is definitely worth the read.

2. Testers are designers
From John Cutler’s keynote: “Design is the rendering of intent”. Anybody who makes decisions that affect that rendering is a designer. What you test, when you test, who you include, your mental models of the customer, and how we talk about what we find all factor in. UX researchers to have to know that they aren’t the user, but often the tester has used the product more than anybody else in the group. There’s no such thing as a design solution, but there is a design rationale. There should be a narrative around the choices we make for our products, and testers provide a ton of the information for that narrative. Without that narrative, no design was happening (nor any testing!) because there’s no sense of the intent.

3. Knowledge goes stale faster than tickets on a board do
From Liz Keogh’s lightning talk: If testers are at their WIP limit and devs can’t push any more work to them, your team will actually go faster if you make the devs go read a book rather than take on new work that will just pile up. You lose more productivity in catching up on work someone has already forgotten about than you gain in “getting ahead”. In practice, of course, you should have the devs help the testers when they’re full and the testers help the devs when they have spare capacity. (By the way, it’s not a Kanban board unless you have a signal for when you can take on new work, like an empty slot in a column. That empty slot is the kanban.)

4. “No matter how it looks at first, it’s always a people problem” – Jerry Weinberg
I don’t recall which session this quote first came up in, but it became a recurring theme in many of the sessions and discussions, especially around coaching testing, communicating, and whole-team testing. Jerry passed away just before the conference started, and though I had never met him he clearly had a strong influence on many of the people there. He’s probably most often cited for his definition that “quality is value to some person”.

5. Pair testing is one of the most powerful techniques we have
Though I don’t think anybody said this explicitly, this was evident to me from how often the concept came up. Lisi Hocke gave a talk about pairing with testers from other companies to improve her own testing while cross-pollinating testing ideas and skills with the wider community. Amit Wertheimer cited pairing with devs as a great way to identify tools and opportunities to make their lives easier. Jose Lima talked about running group exploratory testing sessions and the benefits that brings in learning about the product and coaching testing. Of course coaching itself, I think, is a form of pairing so the tutorial with Anne-Marie Charrett contributed to this theme as well.  This is something that I need to do more of.

6. BDD is not about testing, it’s about collaboration
From Liz Keogh again: “BDD is an analysis technique, it’s not about testing.” It’s very hard to refactor English and non-technical people rarely read the cucumber scenarios anyway. She says to just use a DSL instead. If you’re just implementing cucumber tests but aren’t having a 3-amigos style conversation with the business about what the scenarios should be, it isn’t BDD. Angie Jones emphasized these same points when introducing cucumber in her automation tutorial as a caveat that she was only covering the automation part of BDD in the tutorial, not BDD itself. Though I’ve worked in styles that called themselves “behaviour driven”, I’ve never worked with actual “BDD”, and this was the first time I’ve heard of it being more than a way of automating test cases.

7. Want to come up with good test ideas? Don’t read the f*ing manual!
From Paul Holland: Detailed specifications kill creativity. Start with high level bullet points and brainstorm from there. Share ideas round-robin (think of playing “yes and“) to build a shared list. Even dumb ideas can trigger good ideas. Encourage even non-sensical ideas since users will do things you didn’t think of. Give yourself time and space away to allow yourself to be creative, and only after you’ve come up with every test idea you can should you start looking at the details. John Cleese is a big inspiration here.

Bonus fact: Anybody can stop a rocket launchThe bright light of a rocket engine taking off in the night
The “T-minus” countdown for rocker launches isn’t continuous, they pause it at various intervals and reset it back to checkpoints when they have to address something. What does this have to do with testing? Just that the launch of the Parker Solar Probe was the weekend after CAST. At 3:30am on Saturday I sat on the beach with my husband listening to the live feed from NASA as the engineers performed the pre-launch checks: starting, stopping, and resetting the clock as needed. I was struck by the fact that at “T minus 1 minute 55 seconds” one calm comment from one person about one threshold being crossed scrubbed the entire launch without any debate. There wouldn’t be time to rewind to the previous checkpoint at T-minus 4 minutes before the launch window closed, so they shut the whole thing down. I’m sure that there’s an analogy to the whole team owning gates in their CD pipelines in there somewhere!

CAST 2018 Debrief

Last week I was lucky enough to attend the Conference of the Association of Software Testing, CAST 2018. I had been to academic conferences with collaborators before, and a local STAR conference here in Toronto, but this was my first time travelling for a professional conference in testing. The actual experience ended up being quite trying, and I ended up learning as much about myself as about testing. I don’t feel the need to detail my whole experience here, but I will highlight the top 5 lessons I took away from it.

1. “Coaching” is not what I hoped it was

I’ve been hearing a lot about “coaching” as a role for testers lately. I went to both Anne-Marie Charrett‘s tutorial and Jose Lima‘s talk on the subject thinking that it was a path I wanted to pursue. I went in thinking about using as a tool to change minds, instill a some of my passion for testing into the people I work with, and building up a culture of quality. I came away with a sense of coaching as more of a discussion method, a passive enterprise available for those who want to engage in it and useless for the uninterested. I suspect those who work as coaches would disagree, but that was nonetheless my impression.

One theme that came up from a few people, not just the speakers, was a distinction between coaching and teaching. This isn’t something I really understand, and is likely part of why I was expecting something else from the subject. I taught university tutorials for several years and put a lot of effort into designing engaging classes. To me, what I saw described as coaching felt like a subset of teaching, a particular style of pedagogy, not something that stands in contrast to it. Do people still hear “teaching” and think “lecturing”? I heard “coaching testing” and expected a broader mandate of education and public outreach that I associate with “teaching”.

Specifically, I was looking for insight on breaking through to people who don’t like testing, and who don’t want to learn about it, but very quickly saw that “coaching” wasn’t going to help me with that. At least not on the level at which we got into it in within one workshop. I am sure that this is something that would be interesting to hash out in a (meta) coaching session with people like Anne-Marie and Jose, even James Bach and Michael Bolton: i.e. people who have much more knowledge about how coaching can be used than I do.

2. I’m more “advanced” than I thought

My second day at the conference was spent in a class billed as “Advanced Automation” with Angie Jones (@techgirl1908). I chose this tutorial over other equally enticing options because it looked like the best opportunity for something technically oriented, and would produce a tangible artefact — an advanced automated test suite — that I could show off at home and assimilate aspects of into my own automation work.

Angie did a great job of walking us through implementing the framework and justifying the thought process each step of the way. It was a great exercise for me to go through implementing a java test suite from scratch, including a proper Page Object Model architecture and a TDD approach. It was my first time using Cucumber in java, and I quite enjoyed the commentary on hiring API testers as we implemented a test with Rest-Assured.

Though I did leave with that tangible working automation artefact at the end of the day, I did find that a reverse-Pareto principle at play with 80% of the value coming from the last 20% of the time. This is what lead to my take away that I might be more advanced than I had thought. I still don’t consider myself an expert programmer, but I think I could have gotten a lot further had we started with a basic test case already implemented. Interestingly Angie’s own description for another workshop of hers say “It’s almost impossible to find examples online that go beyond illustrating how to automate a basic login page,” though that’s the example we spent roughly half the day on. Perhaps we’ve conflated “advanced” with “well designed”.

3. The grass is sometimes greener

In any conference, talks will vary both in quality generally and how much they resonate with any speaker specifically. I was thrilled by John Cutler‘s keynote address on Thursday — he struck many chords about the connection between UX and testing that align very closely with my own work — but meanwhile Amit Wertheimer just wrote that he “didn’t connect at all” to it. I wasn’t challenged by Angie’s advanced automation class but certainly others in the room were. This is how it goes.

In a multi-track conference, there’s an added layer that there’s other rooms you could be in that you might get more value from. At one point, I found myself getting dragged down in a feeling that I was missing out on better sessions on the other side of the wall. Even though there were plenty of sessions where I know I was in the best room for myself, the chatter on Twitter and the conference slack workspace sometimes painted a picture of very green grass elsewhere. Going back to Amit’s post, he called Marianne Duijst‘s talk about Narratology and Harry Potter one of the highlights of the whole conference, and I’ve seen a few others echo the same sentiment on Twitter. I had it highlighted on my schedule from day one but at the last minute was enticed by the lightning talks session. I got pages of notes from those talks, but I can’t help but wonder what I missed. Social Media FOMO is real and it takes a lot of mental energy to break out of that negative mental cycle.

Luckily, the flip side of that kind of FOMO is that asking about a session someone else was in, or gave themselves, is a great conversation starter during the coffee breaks.

4. Networking is the worst

For other conferences I’ve been to, I had the benefit either of going with a group of collaborators I already knew or being a local so I could go home at 5 at not worry about dinner plans. Not true when flying alone across the continent. I’ve always been an introvert at the best of times, and I had a hard time breaking out of that to go “network”.

I was relieved when I came across Lisa Crispin writing about how she similarly struggled when she first went to conferences, although that might have helped me more last week than today. Though I’m sure it was in my imagination just as much as it was in hers at her first conference, I definitely felt the presence of “cliques” that made it hard to break in. Ironically, those that go to conferences regularly are less likely to see this happening, since those are the people that already know each other. Speakers and organizers even less so.

It did get much easier once we moved to multiple shorter sessions in the day (lots of coffee breaks) and an organized reception on Wednesday. I might have liked an organized meet-and-greet on the first day, or even the night before the first tutorial, where an introvert like me can lean a bit more on the social safety net of mandated mingling. Sounds fun when I put it like that, right?

I eventually got comfortable enough to start talking with people and go out on a limb here or there. I introduced myself to the all people I aimed to and asked all the questions I wanted to ask… eventually. But there were also a lot of opportunities that I could have taken better advantage of. At my next conference, this is something I can do better for myself, though it also gives me a bit more sensitivity about what inclusion means.

5. I’m ready to start preparing my own talk

Despite my introverted tendencies I’ve always enjoyed teaching, presenting demos, and giving talks. I’ve had some ideas percolating in the back of my mind about what I can bring to the testing community and my experiences this week — in fact every one of the four points above — have confirmed for me that speaking at a conference is a good goal for myself, and that I do have some value to add to the conversation. I have some work to do.

Bonus lessons: Pronouncing “Cynefin” and that funny little squiggle

Among the speakers, as far as notes-written-per-sentence-spoken, Liz Keogh was a pretty clear winner by virtue of a stellar lightning talk. Her keynote and the conversation we had afterward, however, is where I picked up these bonus lessons. I had heard of Cynefin before but always had two questions that never seemed to be answered in the descriptions I had read, until this week:

A figure showing the four domains of Cynefin

  1. It’s pronounced like “Kevin” but with an extra “N”
  2. The little hook or squiggle at the bottom of the Cynefin figure you see everywhere is actually meaningful: like a fold in some fabric, it indicates a change in height from the obvious/simple domain in the lower right from which you can fall into the chaotic in the lower left.

Little Alchemy: A silly little test exercise

Lately I’ve been entertaining myself with a silly little mobile app called Little Alchemy 2, a simple revamp of an earlier concept game. It distracted me enough this week that I didn’t prepare a respectable blog post. While questioning my life choices, I started to convince myself that maybe I had been playing at testing this whole time. Testers, myself included, love to make analogies to science, and this is a game that simulates the precursor to modern chemistry. Of course it can be used as a tortured metaphor for software testing! Follow me on my journey…

Exploration

Screenshot of Little AlchemyThe main concept of the game is to drag icons of real life objects on top of each other, combining them to make new items.

At first there are only a few options: Water, Earth, Fire, Air. The classics. Not hard to just play around with different combinations and see how this game works.

Fire + Water = Steam. Makes sense.

Water + Earth = Mud. Great.

Now I wonder what happens if I combine steam and mud….

After a while you start to get a sense for how this universe works.

Developing heuristics

After a while I start to see patterns. If I combine the same thing with itself I get a bigger version of that thing; can I do that with everything? Fire seems transform most items. When I find out how to make paper or wood, I’m probably going to try to throw those on the fire too.

Combinations quickly multiply.

Before long I have hundreds of items of all sorts. Too many to keep in my head. I start to think I’m probably missing combinations, or I’ve forgotten what combinations I’ve already tried. I know I tried putting wood into fire but I did I put wood into the campfire? Am I missing some obvious combination that’s right in front of me?

Using an Oracle

That’s when I start googling around for hints and suggestions. This part gets a bit cheaty, but at the time it was what I needed to do to get a handle on the problem and keep making progress. I found a site that would randomly show me an item, and I could use that to see if I had the pieces I needed to make it. No more guessing, I was given the right answer.

I suppose this is where I could veer off into automation, but where’s the fun in that? After a while, I started to exhaust the hint system anyway; with only random items the ratio of new to known items started to decline. ROI went down. My oracle was getting less useful, not telling me anything new.

Brute force

There were still an intractable number of items and I had seen enough unexpected combinations that I didn’t trust myself to reason them all out myself. So instead, I turned to brute force.

First item on the list. Try to combine with every item below it. Repeat.

Now I should really think about automation if my goal was to just find all combinations. This is a pure algorithm with finite steps since the game promises a finite number of items. But things start to change after using this for a few iterations manually. Happily the game removes items that have no undiscovered combinations, so in theory the complexity will peak and game will start get simpler again. (Wouldn’t it be nice if software always got simpler with time?) Moreover, a little bit of brute force starts to show patterns that I hadn’t been aware of before. I start to skip ahead: “aha! if that combination works, then I bet this other one will too…” One new strategy begets another!

Inference and deduction

I reach a breaking point where items are being removed from play more and more often. This feels like the home stretch but brute forcing it will still take ages. Often, though, an item only has one or two combinations left to discover before it’s exhausted. I use this to my advantage.

Enter another oracle of sorts; this time, it’s the documentation of everything I’ve done so far. For any item still on the board, the game tells me how many undiscovered combinations an item has, items that have been used to make it, and all the other items it has been used to make so far. This is all data I can analyse to look for yet more patterns, and spot gaps in patterns that have been missed so far. The rate at which I clear items off the board goes way up, and I’m still using the same manual interactions I’ve used all along.

End game

I’m not there yet. Do I still have more to learn? Is another strategy going to make itself obvious before I finish the game, or an old one resurface as the dynamics change?

And what am I going to explore next?

Debating the Modern Testing Principles

Last week I had the opportunity to moderate a discussion on the Modern Testing Principles being developed by Alan Page and Brent Jensen with a group of QA folks. I’m a relative late-comer to the AB Testing podcast, having first subscribed somewhere around Episode 60, but have been quite interested in this take on testing. Armed primarily with the 4 episodes starting from their initial introduction and some back-up from the article by the Ministry of Testing, we had a pretty interesting discussion.

Discussing the seven principles

After giving a bit of a preamble based on the mission statement—“Accelerate the Achievement of Shippable Quality”—we went through the principles one by one. For each one I asked (roughly) these questions:

  1. What do you think this statement means?
  2. How do you feel about this a core principle of testing?
  3. How well (or not) does this describe your approach to testing?
  4. Is this a principle you would adopt?

For the first four principles, there was a lot of agreement. We discussed building better products versus trying to assure the product’s quality, the importance of prioritization of tests and identifying bottlenecks, leaky safety nets, data-driven decisions, and the easy alignment with a whole-team Agile mindset. Then it started to get a bit more interesting.

Disagreement one: Judging Quality

The fifth principle started to get problematic for some people:

5. We believe that the customer is the only one capable to judge and evaluate the quality of our product.

There was a lot of debate here. Although a couple people were on board right away, the biggest question for most in the room was: who is the “customer”? Lots of people could fall into that category. Internally there are stakeholders in different parts of the business, product owners in our team, managers in our department, and team itself to some degree. We also have both our current end users and the people we want to attract into regular users. Some of you may have simpler environments with a clear cut individual client, but others could be even more complicated.

What we did agree on was that you have to use the product to be able to judge it. The people testing have to think like the customer and have a good idea of what their expectations are. Interestingly, when we changed “customer” to “anybody who uses the product”, everybody around the table could agree with the principle as a whole.

I suspect, though, that if we only say “anybody who uses the product is capable of judging and evaluating the quality of the product”, the statement loses its power. My feeling is that if this principle feels problematic in its original form, you may just not have a firm idea of who your customer really is. This just highlights for me how important it is to ask who’s opinion, at the end of the day, is the one that counts.

Disagreement two: The dedicated specialist

It’s likely unsurprising that a principle suggesting the elimination of the testing specialist would raise a few eyebrows in a group of testing specialists.

7. We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.

There was no disagreement with the first clause. Many people immediately connected it with the 4th principle, to “coach, lead, and nurture the team towards a more mature quality culture”. Surely endeavouring to “expand the testing abilities and know-how across the team” is a good way to achieve that. When the group initially discussed the 4th principle, we were all in agreement that we wanted to drive a culture of quality and a whole-team approach to testing.

I am still unsure whether the disagreement with eliminating the dedicated specialist was just a knee-jerk reaction or not. I tried to use an analogy of the tester-as-Mary-Poppins: She stays only as long as she is truly needed, and then takes to the wind again to find a new family in need. It didn’t seem to sell the point. We agreed that our teams should be able to function without us… temporarily. There was one assertion that QA was the most important part of the whole process and therefore could not be eliminated. Another one that the skills are different from other roles. And yet another that not everybody wants to be a dev. (Although, of course, the principle doesn’t end with “… so that they can become a developer.”)

Additional context from Alan and Brent helps here too. In some of the episodes after the principles were first introduced, they do talk about now not every tester needs to be a Capital-M Capital-T Modern Tester. I don’t believe the intent is to eventually eliminate the need for testing specialists full stop. It’s not even a given that the specialist would be eliminated on a particular team, just that the need for a specialist should be reduced. To me this principle is a corollary of reducing bottlenecks and building the testing know-how on the team, albeit phrased more provocatively.

Nonetheless, the closest we got to agreement on this was to say we could eventually eliminate the singular position of a testing specialist, but not eliminate the function.

Is that any different or just an easier pill to swallow?

Wrapping up

Both of these, the two biggest objections to the Modern Testing Principles, have a common theme. The 4th principle asserts that testers aren’t the judge of quality or even truly capable of evaluating it. The 7th pushes the idea that given the right expertise and know-how, a testing specialist may not even be needed. Both of these can feel like a threat. Both speak to a fear of losing agency. Alan and Brent also talked about this in the podcasts: one of the motivations for formulating these principles was to prepare people for how testing is changing so that we aren’t all caught unprepared. While I have doubts that there’s an apocalyptic testing singularity coming—something I plan to write on in another post—it does emphasize how important it is to be prepared for new ways of thinking in the industry.

To wrap up the discussion, we did a quick summary of the words and concepts that had come up as common themes in the principles. Then, to compare, I asked for testing concepts or buzzwords that had been conspicuously absent. Chief among the latter were automation, defect tracking, reporting, traceability, documentation, and not once did we talk about writing a test case. Highlighting what was not explicitly mentioned in the principles seemed to be a great way to highlight what makes this a different approach compared to a lot of our day-to-day experience. Though some of those “missing” elements may come out naturally as tools necessary to really embrace these principles, I felt it important to highlight that they were not the goal in and of themselves.

In the end, these differences and the disagreements were the most interesting part of the Modern Testing Principles. Alan described presenting the principles at Test Bash in much the same way—it’s not much fun if everybody just agrees with everything! Hopefully the discussions sparked some new ways of thinking, if only a little bit.

Testing is like a box of rocks

I was inspired today by Alan Page’s Test Automation Snowman. He makes all good points, but let’s be honest, the model is the same as the good ol’ test pyramid. The only difference is that he’s being explicit about tests at the top of the pyramid being slow and tests at the bottom being fast. Ok, so maybe the snowman thing is a joke, but it did make me think about what might make a better visualization. I quickly doodled something on a sticky note:

A sticky note with a lot of tiny circles in the bottom third, medium circles in the middle third, and a few large circles in the top third.

If the point we want to emphasize is that UI tests are slow (and therefore take a lot of time), we should include that in the visualization! The problem with the pyramid (and the snowman) is that the big tests take up the least amount of space; the small triangle at the top makes it look like having fewer UI tests also means you do less UI testing.

It doesn’t.

At least, not proportionately. If you had an equal number of UI and unit tests, it’s a safe bet that you’re going to spend more of your time working on the UI tests.

So instead, let’s say testing is like a box of rocks. Each rock is a test, and I have to choose how to allocate the space in that box to fit all the rocks that I want. A few big rocks are going to take up a lot more space than a bunch of tiny pebbles. Unless I have a good reason why that big boulder is a lot more interesting than the hundred little rocks I could put in its place, I’m going to go for the little rocks! If I have to add a new rock (to kill a new bug, say) I probably want to choose the smallest one that’ll still do the job.

You can still think about the different levels (unit vs API vs UI, for example) if you picture the little rocks at the bottom forming a foundation for bigger rocks on top. I don’t know if rocks work like that. Just be careful not to get this whole thing confused with that dumb life metaphor.

Ok, it might not be the best model, but I’ll stick with it for now. And like the Alan’s snowman, you’re free to ignore this one too.