Respecting the house rules

I just watched Angie Jones’s keynote from this year’s Selenium Conference using elements of game design to talk about test automation. I really like what she said about the rules of the game. Broadly, there are the rules that are part of the game—the best practices and patterns that are well established parts of the trade—but there’s also the house rules.

These are the rules that you make up with your team members… In the automation game, we’re not really accepting of the house rules. People judge you. They think that you’re clueless if you tell them you’re playing by the house rules. I don’t like this. Can we stop doing this? Let’s stop judging people and assuming that everyone is playing with all of the game pieces and that they can do everything by these outside-the-box rules that technically we made up ourselves. Can we allow people to implement some cultural variances to their game to get ahead? People out here are doing the best that they can.

— Angie Jones at SeConf 2018

She goes on to tell an example about a team that knows they aren’t doing things by the book, but not for lack of trying. People have to work with what they’ve got. And yes, they can do their best to change it, but even if change is possible it doesn’t happen over night and you can’t build perfect (if you even know what perfect would be) over night.

This is good to remember not just for talking about how other people approach test automation, but within our own teams as well, and any time you’re looking at the work of another person. Many times I’ve heard someone complain about how insane or ugly or ridiculous some piece of code is, without considering why it might have made sense to write it like that in the first place. Why it might still make sense to write it like that. No design is sacred, but Angie also makes the point later in her talk that we shouldn’t just be pointing out how not to do things, we should talk about how we can do better. Simply declaring how awful something is showcases the ignorance of the person complaining at least as often as the original author’s.

Worth repeating: “People are out here doing the best they can.”

The full video is below. The house rules part is at about 24:20.

99 character talk: Unautomated automated tests

Some of the testers over at testersio.slack.com were lamenting that it was quiet online today, probably due to some kind of special event that the rest of us weren’t so lucky to join. Some joked that we should have our own online conference without them, and Daniel Shaw had the idea for “99 character talks”, an online version of Ministry of Testing’s 99 second talks. Several people put contributions on Slack, and some made it into a thread for them the MoT Club.

The whole concept reminds me of a story I once heard about Richard Feynman. As I remember it, he was asked the following: If all civilization and scientific knowledge was about to be destroyed, but he could preserve one statement for future generations, what would he say to give them the biggest head start on rebuilding all that lost knowledge? His answer: “Everything is made of atoms.

Luckily the 99 character challenge wasn’t quite so dramatic as having to preserve the foundations of software testing before an apocalypse. So instead, for my contribution I captured a simpler bit of advice that has bit me in the ass recently:

Automated tests aren’t automated if they don’t run automatically. Don’t make devs ask for feedback.

99 characters exactly.

But because I can’t help myself, here’s a longer version:

The problem with automated tests*, as with any test, is that they can only give you feedback if you actually run them. I have a bunch of tests for a bunch of different aspects of the products I work on, but not all of them are run automatically as part of our build pipelines. This effectively means that they’re opt-in. If someone runs them, actively soliciting the feedback that these tests offer, then they’ll get results. But if nobody takes that active step, the tests do no good at all. One of the worst ways to find a bug, as I recently experienced, is to know that you had everything you needed to catch it in advance but just didn’t.

If your automated tests aren’t run automatically whenever something relevant changes, without requiring someone to manually trigger them, then they aren’t automated at all. An automated test that is run manually is still a manual test. It’s one that can be repeated reliably and probably faster than a human could do, sure, but the onus is still on a fallible bag of meat to run it. Build them into your pipeline and loop the feedback back to the programmer as soon as something goes wrong instead. More and more lately I’m realizing that building reliable and efficient pipelines is a key skill to get the most out of my testing.

Now to put my own advice into practice…


* The “tests vs checks” pedants in the room will have to be satisfied for the moment with the fact that writing “automated checks” would have make the talk 100 characters and therefore inadmissible. Sorry, but there were zero other ways to reduce the character count by 1.

CAST 2018 Debrief

Last week I was lucky enough to attend the Conference of the Association of Software Testing, CAST 2018. I had been to academic conferences with collaborators before, and a local STAR conference here in Toronto, but this was my first time travelling for a professional conference in testing. The actual experience ended up being quite trying, and I ended up learning as much about myself as about testing. I don’t feel the need to detail my whole experience here, but I will highlight the top 5 lessons I took away from it.

1. “Coaching” is not what I hoped it was

I’ve been hearing a lot about “coaching” as a role for testers lately. I went to both Anne-Marie Charrett‘s tutorial and Jose Lima‘s talk on the subject thinking that it was a path I wanted to pursue. I went in thinking about using as a tool to change minds, instill a some of my passion for testing into the people I work with, and building up a culture of quality. I came away with a sense of coaching as more of a discussion method, a passive enterprise available for those who want to engage in it and useless for the uninterested. I suspect those who work as coaches would disagree, but that was nonetheless my impression.

One theme that came up from a few people, not just the speakers, was a distinction between coaching and teaching. This isn’t something I really understand, and is likely part of why I was expecting something else from the subject. I taught university tutorials for several years and put a lot of effort into designing engaging classes. To me, what I saw described as coaching felt like a subset of teaching, a particular style of pedagogy, not something that stands in contrast to it. Do people still hear “teaching” and think “lecturing”? I heard “coaching testing” and expected a broader mandate of education and public outreach that I associate with “teaching”.

Specifically, I was looking for insight on breaking through to people who don’t like testing, and who don’t want to learn about it, but very quickly saw that “coaching” wasn’t going to help me with that. At least not on the level at which we got into it in within one workshop. I am sure that this is something that would be interesting to hash out in a (meta) coaching session with people like Anne-Marie and Jose, even James Bach and Michael Bolton: i.e. people who have much more knowledge about how coaching can be used than I do.

2. I’m more “advanced” than I thought

My second day at the conference was spent in a class billed as “Advanced Automation” with Angie Jones (@techgirl1908). I chose this tutorial over other equally enticing options because it looked like the best opportunity for something technically oriented, and would produce a tangible artefact — an advanced automated test suite — that I could show off at home and assimilate aspects of into my own automation work.

Angie did a great job of walking us through implementing the framework and justifying the thought process each step of the way. It was a great exercise for me to go through implementing a java test suite from scratch, including a proper Page Object Model architecture and a TDD approach. It was my first time using Cucumber in java, and I quite enjoyed the commentary on hiring API testers as we implemented a test with Rest-Assured.

Though I did leave with that tangible working automation artefact at the end of the day, I did find that a reverse-Pareto principle at play with 80% of the value coming from the last 20% of the time. This is what lead to my take away that I might be more advanced than I had thought. I still don’t consider myself an expert programmer, but I think I could have gotten a lot further had we started with a basic test case already implemented. Interestingly Angie’s own description for another workshop of hers say “It’s almost impossible to find examples online that go beyond illustrating how to automate a basic login page,” though that’s the example we spent roughly half the day on. Perhaps we’ve conflated “advanced” with “well designed”.

3. The grass is sometimes greener

In any conference, talks will vary both in quality generally and how much they resonate with any speaker specifically. I was thrilled by John Cutler‘s keynote address on Thursday — he struck many chords about the connection between UX and testing that align very closely with my own work — but meanwhile Amit Wertheimer just wrote that he “didn’t connect at all” to it. I wasn’t challenged by Angie’s advanced automation class but certainly others in the room were. This is how it goes.

In a multi-track conference, there’s an added layer that there’s other rooms you could be in that you might get more value from. At one point, I found myself getting dragged down in a feeling that I was missing out on better sessions on the other side of the wall. Even though there were plenty of sessions where I know I was in the best room for myself, the chatter on Twitter and the conference slack workspace sometimes painted a picture of very green grass elsewhere. Going back to Amit’s post, he called Marianne Duijst‘s talk about Narratology and Harry Potter one of the highlights of the whole conference, and I’ve seen a few others echo the same sentiment on Twitter. I had it highlighted on my schedule from day one but at the last minute was enticed by the lightning talks session. I got pages of notes from those talks, but I can’t help but wonder what I missed. Social Media FOMO is real and it takes a lot of mental energy to break out of that negative mental cycle.

Luckily, the flip side of that kind of FOMO is that asking about a session someone else was in, or gave themselves, is a great conversation starter during the coffee breaks.

4. Networking is the worst

For other conferences I’ve been to, I had the benefit either of going with a group of collaborators I already knew or being a local so I could go home at 5 at not worry about dinner plans. Not true when flying alone across the continent. I’ve always been an introvert at the best of times, and I had a hard time breaking out of that to go “network”.

I was relieved when I came across Lisa Crispin writing about how she similarly struggled when she first went to conferences, although that might have helped me more last week than today. Though I’m sure it was in my imagination just as much as it was in hers at her first conference, I definitely felt the presence of “cliques” that made it hard to break in. Ironically, those that go to conferences regularly are less likely to see this happening, since those are the people that already know each other. Speakers and organizers even less so.

It did get much easier once we moved to multiple shorter sessions in the day (lots of coffee breaks) and an organized reception on Wednesday. I might have liked an organized meet-and-greet on the first day, or even the night before the first tutorial, where an introvert like me can lean a bit more on the social safety net of mandated mingling. Sounds fun when I put it like that, right?

I eventually got comfortable enough to start talking with people and go out on a limb here or there. I introduced myself to the all people I aimed to and asked all the questions I wanted to ask… eventually. But there were also a lot of opportunities that I could have taken better advantage of. At my next conference, this is something I can do better for myself, though it also gives me a bit more sensitivity about what inclusion means.

5. I’m ready to start preparing my own talk

Despite my introverted tendencies I’ve always enjoyed teaching, presenting demos, and giving talks. I’ve had some ideas percolating in the back of my mind about what I can bring to the testing community and my experiences this week — in fact every one of the four points above — have confirmed for me that speaking at a conference is a good goal for myself, and that I do have some value to add to the conversation. I have some work to do.

Bonus lessons: Pronouncing “Cynefin” and that funny little squiggle

Among the speakers, as far as notes-written-per-sentence-spoken, Liz Keogh was a pretty clear winner by virtue of a stellar lightning talk. Her keynote and the conversation we had afterward, however, is where I picked up these bonus lessons. I had heard of Cynefin before but always had two questions that never seemed to be answered in the descriptions I had read, until this week:

A figure showing the four domains of Cynefin

  1. It’s pronounced like “Kevin” but with an extra “N”
  2. The little hook or squiggle at the bottom of the Cynefin figure you see everywhere is actually meaningful: like a fold in some fabric, it indicates a change in height from the obvious/simple domain in the lower right from which you can fall into the chaotic in the lower left.

If you didn’t test it, it doesn’t work

Gary Bernhardt has a great talk online from 2012 called Boundaries, about how design patterns influence, for better and worse, the testing that can be done. In particular he advocates a “core” and “shell” model, having many independent functional cores and one orchestrating shell around them. The idea is that each functional core can be tested independently from the others more effectively by not having to worry about mocks and dependencies. The tests of each core can be greatly simplified, while focusing on what it’s actually supposed to do.

The concept is definitely solid, but I do take issue with one thing he says towards the end. I’ll acknowledge upfront that I’m drawing much more meaning from his choice of words than I think was intended, it just happened to strike a nerve. It happens at about the 29m30s mark, when he’s describing how small the tests for the functional cores became in his example program. As for the shell, the orchestrating code responsible for connecting the wires between these cores, he says:

The shell is actually untested. The only conditional in the shell is the case statement that we saw that was choosing between keys. There’s no other decision made in the entire shell. So I don’t even really feel the need to test it. I fire the program up, I hit ‘j’, as long as that worked, pretty much the whole thing works.

The obvious objection here is that just because the cores work independently doesn’t mean that the shell manipulates those cores correctly. Conditionals aren’t the only things that can have bugs in them. But that’s not my objection. It’s this:

Firing the program up and hitting ‘j’ to see if that works is testing it.

Are there other tests he could do? Sure. At a minimum, ‘j’ isn’t the only keystroke the program supports. So he’s made a risk assessment that the shell is simple enough that this is the only test he needs. It might be an informal assessment, even an unconscious one, but it’s still based on the foundation of unit tests, his knowledge of how the program works, and how it’s going to be used. There’s absolutely nothing wrong with that.

Has he automated his test? No. But is something only a test if it’s automated? Also no. Much like he made a risk assessment that he only needs one test, there’s also a judgement call (perhaps implicit) that this one test isn’t worth automating. There could be any number of reasons for making that call. That doesn’t mean the sequence of firing it up and trying it out is something outside of “testing it”.

I think there is a tendency these days to think that a test only counts if it’s automated, but we should know better than that. It’s been argued that the parts we automate shouldn’t even be called “tests”. Whether or not you agree with the terminology, it has to be acknowledged that testing encompasses a lot more than automating some checks.

I think this same sentiment is what motivates the #NoTesting hashtag on Twitter. If you have a traditional, waterfall style, gated, or heavily regimented picture of what software testing is, you’ll exclude from that umbrella all kinds of other activities that people naturally do to poke at their products. “We don’t want that stuffy bottleneck-creating deadline-breaking time-suck kind of testing around here”, they say, “so therefore we don’t want testing at all.” I bring this up not to put Gary into that camp—in the context of the talk this quibble of mine wouldn’t affect the point he was making—but to point out that people do take this idea too far.

Coincidentally, the same day I came across this talk, I also read The Honest Manual Writer Heuristic by Michael Bolton. It ends with a line that captures this idea that people do a lot more testing than they realize:

Did you notice that I’ve just described testing without using the word “testing”?

If you didn’t test it, it doesn’t work.

or, to make it perfectly clear what I’m getting at, let’s phrase it like this:

If you know it works, you must have tested it.