Fog Creek

The Abuse and Misuse of Test Automation – Interview with Alan Page

Looking for audio only? Listen on and subscribe

Up Next
How to Find, Hire, and Retain Developers

We’ve interviewed Alan Page, author of ‘The “A” Word: Under the Covers of Test Automation’. We discuss the abuse and misuse of test automation, when to automate a test, the problems of GUI test automation as well as good test design factors. Hear more about testing from Alan over on his blog and the AB Testing podcast.

Content and Timings

  • Introduction (0:00)
  • About Alan (0:28)
  • The Overuse and Abuse of Test Automation (0:53)
  • When to Automate a Test (4:06)
  • Problems with GUI Testing Automation (6:50)
  • Good Test Design Factors (9:50)
  • What Kinds of Tests Are Suited to Automation (12:37)
  • Recommended Resources (14:12)



Alan Page has more than 20 years of software and software testing experience, primarily at Microsoft. Fittingly, he was lead author of the book “How We Test Software at Microsoft.” He has recently published “The A Word: Under the Covers of Test Automation”, a collection of blog posts about the appropriate application of automation in testing. Alan, thank you so much for taking the time to join us today. Do you have a bit to share about yourself?

About Alan

I’m really excited to be here. I’ve been a big fan of Fog Creek and Joel and everything you guys do so I’m excited to be part of this interview. Thank you for having me. You mentioned over 20 years in software testing and my 20 year anniversary at Microsoft is coming up in just two weeks from Friday. Big, big milestone. I never thought I would ever work at any company, let alone Microsoft, for this long. It’s pretty remarkable.

“You should automate 100% of the tests that should be automated”

The Overuse and Abuse of Test Automation

You say that “Automation is overused and overvalued in software.” Why is this and why do you think this came about?

Probably for a lot of reasons. This is a traditional systems problem. I think some of the main reasons are people wanted to use, I think initially automation as a way to replace a lot of human testing. Where those two approaches really compliment each other. I still see a lot of manual test cases, or manual testing converted to automation. I’ve been at teams at Microsoft, where one of the things we tracked early on was to track the value of our test team was the manual to automation conversion. At the time I thought this just doesn’t seem right. But I was too junior to figure out what to say about it. Over time I figured out, you know what, these are actually two different kinds of testing.

You need sort of the human… Every team has this. Especially your audience knows that these guys are just good at walking through things and going that’s not right, that’s not right, that’s not right and automation doesn’t catch any of those things. On the other hand, automation is really good for a lot of other types of testing. We just tend to in general, we, the software industry, tend to apply automation as this blanket way to do all testing when it’s really especially suited for several specific kinds of testing.

The book entitled “The A Word” focuses on the abuse and misuse of test automation in software development. What abuses and misuses have you seen?

One of the abuses is I think people trying to automate too much or automating the wrong things and there’s a quote I use. I’m sure it’s in the book because I say it all the time is that you should automate 100% of the tests that should be automated. The challenge in test design is to figure out which of those tests are best suited to be automated versus this is really going to be really error prone.

Probably the biggest abuse I’ve seen is just going crazy trying to automate too much, and even worse being very proud that you have 10,000 or 100,000 or a million automated tests when you don’t have a lot of trust over the quality of those tests or the quality of the infrastructure they’re running on or the product. I think a lot of teams get really stuck. They get… They have test failures and I believe that test failure 100% of the time should indicate a product failure. Failure of the product on your test. We all know that the first thing we look at is the test correct and did it do… What else could have gone on here? Maybe if I run it again.

Another abuse is the test failed, I run it again, they pass. Okay, everything’s fine, which just drives me nuts because you don’t know… Something was flaky. I don’t think you even have to be anal retentive to get mad about that. I am really, really scared of flaky things because that flakiness that happens once or twice inside your walls, once you scale and deliver that to 100,000 or a million people that may happen a lot more. You want to be really careful about those. I’m just really a big fan of focusing the automation on things you can really trust and verify the automation. I think the biggest misuse, just trying to automate too much and being careless about the quality of that automation and the means of doing that.

When to Automate a Test

When should you automate a test?

When it’s necessary. My rules for automation a test, well let me draw a parallel here. I coach a soccer team of 12 year olds. One of the things I talk about, and even with 12 year olds when they’re playing defence, when we get the ball in our half, I say either clear it or make 100% pass. Meaning you clear the ball hard outside or you make the pass you absolutely is going to get to your teammate accurately where they can receive the ball. I think there’s a parallel in automation. When I… The things I want to automate is I want to be 100% confident in this test. I can write this test and I can trust the value, pass fail, 100% of the time.

There’s going to be a caveat on that for a minute I think. We’re talking about unit test, functional test, regression test, the tests I want to run every day across hundreds of platforms. That is absolutely the right answer. I think there’s a little bit of a caveat when I want to write what I’ll call exploratory automation. I’m going to go what happens if I write a monkey test? Somebody just randomly clicks on buttons, etc. etc. etc. Just as a stress test. Totally fine. I’m not going to run that everyday on every build and trying to investigate every single failure. I’m using that to flush out some of the really tricky weird things. There’s all kinds of examples of that.

For those tests that I want to run everyday across lots of devices, or lots of platforms I want 100% confidence that when that test passes functionality is correct and when it fails there’s a product bug that I need to go address. My heuristic for automation is I automate when I’m bored. So when I very first started at Microsoft 20 years ago, and I was actually a contractor for six months first so this was over 20 years ago. I was handed a spreadsheet full of test cases. I knew how to code. I said, I asked my manager, “When do you need these automated?” He said, “We don’t have time to automate, you just need to run these everyday.”

I got bored about two days in and automated them anyway because I could and I could do it really quickly. I was fairly confident in my ability and I was bored. I found other things to do to fill my time. There’s always lots of work to do. I could run these automated tests and sort of a really poor man’s regression test because these lists of test cases weren’t that good anyway. I was bored. That’s one of my biggest heuristics. I was bored running the test cases. There’s a repetitive task in … I just did Lync which is now Skype for business for awhile and if I wanted to add a thousand users to see what was going to happen, I’m going to automate that. I’m never going to do that manually. I could even, I should be pretty confident that should work everyday too. So that falls into 100% category as well.

“I think when you hear the phrase ‘it’s just test code’. To me that’s a code smell”

Problems with GUI Testing Automation

While on automating things, a quote from your book says, “For 95% of all software applications directly automating GUI is a waste of time.” Why is this?

I think it’s really hard to get right. I think that 100% rule I have where you can really trust those tests is very difficult. I think there’s a lot of promise with automation tools is to automate end to end scenarios. For me, I think looking at end to end scenarios is very much a great activity for humans to do. We want to see how the software works end to end. Where I have seen success with automation at the GUI level is very short tests because the longer you go the more weird things can happen and the more chance for flakiness you have in that test.

Another reason, and I’m sorry to say this, is often we have is this, really scary when you think about it, to me at least… We often have our most junior developers, our brand new testers who are learning to code or maybe some company that’s where you start people. You get these very junior people without a lot of design knowledge for software and a lot of experience. You tell them write this GUI automation. You have junior people doing something that’s much, much harder than most people admit and it just makes it more prone for failure.

I think when you get success is you have developers who know what they’re doing, more senior developers. It doesn’t have to be your top developer but you have experienced people who know what they’re doing. They understand design and they’re very careful about writing tests that they can trust to show product status. That’s the case that GUI automation can actually work.

There’s a friend of mine, Chris McMahon over at Wikipedia. Every time I mention GUI automation, brings up the fact that he’s had great success doing it but the difference is Chris knows what he’s doing and knows how to be careful with it. I think a lot of people trying to write GUI automation just don’t have those skills or experience to handle the complexity of how difficult that is.

What about the 5% of cases where GUI automation does make sense?

I’m sure it’s obvious to the listeners that these numbers are not based on actual data, they’re made up for me to make a point. Anecdotally, probably about right. When you’re confident that issues will… Failure is product bugs, absolutely. Regression test, very good for quick tests, not end to end. Be careful of that. I think testing those small tasks, making sure the functionality is correct so that you can do some accurate end to end sort of testing.

I think one way to think of it is to ensure the product can be tested thoroughly by humans and those humans can be internal testers or customers versus trying to test the product end to end. I think when you try to, really when you’re trying to replace or try, you can cover all of this end to end scenarios in automation you get into trouble. I think for a series of quick tasks, verifications, making sure that the functionality across the product is correct, absolutely good places to apply automation, both at the GUI and below the GUI level.

Good Test Design Factors

At the heart of a number of test automation issues is poor test design. What are some of the important factors to consider when designing an automated test?

One is just your basic design principles for code which I think as I mentioned before when you have junior people doing that, sometimes they just don’t have that experience to apply those. One of the things I did is I was on the Xbox One team before I joined this project team at Microsoft and I was really adamant about doing something that had never been done on that team before, which is I held our test code to the exact same standards as our product code. The exact same static analysis tools, the exact same rules for code review etc. We had a really high bar for our test code. When you’re trying to ship a console to a million people and you’re trying to run massive numbers of tests across a huge number of consoles, we did not want to waste our time investigating false failures.

I think when you hear the phrase ‘it’s just test code’. To me that’s a code smell. Whenever I hear someone say that I know I need to go dive in deeper and figure out what’s going on. Some design things, like I think one big one that I apply right away is that tests do one thing. Don’t have a test function called, you know, download app, install, and make sure it works or something like that. Have a function do one thing. For developers it’s pretty obvious. You’re writing a unit test, you usually have one assertion or maybe a few if it makes sense in the logic. You’re not trying to have one test method do a whole bunch of different things.

Minimize your assumptions. The typical things around magic numbers for screen size or assuming that somebody will never suspend while running your app or running your test. Just minimize how many of those are and guard for those. Then one huge thing I look for in test code is really good logging on what’s going wrong.

Another big thing I push is when there’s a test failure, I want to be able to know… Ideally I want to know what the problem is and what the fix is or what exactly what went wrong at least, from the logging. I think another social code smell for me is well let me run that again and see what happens or let me run it under the debugger. We all know how to run debuggers and we all know how to run the test again and fiddle through of what’s going on. You save a ton of time if you put that care upfront, get the right logging in place. I really push that. I think, one phrase I use here a lot is once you have to detach the bugger you lose. It just wastes too much time. I like to get that logging right the first time. Especially when we’re deploying to a lot of people and maybe we want to pull these logs from, or maybe we have a customer run a test because they have a weird thing going on. I really push accurate logging for those tests.

“In automation we just don’t leverage the power of the loop”

What Kinds of Tests Are Suited to Automation

You say that using automated tests just for regression testing is short sighted. What other types of testing is automation useful?

When I look at the literature, when I talk to teams doing automation, they’re automating those functional scenarios, end to end things that we’ve already talked about but one of the areas I think a lot of teams miss is using automation for what I called earlier exploratory automation. Writing that quick perf test. You don’t need to be a perf expert and figure out what are our deltas and deviations and all these things, but it’s as a nice perf stress test, I’ll use the Skype example I gave earlier is what happens when I add 1,000 contacts? I can just write a quick little test that there’s a snapshot of memory. Add 1,000 contacts, checks memory, okay, looks good. Really that’s maybe 15 minutes to write that code. Probably faster if you already have some frameworks in place.

It’s a very valid test, and then I can think of how about I connect to a call and disconnect. Again very short steps, but I think we err towards, again it’s a general we. We err towards trying to hit those user scenarios versus the things that automation and programming are great for are looping and do things a whole bunch of times seeing what happens. I think we lose a lot of value by not going oh yeah, this is a thing that’s really awful and impossible to do by hand but easy and fun and quick and often valuable to do using code. I found a lot of times in automation we just don’t leverage the power of the loop.

Recommended Resources

What are some useful resources for those wanting to learn more about a holistic approach
to software tests?

My favorite book, and I said this many times before on sort of the functional testing part… This is good for developers as well as testers, like kind of getting into the idea of everything from the basics of functional testing, beyond unit testing, but functional testing to even getting into model based testing a little bit is Lee Copeland’s book “Practitioners Guide to Software Test Design” I always like that for a beginning book. It’s very quick and to the point and Lee’s kind of a funny guy. There’s some humor in there. It reads really well for me. On the other end, for thinking more about the systems thinking part of what software testing is, sort of the philosophical side is Jerry Weinberg’s book “Perfect Software and Other Illusions”. It kind of gives you that other end of it.

In between, there’s another line I use, not in the book. It said, when I first started studying software testing I read one book and thought I knew everything, and then I read another book and was confused. It wasn’t until I read my third, fourth, and fifth book and started reading blog posts and articles that I began to form my own opinions. There is so much out there and you don’t learn what to throw away I think until you begin to form your own opinions.

Thank you so much for taking your time to join us.

It was a lot of fun. Thanks for having me.