Archive for the ‘General’ Category

Embedded Testers In Development Teams – Interview with Trish Khoo

December 17th, 2014 by Gareth Wilson


In this interview with Trish Khoo, Test Engineering Team Lead for Google My Business, we discuss the changing role of the tester in software development teams. She discusses the trend of embedding test specialists within development teams, benefits of the approach, common misconceptions developers have of testing and how to successfully introduce these changes into your organization.

You can read more about this and other testing-related topics on her blog.


Content and Timings

  • Introduction (0:00)
  • About Trish (0:30)
  • The Changing Role of the Software Tester (1:43)
  • Separation of Developers and Testers (6:30)
  • Maintaining Test Quality in Mixed Teams (10:48)
  • Getting Management Buy-In (12:54)
  • Ways of Incorporating Testing in Development Teams (14:27)
  • Recommended Testing Resources for Developers (15:48)




My name is Derrick, I’m joining from Fog Creek Software where for quite a while I held the role of support engineer. Now I’m helping with the sales team. So, Trish Khoo thank you very much for joining us today from Google out in sunny California where you’re the Tech Lead and Manager for the Test Engineering Team at Google.

At Google My Business yeah.

About Trish

Great. So you were previously also in testing and development roles at Campaign Monitor and Microsoft, founded the Sydney Tester Meetups, and run the TestCast podcast, you write about testing on your blog at and you’re a contributor to the Testing Planet and Testing Circus.

Yeah, so that’s all true.

All true, yeah, you’re quite heavily involved. That’s fantastic.

I’ve been moving countries a lot lately. I’ve just moved to San Francisco to join the Google headquarters. Prior to that I was working in London, also at Google, for about two years. Keep going back in time, I was in Sydney for about 7 years before that working at various places like you mentioned, Campaign Monitor, I also worked at a place called Salmat, Microsoft before that, I had a consultancy as well. So this is the journey that has lead me to be in Google working out of Mountain View. And I often go around speaking at a lot of conferences, mainly testing conferences, mainly speaking about software testing and software engineering and how the two can meet in the middle.

The Changing Role of the Software Tester

Over the last ten years have you seen the role of the tester change at Google and elsewhere?

Yeah, so I guess for me personally I’ve seen software testing roles change in my own kind of role. Like, I’ve always worked in a siloed testing role and I think as I’ve gained more experience I feel like the roles I’ve had have been more overlapping roles in terms of, not just being specialised in a testing silo so much but being able to take on more programming skills, different tasks, take on things like release work, more software development work and just seeing the value of that. In terms of how Google has seen the software tester change, definitely Google over the last five years has gone through a huge change in terms of how the testing role was defined. Only as short as five years a go most of the software testing at Google was actually done manually. And there’s been a massive shift in order to take the software testing activity to being in software engineering teams and there was a new role created called Software Engineer in Test, which meant it was exactly what it sounds like, a Software Engineer who actually knows something about testing as well. Who was creating tools and increasing developer productivity from a testing perspective, so that means making software more testable, so that it’s easier to test and being able to get that fast feedback back to developers.

Why do you think these changes came about?

I think there’s a few reasons. Like, first of all I think that if you have the separate roles of a manual tester and developer it becomes quite an inefficient process overall. Separating out the testing activity from the software engineering activity is, it creates this massive feedback loop. So say you finish implementing something in code, you send it to a tester to manually test. It takes them some time to notice it, get acquainted with what you’ve written, actually test it out and then give you that feedback. Whereas if you found a faster way to do that yourself, then that’s actually saved you and the project a lot of time. So, that’s one reason, just reducing that inefficiency in the process. I think another is that when software developers know more about testing it actually influences the way that they are actually creating software in the first place. They’re designing it for testability, and thinking in terms of what could go wrong and that has a huge impact on the robustness of the software that is created.

I think that software testing as a profession is a specialised field, especially when approached from a technology perspective, that sometimes is seen as a lesser role than the Software Engineering role. In terms of compensation, in terms of just status, and we really wanted to elevate that role to something that is better recognised and gets the recognition it deserves, in terms of being an area of expertise that actually takes a lot of time and study and practice to become proficient in. So that was another reason to just raise the bar in terms of bringing people in to that field and ensuring they’re recognized for what they do.

So you mentioned quite a few benefits to this role some of becoming more prominent and what these changes have introduced

Yeah, I think that there, not so many benefits in terms of, like, I think that there are a lot of benefits in making the role a more technical role and a more well-rounded role in terms of skills, rather than siloing, you’re a manual tester now. There are a lot of benefits in terms of recognizing that both of these roles have overlapping goals. In terms of, saying not so much ‘you’re in charge of creating the software’, ‘you’re in charge of making it actually work properly’, but saying that both of you are in charge of making that software and making it work properly, and the goal is to create and ship product. No what skills do you both have in terms of reaching that goal now. You perhaps are a better architect in terms of creating the software, you perhaps are better at coming up with the test approach that’s going to help that get to a good place, but in terms of every day tasks and what you can do, there’s a lot more leeway in terms of the scope of what each person can contribute to that team and in terms of value.

Separation of Developers and Testers

There’s a traditional way of having QA, or testers. Devs just don’t have the right mentality to effectively test their own code – ‘devs build stuff, testers break stuff.’ Do you agree with that things should be separate?

Yeah, I’ve actually heard that excuse quite a lot and I don’t really agree with it. I think that, I do hear quite a lot of excuses from developers who aren’t used to testing, in terms of why they don’t think that they need to take a more active role in it. And I think most of the time the first mistake that developers make is in thinking that testing is supposed to be easy. And then they think ‘well it doesn’t seem that easy, so it’s probably just something that I’m not supposed to do or something that is impossible to do right.’ But testing is in fact very, very hard, especially in terms of, if you haven’t done a lot of it before in your project and there’s not a lot of tests there, the code just isn’t written in a testable way. So there is actually a mountain of technical debt to be cleaned up there that needs to be recognized, before you can say ‘ok, lets start testing now.’ And another common excuse I hear is that if I start testing then it will actually take more time, so I just have to let my project manager or boss know that if I do testing then it’s going to slow down your project. But in fact, that’s also a fallacy, because yes ok, if there’s a mountain of technical debt to clean up then you are going to slow down in the short term, but in the long term, in terms of project time, the additional time you’re writing tests during code creation surprisingly makes it faster in project time, because by the end of the project you aren’t going to find as many bugs. So you’re not going to waste any time debugging, and fixing, and delaying the project right at the end, which is the most stressful time of the project.

It sounds like you’ve come across developers who haven’t wanted to test, how do you go about handling that?

I think it’s difficult if you’re in a siloed role, as I have tended to be, just about the entirety of my career in testing. It’s really difficult to go to somebody in another siloed role and say ‘hey, this is how you should do your job,’ I think that it’s best to be able to find developers on your team who already embrace that and get them to be advocates for this new approach. Ideally in has to be top-down, it has to come from management in order to really convince people of the benefits. It’s not so much about getting the stick out and saying ‘you have to write tests or else,’ but just showing them why this is useful right. Like for some people it has a been an easy sell, because they just haven’t known how to do it and you can show them ‘you think this is difficult, and it is difficult, but here’s what it is going to achieve in the end.’ I had a great quote from a developer I work with who said that he thinks that ’tests are protection of his code from other developers,’ which I thought was a great way of putting it.

That’s an interesting way of putting it for sure, yeah.

I think the other mistakes that developers often seem to make is when they think about testing, is that they think that testing is for finding bugs because that is one side effect of testing. But, testing is just as much about describing the behavior of the code and also just setting expectations, and most of all preventing bugs. So I often hear the excuse in terms of ‘I haven’t ever seen any bugs that would have been caught by the test that you are telling me to write,’ and you know, that’s true for the moment, but what’s to say that it won’t happen in the future, your projects going to go through changes. And I’ve had developers to come to a new code base and ‘well, how am I supposed to know what any of this does because there are no tests in it that describe what it is meant to do? I can figure out what the code does, but I don’t know what it’s purpose is.’ So I think that there is a big mentality shift there, and there’s a stark difference between developers who are used to testing and see the benefits and developer who aren’t used to testing don’t quite understand it.

Maintaining Test Quality in Mixed Teams

So having the roles sort of change now, where the developers also test, if you will call it a sort of ‘multi-skilling’, some people might say it’s a ‘Jack of all trades, master of none’ type scenario, where the depth and quality of testing is reduced over time. What’s your opinion on that?

There’s two parts of testing here that really need to be defined. I’ve heard some people describe it as testing and checking. There is testing, which we can describe as exploratory testing, where you are searching for answers to very broad questions. And there is checking, which is answering questions we already know to ask, so very binary things like ‘can I log in or not’, ‘if I click this button does the page explode?’ that kind of thing. But exploratory testing questions are more like ‘now that I’ve implemented this feature, does this existing feature work right?’ or ‘how do these two things work together?’ or ’what happens if I do this?’ I’ve got no idea, right. So this is a manual testing kind of task. And I think that people who are used to it, if they’ve practised it, they become more experience with it, and people find testers who are very, very good at that. And I think that it’s good to have these specialists on the team and I think it’s ok to recognize that we have generalists and specialists, in terms of people who are very good at one thing and OK at a lot of other things. So it’s almost like an old Dungeons and Drgaons game, where you’ve got your skill set and you say ‘I’m really good at this skill, I’m OK at this skill and really bad at this one.’ But I think that the thing to recognize is that testing needs to be a core competency of software engineering. It’s kind of OK to be OK at it, and have someone else on the team be spectacular at it, but the idea is that you should be at a minimum of OK at it.

Getting Management Buy-In

Earlier we sort of talked about selling testing to different people and you mentioned the importance of testing coming from the top-down. How do you get that buy-in and why should they take it seriously?

I think that it can be really difficult and of course it’s going to depend on the individuals that are within your organisation. I guess that goes without saying. But the thing that I usually focus on is time to launch and the quality of launches. So if you do post-mortems in your company, and well, even if you don’t. You certainly know the latest disasters that have happened in production. Just pointing to them, pointing to your last stressful launch and just say that this can help avoid all of that, right. It’s going to make your process more efficient and just focus on the fact that you can have more efficient launches as a result of this. And pointing to big companies like Google, like Microsoft, like other large companies as well that already follow these practices and seeing the results from them. And just pointing them as example to say ‘hey, these guys are seeing results in this way, may be we can start following them as well.’ One thing that’s really good to illustrate, are the feedback loops. And having a separate manual testing role and separate development role, and how much time that actually takes in order to get a bug fixed. As opposed to getting that feedback loop shortened, and earlier in the cycle, perhaps even getting it automated, and just show the difference there in terms of project time.

Ways of Incorporating Testing in Development Teams

What are some of the ways one can start to move towards this testing change in your organization? Could they just switch or is there more to it?

I think it is more of a cultural change than it is a process change and that’s what makes it very hard. A wise person said to me that she actually thinks that you can only really achieve this change once you have parity in your team between developers who fully embrace and want this to be the way forward and developers who aren’t there yet. So once you’ve got a 50/50 and you can tip the balance slightly over, then you can start to see change with a lot of pairing and a lot of advocacy from the side that wants this to happen. That was one person’s opinion, and another person just said that you just have to start doing it. It’s important for management to be very transparent – why is this happening, what the benefits are, and just say just try it for one project. This was done at Microsoft in that way, and there wasn’t a lot of panicking, there was some that were a little skeptical, but no-one was freaking out and when they saw the benefits, then they saw that this is a good idea.

Recommended Testing Resources for Developers

Are there any additional resources you might recommend to developers, who see this and want to do it. How would they start to learn more about testing?

About testing, I can recommend a couple of books actually for starters. There is one called ‘Driven by Tests,’ which is very much focussed on the unit testing side of things, but it’s a very good book, I’ve had that recommend to me. Then on the exploratory testing side of things, there’s a book called ‘Explore It’ by Elisabeth Hendrickson, which is aimed as an exploratory testing guide specifically written for developers. So I heartily recommend that book as well.

Trish Khoo thank you very much for joining us today.

Great, thanks very much for your time.

Taming a Wild, Testless Code Beast – 4 Steps To Improving Test Coverage

December 16th, 2014 by Andre Couturier

Whether you’re working on an existing or new application, all too often you’ll find yourself playing catch up when it comes to tests. Soon deploying code changes feels like poking at some ugly, sleeping code monster – you aren’t sure what’s going to happen, but you know it won’t be good. So you pussyfoot around the old code, hoping it won’t stir.

Virgil_Solis_-_Apollo_Python (1)

Now, this beast may have been of your own creation – you started out with the best of intentions, but development life got in the way and writing tests just never made it to the top of the To-Do list. Or it may have been unleashed on you. You got dumped on to a legacy project with barely any documentation let alone tests. Regardless of how you got there, when faced with a bunch of code and few, if any, tests, at some point you’re going to have to try and tame the wild, testless code beast.

Here are the 4 things you should do first to tame the beast and improve test coverage:

1. Add the Right Tests

Start by adding tests in the areas where it is easiest. It’s important to consider the order in which you do this, to make sure you get the most out of your scarce resources. You want to start adding tests in the following order:

  • i. Create tests as you fix bugs

    Add tests to prove that your specific fix is working and keep them running to show that this does not break again. This benefits from being somewhat targeted – you are creating tests in your weakest areas first. The weaker it is (i.e. more bugs) the faster you will build up tests.

  • ii. Create tests with all new features

    All new features will need to have tests created to prove that the feature works as expected. If you’re covering the new aspects of your application, then at least things aren’t getting worse.

  • iii. Create tests as you add to old features

    When updating old features, add tests as you go to show that the older functionality is not breaking in unexpected ways.

  • iv. Create tests in identified risk areas

    Talk to the Devs and Testers on your team and ask them to point out any weak areas or issues they have experienced. Talk to your support team too – they are an excellent resource with a direct line to the customer. They’ll know the areas where your product frequently has issues.

74 (1)

2. Turn on Code Coverage

Code coverage is a tool included in most Continuous Integration systems (or one that can be added with a plugin). These tools will instrument and monitor your code as your tests run to determine how much of your code the tests use. For this to be useful, follow these steps:

  • Start running code coverage against all your code

  • Get a baseline

    Find out what the tool can see, where you are currently at etc.

  • Determine areas that you want to exclude.

    There are likely areas of your code that you don’t want to cover – third-party tools, ancient code untouched in years etc.

  • Determine coverage goals

    Sit down with your team and discuss what your current coverage is and what your ideal can realistically be (usually 90% or above).

  • Work-out steps to improve your coverage

    You aren’t going to fix this problem overnight. Put in place some specific tasks which are going to help you achieve your goals over time.

  • Determine your pass/fail criteria

    Is staying the same OK, or should it always go up? Is any drop a fail?

  • Run Code Coverage constantly

    Use automation to run your coverage tests, use the criteria you defined as a team to report a pass/fail and do this constantly. It is a lot easier to add tests when the code is still front and center in your mind than later on.

51 (1)

3. Run your Tests on a Scheduled Basis

You should run your tests regularly, on several schedules:

  • Run them on every check-in

    Use CI tools like Jenkins to run (at least) your unit tests on every check-in. Look at making them run in parallel if they are taking too long to run at this frequency.

  • Run them on every build (package)

    Depending on how your systems work, your CI infrastructure can help you with this. This could be on every check-in if you are on Continuous Deployment, or every day/week/month that you use. This should be a clean install on a test environment and a full run of all your automated tests.

  • Run them on every deploy

    You should run all your automated tests against your environments immediately after a deploy.

  • Run them every X days/hours/minutes

    Run your automation suite as often as you can against your constant environments (Production, Staging etc). This should be daily at least and during ‘off-peak’ times when it won’t interrupt others too much. You can increase the frequency further if your tests are short, don’t add too much load to the system etc.

4. Provide a Button to Run the Tests

Again, use a tool like Jenkins to make test runs a self-serve operation. A developer should not have to be delayed by asking a QA person to run a test for them. Get a system in place where your tests will run and give them a button to press. Remove as many barriers for everyone to run the tests as possible. The easier you make it to run them, the more likely they are to be used and run when they should be.

If you follow these steps, then you’ll find that over time you’ll be able to turn an unwieldy application into something more manageable. By adding tests to the key areas first, then making things easier when you can, you can build confidence around your code changes and deploys.

A Build First Approach to JavaScript Application Design – Interview with Nico Bevacqua

December 10th, 2014 by Gareth Wilson


In this interview with Nico Bevacqua, author of ‘JavaScript Application Design: A Build First Approach,’ we discuss what is a Build First approach and how it can help you to maximize the maintainability of your applications and improve your productivity by automating common tasks.


Content and Timings

  • Introduction (0:00)
  • About Nico and Build First book (0:40)
  • What’s a Build First approach (3:48)
  • What application sizes does it work for (7:00)
  • Why front-end development has eschewed common back-end best practices (8:10)
  • Ways to think through complexity in your apps (10:08)
  • What those with existing apps can learn from Build First (12:00)
  • How to better handle modularity and dependencies (14:14)




Today we have Nico Bevacqua from He’s also the author of ‘JavaScript Application Design: A Build First Approach.’

I’m your host, my name is Derrick, I’m formerly Support Engineer for Fog Creek’s FogBugz and Kiln Developer Tool products. So Nico, thanks for joining us. Can you give us a little introduction about yourself.

About Nico and Build First book

Sure, yeah, I’ve been blogging for about two years at, like you said. It’s a blog and lately I’ve been writing about Progressive Enhancement, performance in web applications and JavaScript in general. I’ve just finished writing a book actually, like you said, it’s ‘JavaScript Application Design: A Build First Approach.’ It focuses mostly on correctly designing your applications for the front-end, because I noticed that there was kind of a need for a book to discuss those concepts in the context of JavaScript. Because, yes, we do have books that talk about quality in programming, but not really for the web, you don’t see much around that topic you see more about specific frameworks, or the language itself and its quirks, but not developing and automated build process or learning how to design an application that really works well, and is maintainable, and easy to test and so on, besides actually learning the language.

It seems like it’s becoming more and more important, because JavaScript is a much larger part of the web applications that we have available to us today. You’re also focussed on

Yeah, that’s right, I actually started the blog because I read the ‘Pragmatic Programmer’ and among the many things it says, it tells you to to get out of your comfort zone, to do something that you’ve been meaning to do but didn’t really get around to. And for me that was learning a new language. I was a C# developer before. So I started working on that and I actually wrote the blog during that time. I started blogging my experience as I learned the language. I’ve been doing a lot of Node JS open source contributions too since I started doing that.

Yeah, so I kind of wanted to touch base on the Build First book again. It sounds like the technique that you’re writing about is in the name, ‘Build First.’ Can you elaborate on the Build First approach and why it’s important?

What’s a Build First approach

The Build First approach means you’re going to be more productive. Build First really means that you care about automation. No matter how small a step is, if you’re planning on repeating it then you’re wasting your time. So Build First gives you more productivity. Not necessarily whilst you’re deploying, because may be you’re deploying like once a year or week or something. There’s not really much to gain in productivity there. But during development you’re probably, at some point if your app is complex enough, you’re going to have to have builds during development. So a Build First approach makes sure you don’t have any set backs as you’re growing the application as long as you take an incremental approach in to Build First. What I kind of get a lot is ‘well I have a full blown build process before I even write one line of code’ and that’s not really what it is about, it’s more about having a list of rudimentary tasks around your application so you’re at least about to run a linter with automation and then add more tasks as you need them. And then once you actually get to somewhere where you actually want to deploy your application, you already have in place the processes that allow you to systems integration and continuous deployment. It doesn’t really take a lot more effort to do that to add more tasks in to that build process and so on. Whereas if you weren’t doing any build stuff before that it will probably be a pain point for you when you decide ‘ok, this is a real thing, I want to move in to production’ and I want to setup all of these automated processes and you don’t really have anything like that setup. So you probably have to roll back a lot of stuff that you did during development like dynamically bundling stuff within the application running and things like that. I thing this helps design better applications too because you’re not just baking development practices in to the application itself but you’re abstracting that out of the application itself and making sure that you have different flows for different environments.

What application sizes does it work for

Does it matter on the type or size of the project when you shouldn’t follow a Build First approach?

Yeah there’s definitely a couple of places where following a Build First approach doesn’t really have a great impact. And that’s probably, a weekend hack or something like that, that’s just a bit of fun, that you know isn’t going to go very far. You’re not really going to get the benefits, because it’s not a long term thing that you’re developing. In a similar vein, if you’re developing a prototype of some type then it probably won’t make any sense either, because you’re probably going to scrap that anyway and build a real thing afterwards.

Why front-end development has eschewed common back-end best practices (8:10)

A lot of what the Build First approach covers is in many respects an attempt to apply the discipline and known best practices common in backend development to front-end development. Why do you think these practices are commonly ignored in front end development?

That’s a great question, and I’ve thought a lot about that. I think that the reason for that is we got here because of how popular the web really is. If you think about it, in most other environments or languages, you don’t really get a way with missing semi-colons and missing closing tags, and missing CSS valid properties and the web is really built for that. You know it doesn’t really matter if you make mistakes. It’s an environment where, I wouldn’t say you’re encouraged to make mistakes, but it’s ok to make mistakes in a way. The point here is that it is trying to build more quality in to the applications that you are building. And it’s kind of ironic that Node and Angular are trying to be more backend-ish. You know, they are bringing these complex architectural designs in to JavaScript. But they’re kind of going the other way and breaking the web, and letting go of progressive enhancement, which was like the building block of the web when it was first getting started. Nowadays we have to constantly remind people that progressive enhancement is something that we should be taking very seriously and we’re really not.

Ways to think through complexity in your apps

Build First seems to be thinking through the complexity of your project ahead of time, what techniques can people use to help them think through that complexity?

So I think one of the key aspects is really one that we’ve already touched down on, and that’s prototypes. And the problem is that you should definitely build a prototype properly, and validate your product ideas before you build a full blown application. But you should also ditch that prototype as it is proven and that you know that it’s going to become a serious application and that’s because you’re probably not taking a Build First approach in a prototype and you’re not considering the design impact that building that prototype will have on the final product. And once you know how the product is going to be designed you’ll have a much better idea of the kind of application design that you need to put in to it.

This kind of goes to designing an application with maintainability in mind. Making sure that the application is modular and testable and actually tested. And this is exactly the opposite of what the prototype is going for. Which is about building something as fast as possible, not really caring about maintainability because you know it’s something you’re going to ditch in the end. But it really, really becomes a problem if you don’t do that. If you go and build on top of that.

What those with existing apps can learn from Build First

Build First makes a lot of sense for new projects, but what if anything that you can think of, should those with existing projects learn and apply using this Build First approach?

Yeah, that’s another great question too. I think there’s two aspects that you can better from the most. And that’s that, firstly, you should consider testing to be of the upmost importance. That at least you should try and get the most code coverage as you can in your programs application. And if you run in to bugs write tests for them so that they don’t happen again. And the other aspect is to try and get as many continuous processes in to your application as you can. This means that if you’re doing manual deployments then you should be doing them continuously or at least programatically if you don’t want to implement continuous deployment. But at least by doing the release programatically you’re at least saving yourself from a lot of issues that stem from doing deployments. Deployments are really hard and you can lose your company around doing deployments as some people have. And really continuous all the things because during development there’s also these builds are small, or refreshing the browser, or resetting your files in the text editor, which is a waste of time if you have to do it over and over again and that’s kind of the philosophy behind Build First to continually improve the quality of what you’re building now and reduce the risk.

How to better handle modularity and dependencies

Modern JavaScript makes extensive use of libraries and pre-built components, how can people better handle the complexity of modularity and dependencies?

Yeah, first off they would have to use some sort of module system, like npm or whatever, there’s many other. But some sort of system that automates for you the handling and management of dependencies so that you don’t have to download some stuff by hand. And also in that same vein you should use something like CommonJS or request js or some other type of module that allows you to develop applications in a modular way without having to roll your own management system for packages. And one that allows you to interact with third-party modules as well. And lastly I think that you should develop small modules that do one thing well, that’s kind of the philosophy that node js stands for and think it’s the golden rule that you should follow in software in general, and with the web in particular. And we should be striving to develop small pieces of software that work really well with others rather than monoliths that try and do everything at the same time.

Sort of listening to the Build First approach has me thinking about how I would approach things differently with a little test Ember JS application that I’ve been working on. Nico Bevacqua thanks very much for joining us all the way from Argentina today, again from

Thank you for having me, I’m glad that we did this, it was a pretty nice interview, so thanks.

How to Organize a Hackathon

December 9th, 2014 by Gareth Wilson

So you want to run your own Hackathon? Great! Hackathons are a good way to meet and exchange ideas with fellow developers and creatives. They give attendees a focused way to learn and make something in just a few hours or days. They also force people out of their comfort zones, so can be a great way to get people to work on different projects or with new technologies and programming languages. However, these events take a significant amount of planning and preparation to be successful. At Fog Creek, we’ve run many such events for developers over the years. Here are our tips for organizing your own Hackathon.


Pick An Inspiring Theme

There are plenty of events for developers. Pick an interesting theme for your Hackathon to help your event stand out and increase attendance.

It could be a community event based around a specific language or tool. A corporate event for an API or product. Perhaps an internal event to encourage innovation (Facebook’s Like button was first demoed at their own internal Hackathon). Or maybe one based around some special interest topic, such as a charitable cause or hot topic. Whatever it is, it’s worth putting in the time to come up with a creative spin that sets it apart from any others.

Set Event Goals and Define Success

Having a clear idea of what you want to get out of your event will allow you to focus on what matters. You might define success in number of attendees, submissions gained, press mentions etc. But think about this upfront to simplify the planning process.

64 (1)

Work Out Who You Need to Involve

Knowing your goals will help you begin to understand the scale of your event and what you need to focus on. If it’s maximizing attendees, then you’re going to want to go big. Big often means expensive and so you might want to get sponsors involved to cover the cost. If the Hackathon has a competitive edge, then that means judges. Maximizing submissions? Then you might want to think about offering prizes. Doing it for the coverage? Then start reaching out to your media contacts early.

Hackathons also need a lot of little things taken care of, and often all at same time. Since you can only be in one place at once, you’re going to need helpers. You’ll want an MC to keep things organized and the event flowing. Then there’s people on the doors to help get attendees registered. To get the most value out of your event, you should consider recording it and taking lots of pictures. So perhaps a photographer or at least a friend/colleague or two. You know, the ones with the fancy cameras that they carry with them everywhere they go. If you’re doing demos, then you’ll also need people to help with the A/V equipment and be on hand to offer tech support. Make it clear on the day who is there to help with colored t-shirts.

Choose a Date and Time That Works

You need to pick a date and time that will work for your crowd. Make sure you leave enough time to plan it all and consider the day of the week that will maximize attendance.

For work-based Hackathons around a product or service, weekday events are OK. You’ll need to provide enough notice for attendees to get approval for time out of the office. Also stick to typical office hours like 9 to 5. For other types of event, weekends are better for longer events. Weekday evenings, straight after work, can work well for shorter events – just remember to keep the drinks flowing.

Before you settle on your date, check out event sites like Eventbrite, Meetup and Lanyrd to rule out clashes with other events. Starting 8-10 weeks in advance is usually about right and remember to at least send out a ‘save the date’ blog post or mail once you’ve picked it.

87 (1)

Find An Awesome Venue

This will probably be your biggest expense but it’s not where you want to try and save money. Location can be a key factor for attendees when deciding whether to come. It needs to be easy to get to, big enough for all the attendees and with the facilities to support them. That means enough space, WiFi and power. If your event runs overnight, then you’ll also need space for people to sleep, blankets, stuff to lie on and maybe showers.

Make it as easy as possible for people to get there. Provide comprehensive directions, maps and transport details. Don’t forget to spell out what to do once they get there too – signing in at reception, getting past security etc. Print big signs to guide people.

Get the Kit

You’ll need A/V equipment like a projector and microphone to allow for slides and demos of applications. Test this ahead of time. The first time you set this stuff up shouldn’t be on the day of the Hackathon.

It’s often easier for attendees to present on their own hardware. Yet, you need to ensure a quick turnaround between presenters. To help with this have a couple of stations setup which connect to the projector. This way, while one team is presenting the next can be setting up.

If you’re filming the event, and you should, then you need camera equipment. But unless it’s going to be a regular thing, you should hire the equipment or a photographer/videographer who already has it.

Get your swag on. Have a bunch of t-shirts available for giveaways and thankyous. Make sure whatever design you come up with that it doesn’t have a date on it. If it does, it just means that you can’t re-use any leftovers later and no-one wants to wear an old shirt.

Make sure you have plenty of spare cables, USB drives, socket adapters and extenders available. Cover all connection types – Thunderbolt, DisplayPort, VGA etc.

59 (1)

Have Killer WiFi

This is super important, but like with Hotels, WiFi sucks at a surprising number of event venues. Check this out before the day and make sure the venue knows how important this is for you. Better yet, pick a venue where the WiFi has been thoroughly battle-tested by previous dev-related events. Make sure the venue has plenty of power sockets too. You want approximately 1.5 per attendee to cover all the laptops, tablets, phones etc.

Get More Than Enough Food and Drink

Don’t skimp on the food and drink. Nothing sends people home quicker than being hungry or thirsty. Get a mixture of stuff – it can’t be all Red Bull and Oreos. So soft drinks, tea, coffee, water and juice. Arrange for breakfast, lunch and dinner. Have snacks available whenever people want them and include both healthy and junk food options. Don’t run out! Make trips to the local shops if you’re getting short during the event itself. You always want to end up with too much than risk having too little. I once ran out of tea at a conference I was running in London. Stuff Got Real.

Communicate About the Event Regularly

Blog regularly before and after the event. Hit up any press contacts and influencers that you know to spread the word on Twitter and elsewhere. Keep in regular contact with your sign-ups – once people know about it, you want to make sure it stays on their radar. To help with this, setup a dedicated mailing list. Drip out information like venue confirmation, sponsors, judges and guests. Remember to follow up after the event too as this is a good way to keep the mailing list fresh and ready for your next event.

You also want to setup a dedicated web page or site for the event, which collects all the key info together (date, location etc). For registration, don’t re-invent the wheel, just use Eventbrite, Meetup or similar.

Have Fun

Once it’s all over, remember that it was in fact a ton of fun, and start prepping for the next one…

A Guide to Open-Sourcing Your Project at Work

December 4th, 2014 by Gareth Wilson

Congratulations, you’ve written something at work that is amazing and you want to share it with the world! This guide covers three key areas that you should consider before making the leap: Why, when and how to do it.

Why Should I Open-Source My Work Project?

Open-sourcing your project at work can be a great idea. It can:22

  • Help you to build a developer-friendly brand

    From those with a developer-focussed product, like Stripe and Twilio, to those with APIs, like Facebook, Google and Square. Open-sourcing your code can be a good way to build your company’s relationship with developers.

  • Allow you to give back to the community

    Just think of all the libraries and software you use on a daily basis that make use of open-source code. Adding your own is a good way of paying it forward, so that others can benefit as you have done. We’ve open sourced a number of libraries and even whole products.

  • Help you to recruit

    Take Yahoo and LinkedIn for example. They’ve found that through their commitment to Open-Source projects (like Hadoop and Kafka), that they’ve been able to encourage developers to join them, who otherwise might not have.

  • Gain more contributors than your project ever would have in-house

    Like for example Square’s Dagger, a dependency injector for Android and Java. Having released it, many developers are contributing to it, including those at Google. In fact, Google developers now make more contributions than Square’s developers do themselves.

When Should I Open-Source My Work Project?

There are two conditions that you want to meet before open-sourcing your project. You want to make ensure that:

  • It won’t hurt your business

    It may be an impressive, complicated bit of code that would be useful to other products beyond your own. Yet if that development is your secret sauce, then giving it away would be bad for business. Likewise, if your library is an integral part of what makes your product unique or even what makes it possible, then you might want to keep it in-house.

  • Your code is helpful to others

    Consider whether any one else would actually want what you’ve created. Is it so uniquely tied to your workflow or infrastructure that it wouldn’t be usable by others? As a rule of thumb: if making it suitable for general consumption would make it less useful for yourself, then it’s probably not worth the effort.

Ok, so you’ve met those two requirements? Then let’s get onto the mechanics of open-sourcing some code.

How Do I Open-Source My Work Project?


  • Step 1: Audit your code for security leaks

    Chances are higher than you might like to admit that you or a colleague have left some passwords, user names, IP addresses, machine names, personal contact information, or other security hazard somewhere in your code. Keep in mind that this applies not only to your final code on master, but also to all the changesets you’ve had in the past.

    For that reason, we recommend you do two things:

    • 1. Make a brand-new repository

      Chop off all the history of the code up to that point. There will be new history and it saves you having to audit all the historical versions of your code. Plus, no one needs to know that it took you two weeks to wrap your head around C++11 lambda syntax.

    • 2. Audit the code for security problems

      This will take a lot less time than you think. Look especially at test suites and any places that are near connection points to other systems.

  • Step 2: Strip your code of profanity and immature pot-shots

    While you’re in there, also rip out anything inappropriate that makes you sound more like a teenager than a professional. This doesn’t mean you can’t have any humor in your source code. But it does mean that jokes made at the expense of your competitor, a customer, or the decrepit browser you’re forced to support, might not be appropriate.

    If in doubt, think about whether you’d feel comfortable reading your code aloud to those beyond your team.

  • Step 3: Make sure your code adheres to best-practice naming and formatting

    You’ll want your open-source code to be examples of your best work. Make sure you are using good, standardized naming conventions and formatting. Use tools like pyflakes/pep8, jslint, gofmt, ReSharper, and others to help.

    Also, keep in mind that if you’ve been wanting to do the One True Naming Standardization for your project, now’s a good time. Once you open-source your code, there will be a lot of inertia to avoid breaking changes. Get those done before you release. It’ll also make it easier for other contributors to get started with your code.

  • 96

  • Step 4: Document it

    You don’t have to write ninety pages of info docs, but you should at least have a nice Markdown-formatted in your root directory that explains what your software is, how to use it, and (if applicable) how to build it.

    If you’re releasing a library, you should also make sure your code has docstrings/JavaDoc/whatever so that you can generate API documentation.

  • Step 5: License your code

    You’ll want to get some proper legal advice on this. But before releasing your code you should pick a license. Unless you have a compelling reason to do otherwise, the MIT license will probably suffice. It’s short, sweet, well-understood, liberal, and makes integrating third-party changes back into your own products headache-free. But if you’re contributing code that you want to be included in a project that already has its own license, you might want to use that license instead. Here’s a useful overview of license types for more info.

    You’ll want to put a LICENSE file in your repository and have a copyright notice somewhere prominent — either in that file, or in the README. Such as ‘(C) 20XX Your Name. All rights reserved.’

  • Step 6: Name your library or tool

    Pick a name. Make sure it’s not offensive and avoids the same name as other existing libraries and trademarked products.

  • Step 7: Push your code

    Put it on GitHub, creating your own organization, repository and pushing your code.

    Do keep in mind that some communities have secondary systems that you should consider pushing to as well. If you’re writing .NET, then that might be Codeplex. If it’s Ubuntu-specific then a Bazaar mirror on Launchpad etc.

  • Step 8: Publish your package in the appropriate package archive

    If you’re publishing a library, submit it to the appropriate package manager. For .NET, that’s NuGet; for Python, it’s PyPI; for Perl, it’s CPAN; for Ruby, it’s RubyGems; for Node, it’s NPM; and so on. Also make sure that someone else at your company, such as a sysadmin, has the ability to continue maintaining the library in the unfortunate circumstance that you get hit by a bus.

  • Step 9: Announce your code

    You’re all good! Time to announce. You’ll want to blog and tweet it out. You should also consider publishing on /programming on Reddit and Hacker News etc.

And that’s it! You’re all done!


…well, nearly.

  • Step 10: Don’t forget about your code

    Just because you’ve published it doesn’t mean you’re done. You unleashed a new child unto the world; you need to take care of it. Monitor pull requests and bug reports on your new project. If you realize that keeping your project going is overwhelming, then a hearty congratulations. But remember that it is your responsibility to at least find an extra or replacement maintainer. It’s okay if your project ultimately forks, but it’s best not to do so just because you dropped the ball incorporating freely submitted improvements to your code.

That’s it. For real this time. So go out, contribute, and have fun!

Building with Best Practice at Button – Interview with Chris Maddern

December 3rd, 2014 by Gareth Wilson


In this interview with Chris Maddern, Button Co-Founder, we discuss how they have taken a Best Practice lead approach to building Button. We dive in to what best practices they have implemented, including things like Code Reviews and building as if for open source. He tells us about the impact of this approach on velocity and morale and what benefits they’ve seen.


Content and Timings

  • Introduction (0:00)
  • About Button (1:10)
  • Software Development Best Practices (4:10)
  • Code Reviews (5:50)
  • Building Like It Is Open Source (9:40)
  • Failure and Morale (11:30)
  • Moving Faster with Best Practices (12:42)
  • Benefits of a Best Practice Lead Approach (15:53)




Today we have Chris Maddern from Button, Co-Founder at Button, formerly engineering lead at Venmo, a payments startup which was acquired by Braintree in 2012. So Button’s a mobile acquisition and retention platform. Chris is going to talk to us about what the engineering team at Button have put together as Best Practices. I’m your host Derrick, formerly a Support Engineer for Fog Creek’s FogBugz and Kiln Developer Tools products. So Chris, thanks very much for joining us today.

So yeah, I’m Chris. Originally from London and moved here [New York] about four years ago. Prior to moving here I was involved in building a company called iCarCheck, which is essentially a Carfax product for the European market. So when I moved out here, I started really focussing down on mobile. Spent a year and a half with a company called Animoto, building out their mobile product. Moved to Venmo and spent a little under a year and a half there heading up mobile engineering at Venmo and then Co-founded Button about six months ago.

We’re the only way of that I know to monetize that actually makes your experience better

About Button

And what is Button?

The way that I look at Button is, we’re building a framework for apps to work together. At a really high-level I like to say, that we’re the only way of that I know to monetize that actually makes your experience better. So the idea is how do we help these apps to work together in a way that extends one apps functionality in to another app while driving users, installs and commerce in the second app from the first app.

So like for example, if I make a reservation in Resy, it’s intuitive that I need to book an Uber to get to my reservation. So we provide Resy with the ability to directly productise offering an Uber ride inside of Resy and then when they get driven over to Uber obviously there’s economics in driving that ride.

That’s very cool there, you know it sounds like it’s offering an extra level of service. And you said that the other product that it’s integrating with is called Resy?

Yeh, so Resy is one example, Resy is a great little app that lets you pay to reserve tables that otherwise you and I would find it kind of difficult to get tables at. So like a table at Rosemary’s, which doesn’t really take reservations. A table at , which is typically pretty packed. And so the idea is that we help them to give a value-add piece of functionality which encourages you to use them again, which also gives them a monetisation to use. And we’ve been super fortunate to be able to work with Uber so early on, and they’ve been very very supportive and helpful with us, in trying to bring these amazing productisations to market.

Yeh, that Uber integration is pretty new right for you?

Yeh, so we launched this, this is kind of our second product to market, we launched it around four or five weeks ago, so we’re just getting to see the first data coming in to that.

The integration, which is sort of just putting all of the pieces together, there are not too many mysteries there, right?

Yeh, so the kind of generic technology that we’ve been trying to build is how do you move a user from one application to another application in a totally attributed way and be able to understand and build our around that. And then in the future, potentially tie that back in to our original loyalty card, so say when you move from one app to another app you get some kind of points incentive, or you get some other incentive around that. So yeh, in terms of like what was actually new. We built an SDK around Google’s API, we built really probably one of the things that I’m proudest of when I say we’ve built it, is the ride picker that we drop in to our partners’ apps, so a really beautiful way of choosing the Uber ride that you want to take and then sitting basically on top of the rails that we’ve built for everything else.

Software Development Best Practices at Button

Referring to your engineering blog post, there was a blog post from October about some best practices. What kind of best practices have you adopted at Button?

Yeh, so I was super excited that you guys reached out about this. I think that one of the best things that you can do when you when you try and adopt best practices is actually to just try and define what means to you. And to write it down, maybe share it publicly, maybe share it privately, or just like set it as your desktop wallpaper. But really commit to what that means to you. And so that blog post was originally an internal GitHub Wiki that I’d written basically for myself as the only person writing code at Button at the time. And then it translated to the blog post that you see today. So in terms of the best practices that we have right now, we try and start with really lofty goals that I’ve outlined in that blog post. Like 100% code test coverage, everything being code reviewed. Kind of stuff that that you hear that’s very standard, but being dogmatic about it is rare. And starting with the goal of being extremely obsessive with it, with the acceptance that, you know, you’ll only get 80-85% there because sometimes you just need to ship something. You know honestly, it’s terrible to say this, but sometimes you just can’t be bothered, like sometimes you just can’t be bothered to spend that extra five hours to get that extra 10% code coverage or something. But you set these goals and if you make it 80% of the way there then you feel like you’re going to be better off. So one of the most significant of the ones we follow absolutely religiously is code review. So we code review everything, no matter how small the change is. Just to get two sets of eyes on it. I would ask pretty much anyone to code review my code who has any context on coding at all. Because it’s amazing what new people will see that you just don’t. Even if they don’t have much context on what it is that you’re trying to do. Some things are just obvious, but you just don’t see them.

I would say somewhere between 30% and 40% of new code that was being written at Venmo was actually being contributed to open-source

Code Reviews

Yeh, that’s one of the great benefits of code reviews. So it sounds like everything gets code reviewed for you there, you know, maybe not tagging specific changesets. Would you say something like that gets code reviewed? Probably not

So you mean actually like creating the tag. So for that stuff we try and follow a pretty strict branching process. So anything that’s being marked as ready for a particular release will go down to a feature-release branch for that feature and then that branch will be end-of-lifed with the last commit that increments the version number and custom tag created. So we try and create some process around that because it’s kind of amazing how many times you follow a really robust process to 99% of the way there, and then a tiny little thing at the last minute. Like, how we do code signing, how you implement a version number, causes an issue that’s very real. So we try and go that whole mile.

It sounds like you just do 1 on 1 code reviews, in the sense that there’s one other person looking at it. Do you ever find yourself in the situation where you’re using more than one person?

Yeh, so this is a new and unique problem to us now, now that we’re actually growing. So originally there were two of us. And so if we were reviewing code, it would be the other person. We’re now at 4 and looking to grow more in the coming months too. And so we try and create sideways visibility wherever we can. So have some concept of assigned reviewers, but by default everything communicates via groups that go in to all of our engineering teams, so people know what pull requests are open, a room for new requests to people can review things. And we try and encourage just an awareness of what’s going on and the code culture in each of the codebases. In terms of people who aren’t actually working on that codebase actually contributing to code reviews is really low. And I think that’s simply because everyone is so heads-down. I would hope to see, and certainly whilst I was at Venmo, we would massively encourage pull requests which would have half a dozen people commenting on. And what’s interesting is, your pull requests then become one of your biggest signals in to your style guides and in to your conventions. Because you’ll start discussing things and you’ll realise that there is no codified way of doing that. And it’s something clearly people feel strongly about to pull down the pull request, so you should start to think about how you codify that in to your style guides.

It’s really nice to hear that you value code reviews, it’s something that a lot of people, you know, aren’t sure what to do yet about.

The thing is just to start. Have someone look at your code, and then as soon as you take off the hat that says ‘I want to get this committed as soon as possible’ and put on the hat that says ‘I want to protect the codebase’, which is the hat that you should wearing when code reviewing, then you just start seeing benefits. Even if it’s just you who can take off the hat and put on the other hat. I’ll routinely review my own pull requests before anyone else, to make sure there’s nothing that I’m going to be embarrassed about.

I would rather have a 70% completion rate from sprints with zero hot fixes, than 100% completion with even one hot fix

Building Like It Is Open Source

It sounds like your experience in the past has helped you to move really quickly here at Button. Is there something sort of specific from your experience at Venmo that has helped you here?

Yeh, so at Venmo we built with open-source in mind with pretty much anything that we would do. It’s something of an off-the-cuff estimate, but I would say somewhere between 30% and 40% of new code that was being written at Venmo was actually being contributed to open-source. Whether that be in one of our container libraries, one of our core SDKs. We tried to very much run a kind of eat your own Dogfood principle with our SDKs. So the Venmo app is built on the Venmo SDK that anyone can build on top of. So when you start thinking about open source, it guides many of the principles that you use. Because honestly then you have broad visibility of your code and your code quality, open standards, documentation inside of code are kind of a must have, rather than a nice to have. And then, the other thing, and this sounds kind of hyperbolic, but just an absolute focus on quality. You just can’t ship anything that’s not ridiculous high quality because there are always quality problems that you’re not aware of, so you need to find all of the ones that you can and fix them.

Failure and Morale

Yeh, as you’re pretty new in this space, quality is important to your reputation.

Yeh, and it becomes a lot more apparent when you have problems, like I remember looking in to Crashlytics at Venmo and you need a very, very small percentage of sessions that crash, before that number becomes very depressing. And you have to stop everything and you have to get in to hot fix mode.

Yeh, right and we all know the sort of snowball effect hot fix mode has.

Like not just on software quality and velocity, also just on morale. It sucks to be hot fixing stuff.

Yeh, because you know, then you have monitoring 24/7, your cellphones going off and you’re not getting family time or whatever. Yeh, so you’re right, it’s not just code but it’s team morale which is important too.

Yeh. One of the things that I think commonly gets mistaken as failure inside organisations that run Agile, is moving things off at the end of a sprint. I am of course tending to agree with that, but what I do say is when you have to hot fix that’s failure. I would rather have a 70% completion rate from sprints with zero hot fixes, than 100% completion with even one hot fix.

Moving Faster with Best Practices

Starting with best practices early on, did that sort of restrict how fast you can move at all?

Yeh, so there’s an overhead. So for the first several weeks of trying to build Button, I didn’t really build anything. The truth is that once you’ve done that a few times, once you’ve set up CI, and you’ve setup the tools of the trade, that stuff isn’t so expensive. You should definitely be doing it inside of new projects. The truth it, for every piece of overhead, or for every piece of friction that exists along the way, that makes you a little bit slower as you’re doing things, it more than gets made up for at the end. So, I mention this in the post, but this is something that I’ve noticed in software projects where it’s true that often the last 20% really does take 80% of the time. And I’ve found that best practice can reduce that 80% to something more like 40% or 50%. Because when you’re not building with best practices everything you’ve moved across the Kanban board, I’ve found, is never truly done. It’s either not quite to spec, not quite right. Or, there’s a quality issue – it’s not working 100% correctly, or there’s you know, something you’ve missed in code coverage. The savings at the end allowed us to get our produce to market faster than I think if we had just started coding, never done a test and never reviewed any pull requests, because it does pay off a lot.

Building with best practices and emphasising the time to do it, emphasising that the last 20% of the task is really 80% of the work. Did you find that you sort of had to sell that to the rest of the business?

There are always moments, where it does seem to other people that you could just get on with it and stop worrying about this stuff. I got really fortunate, in that both of my co-founders, or all of my co-founders, kind of get it and are genuinely respectful of deferring to us on matters of engineering. So I think we have a really healthy balance between engineering and business. I’m really fortunate to have partnered with a couple of big business heavyweights and fortunately we haven’t become this big business run company where we build stuff to spec and don’t care about engineering. Our platform is our product and our SDKs are our product, and we invest very heavily in things that we want to maintain for years to come.

Right, and that’s sort of, the reason that you sort of have that perspective is to not only be around as a business but because you also have that open source mentality. You know, we’re building for the open source in a sense.

Yeh and while we haven’t gotten the chance to do a lot of that yet, the way that we’ve designed a lot of what we have built, allow us to be able to very soon start to open source quite a number of projects.

Benefits of a Best Practice Lead Approach

6 to 7 months in now for button, it sounds like treating things as open source, kind of putting in those preferred best practices is really working for you.

Yeh, we’ve had lots of benefits. I’d say two keys. So there’s a difference between building a product and taking a product to market. And so once you’ve built the product you have to take it to market. And so over the last couple of months we’ve been talking to a lot of partners, creating a lot of integrations and I think that that way that we built really help us. Firstly, when you step back and think about things first, your design is much more modular and intuitive. And secondly, it’s fully documented. So creating our integration guides for the documentation that we needed to be able to share with the developers in order to do integrations was significantly easier because we have a lot of that stuff built already. The other kind of main way, is that with any software project you change along the way. You know, you switch focus a bit, new shiny things come in at you, old shiny things become less shiny. And along the way we’ve made several course corrections, in what we really want to build and what we think is really important. And it turns out that we’ve built a very modular framework, so the core of what we created never stopped being useful. And I think that when you think about building open source products, think about the layers of maintainability, that this approach lends itself to doing that. And so I genuinely believe that if we hadn’t gone down this path, then some of the shifts in, like what we were thinking is important or what we think is the highest priority would have basically meant we would have had to start again. If it weren’t for the fact that the way that we approached it gave us this very modular, layer-based approach meant that we could just think about changing the interface tools versus, we need to re-do the whole thing. So yeah, like massively beneficial. Which is why I chose to share the document publicly, because originally when you start and you’re like, I kind of have an idea, you think, if I live really anally by these things then my life is probably going to be better. And in the end I was like ‘oh wow’, it really kind of was better.

Thank you, for your time today, I really appreciate it!

Awesome, thanks guys, it has been a pleasure.

Likewise Chris, take care.


Looking for ways to review your code? Try Kiln Code Reviews.

9 Integration Testing Do’s and Don’ts

December 1st, 2014 by Andre Couturier

integration tests check systems work together
Integration tests check that your application works and presents properly to a customer. They seek to verify your performance, reliability and of course, functional requirements. Integration tests should be able to run against any of your developer, staging and production environments at any time.

Writing good tests that prove your solution works can be a challenge. Ensuring that these tests perform the intended actions and prove the required outcomes requires careful thought. You should consider what you are testing, and how to prove it works – both now and in the future. To help you create tests that work and are maintainable, here are 9 Do’s and 9 Don’ts to consider:

When Creating Integration Tests Do…

test with customers

1. Consider the cost/benefit of each test

Should this be a unit test? How much time will it save to write this test over a manual test? Is it run often? If a test takes 30 seconds to run manually every few weeks, taking 12 hours to automate it may not be the best use of resources.

2. Use intention revealing test names

You should be able to work out what a test is doing from the name, or at least give you a good idea.

3. Use your public API as much as possible

Otherwise it’s just more endpoints and calls to maintain when application changes are made.

4. Create a new API when one isn’t available

Rather than relying on one of the Don’ts

5. Use the same UI as your customers

Or you might miss visual issues that your customers won’t.

6. Use command line parameters for values that will change when tests are re-run

Some examples include items like site name, user name, password etc.

7. Test using all the same steps your customers will perform

The closer your tests are to the real thing, the more valuable they’ll become.

8. Return your system under test to the original state

Or at least as close to it as you can. If you create a lot of things, try to delete them all.

9. Listen to your customers and support team

They will find ways to use your systems that you will never expect. Use this to your advantage in creating real world tests.

When Creating Integration Tests Don’t…

integration testing

1. Write an integration test when a unit test will suffice

It’ll be extra effort for no benefit.

2. Use anything that a customer cannot use

Databases, web servers, system configurations are all off limits. If your customer can’t touch it, your tests have no business touching it either.

3. Access any part of the system directly

Shortcuts like this just reduce the quality of your tests.

4. Use constants in the body of your tests

If you must use constants, put them in a block at the top of your test file, or a configuration file. There is nothing worse than having to search through all your source files because you changed a price from $199.95 to $199.99.

5. Create an internal only API

Unless necessary for security or administration.

6. Create an internal only UI

You’re supposed to be testing what the customer will see after all.

7. Make your test too complex

No matter how brilliant your test is, keep it simple. Complexity just breaks later. If you are finding it hard to write, it will be hard to maintain too.

8. Test more than one thing

Stick to what you need to test, if you try to do too much in one test it will just get more complex, and more fragile.

9. Leave the test system in a bad/unknown state

This means a broken or unusable site, database or UI.


To learn more about Testing and QA at Fog Creek, watch the following video:

How Fog Creek Got Started

November 20th, 2014 by Gareth Wilson

Starting out as a consulting company in 2000, Fog Creek was founded with the goal of creating the best place for developers to work. The video covers the early years of Fog Creek. Hear from our founders, Joel Spolsky and Michael Pryor, how they navigated the Dot-com crash and bootstrapped the company in to a growing, product-based business.

Effective Code Reviews – 9 Tips from a Converted Skeptic

November 17th, 2014 by Gareth Wilson

I knew the theory. Code reviews help to:

  • Catch bugs
  • Ensure code is readable and maintainable
  • Spread knowledge of the code base throughout the team
  • Get new people up to speed with the ways of working
  • Expose everyone to different approaches

Or, they’re just a giant waste of time. At least, that was my first impression of code reviews.

I was the new guy, a recent grad, developing plugins for a software company in London.

Over time I had to submit blocks of identical or similar code. They would get reviewed by the same poor, put upon guy (“he’s the best at it” my manager told me. No good deed…). Yet each review would come back picking at something different. It seemed needlessly picky and arbitrary.

Worse still, reviews would take days, if not weeks. By the time I got my code back I could hardly remember writing it. It wasn’t the guy’s fault. He’d asked for a senior dev, but had gotten me. He was sick of dealing with the issues every inexperienced developer makes, and code reviews were his way of exorcising that frustration.

Add to this the time lost in syncing the different branches, the context-switching… I was not a fan, nor were the rest of the team and it showed.

Skip forward a few years though and I find myself nodding along whilst reading a tweet quoting Jeff Atwood:

“Peer code reviews are the single biggest thing you can do to improve your code.”

What I had come to appreciate in the intervening years is that it wasn’t that code reviews were bad. Code reviews done badly were. And boy, had we been doing them badly.

I had learned this the hard way. And it certainly didn’t happen over night. Although on reflection, code reviews have saved me from more than a few embarrassing, build-breaking code changes! But after I had worked elsewhere, I gained experience of different and better ways of working. This gave me opportunity to see first-hand the benefits of code reviews that I had dismissed before. So now I consider myself a converted skeptic.

So that you can avoid such pains, check out our video and then read on for tips that will skip you straight to effective code reviews.

9 Code Review Tips

For everyone:

  • Review the right things, let tools to do the rest

    You don’t need to argue over code style and formatting issues. There are plenty of tools which can consistently highlight those things. Ensuring that the code is correct, understandable and maintainable is what’s important. Sure, style and formatting form part of that, but you should let the tool be the one to point out those things.

  • Everyone should code review

    Some people are better at it than others. The more experienced may well spot more bugs, and that’s important. But more important is maintaining a positive attitude to code review in general and that means avoiding any ‘Us vs. Them’ attitude, or making reviewing code burdensome for someone.

  • Review all code

    No code is too short or too simple. If you review everything then nothing gets missed. What’s more, it makes it part of the process, a habit and not an after thought.

  • Adopt a positive attitude

    This is just as important for reviewers as well as submitters. Code reviews are not the time to get all alpha and exert your coding prowess. Nor do you need to get defensive. Go in to it with a positive attitude of constructive criticism and you can build trust around the process.

For reviewers:


  • Code review often and for short sessions

    The effectiveness of your reviews decreases after around an hour. So putting off reviews and doing them in one almighty session doesn’t help anybody. Set aside time throughout your day to coincide with breaks, so as not to disrupt your own flow and help form a habit. Your colleagues will thank you for it. Waiting can be frustrating and they can resolve issues quicker whilst the code is still fresh in their heads.

  • It’s OK to say “It’s all good”

    Don’t get picky, you don’t have to find an issue in every review.

  • Use a checklist

    Checklists ensure consistency – they make sure everyone is covering what’s important and common mistakes.

For submitters:

  • Keep the code short

    Beyond 200 lines and the effectiveness of a review drops significantly. By the time you’re at more than 400 they become almost pointless.

  • Provide context

    Link to any related tickets, or the spec. There are code review tools like Kiln that can help with that. Provide short, but useful commit messages and plenty of comments throughout your code. It’ll help the reviewer and you’ll get fewer issues coming back.


Register Now for ‘Code Reviews in Kiln’ Webinar

Join us for our next live ‘Code Reviews in Kiln’ webinar. This webinar will help first time or novice users learn the basics of Code Reviews in Kiln.

We’ll cover:

  • What are Code Reviews
  • Why use Code Reviews
  • When use Code Reviews
  • What to look for during Code Reviews
  • Creating a Code Review
  • Commenting and Replying on a Code Review
  • Working with Existing Code Reviews
  • Code Review Workflow

Secure your spot, register now.

Scaling Customer Service by Fixing Things Twice

November 10th, 2014 by Gareth Wilson

As a bootstrapped company we’ve always had to work within a budget and avoid unnecessary costs without harming our mission. One area that has the potential for suffering in the face of limited budgets is Customer Service. When a company is growing, customer service can suffer or it can grow to consume a considerable amount of your budget. We’ve been able to grow our customer base by more than 10 times, maintaining a high level of customer service while keeping our support costs manageable.

How? By Fixing Things Twice using the 5 Whys.

Check out the video and read more about how these techniques can help you scale customer service without the cost.

Fixing Things Twice

When a customer has a problem, don’t simply resolve their issue and move on – but rather take advantage of the issue to resolve its underlying cause. This is Fixing Things Twice.

We think that for each customer issue, we have to do two things:
1. Solve the customer’s problem right away
2. Find a way to stop that problem from happening again

How we solve the first depends on the specific problem at hand, but to resolve the second we use the 5 Whys method.

Resolving Root Causes with 5 Whys

The 5 Whys is a problem-solving method and form of root-cause analysis. It involves recursively asking the question ‘why?’ five times when faced with an issue. Doing so enables you to get to the bottom of it, allowing you to fix and stop it from happening again.

Just ask why

This technique was recently popularized by Eric Ries in his book ‘The Lean Startup’. Yet it was developed by Sakichi Toyoda in the late eighties at Toyota. Over the years its use has spread beyond the motor industry to software development and other areas such as finance and medicine. It’s a recommended technique used by the UK’s National Health Service for example.

Lets take a look at how it works using a hypothetical situation:

  • The initial problem – The machine won’t start
  • 1st Why? – There’s no power
  • 2nd Why? – The power supply is not working
  • 3rd Why? – The capacitor has broken
  • 4th Why? – The capacitor was old but had not been replaced
  • 5th Why? – There’s no schedule to check for ageing parts in the power supply units. This is the root cause.

This technique is especially useful when you’re able to focus on processes as causes, like in the example above. Whilst other factors like time, money and resources might play their part, they’re beyond our immediate control. 5 Whys quickly exposes the relationship between the various causes of a problem – yet it does not require any analysis tools. This means it can be adopted and applied throughout an organization. We’ve extended its use beyond Support to include System Administration too. Watch the video above to see an example.

What does Fixing Things Twice mean for Support?

Here’s a couple of examples of using Fixing Things Twice provided by Adam Wishneusky, Technical Support Engineer here at Fog Creek:

When we had people asking a lot about our security practices. We fixed the problem once by giving the customer an answer. Then fixed it twice by putting up public docs at

Another example from a few years ago, is when we found that FogBugz wouldn’t let you create a new LDAP-authenticated user if one already existed in the database with the same LDAP UID even though that user is deleted. We showed customers how to manually fix the data to get them working, but we also pushed the dev team to fix the bug.

support and development
From this example you can see that Support must have access to the Development team. It’s often the only way the underlying issues will get fixed.

It takes commitment too – it’s easy to skip the second fix and it’s tempting to do so as it means spending more time on any one issue. But it’s a false economy to do so. When the issue crops up again and again, you’ll have to spend even more time on it.

If you stick to Fixing Things Twice then over time all the common problems get resolved. This frees up your Customer Service team to look in to the unique issues that need more time. Resolving your most frequent issues overcomes the support overhead that typically comes with adding new customers.

Looking for more?

Visit the Archives or subscribe via RSS.