Archive for the ‘General’ Category – Interview with Salvatore Sanfilippo

March 2nd, 2015 by Gareth Wilson

In, we chat with developers about their passion for programming: how they got into it, what they like to work on and how.

Today’s guest is Salvatore Sanfilippo, an Open-Source Software Developer at Pivotal, better known as the creator of Redis, the popular data structure server. He writes regularly about development on his blog.

Location: Catania, Sicily, Italy
Current Role: Developer at Pivotal

How did you get into software development?

My father was a computer enthusiast when I was a small child, so I started to write simple BASIC programs emulating him. My school had programming courses and it was great to experiment a bit with LOGO, but I definitely learned at home. The first computer I used to write small programs was a TI 99/4A, however, I wrote less-trivial programs only once I got a ZX Spectrum later. I got a bit better when I had the first MS-DOS computer, an Olivetti PC1, equipped with an 8086-clone called Nec V30.

I remember that in my little town there were free courses about programming in the public library. One day I went there to follow a lesson and learned about binary search. It was a great moment for me – the first non-trivial algorithm I learned.


Tell us a little about your current role

Pivotal is very kindly sponsoring me to write open-source software, specifically to work on Redis. So my typical day involves getting up at 7:15 am, taking my son and/or daughter to school, and then sitting in front of a computer from 8:00 am.

What I do then depends on what is urgent: handling the community, replying to messages, checking issues or pull requests on Github, fixing serious problems, or from time to time, designing and implementing new features.

I don’t have a fixed schedule. Since there aren’t many developers contributing to Redis, I’ve found that I can be most effective for the userbase if I’m flexible. Moreover, when I feel tired or annoyed, I use the trick of working on something about Redis that is exciting for me in a given moment. So I likely end the day with something good, even if it was a “bad day”, like when you slept too little for some reason and need a bit more motivation.

Currently I’m working on Redis 3.0.0, that is in release candidate stage right now. This release brings clustering to Redis, and it was definitely challenging. It’s non-trivial to find a set of tradeoffs in order to make a Redis Cluster that looks like Redis single-node, from the point of view of the use-cases you can model with it. In the end, I tried to overcome the limitations by sacrificing certain other aspects of the system, in favor of what looked like the fundamental design priorities.

I’m also working on a new open-source project currently – a distributed message queue. It is the first program I’ve written directly as a distributed system. I’m writing it because I see that many people use Redis as a message broker for delayed tasks, even if it was not designed for this specifically and there are other systems focusing on this task. So I asked myself ‘if Redis is not specifically designed for this, what makes people happy to use it compared to other systems?’ Maybe I can extract what is good about Redis and create a new system that is Redis-like but specifically designed for the task. So Disque was born (code not yet released currently).

When are you at your happiest whilst coding?

My preferred moment is when I write a lot of code, maybe without even trying to compile at all for days. Adding new abstractions, structures, functions, with an idea about how to combine all this to create something. Initially everything is a bit of a blur and you don’t really know if it’s going to work. But, eventually every missing piece starts to be filled with code, and the system makes sense, compiles, and suddenly you see it working on your computer. Something magical happened, something new was created where there was nothing a few days ago. I believe this is an aspect to love about programming.


What is your dev environment?

I use a MacBook Air 11″ and a desktop Linux system that I use mostly via SSH. I have an external monitor and keyboard that I attach to the MBA when I’m at home. I work at home in a room that is half office and half gym. I code sitting, doing small “sprints”, of 45 minutes to a max of 1.5 hours. Then I get up and either grab a coffee, exchange a few words with my Wife, go to collect one of my children or alike.

I don’t like music when I code, as I love the kind of music that needs pretty active listening, like Square Pusher, Venetian Snares, Classical music, and so forth. I don’t stop too late, maybe 6pm, but if I did not reach 8 hours of work, then I work again when my family goes to bed.

I use OS X on the MBA, with iTerm2 and the usual command line tools: Clang and Vim. On the Linux system, I use Ubuntu with GCC, Vim, and Valgrind. Vim and Valgrind are two fundamental tools I use every day.

From the point of view of desktop software, I use Google Chrome, Firefox, and Evernote to take a number of notes while I write code – kind of like a TODO list to avoid losing focus if I think about something the code needs.

I use Twitter a lot for work with the only Twitter client I actually like, and I wish it was more actively developed: YoruFukurou. For IRC, I use Limechat. For messaging, Telegram.

What are your favorite books about development?

A long time ago I enjoyed Structure and Interpretation of Computer Programs (SICP), but I never read it again. Books on algorithms were the ones that turned me from a Spaghetti-coder into something a bit better, probably. I’m not a fan of Best Practices, so I don’t like a lot of books like Design Patterns or The Pragmatic Programmer. Recently I re-read The Mythical Man-Month (TMMM) and The Design of Design, however, I find the latter less useful. I would love to read more actually detailed accounts of software project experiences and TMMM is more like that.


What technologies are you currently trying out?

I want an excuse to try Rust and Swift in some non-trivial software project. I don’t see Go as something that is going to serve as a “better C”, so I’m waiting for something new.

I’m not a big fan of new things in IT when they have the smell of being mostly fashion-driven, without actually bringing something new to the table. So I tend to not use the new, cool stuff unless I see some obvious value.

When not coding, what do you like to do?

Spending time with family, being a Father, is something I do and enjoy. I also try to go outside with my Wife without the children when possible, to have some more relaxed, fun together. I enjoy doing power-lifting and I also run from time to time. I like theatre – I used to watch many pieces, and recently I got interested in it again and I’m attending an acting laboratory.

What advice would you give to a younger version of yourself starting out in development?

Don’t fragment your forces into too many pieces. Focus on the intersection of things that at the same time you find valuable, and many people find valuable. Since niches are a bit of a trap.


Thanks to Salvatore for taking the time to speak with us. Have someone you’d like to be a guest? Let us know @FogCreek.


Previous Interviews

Jon Skeet
Bob Nystrom
Brian Bondy
Jared Parsons

A Developer’s Guide to Growth Hacking – Tech Talk

February 27th, 2015 by Gareth Wilson


Given the media hype that surrounds the term ‘Growth Hacking’, you can be forgiven for dismissing the whole thing as another marketing buzzword. But what can get lost in the hubbub are some useful, development-inspired, working practices that can help a team focus on maximizing growth.

In this Tech Talk, Rob Sobers, Director of Inbound Marketing at Varonis, tells you all you need to know about Growth Hacking. Rob explains what Growth Hacking is and describes the processes key for it to be effective – from setting goals, to working through an experimentation cycle and how it works in practice.

Rob was formerly a Support Engineer here at Fog Creek, and is the creator of his own product, Munchkin Report. He writes on his blog about bootstrapping and startup marketing.


About Fog Creek Tech Talks

At Fog Creek, we have weekly Tech Talks from our own staff and invited guests. These are short, informal presentations on something of interest to those involved in software development. We try to share these with you whenever we can.


Content and Timings

  • What is Growth Hacking (0:00)
  • People (2:34)
  • Process (3:22)
  • Setting Goals (5:25)
  • Experimentation Cycle (6:12)
  • How It Works In Practice (12:03)



What is Growth Hacking

I was a developer, started out my career as a developer, kind of moved into the design space and then did customer support here, and then now I’m doing marketing. I’ve been doing marketing for the past, I don’t know, two and a half three years almost. This sort of like, phrase, growth hacker kind of cropped up. I kind of let the phrase pass me by. I just didn’t discuss it. I didn’t call myself a growth hacker. I stayed completely out of it, mainly because of stuff like this.

It’s just overwhelming. Like Google ‘growth hacking’, you’ll want to throw up. What it really comes down to is that growth hacking is not at all about tactics. It’s not about tricks. It’s not about fooling your customers into buying your software or finding some secret lever to pull that’s hidden that’s going to unlock massive growth for your company. It’s really about science. It’s about the process. It’s about discipline. It’s about experimentation. Tactics are inputs to a greater system.

If someone came up to you, you’re a Starcraft player and said, “What tactic should I use?” You would have a million questions, “Well what race do you play? Who are you playing against? Who’s your opponent? What does he like to do? What race is he playing? Is it two vs. two or is it three vs. three?” There’s so many different questions. So if someone comes up to me and says, “What tactics? What marketing channels should I use for my business?” You can’t answer it. The answer is not in the tactics.

So this is what Sean Ellis, this is how he defines growth hacking. He says, “Growth hacking is experiment driven marketing.” You walk into most marketing departments, and they’ve done a budget, and they sit in a room, and they decide how to divvy up that money across different channels. “Okay, we’ll buy some display ads. We’ll do some Google Ad Words. We’ll invest in analyst relations,” but they’re doing it blind. Year after year, they’re not looking at the results, looking at the data, and they’re not running experiments. So it’s really kind of blind. So this is really the difference.

I took it one step further. I said growth hacking is experiment-driven marketing executed by people who don’t need permission or help to get things done, because I think growth hacking’s a lot about the process. And it’s about culture, and embracing the idea of doing a whole bunch of marketing experiments week over week. But if you have a team that is only idea-driven, and tactic driven, and then they have to farm out all of the production to multiple other stakeholders in the business like teams of Devs or Designers, then you’re not able to iterate. So to simplify it I just said, “Growth hacking equals people, people who have the requisite skills to get things done from start to finish, and process.”


So let’s talk about people. You don’t just wake up in the morning and just say like, “Let’s do some marketing.” You have to know what your goals are and then break it down into little pieces, and then attack based on that. So this is a system, that was devised by Brian Balfour at HubSpot. I call it the Balfour method. A good way to measure a person, when you’re hiring to be a growth hacker and run growth experiments, is to show them this chart and say, “Well how far around the wheel can you get before you need to put something on somebody else’s to-do list?” Now granted you’re not going to always be able to hire people that can do everything. I’ve seen it work where people can do bits and pieces, but it sure is nice to have people who can do design and development on a growth team.


So before you begin implementing a process at your company, what you want to do is establish a method for testing. And then you need analytics and reporting. I’ve seen a lot of companies really miss the boat with their analytics. They’ve got it too fragmented across multiple systems. The analytics for their website is too far detached from the analytics that’s within their products. Because you don’t want to stop at the front-facing marketing site. It’s great to run A/B tests, and experiment on your home page, and try and get more people to click through to your product page and your sign-up page, but then also there are these deep product levers that you can experiment with your onboarding process, and your activation and your referral flow.

So what you’re really looking for, and the reason why you establish a system and a method, is number one to establish a rhythm. So at my company we were in a funk where we were just running an A/B test every now and then when we had spare time. It’s really one of the most high-value things we could be doing, yet we were neglecting to do it. We were working on other projects. The biggest thing we did was we had implemented this process which it forces us to meet every Monday morning and discuss, layout our experiments, really define what our goals are, and establish that rhythm.

Number two is learning, and that basically is that all the results of your experiments should be cataloged so that you can feed them back into the loop. So if you learned a certain thing about putting maybe a customer testimonial on a sign-up page, and it increases your conversion by 2%, maybe you take a testimonial and put it somewhere else where where it might have the same sort of impact. So you take those learnings and you reincorporate them, or you double down.

Autonomy, that goes back to teams. You really want your growth team to be able to autonomously make changes and run their experiments without a lot of overhead. And then accountability, you’re not going to succeed the majority of the time. In fact you’re going to fail most of the time with these experiments. But the important thing is that you keep learning and you’re looking at your batting average and you’re improving things.

Setting Goals

So Brian’s system, it has a macro level and a micro level. You set three levels of goals. One that you’re most likely to hit. So 90% of the time you’ll hit it, another goal at which you’ll hit probably 50% of the time, and then a really reach goal which you’ll hit about 10% of the time. And then an example would be let’s improve our activation rate by X%. This is our stated goal. Now for 30 to 60 days let’s go heads down and run experiments until the 60 days is up, and we’ll look and see if we hit our OKRs with checkpoints along the way. So now you zoom in and you experiment. So this is the week-by-week basis. So every week you’re going through this cycle.

Experimentation Cycle

So there’s really four key documents as part of this experimentation cycle. The first is the backlog. That’s where you catalog. That’s where you catalog all your different ideas. Then you have a pipeline which tells you what you’re going to run next, as well as what you’ve run in the past. So that somebody new on the team can come in and take a look and see what you’ve done to get where you are today. Then is your experiment doc which serves sort of as a specification.

So when you’re getting ready to do a big test, like let’s say you’re going to re-engineer your referral flow, you’re going to outline all the different variations. You’re going to estimate your probability for success, and how you’re going to move that metric. It’s a lot like software development as you’re estimating how long somethings going to take, and you’re also estimating the impacts. And then there’s you’re play-books, good for people to refer to.

So with Trello it actually works out really well. So the brainstorm column here, the list here, is basically where anybody on the team can just dump links to different ideas, or write a card up saying, “Oh, we should try this.” It’s just totally off the cuff, just clear out whatever ideas are in your head and you dump them there. So you can discuss them during your meeting where you decide which experiments are coming up this week.

The idea is that you actually want to do go into the backlog. The pipeline are the ones that I’m actually going to do soon, and I’ll make a card, and I’ll put it in the pipeline. And then when I’m ready to design the experiment, I move it into the design phase, and then I create the experiment doc. And then I set my hypothesis, “I’m going to do this. I think it’s going to have this impact. Here are the different pages on the site I’m going to change, or things within the product I’m going to change.” And then later in the doc, it has all of the learnings and the results.

So one key tip that Brian talks about is when you’re trying to improve a certain metric, rather than saying, “Okay, how can we improve conversation rate?” You think about the different steps in the process. It just sort of helps you break the problem in multiple chunks, and then you start thinking a little bit more appropriately. And this is actually where the tactics come into play when you’re brainstorming, because this is where you’d want to actually look to others for inspiration. If you’re interested in improving your referral flow, maybe use a couple of different products, or think about the products that you use where you really thought their referral flow worked well, and then you use that as inspiration to impact yours. You don’t take it as prescription. You don’t try and like apply it one-to-one, but you think about how it worked with their audience and then you try to transfer it over to how it work with yours.

Prioritization, there’s really three factors here. You want to look a the impact, potential impact. You don’t necessarily know but you want to sort of gauge the potential impact should you succeed with this experiment. The probability for success, and this can be based on previous experiments that were very close to this one. So like I mentioned earlier, the customer testimonial you had a certain level of success with that in one part of your product on your website, and you’re going to just reapply it elsewhere. You can probably set the probability to high, because you’ve seen it in action with your company with the product before.

But if you’re venturing into a new space, let’s say like Facebook ads. You never run them for your product before. You don’t know what parameters to target. You don’t know anything about how the system works and the dayparting and that, then you probably want to set the probability to low. And then obviously the resources. Do I need a marketer? Do I need a designer, a developer and how many hours of their time?

So once you move something into the pipeline, I like to have my card look like this. I have my category, my label. So this is something with activation, trying to increase our activation rate. And then I say, “It’s successful. This variable will increase by this amount because of these assumptions.” Then you talk with your team about these assumptions, and try and explain why. So the experiment doc, I had mentioned before, this is sort of like your spec. I like doing this rather than implement the real thing upfront, if you can get away just putting up a landing page, and then worrying about the behind the scenes process later, do that. Like if you’re thinking about changing your pricing. Maybe change the pricing on the pricing page, and then not do all the accounting, billing code, modifications just yet.

Implement, there’s really not much to say about that. The second to last step is to analyze. So you want to check yourself as far as that impact. Did you hit your target? Was it more successful than you thought, less successful? And then most importantly, why? So really understanding why the experiment worked. Was it because you did something that was specifically keyed in on one of the emotions that your audience has? And then maybe you carry that through to later experiments.

And then systemize, another good example of systemizing that actually comes from HubSpot is the idea of an inbound marketing assessment. It’s actually their number one Lead Gen channel which is they just offer, for any company that wants, they’ll sit down one-on-one and do a full assessment of their website, of their marketing program, et cetera. When they were doing these one-on-one discussions those became their best leads most likely to convert.

So they made something called Website Grader which is something you could find online, and it’s sort of like the top of the funnel for that marketing assessment where someone’s like, “Ah, I don’t know if my website’s good at SEO, and am I getting a lot of links?” like that. So they’ll plug it into the marketing grader. It’ll go through and give them a grade. They’ll get a nice report, and then a sales rep in the territory that that person lives will now have a perfect lead-in to an inbound marketing assessment. Which they know is a high-converting activity should someone actually get on the phone with their consultant. So it’s a good example of productising.

How It Works In Practice

So this is just sort of like how the system works. So Monday morning we have … It’s our only meeting. It’s about an hour and a half, and we go through what we learned last week. We look at our goals, make sure we’re on track for our OKR, which is our Objective and Key Result. And then we look at what experiments we’re going to run this week, and then the rest of the week is all going thought that rapid iteration of that cycle of brainstorming, implementing, testing, analyzing, et cetera.

So you kind of go through these periods of 30 to 90 days of pure heads-down focus, and then afterwards you zoom out, and you say, “How good am I at predicting success of these experiments? Are we predicting that we’re going to make a big impact, or making small impacts? Is our resource allocation predictions accurate?” And then you want to always be improving on throughput. So if you were able to run 50 experiments during a 90-day period, your next 90 days you want to be able to run 55 or 60. So you always want to be improving.

Help Work to Flow – Interview with Sam Laing and Karen Greaves

February 25th, 2015 by Gareth Wilson

Looking for audio only? Listen on


We’ve interviewed Sam Laing and Karen Greaves, Agile coaches and trainers at Growing Agile, in Cape Town, South Africa. Together they wrote ‘Help Work to Flow’, a book with more than 30 tips, techniques and games to improve your productivity. We cover how software developers can improve their productivity and manage interruptions, why feedback and visible progress are important to staying motivated and how teams can hold better meetings.

They write about Agile development techniques on their blog.


Content and Timings

  • Introduction (0:00)
  • Achieving Flow (2:06)
  • Importance of Immediate Feedback (3:07)
  • Visible Progress (4:27)
  • Managing Interruptions (5:42)
  • Recommended Resources (8:50)




Today we have Sam Laing and Karen Greaves who are Agile coaches and trainers at Growing Agile based in South Africa. They speak at conferences and write about Agile development. They’ve written 8 books between them, including their latest book, Help Work to Flow, part of the Growing Agile series. Sam and Karen, thank you so much for taking your time to join us today all the way from South Africa. Why don’t you say a bit about yourselves?

We both have worked in software our whole careers and we discovered Agile somewhere along the line about 8 years ago and figured out it was a much better way to work. Then in 2012, so just over 3 years ago, we decided that’s what we wanted to do, was help more people do this and figure out how to build better software. So we formed our own company, Growing Agile, it’s just the 2 of us and we pair work with everything we do, so you’ll always find us together.

Really what we do is just help people do this stuff better and use a lot of common sense. What’s been quite exciting in the last few years is now being business owners of our own small business. We’re getting to try lots of Agile techniques and things for ourselves and it’s lots of fun and a constant journey for us.

I wanted to touch on the book, Help Work to Flow, what made you want to write that?

What actually happened is we joined the meetup called the Cape Marketing meetup and it’s for small business owners to learn about marketing techniques and how to market their small businesses. In doing that we realized that a lot of them are very, very busy and think they need to hire people to help them with their admin. We ran a mini master class with them on Kanban on how to use that as a small business owner. They absolutely loved it. They’re still sending us pictures of their boards and how it’s helping them with their workflow. We are like, well actually we have a lot of these tips that could help other people so let’s just put them together into a book.

Achieving Flow

I want to touch on flow for a bit, it’s often elusive to developers. How can developers help themselves achieve flow?

The first thing that pops in is pairing. To help with flow, having a pair, this is my pair, helps a lot. When someone’s sitting next to you, it’s very difficult to get distracted with your emails and with your phone calls and with something else because this person right here is keeping you on track and keeping you focused.

Another one would be to avoid yak shaving. I’m an ex-developer, I know exactly how this happens. You’re writing a function for something and you need to figure out some method and you go down Google and then you go into some other chat room. Next thing you’re on stack overflow and you’re answering questions that other people have asked that has nothing to do with what you were doing. Again, to have a pair to call you on that yak shaving is first prize but otherwise to recognize when you personally are yak shaving, bonus points.

Importance of Immediate Feedback

Enabling immediate feedback is said to be key to achieving flow. How can this be applied within the context of software development?

What we see lots of software teams do, is they all understand that code reviews are good and if they’re not doing pair programming then they should do code reviews. You even get teams where testers get their test cases reviewed but often what we see is teams leave that kind of to the last minute. We’ve done everything now let’s just quickly do the review.

One of the things we teach teams is instead of thinking of it as a review which is a big formal thing you do at the end, we use something that’s just called show me. I think we got it from Janet Gregory who wrote the book on Agile Testing. Literally as soon as you’re done with a small task, something that took you an hour maybe 2, you grab some else on your team or working with you and you show them for 5 minutes.

Like “Hey, could you just come and see what I did?” They quickly look at it, you get immediate feedback. Does it meet what their expectation was? Are there any obvious issues. The great thing about reviews where you get to explain what you did, sometimes you find the issues yourself. Definitely using that show me idea, versus making reviews a big thing. We’ve seen that have a radical changes in feedback for teams.

Visible Progress

The book highlights the importance of clear goals and visible progress. What do you mean by visible progress and why is it important to flow?

When you physically write down what’s top of mind, it kind of enters your brain so you don’t have to worry about those things anymore because you’ve written them down. If you’ve got, imagine a board in front of you with all these things that have been popping off of your mind, when you move those stickies across to done, you get this sense of achievement.

You automatically sending information out to those around you and to yourself on how much you’ve done versus how much you’ve still got to do. It helps you to get into the flow of getting stuff done. It also helps other people realize how busy you are and whether they should or shouldn’t add more to your workflow.

It’s really interesting, earlier today our personal task board, which is a physical board that sits between us, was like overwhelmed with too much stuff. I was like, “I just feel like we’re not getting anything done.” Sam took all the stickies and she put them on another piece of paper and said, “Okay, here are the 8 we have to do today.” Just looking at a board with 8 things, I said, “Okay, I can do 8 things today.” Before I was like, “I can’t even start anything because there’s so much stuff here.” That really, really helped.

Managing Interruptions

Here at Fog Creek every developer has a private office, to help them sort of reduce interruptions. What other techniques can you recommend to help developers manage interruptions?

Firstly, you’re quite lucky, most of the teams we encounter here are using open-plan offices and personal offices are quite rare but one of the teams we worked with had this great idea and it’s in the book and its called the point of contact for emergencies and it’s pronounced as POCE. What happens with that is they, in the team they agree who’s the person for this week who’s going to be the interrupt-able person and they even put a sign on their doors, going if you’ve got an urgent issue speak to Adrian, Adrian is going to be the POCE this week.

They rotate that within the team and so that persons going to have a lot of interruptions that week but everyone else is a lot less interrupt. The rule is, that person can then talk to team members if they can’t solve the problem but at least then team members are being interrupted by one person who’s part of their team and probably not as often as outside. That’s one idea that we can use.

Another one is, if you do get interrupted, don’t immediately do that interruption. Add it to your task list, add it to the list of work that you think you’re going to do that day and prioritize it accordingly. Often if you get disrupted with, do this and do that and do that, you’ve pushed your whole day out of sync and they’re usually not as important as other things.

We’ve touched on ways that an individual developer can help themselves but how can dev management go about encouraging flow across whole teams or organizations?

One of the biggest areas that we see waste are meetings. Often you have a lot of people involved in a meeting, it’s poorly facilitated or not facilitated at all. Also the actual goal of the meeting, sometimes is just to talk about what’s going to happen in the next meeting, which is atrocious and such a waste of time.

For management to encourage having facilitators, encourage people to get facilitation skills so that these meetings aren’t such a waste of time would already be a huge money and time saver. Also to look at having meeting free days so that people aren’t rushing from one meeting to the next or only have one hour of work between 3 meetings. Rather have one day that you are completely immersed in meetings and not going to do any work and then 2 days of no meetings, where you could focus.

I mean, we came up with that because we found with us being coaches and trainers, we’re often meeting new clients, giving them proposals or whatever. We go around quite a lot and sometimes we do onsite work and then we’re in the office. We just felt like we were getting nothing done. We did it the other way around, we came up with out-of-office days, so on 2 days a week we’ll be out of the office and everything has to fit into one of those 2 days if we’re meeting someone else. Then the other 3 days we had in the office and we actually got a lot of things done. That’s where we identified that and it still works really, really well for us.

Recommended Resources

What are some resources that you can recommend for people wanting to learn more about some of these types of techniques?

Really if you just Google productivity tips, you’ll find a whole bunch. The trick is that they’re not all going to work for you, so try a couple.

Even the ones in our book, not everything is going to work for you and it’s definitely not all going to work at the same time.

Our advice is find tips wherever you can, try them and then keep the ones that work.

Sam and Karen, thank you so much for joining us today.

Thank you.

Thanks very much.

Why Your Retrospectives Have Become Unproductive

February 24th, 2015 by Gareth Wilson

Retrospectives provide teams with an opportunity to reflect. They’re an opportunity to discuss what is working, and what isn’t, with the goal of iterative improvement. The meetings themselves should be a safe place for team members to share and constructively discuss processes and practices. Then, as a team, come up with actions to resolve problems or otherwise improve how the development team functions.

Yet, often this isn’t the case – retrospectives break down, become unproductive or just don’t happen at all.

Here are 3 core failings with retrospectives, along with potential causes and remedies:

1. Retrospectives That Don’t Lead to Real Change

retrospective (1)
At the heart of retrospectives is the desire for continuous improvement. Feedback gathered during the meetings should result in action items. These actions items, upon completion, should deliver positive change. But if action items aren’t completed, or the true cause of problems is not identified, then belief in the process can wane.

This can come about for a few reasons:

  • Too many action items

    It’s important that you don’t try and tackle too much and fail to make real progress with any of them.

  • Items are vague or have no clear resolution

    The action items you create need to be specific and have a definitive end point. Items like ‘improve test coverage’ or ‘spend more time refactoring’ lack specificity and need to be quantified. Concrete action items provide demonstrable results – allowing the team to see and feel the improvements achieved by following the process.

  • A lack of responsibility for actioning items

    Often the facilitator can end up with all the issues, or items are assigned to groups of people. This is a mistake – each item should have a clear owner who is charged with ensuring it gets done, even if a team will be completing them.

  • Too much emphasis on technical issues

    Working with tech is what we do, so identifying problems about systems, servers, libraries and tooling is easy. But you need to ensure that you give just as much attention to working practices, communication, and people problems. Otherwise, these key impediments to improvement will hold you back.

Whatever the reason, it’s important that you’re completing action items. So prioritize them, and focus on just a handful of items that you know can done before the next retrospective. Break down larger issues so that you can begin to make progress with them too. Track items raised at previous retrospectives and review results each session. This sense of momentum helps to build and maintain a belief in the process and fuel future improvements.

2. Retrospectives That Don’t Occur Often Enough

retrospective2 (1)
If retrospectives don’t happen often enough, it can cause a number of knock-on effects:

  • Too much to go over in any one retrospective

    This results in meetings that fail to get to the cause of issues. Or due time isn’t spent on issues important to attendees, which can be disheartening.

  • Previous action items are no longer relevant

    So much has changed since the items were identified that the issues raised are no longer a priority. This doesn’t mean they aren’t important. More often, it just means you’re compounding them with others and you’re missing an opportunity to improve.

3. A Lack of Participation in Retrospectives

retrospecitve3 (1)
This can often happen if the meetings aren’t managed effectively:

  • Sessions are long-winded

    You should decide on the agenda before the meeting to avoid straying off topic and unfocused discussion. You might want to consider time-boxing sections of the meeting. This helps to ensure discussion on one section or type of problem doesn’t consume all available time and attention, and that all areas get adequate attention.

  • Sessions have become stale

    Change things up. Try removing seating for one session, so people don’t just sit back and switch off mentally. Or change the format so you aren’t just repeating the same old questions. There are plenty of different techniques: from the Starfish and 4Ls to FMEA if you decide to deep-dive on a specific issue. Or just pair off and discuss items to bring back to the group. Some people open up better in smaller groups. And one-to-ones force a conversation, otherwise things gets awkward.

  • There’s a lack of trust

    A lack of participation can also result from a breakdown in trust. You should only invite team members to take part. Observers, especially management, despite noble reasons for attending, should be dissuaded from doing so. It may seem to make sense to share feedback so that other teams can learn from it too. But the specifics should be kept in the room. People might not contribute if they know there will be a long-term record of it, or if attributable comments are shared. Just share the action items or areas you’re looking to improve.

  • Sessions are too negative

    Retrospectives should encourage introspection and improvement. But this doesn’t mean it’s just a time to moan. It can be too easy to focus on the things that aren’t working or you failed to do. So it’s important to make an effort to highlight improvements, and not just from the last iteration but over time too.

With a few changes and a renewed commitment to the process, retrospectives can be a great way of ensuring you’re constantly improving. They can be an important part in making the working lives of your developers better and more productive. – Interview with Jared Parsons

February 23rd, 2015 by Gareth Wilson

In, we chat with developers about their passion for programming: how they got into it, what they like to work on and how.

Today’s guest is Jared Parsons, Principal Developer on the C# Language Team at Microsoft. He is also the creator of VsVim, a Vim emulator for Visual Studio. He writes regularly about development on his blog.

Location: Kirkland, WA, US
Current Role: Principal Developer at Microsoft

How did you get into software development?

I didn’t really get into computers until high school and even then it was just to sneak into the computer lab to play Civilization II. Games weren’t really allowed, but the teacher agreed to let me keep playing if I’d agree to take the AP programming class next year when I was a senior. She showed me a bit of Pascal in preparation. I thought it was interesting, but I wasn’t truly engaged by programming at the time.

That all changed after a prank my friend, Sam, and I pulled our senior year. Sam was much more into computers than me and had found out about WinNuke. It was a simple command line tool that would blue screen any computer running Windows. All we had to know was the IP address and we could crash a machine a friend was working on from across the room. This of course required us to physically log into the computer and find the IP address. We decided to map out a couple of machines in preparation for pranking friends the next day. After mapping a few machines, I noticed a pattern: the addresses all had the same three numbers at the start, only the last number differed and it was basically sequential. I wondered if all the computers in the school followed that pattern. Sam, one step ahead, used this tool I’d never heard of called Perl to call WinNuke with 256 different addresses. As he hit enter we looked up in amazement as every computer in our room and across the hall blue screened… and then we made a quick exit to avoid getting into trouble.

It was a silly prank, but it was also the first time I’d seen programming applied to a real world scenario. The idea that you could affect things with code was amazing and really peaked my interest in programming. Eventually, I ended up majoring in Computer Science and I’ve been obsessed with programming ever since.


Tell us a little about your current role

I’m a developer on the C# language team. I’ve been working on all aspects of programming for the last nine years at Microsoft: everything from the compiler to the debugger. The last four I spent working in research on an extension to the C# language focused on performance and reliability. Recently though I rejoined the C# team and am looking to transfer some of the positive outcomes of the research back into the shipping C# language.

In my spare time I develop a number of tools, my favorite of which is VsVim. It is a Vim emulator for Visual Studio written in a combination of F# and C#. I started the project about four years ago and it still takes up a good chunk of my free time.

You’ll also occasionally find me hanging out on

When are you at your happiest whilst coding?

When I’m refactoring or cleaning up old hacks in my code. It’s extremely satisfying. I often have to drag myself out of large cleanup operations just to get real work done.


What is your dev environment?

The one tool I cannot live without, no matter what platform I’m working on is Vim. Either using Vim directly or installing a Vim emulator into whatever editor I’m using at the moment. I started using Vim in my sophomore year of college and now, years later, the key bindings are simply burned into my brain. Sure the keystrokes necessary to do simple edits look like gibberish at first, but once you get past that initial learning curve it becomes a big productivity enhancer.

My biggest quirk is that I prefer an office to a team room environment. Not because I’m anti-social or subject to distraction (I’m actually quite chatty). It’s just that every now and then my computer doesn’t behave correctly and it needs a stern talking to. In a private office, I can close my door and tell my computer what I really think about it. In a team room environment, I’m just not sure how that would work.

What are your favorite books about development?

These are the books I recommend to all programmers:

  • The Pragmatic Programmer by Andrew Hunt: Overall solid book on how to approach programming, in general. It’s a book every programmer should read.
  • Purely Functional Data Structures by Chris Okasaki: Like many developers I learned programming on mutable data structures and imperative languages. It wasn’t until I left college and picked up this book that I was introduced to immutable data structures and functional programming. Learning these techniques gave me a whole new perspective on programming and fundamentally changed how I approach algorithms. It’s a must read for anyone without a strong functional background.

What technologies are you currently trying out?

I’m a language geek and every year or so I try and pick up a new language. It’s a great way to keep a fresh outlook on programming. Every popular language is great for something and occasionally I can find ways to incorporate that something back into the work I’m doing.

This year I’m going to learn Ruby. I’ve spent the last year or so on really tough static type system issues in C# and I feel the need to branch out. Ruby is about as far away from those problems as I can get and still be learning a new language.


When not coding, what do you like to do?

When not in front of a keyboard, I’m a very active person. I play soccer year round, run a couple of times a week and occasionally bike to work. In the summers, you’ll find me hiking around the Pacific Northwest with my wife Emily and our dogs.

What advice would you give to a younger version of yourself starting out in development?

There is a couple of pieces of advice:

  • Don’t ever settle for one programming language or technology. Keep trying out new things and pushing your boundaries.
  • Get involved with the developer community: contribute to an OSS project, answer questions on, join a local user group, etc. Constantly engaging with other people will help push you to grow as a developer.
    Oh and read, read and then read some more.


Thanks to Jared for taking the time to speak with us. Have someone you’d like to be a guest? Let us know @FogCreek.

Intro to Electronics – Tech Talk

February 20th, 2015 by Gareth Wilson


In this Tech Talk, Lou, a Support Engineer here at Fog Creek who runs our internal weekly Maker Space meet-up, gives an introduction to Electronics. He explains how electronic components like resistors, capacitors, diodes and switches work, along with key laws like Ohm’s Law, Kirchoff’s Voltage and Current Laws and how series and parallel circuits differ.


About Fog Creek Tech Talks

At Fog Creek, we have weekly Tech Talks from our own staff and invited guests. These are short, informal presentations on something of interest to those involved in software development. We try to share these with you whenever we can.


Content and Timings

  • What is Electricity (0:00)
  • Electric Circuit Water Analogy (1:47)
  • Resistors (3:10)
  • Ohm’s Law (3:43)
  • Kirchoff’s Voltage Law (6:00)
  • Kirchoff’s Current Law (6:40)
  • Capacitors (8:40)
  • Recommended Resources (12:20)



What is Electricity

Electricity is a fundamental form of energy observable in positive, yadda, yadda, yadda, alright, we’ll just focus on a little bit at a time. Electricity is a fundamental form of energy according to our current models of how we see the world around. This is as low as it gets. This will probably change over time but that’s how we’re representing it for the purposes of this. It’s observable in positive and negative forms, ok, there’s plus and minus. It occurs naturally like lightning, or is produced as in a generator. We all know this, this is great. And that is expressed in movements and interaction of electrons and I think that’s really the key point, that what we’re doing is pushing around electrons. They’re doing all of the heavy lifting. Simple atomic model – let’s go with hydrogen right here. A simple review, we have a proton in the center and it’s got that positive charge unit electron out in the orbitals. It’s carrying that negative charge. So what does that mean, how do we use that knowledge? When we look at our conductors, our metals, right. Copper’s a great conductor. And if you start looking at how the electrons are organized, they’re organized into shells. Shells are filled from the lowest energy level to the highest and every time you fill a shell and step up to the next shell, there’s a jump in energy levels. We don’t really need to know about the energy levels to do electronics, but the neat thing about it is as the atoms grow and you go from to larger protons in the electrons, there’s more pressure or attraction being put on all of the electrons. So when you jump an orbital and start building a new orbital, that’s where you get a weak interaction with the outermost electrons so they’re the easiest to bully. And they’re the ones we’re going to pick on.

So we want to make them work. We talk in terms of voltage and current. Voltage is the pressure that you put on to electrons. They amount of motive force you’re pushing behind them, whereas current is a reflection of how many electrons you’re actually putting to work. So the number of electrons that you’re pushing past your point is current, and the amount of pressure that’s on those electrons, or for lack of a better term the speed they are travelling at, is the voltage. That’s not accurate, but that’s kind of an analogy.

The water analogy, if you’ve ever looked at electronics is probably something you’ve seen before. Voltage is like water pressure in this analogy and current is like water volume. So the amount of water you have flowing past a point is your current. We talk about the amount of water that flows through a point or current through a piece of electronics component. Pressure we talk about the amount of pressure across, or the pressure gradient of water from the start to the finish of a component.

Just to give a picture of some of the things we’ll be looking at – batteries, that’s your voltage or your pressure source. We talk about ideal voltage sources because it’s really easy to pretend that that battery always puts out 9 volts. In practice, that’s not true, but just we’re going to pretend that it always puts out at a fixed 9 volts as it makes things a lot easier to understand as you’re learning it. And that’s going to be like our water pump, that’s how you give that motive force to that water.


Wires they’re like pipes, they’re going to be carrying our electrons, that’s how we guide them and send them to where we want them to go. Resistors are components that restrict pipe size, so with a resistor we can determine how many electrons can get past the point. This is to limit current. Everything has resistance, resistors are specially designed to bring a larger amount of resistance than normal at fixed levels so that we can sort of control that pressure. So we treat everything else as ideal in not having resistance because it makes it easier. So we pretend that the only things that resistance are these resistors.

Ohm’s Law

Ohm’s Law tells us that voltage is equal to current times resistance. If you double, if you increase your current, which is confusingly named I, you have to reduce your resistance to get the same amount of voltage. If you don’t and you increase your current, and you keep the same resistance then you’re going to need a bigger voltage supply to compensate for that. And that’s really helpful because again, resistors are how we control current and this is why. For a fixed voltage supply, this Ohm is the thing you’ll be adjusting because you’ll pick your resistors accordingly. And likewise if you pick a bigger resistor then you’ll get less current.

How do we know how much work is being done across the circuit? How do we know that the LED is going to light up? We have to account for all of the voltage. In a complete circuit the pressure from your power supply, your voltage, is going to drop across all of the components. So we have to figure out how much voltage is crossing across each component. Because ideally what we want to find out is how much voltage is dropping across this resistor, because then we can use the voltage and the resistance to determine the current that’s flowing through the circuit. Why is that important? If you throw too many electrons through the LED it blows up. Well, it doesn’t blow up but it smokes and doesn’t work ever again. So I did the math for us, the arithmetic for us, because no-one like to do arithmetic. LEDs have forward voltage, it’s like a pressure gate on a water valve. You need to put a certain amount of pressure across the LEDs or else it doesn’t turn on. Voltages vary as you go from red to blue, white, the forward voltage tends to increase. I picked 2.2 volts because that’s pretty common for a red LED. The remaining voltage is that 6.8 volts since we have 2.2 volts crossing across the LED all 6.8 volts have to drop across that resistor to make a total 9 volts. So what does that mean? If you do the arithmetic we don’t have to concern ourselves with it because computers can do arithmetic, they’re great at that. What you get is .021 amps, so it’s 20 milliamps roughly. 20 milliamps happens to be a great value for getting a lot of brightness out of a LED without over driving it. It’s safe for the LED and the 330 Ohm resistor if you’ve ever bought like an Arduino kit, or anything similar, you see a lot of 330 Ohm resistors. That’s not uncommon when you’re dealing with 5-9 volt power supplies. The math should work out that the current stays in a pretty safe place.

Kirchoff’s Voltage Law

Kirchoff’s Voltage Law, the plain english translation is that the sum of all voltages in a loop is 0. Resistance in a series circuit sums. We have a 330 Ohm resistor, and 3.3K Ohm resistor, we’re going to have 3630 Ohms. So now we no that we could model this as one resistor if we wanted to make this simpler. You could take those two resistors out and put in a 3630 resistor. You’re not going to find one of those, which is why the resistor values are all weird, they’re designed that way so you can build pretty much any resistor value you can imagine. Current is reduced by the increased resistance, because remember, resistors restrict the flow of electrons.

Kirchoff’s Current Law

So what we have, we have two loops. You can examine each loop independently. Kirchoff’s Current Law tells us that the sum of all currents as a node is 0. Or you don’t get nothing for free. So we know from looking at each of these individual circuits, we can just wipe out the bottom loop here and just deal with the LED and a 330 Ohm resistor, so we know that it’s going to get 20 milliamps. The other circuit is exactly the same. Same resistance, same LED. They’re both going to draw 20 milliamps. If you want 20 milliamps for one circuit and 20 milliamps for the other circuit, you’re going to need 40 milliamps out of your power supply. You don’t get nothing for free.

In parallel circuits, parallel resistances sum inversely. This might be intuitive, because when you think about it, what did we just say? The whole circuit is now taking twice as much current. Now the same amount of current is only flowing through each individual branch. But if you reduce this down to a single equivalent circuit, then you’d be looking at a total resistance for the circuit of 165 Ohms. Current has increased circuit wide. Voltage drops are independent across each loop. The pressure doesn’t drop, there’s no, we’re going to treat it like there’s no internal resistance at nodes, things like that. Again, in practice things can be a little bit different but in ideal analysis, this is what we, this is how we treat everything. And that’s pretty much how you’ll see it if you start playing around with meters and looking at this deeper detail.

Ok, you’re now an expert on making things light up. You know how to make things light up by themselves, you know how to make things light up in loops and in parallel. If you want you can throw more LEDs in series, just keep in mind that with that forward voltage. Lets say you have a 2.2 volt LED, right. You put two in series it’s going to be a 4.4 volt drop. You put 4 in, it’s going to be a 8 volt drop. You put that fifth one in, it’s going to go over 9 volts, if you can’t put enough pressure across them then they don’t open at all, LEDs don’t light up, it’s like you cut a circuit. Nothing will light up.


Capacitors are a curve ball. They operate a little bit differently. Capacitors are like a storage device, in a water analogy they would be, you know, one of these nice big water towers. So the size of the capacitor determines how many electrons it can hold. Capacitors are measured in farads, and farads are one coulomb volt. So if you have electrons coming out from the upper left, again we have parallel circuits, the voltage is going to push electrons on to the capacitor until it’s full. Then it’s going to stop pushing. The voltage is also going to flow our LED and our 330 Ohm resistor. Again so we don’t blow it up. Then we’re going to come back and go to the battery. One the upper right corner is our switch button. By default that push button is off. So again, that’s going to break the circuit. No electrons are going to flow until we’ve pushed that button. That’s how you control things. Once you push that button, electrons are getting through, they’re getting dumped on to the capacitor. When you release that button what you’re going to have is electrons flowing from the capacitor, that negative plate through the LED and through the resistor, because now there’s no pressure from the battery holding electrons in the capacitor. So they’re going to flow around to the other side of the capacitor. The capacitor in a way will work like a battery, but the storage tank analogy isn’t great because the water doesn’t refill its own storage tank. You have a complete circuit, with a capacitor that’s acting like your voltage source. That’s providing the pressure and then the complete circuit is the LED and the resistor. Everything up top goes away. Because remember when we switch that switch off, the battery can’t take those electrons so it’s not a circuit. You push that button electrons are racing out as fast as they can at 9 volts of pressure, filling the capacitor very quickly and keeping the light lit. When you let go of that button, there’s no pressure holding those electrons back in the capacitor any more, it’s like you opened the valve on a water tank and out comes all the water. They have to flow through the light, they have to flow through the resistor but you’re going to lose, there’s only a limited number of electrons, or a limited capacity on that capacitor. Once that charge is expended the light goes out. But because the capacitor has a certain amount of pressure on it, as it’s expelling the electrons, it’s losing voltage, it’s losing the ability to maintain that pressure. The same thing happens in batteries, but it’s a little bit different. This happens very quickly in the smaller capacitors and as the voltage drops, remember your circuit analysis. You have to have a certain amount of voltage to keep that LED open and when you’re dropping that voltage there your voltage on the capacitor drops, your pressure drops, the voltage across the resistor changes which changes the current, which changes the speed at which your capacitor drains. And when you change the current, remember the current to a certain degree you’re doing to see dimming in the bulb, the LED.

So the higher the Farad rate of the capacitor, the longer it takes to charge because it can hold more electrons. The time constant is pretty interesting. I find that it never really works as planned, but in general it holds up pretty well. You’re going to be looking at the resistance on Ohms, the capacitance in Farads if you multiply them together this is going to give you the two thirds mark of the discharge of the capacitor. If you fully discharge a capacitor that’s what that number is going to look like. Again with LEDs you have that forward voltage, it cuts off at 2.2 volts so it gets a little wonky, but you get some idea at how long, how much charge that capacitor has.

Recommended Resources

It all gets more complicated from here, but it’s a lot more fun. A few places to get stuff from. SparkFun is great, Fritzing is awesome it lets you build all of the schematic diagrams, and all the drawing diagrams. All that’s at Fritzing. Ada Fruit is local to New York City, also a great place to get stuff and they have amazing tutorials. So I recommend checking that out as well.

Becoming a Tech Lead – Interview with Pat Kua

February 18th, 2015 by Gareth Wilson

Looking for audio only? Listen on


In this interview with Pat Kua, Principal Consultant and Tech Lead for Thoughtworks, we discuss becoming a Tech Lead and the challenges faced by those new to the role. Pat is the author of the book ’Talking with Tech Leads’, in which he spoke to more than 35 Tech Leads about their experiences. We cover common issues faced by Tech Leads, the nature of the Tech Lead role, why he thinks Tech Leads should spend at least 30% of their time coding, and how to find the time to do so! We finish by discussing common mistakes new Tech Leads make and with Pat’s recommended resources for those wanting to learn more about being a successful Tech Lead.

He writes about topics like this on his blog.


Content and Timings

  • Introduction (0:00)
  • About Pat (0:24)
  • Talking with Tech Leads (1:07)
  • Common Issues Faced by Tech Leads (2:22)
  • The Tech Lead Role (7:00)
  • Tech Leads Should Code (8:14)
  • Common Mistakes of New Tech Leads (12:28)
  • Recommended Resources (15:02)




Today we have Patrick Kua, principal consultant and Tech Lead for ThoughtWorks based in London, UK. He speaks and writes about Agile development, Continuous Delivery and is the author of two books, ‘Talking with Tech Leads’ and ‘The Retrospective Handbook.’ Pat thank you some much for joining us today, I really appreciate it. Why don’t you say a bit about yourself?

About Pat

Sure, firstly thanks for having me. I’ve been working in the industry for quite a long time and working as a consultant in spaces with Agile teams. One of the interesting things I’ve observed is a gap on leaderships skills for technical people. And I think it’s kind of an important area for developers to look at, particularly if they’re looking towards an architecture career. If they’re looking to lead teams or work with people. And my interest in being in this space of working and developing developers who want to move in to the role of a Tech Lead or an Architect.

Talking with Tech Leads

I wanted to touch on the book, ‘Talking with Tech Leads’. What made you want to write the book?

It was quite interesting actually, because I went through the transition many, many, many years ago and I’ve worked with a lot of clients, I’ve worked with a lot of developers trying to coach them in to this new role and what I’ve really found is that people really struggle to find good materials out there. So, there’s lots of books that teach you how to be a great developer, that teach you how to learn refactoring skills, things about good clean code, how to unit test stuff, but actually trying to build a solid architecture at the same time as convincing people that it is a good architecture and to get developers to contribute towards this – there’s actually not a lot of stuff out there. And I kind of focussed on this idea of talking to different types of Tech Leads because I think, it’s kind of interesting when you talk to other people who have been through a similar position, when you’re a Tech Lead yourself, you’re kind of by yourself. And there’s not a lot of support. You’re kind of thrown in to this world between business people and this frustration of wanting to write code and there’s nobody there to help you. So I kind of wanted to focus on actually collecting the experience of different people going through that transition or have been through that transition to see if there were sort of common themes that would come up.

Common Issues Faced by Tech Leads

So you went and spoke to what, 35 Tech Leads about their roles. Where there any themes or topics that kept coming up?

Absolutely, yeah it was quite interesting talking to people. I had started off my quest starting to try and find people who were quite new to the role. Which actually was a huge challenge by itself because once a Tech Lead has been through it one, they kind of understand what it is, and it’s hard to relive that first step into the role. And then talking to more senior, seasoned Tech Leads, it was kind of interesting. So for the people who were stepping into the role the first time, a lot of the challenges were actually about the people side to the role. So it was kind of being thrown away from this world where you’re surrounded by compilers, test frameworks, new technologies and then suddenly you’re having to worry about who on your team is currently a bit upset, about trying to negotiate an agreement between two very opinionated developers, or maybe more than two… while you’re still trying to get something delivered and at the same time, sort of business folk, are depending on you to come up with plans and deliver on schedule. And that’s quite an overwhelming theme that emerged from the first group. From the more senior, seasoned group, so the people who had been Tech Leading for a while, a lot of the themes were the surprise of having a broader outlook, so your day-to-day thoughts aren’t so much as what features you’re developing, even though you still write code, but it’s kind of thinking more outwardly. So thinking about longer-term plans, are there any bigger, nagging architecture issues that need to be dealt with, or are there the right technologies choices or should we be looking at a new technology and if so how do you kind of introduce that. There was also another aspect, which was around the business side. To dealing with technical ideas or technical topics in the context of a business setting. So, in the book I talk about bridging with business. And a lot of the skills from Tech Leads, that they end up being developed is this kind of translation, where you’re kind of trying to avoid all of the technical terms, and trying to express it in ways that your normal family members might be able to understand who don’t understand technology whatsoever. But you kind of need their buy-in to understand why you’re going a certain direction with things. And the final topic or theme, that emerged was really about dealing with yourself. So I think it’s true for anyone in a leadership position, which is you won’t have enough time to deal with all of the things that you’d like to do, and this is true when you’re being pulled in quite a few different directions. But I think it was really nice to hear some of the experienced from Tech Leads particularly reflect on how they manage their time, how they avoid stress. So one of the Tech Leads talked about meditation about his go to thing about managing himself. Trying to really prioritize time to its best use possible.

Having been a Tech Lead yourself for a number of years, you obviously had some ideas of the challenges that Tech Leads face. Were there any issues that came up that surprised you?

Yeah, I think that the last thing about managing yourself was one of the things that I never really thought of. When I was playing Tech Leads a lot of my time is thought about outside, and actually going through this process has helped me reflect on how I manage my own time, what my own strengths and weaknesses were. And it really made me self-reflect on how I can improve myself as a Tech Lead as well. And that was a really interesting thing. I think understanding my strengths and weaknesses helped me to understand that everyone has their own personal approach to these themes and that’s a really nice aspect to the book as well, it kind of shows how everyone focusses on different things, they think about different things, but it doesn’t necessarily mean that they’re better or worse. It’s just a different way of achieving the same sort of goals. And I think for me that was sort of a surprise and also pleasant message, because rather than trying to mold people into what I see as an ideal Tech Lead it’s kind of gives people a lot more freedom to mold the role into however they see it best fit their own skills.

The Tech Lead Role

On your blog you go into some of the challenges about being a Tech Lead, you often position it as a paradox. Is that how you see the Tech Lead role, as one where you have to constantly balance competing forces?

I think to be a successful Tech Lead it is definitely one of those key things. So, it kind of comes back down to the ‘you can’t do everything at once’, so what do you do? And I think the natural choice is to kind of say ‘well, I’ll focus on one thing’ at the cost of everything else. And this is where the paradox comes in, so one of the examples that I’ve written up on the blog is often talking about we often work with developers in a professional context, so we have to be really focussed on delivering something. At the same time we kind of want the team to be learning new technologies, to sort of spend time on training, to go to conferences, to sort of focus on removing problems and the technical challenges that we have in our environment. But, it’s kind of hard to balance out those two kind of aspects. So the paradox of do you spend more time on learning or do you spend more time on delivering, I don’t think you can ever really say that’s it’s a perfect unbalanced equation, it will wax and wane depending on what things are going on. But it’s a key skill for a Tech Lead to sort of accept the paradox and embrace the paradox and to find a way to do both at the same time.

Tech Leads Should Code

Having transitioned from a Developer to a Tech Lead, you obviously can’t spend as much time coding as you’d like given your new responsibilities. But you recommend Tech Leads spend, what, like 30% of their time coding? Why do you think that that’s so important?

It has been really interesting though because, when I talk to developers going into that Tech Lead role, the biggest thing is ‘do I still get to code as a Tech Lead or do I get to do this sort of post-technical work?’ and it’s a huge concern. And my answer is, to be effective, I think you do need to code. I’ll go into the reasons why in a second. But the next immediate question is ‘well how much time should I actually spend coding?’ in order to be effective. And I guess my reasons, why I think it’s important for people to code in this role, is that, I think developers really respect technical ability. And it’s very difficult for people who want to follow a leader, and you want to respect their leadership decisions, if they don’t have that respect for their ability. And I think I recognize this in myself as a developer as well. I don’t understand how, this sort of the classic ivory tower architect, that makes decisions far removed from the context of the problems that we experience on a day-to-day basis. And I think that’s one of the key reasons why Tech Leads, to be effective, still need to code. Or at least, being at the coal face, where there reading code, being able to work with people on code, is that they really have to understand what the key problems are. They understand the language that developers use, so that they can talk at the same sort of level, and also know when people are maybe making a bigger deal out of something that’s maybe technical not true. So I think that’s a skill that Tech Leads need to find out. I think 30% is a good minimum if I think that of a week, it’s a day and a half. Ideally, it’s great if a Tech Lead can do more. But, there’s also phases of a product or project where if there’s lots of recruiting going on, or if there’s budgeting type cycles, then the Tech Lead will be the first to get pulled out. But it’s important that the Tech Lead thinks about spending some time in the code, so you’re getting the respect across the rest of the team and just to have a true awareness of what’s actually going on.

The Tech Leads I know are constantly interrupted, are there any techniques or ways of working which can help deal with these interruptions and sort of block out time for things like coding?

Tech Leads will always be interrupted. It’s a pain, and it’s quite interesting talking to some of these Tech Leads about how they approached this. So some Tech Leads simply block out time in their calendar so that they can actually spend time without interruptions. I think that’s easier in some environments that others. In a sort of more Agile environments, where the team is all co-located, there’s lots of things going on, even blocking out your calendar doesn’t really solve a lot of that problem because people will just come and walk up to somebody and interrupt them, because they’re available. I think that that’s an easy thing to do. A technique that can help in this type of environment is just having some sort of visible sign that says actually I need some quiet time. So I’ve used personally for instance, a flag on the desk to indicate when it’s no interruption time. And it’s not that I’m not available, but it’s that I needed time to really focus on something and think about things and I’ll be available shortly after as well. Other Tech Leads take themselves out of the working space. So they go off to a quiet room, so they book a meeting room and then they sit in the room by themselves. Just so that they can go through a few things in quiet time. Another way is to maybe off-set your day as well. So either, start early or end early but do the same sort of hours. And this kind of helps you to get through all of those things before everyone else gets there in to the workplace and interruptions start to come.

Common Mistakes of New Tech Leads

So not spending enough time coding seems like a common mistake Tech Leads can make. What other things do you see that people sometimes get wrong?

This is kind of interesting, so particularly when you watch new developers go into this role for the first time, I think there’s two streams of a Tech Lead that I see. So there’s the Tech Lead that believes that the entire team should be self-empowered, every developer should know what they should do, and the Tech Lead wants to have trust in everyone. And so they sort of just sit back and hope for the best. And I think this is where things get a bit dangerous if the team hasn’t gotten into that mode and they don’t already have that strong vision. I think what happens there is that probably the most opinionated developers sort of pipe up and they form the direction themselves. But in these kind of environments things kind of quickly fall apart. Particularly arguments between developers, they go unresolved. People spend a lot of time arguing and the same thing comes up over and over again. In the other extreme, I see Tech Leads who see themselves as I need to demonstrate that I’m a Leader, they’ll want to still make all of the heavy technical choices, so they’ll want to be involved in every technical discussion, and it’s important that they have their own say and they make their stamp on the project. And I think these two extremes are elements that good Tech Leads should be able to play, but they shouldn’t be playing that mode of Tech Lead all of the time. So one of the leadership models that I talk about is the situation leadership model, which kind of talks about situations where more directive behavior is useful. Where maybe more targeted coaching is useful, and where perhaps pure delegation, and just sitting back and trusting the team to do the things that need to be done makes sense. And the great thing about the situation leadership model is that it helps people identify when and where to use these modes of behavior without actually saying it’s good or bad. And without saying you have to act like this all of the time. So I think that takes a lot of time to develop an awareness of when to use which mode, but it’s probably one of the first things that I see with first time Tech Leads.

Recommended Resources

Can you recommend some resources for developers or new Tech Leads who want to learn something about being successful in the role?

I think there’s quite a lot of books that can help Tech Leads, and I think particularly from a people side, that’s one of the hardest things for a Tech Lead to learn and it’s just a skill that you don’t necessarily practice when being a developer. And I think there’s a couple of books that I would recommend.

I think the first one would be a book called ‘Getting To Yes’. The book is about negotiating, and it’s about trying to come up with a good way forward while you’re trying to appease both sets of interests. What’s great about it, is that it’s a very short book and I think it has some really great stories that help people to understand some good skills.

I think the other book around sort of people skills would be ‘Crucial Confrontations’ and ‘Crucial Conversations.’ I think it’s useful because there will always be some tense situation at some point in a project with one or two people in a project. It’s kind of natural, people are unpredictable unlike computers, although computers can also be unpredictable. But I think that the book helps people to understand how do you have a discussion around the topics where they’re emotionally sensitive, or you know that you have a certain position and reaction where you need to deal with a difficult topic and talk to somebody about it.

And I think that those two books are very good starting books for Tech Leads to learn about people skills.

Pat, I want to say thank you so much for joining us today.

Great, thank you for having me.

How Our Support Team Share Knowledge Using FogBugz

February 17th, 2015 by Derrick Miller

Support or Customer Service, at Fog Creek is different. The team is empowered to help our customers and do whatever it takes to solve a problem. We take great pride in the customer service we make available to our customers and often write about it – from seven steps to remarkable customer service, using it as a competitive advantage, dealing with angry people, to our 5-part series about how we do customer service, and how Trello (a product we created and spun off into its own company) use FogBugz to support 4 million members with just one person.

Something we haven’t written much about is how the support team share knowledge among its members. The nature of Support is fast-paced. There is always a constant flow of knowledge – from archival knowledge to the streaming updates of issues and features.

Staying abreast of this information is difficult, but our Support team stays on top of it all with FogBugz, and it’s search, wikis & cases, auto-subscriptions, shared filters, and subcases. We’ve broken down how we use each of these features to help share knowledge here at Fog Creek.

Write Down Your Internal Processes

The team uses Wikis for longer lived documentation. As a Support team member, you know to look in the “Customer Service” wiki before interrupting someone else. This wiki contains archival information for newer hires (FogBugz has nearly 14 years of history) and documentation of internal processes, among other gems. The great thing about the information in the wiki is that it is searchable. There’s also a clear outline and hierarchy to help organize the content and assist someone trying to find a piece of information. For example, if a new Support member was looking for general information to get started, they can see the table of contents and click the “General Customer Service” link.


If a few terms in the article are known, run a search. For example, here a Support member is looking for an article about To-Do items not completing:


Document Your Thoughts

In addition to the Wiki, the Support team leaves artifacts for fellow team members in cases. They know that someone will leverage the power of search to help them with future cases. When working on a case from a customer, they write out their hypotheses or thoughts on how to approach the problem in the case comments. They make sure to include error messages too. If the case changes hands because a team member is out of the office, the information for troubleshooting it is already in the case, saving the new team member from starting over.


In the Support world, log messages are gold. We include them in case comments to improve searchability. Searching helps them develop additional hypotheses or determine ones to avoid. Perhaps another Support member looking at the case above would say “Why wouldn’t we check the Chrome console logs first for any errors?” and add that to their troubleshooting steps.

For the occasional “FYI” situation, the Support team uses the “Notify More Users” field on a case to explicitly let a team member know of new information. This organically grows knowledge throughout the team. Here we have Mary editing an inbox case and notifying Erin:


Stay Updated Automatically with Auto-subscriptions

The auto-subscribe feature is used by the support team to keep up with key Project Areas in FogBugz as well as cases they create.

These key areas being “Known Issues” and “Fix It Twice”. The “Known Issues” area contains cases that quickly document a recently uncovered issue with our production (i.e. live) On Demand service and the communication needed both internally and externally. This could be an unfortunate service interruption, a regression bug or a new feature not working as expected. These issues can be reported either internally or externally. Here, we have an issue reported externally that Mary is creating a known issue case about (you can also do this for “Fix It Twice”) cases:


Once the case above is created, Mary will be subscribed to the case so that she can see when Jamie adds the related subcases. Jamie has the same auto-subscription preferences, so he’ll be able to automatically see updates on the case that Mary makes. This is visible under the “Subscribed To This Case” field on the left-hand side of a case if you have this feature enabled.


Essentially, any time a case is updated or created by someone, the rest of the Support team instantly knows about it. They can use this information to react to any cases they are currently handling.

Bonus tip: You can auto-subscribe to Wikis too!

Don’t Repeat Yourself with Shared Filters

To further take advantage of FogBugz’s search capability, the Support team uses three primary shared filters: one for customer cases due today, one for the “Known Issues” area, and one for the “Fix It Twice” area. This is in addition to their own personal filters for cases assigned to them.


The filter “Support: Next Due”, is very important because of our promise to our customers: We answer all email within one business day. It shows all cases due today. Having these two filters available to every team member without any extra effort (read: creating their own identical filters), saves everyone time so they can focus on their work. New team members briefly look at the list of cases that are due today (and not assigned to them), and subscribe to anything that is new that they might learn something from. They don’t subscribe to everything because that is quite simply just too much information, but cherry-picking some cases to “eavesdrop” helps fill knowledge gaps quickly. The added bonus with eavesdropping on cases is that a new team member will see the culture and tone of the company and use that in his or her own cases.

Keep Things Organized with Subcases

“Who is currently affected by that new Known Issue?” is a question the Support team will ask for every new Known Issue that they get a notification on. The answer is in the subcases. When a case is added to the “Known Issues” area, this case becomes the parent case, and any and all customer cases become subcases. The Support team uses this hierarchical relationship to get a top-level view of who is affected and needs communication about the issue. See “Subcases” on the left-hand side of a case, or click “Case Outline” if there are several.

FogBugz’s search feature is tremendously powerful. The support team uses the available search axes to narrow down cases and/or wikis that contain relevant information. Has a similar issue happened before? How did someone else approach solving the problem? Didn’t someone already report this problem? Among the countless other questions. For example, a full-text search for “todo item not completing” can be run to see what wikis and cases show up:


The key search axes for the Support team to sift through full-text results are:

  • orderby:lastedited
  • edited:”today”
  • edited:””
  • type:case
  • type:wiki

The key takeaway here is that the information exists in cases and wikis because someone took the time to write it down. Don’t worry, the first time you write something down, it doesn’t have to be perfect. You can always go back and edit what you wrote.

Creating an auto-subscription, a shared filter, and writing things down in wikis and cases will put you well on your way to getting the most out of sharing information with your team. The powerful search facility provides one way of accessing the increasingly valuable knowledge base growing inside of your FogBugz. – Interview with Brian Bondy

February 16th, 2015 by Gareth Wilson

In, we chat with developers about their passion for programming: how they got into it, what they like to work on and how.

Today’s guest is Brian Bondy, Senior Software Engineer at Khan Academy. He was previously at Mozilla and is the creator of Code Firefox, a resource site for would-be Mozilla contributors. He writes regularly about development on his blog.

Location: Ontario, Canada
Current Role: Senior Software Engineer at Khan Academy

How did you get into software development?

I grew up on a gardening farm in what seemed to be a completely different world from what I live in now. As a kid my first computer was a Commodore 64, I loved playing games on it but I never programmed for it. I took some programming classes in school, but I’d say I was nearly 100% self taught and didn’t really learn anything in school relating to programming. Computer science yes, programming not so much. I started programming at 11 or 12 years old. My first languages were Turing and QBasic. In particular, I loaded up a program that my brother had made on an 8” floppy disk. The program asked you your age, and would output too old or too young depending on what you entered. It was magic to me and I was hooked from that point on.

The Internet wasn’t as popular at the time and I didn’t have it, but I knew I wanted to learn programming. Magazines mentioned that you needed to know C++. I saved up my money from working on the farm and had my mom call Borland to order me Borland C++ 4.0. About a month later the package came in the mail. I purchased several editions of Programming Windows 3.1 and beyond by Charles Petzold and read those cover to cover. In general, I had more books than I would ever get through. More than 20 years later I’m still learning tons and still find it fun.


Tell us a little about your current role

I’m currently working at Khan Academy as a remote developer. We all make up our own titles, but I’d say my role is that of a senior software engineer. My wife, being a Game of Thrones fan wanted me to pick “King of the North” as my title, but I wimped out.

I write a lot of code every day both at work and in personal projects. I try to work on some personal project at least 30 minutes a day, 7 days a week. Largely credit to John Resig, I’m currently at 202 days on GitHub which pales in comparison to his 448-day streak.

At Khan Academy, I was working in the computer science department building tools to help people learn JavaScript, web pages, and SQL. Recently I’m leading the development for a math competition with leaderboards for a program we’re calling LearnStorm, which is piloting in the Bay Area.

My current main side-project is a Khan Academy app for Firefox OS devices. It was released a couple of months ago so I’m just adding features to it and also trying out new things like Flow type checking in that project.

I also work on Firefox stuff when I have free time. They call people in their community Mozillians. I used to work there before Khan Academy and what they say holds true, once a Mozillian always a Mozillian.

When are you at your happiest whilst coding?

When I’m learning something new. It’s also fun to finally fix something you’ve been struggling to figure out.

I love writing things from scratch. When I was in high school and University I used to read standards specs and implement them. I wrote client/server implementations in C++ for various protocols and file formats just to understand it all at a deep level. For protocols I’ve written client/server implementations for HTTP/HTTPS, FTP, SFTP, FTPS, SMTP, POP3, MSN Messenger, Gnutella and more. For file formats, this was mostly various different image file formats including GIF, JPEG, BMP.


What is your dev environment?

I’ve been a Windows developer for the past couple of decades, but I don’t currently own a Windows computer. I’m working exclusively on a MacBook Pro at Khan Academy.

I mostly use Vi and various plugins for everything, but I prefer to use visual debuggers when I’m tracking down a bug. Built-in browser debuggers for JavaScript, PyCharm for Python, Visual Studio for C#.

For source control, I use Mercurial for Mozilla code and Git for everything else including my personal projects.

I often code on my couch, at the kitchen table, and in a variety of different settings. I always take meetings in my office though.

How do you stay productive and avoid interruptions?

I used to find the whole coding flow concept to be extremely important; however, because of working from home, I think I’ve learned to be super productive while there are noises and kids running around. I’ll occasionally spend a few hours at night cranking out a lot of code with no distractions.

What technologies are you currently trying out?

Historically I’m primarily a C++ developer but I’ve been doing a lot of ES6 JavaScript lately and integrating a lot of the amazing work Facebook has been doing with JavaScript, including React, Flow, and Jest. As well as using things like Browserify and Gulp.

I’d love to do more C++11 if I had infinite time.

If you weren’t a developer, what else would you do?

Possibly an entrepreneur of some kind or a University Prof. I can’t imagine myself being good or loving anything else career wise as much as I do coding though.


When not coding, what do you like to do?

Mostly doing things with the family. I have twin 6-year-olds and a 2-year-old so we’re always busy. My wife and I recently introduced them to Star Wars and they loved it. One of the twins wants to be a Jedi and the other wants to join the Dark Side. Maybe I should be concerned, but he’s in love with Darth Vader and wants to be just like him.

After they go to bed I’ve taken up running. I ran my first marathon a couple of years ago.

What advice would you give to a younger version of yourself starting out in development?

Work on a bunch of personal projects and contribute to open-source projects. Consuming information is one thing but if you don’t build things you won’t remember it.


Thanks to Brian for taking the time to speak with us. Have someone you’d like to be a guest? Let us know @FogCreek.

Go and Artificial Intelligence – Tech Talk

February 13th, 2015 by Gareth Wilson


In this Tech Talk, Tim, a Software Engineer here at Fog Creek and 2-dan amateur Go player, talks about Go (the board game), Artificial Intelligence and attempts to create computer programs that can beat human players. He gives an overview of Go, explains how to play it and why Go AI is hard. He finishes by describing the progress so far with Go AI programs and what the future is likely to hold.


About Fog Creek Tech Talks

At Fog Creek, we have weekly Tech Talks from our own staff and invited guests. These are short, informal presentations on something of interest to those involved in software development. We try to share these with you whenever we can.


Content and Timings

  • Introduction (0:00)
  • Overview of Go (0:32)
  • How to Play Go (4:37)
  • Go Artificial Intelligence (12:22)
  • Progress with Go AI (20:04)




Alright, so I’m going to talk a little bit about the board game Go. I’m sorry to all of you who are hoping for Go Lang discussion here. And also some of the progress that has been made with Artificial Intelligence and trying to get computers to be as good as humans at Go.

So first I’m going to tell you a little about the game, then I’m going to teach you how to play, then we’ll talk about a few things that come up in the game that make Artificial Intelligence for Go difficult and then take a look at the state of the art and progress that has been made.

Overview of Go

So Go has three names, so you probably haven’t heard of the others. But Wei-chi is the Chinese name and Baduk is the Korean name. Go came here from Japan, so most people in the US call it Go. It’s around 3 or 4 thousand years old, and it’s the oldest game that is still played in its original form. It was one of the ‘Four Accomplishments’ that were required of Chinese gentlemen back in the day. So, you needed to be able to do calligraphy, painting, you needed to be able to play the Lute and play Go. So after this talk you’ll be a quarter of the way there at least.

In Japan, the Shogun Tokugawa established 4 Go schools around 1600s. And each of those schools was supposed to work on the game, try to perfect the game and then once a year there would be a large tournament called the Castle Games. It wasn’t large in terms of people, each school was allowed to send one representative, but it was a big deal. And the winner of the Castle Games got a cabinet-level position in the Government. So there was a lot of incentives to improve at Go and this is where Go skill really started to blossom, I guess you could say. And then in the 1900s it became possible to make a living playing Go. In Japan, a professional system was created in the 20s and newspapers started offering large prizes for tournaments. Currently the best you can do at Go if you win a large tournament, is win half a million dollars. So, not too shabby. Professional systems were established later on in Korea and China and it’s more popular there now than any game is in the US.

This is a recent street fair in Korea, and you can see there are a few people who know how to play Go on the street there. I think there are about 500 players, and you can see the people in the middle wearing the yellow sleeves are the professionals that were there. So they are all playing something like 8 or 10 simultaneous games against passers-by who just stopped off to play.

So, what is Go like? It is a two-player strategy game, something like Chess. There’s no hidden information or randomness. Usually we play on a 19×19 board, but there are two smaller sizes that are used to teach beginners, or just for a shorter game. And the basic goal of the game is to surround territory. Also within Go there are rankings and a handicap system. So you start out as a beginner, you start out around 25 kyu, somewhere at the bottom and then you progress upwards. So you can go 24 kyu, 23 kyu, not all of the ranks are shown on this chart. And once you get up to 1 kyu as an amateur, you switch to 1 dan, which is basically like a Black belt, and then you work your way up to 6 or 7 dan. So you can see on the left here the EGF rankings, the European Go Federation uses numbers instead of the rankings. And those correspond roughly to Elo ratings, so if you’re familiar with Chess ratings they’re something like that.

Once you get to the strongest amateur levels then you get professional levels above that. And those are not the same distance apart, so you can see the third-row from the top here is 2700 and that’s a 1 dan professional, and then the top professionals in the world are 9 dan and the different in rankings points is only 240 there. The difference in rankings is determined by the number of handicap stones that a player would need to take playing against someone stronger and weaker. So if you’re three ranks apart, then the weaker player would start with three stones on the board and then you’d be able to have a good game. So that’s actually one of the really cool things about Go, is that you can play someone who is quite a bit stronger or weaker than you and still have a good game and it doesn’t really change the feel of the game very much. In Chess, it’s kind of hard to do that, right. If you’re 400 points weaker than somebody in Chess, they’re going to beat you every time. And you can try something like spotting them a piece or the Queen or something like that, but it sort of changes the game. You’re not really playing the same game anymore. So in Go you can start with a handicap and it still feels the same way.

So this is what a finished game looks like. Your pieces are actually played on the intersections of the board. That’s a common beginner thing, almost every other game we know you play in the spaces, but Go you play on the intersections. And the idea is to surround space. The way it works is players put a stone on the intersection on their turn. So you add a stone to the board, and that’s it. The stones don’t move around. They can be captured, so if they get completely surrounded then you can remove them from the board. But they just stay in place once they are added.

How to Play Go

Alright, so now I’m going to teach you the game. There are only three rules. One is the rule of capture. So if any stone on the board has breathing spaces next to it, or liberties, and you can see the liberties are marked on the diagram. So the stone in the corner has two, the stone in the middle has four and actually stones that are connected horizontally or vertically should share their liberties. So this group of two stones on the right here has six liberties and the one at the top has four. So those stones will live or die as a group.

If you fill the liberties around the stones, they are captured. So here are some stones that are almost captured. This is called being ‘in atari.’ It’s kind of like check in Chess. So if it’s black’s move, black could capture any one of these four groups on the next move and that actually is where the name of the company Atari came from. Nolan Bushnell is a big Go fan. So if it’s black’s move, black could play on the last liberty for any of these white groups. And then whichever one he played would disappear. He obviously couldn’t play all four of those move at once. So these stones would be captured and removed from the board and each one of those stones is worth a point at the end of the game. If it was white’s turn when white got into atari, then white could play another stone connected to the ones that are in danger and gain more liberties for them. So it could extend out of atari and try and get away. So you can see the stones on the top edge here, each now have two liberties – one to the right and one below the last stone. And the ones in the middle that have extended each have three liberties now. So it’s going to be harder to capture those. Alright, so that’s rule one.

Rule two, very simple. You are not allowed to commit suicide. So white cannot play any of these four points because it would cause the stone just played to die.

Third rule is the rule of Ko. Ko in Japanese means eternity, and this is to prevent infinite loops in the game. So if you have a shape like this in the game, black can capture a white stone. And so if he captures that white stone, now you’re in the symmetrical situation and white can capture and take us right back to where we started. And that is not allowed. So we can’t have any infinite loops. The simplest way to say this is that the whole board position cannot be repeated. So after black captures white stone, white cannot recapture immediately. White has to go change something else on the board and then on his next turn can come back and capture the black stone. And then actually leads to some complexity that we will see later on. That’s it. Three rules: you can capture stones by surrounding them, can’t commit suicide and can’t keep repeating the same position over and over. So you’re all ready to go play Go now.

The scoring is based on the territory that you can surrounded. The way the game ends is that both players pass. You’ll get to a position at some point where playing in your own territory would be bad because you’ll be filling up your own points that you have surrounded. And playing in the opponents territory would be bad because the stones that you play on their territory would end up getting captured. And so there’s nothing productive to do and at that point you pass and then if the other player passes, the game is over.

So you count the spaces that you have surrounded, they’re each 1 point, so the empty points here with the white and black squares on them are a point each. And then the black group and white group that have the squares on them are dead, so the players, this is actually one of my games, so the players at the end of the game we agreed that the black in the upper left there is dead and the white group in the lower right. So at the end of the game, here we’ve got a black territory in the lower left and in the upper right. And in the lower right. Then we have some white territories. We would count the empty points, we also take the agreed dead stones off the board and each of those is worth a point to the player that captured them. And then we add that all up and white gets 6.5 extra, which are called komi, and that’s because black has the advantage of going first. So we’re compensating for that advantage. And then whoever has the most points after that wins the game.

OK, so the rules are extremely simple, but there are some interesting things that come out of them. One of which is called a ladder. So here we’ve got a situation where white has some stones that are almost surrounded. He’s got one liberty left, and if it’s white’s move, white could try to run away. So white could extend the stones, now he has two liberties. But in this kind of shape black can play atari again, and then white can try to run away again, and black can play atari again. And you can see where this is heading, right. Well white is going to run in to the wall and then he’s going to have anywhere else to run. So these white stones will eventually be captured. A few moves from now it will look like this. And white has one liberty, at d19 up here and if white plays on that point, he will still only have one liberty so the stones will get captured on the next move by black. So if you have a ladder like this, the stones can run away for a while, but then they’ll run in to the edge of the board and they will die.

Things get a little more interesting if there’s a white stone in the way. So in this case we have a white stone along the path of the ladder, and that’s called the ladder breaker. So this ladder isn’t going to work for black. If white starts running away, black can make him zig-zag back and forth a few times and then white will connect to the ladder breaker. So as this point it’s actually black’s move, white is just connected to the ladder breaker and now black has some serious problems. The two points marked with circles are big weaknesses for black and black only gets to play one stone. So black can only fix one problem or the other. And then white’s going to play the other point. And both of these circle-marked points are going to threaten two black stones at once and black will again be only to save one of the two. And so white is actually going to bust out of this ladder and then all of black’s stones are going to be in trouble. So you really only want to play a ladder if it works. It’s very, very bad to play a ladder that doesn’t. And we’ll see why that’s important a little bit later on.

Here we have a white group that is completely surrounded by black and has one liberty in the middle of the group. And if it’s black’s turn, black is allowed to play in the center of this group. When we play the black stone, the black stone that was just played and the white stones will not have any liberties, but first we will first remove the opponents stones before we check to see if our move was suicide. So we’re allowed to go ahead and play here because the white stones are going to die. And then that will give the black stone breathing space. So this is possible, but if white has a group that looks like this. There’s nothing that black can do. Alright, because, since white has two liberties and either of those moves would be suicide for black. There’s no way to ever capture this white group. OK, so this is where life comes from. If you can get two eyes for your group then your group will be alive. And so there are situations like this, where white has a group that is almost alive. And it depends whose turn it is. In this case, if white plays in the middle, white will have two eyes and the whites will live. If black plays in the middle, white can no longer make two separate eyes, and so this group is going to die.

And then with larger groups, you’ll have situations where you don’t actually play it out. So this white group actually have two points in the middle that white could play on to make the group alive. And if black takes one point, white will take the other. So this is actually the most common situation. You’ll have a group that’s big enough that it can make two eyes no matter what black does. And so, experienced players won’t try to kill this group – they’ll just know that it is alive and leave it alone.

Go Artificial Intelligence

Alright, so lets talk about Artificial Intelligence a little bit. Here are a few games and how we’re doing it, creating AIs to beat people at them. Tic-Tac-Toe, obviously trivial. Checkers has 10^20 positions, and the AI strength is perfect and in the last few years Chinook is a Checkers program that now plays perfect Checkers and they did that by evaluating the entire game tree. And so they know that Chinook can’t be beaten, the best you can hope for is a draw. Othello has more positions, and AI is Super-human at Othello. And then we have Chess and two types of Go. 9×9 Go is Go on a really small board that you would sort of start out as a beginner or just play to have a quick game that lasts maybe 10 minutes. And it has 10^38 positions and computers are now competing with the best people. Chess has more positions than that, and as we all know, Chess computers are better than humans. And then 19×19 Go has an enormous number of positions, so 10^172 and the best computers right now are at the strong amateur level.

The way that Chess AI was approached is through Tree search, an Evaluation function and Alpha-Beta pruning. And that’s how the first Go AIs worked as well. We can look at any Chess game as a tree of possible board states, right. We have the initial state at the top and then we have a branch down from that for each legal move. And then from each of those positions we have more branches for the legal moves from there. And we make some gigantic tree that makes every possible position. And we can search within that tree to find good moves and moves that lead us to good situations and avoid bad situations. And so what we can do then is look at all of the possible positions that we are going to, that moves will lead to, and we try and evaluate them. OK, so we look at a position and we try and decide whether it’s good for white or good for black. And this, basically you get a positive number if it is good for white and a negative number if it is good for black. So this happens to be good for white, maybe. I may have that backwards. So anyway, Chess AIs could evaluate the board and then you take your tree and you basically chop off branches that lead to bad positions for the computer. So you make moves that avoid those branches. So you can prune the tree and then analyse the parts of the tree that you’re actually going to allow to happen by making good moves and try to find a path to a winning position.

So Go AI started off doing this. There were basically two approaches at the beginning. There was sort of opening book, where there would be sort of standard openings that were programmed in to the computers. But Go that doesn’t work very well because the opening probably lasts, I mean the standardized openings might last 15 or 20 moves out of a 200 or 300 move game. And that’s just not a big advantage. And then there was some pattern recognition stuff done for small positions, but then beyond that they basically just attacked the problem using Tree search. And one of the big problems with that is that the branching factor in Go is much than in Chess. So in Chess on average there are something like 45 legal moves and a game last for 50 moves, so a tree isn’t that deep. In Go there are on average 250 legal moves and the tree is 300 levels high, instead of 50. So there’s a much larger number of positions to search if you want to make tree search productive.

But that’s not the only problem. Actually in Go the board gets more complicated as the game goes on. So one of the things that’s nice about Chess is that when you get to an end-game position, depending on what pieces are left, it can just be a solved problem. So you can have an end-game database that just says, you know, this huge class of positions all result in a win for white and if you can reach one of those in your tree search, then you don’t have to do any more work. But in Go actually, as you get closer to the end of the game, the game gets more complicated. So that is not helpful. Another thing is that it’s hard to prune branches. You can’t even do things like, say if a move leads to an opponent capturing a large group of stones, then we’re going to ignore that branch because even sacrificing large groups of stones can be a good strategy. So throwing those options out is not really a good way to get a strong program.

And then the biggest problem is that evaluating a position is dificult. So in Chess it turns out it’s not that hard, you have a simple rule that does a pretty good job of telling you who is winning the game. So, a big piece of that is just which pieces have been captured, right. Which player still has more pieces on the board. And in Go it’s much, much harder to look at a board position and decide if it’s even good for white or black. And I’m going to show you a couple of the reasons for that. So one of the big reasons for that is Go has a large board, right. So there’s 361 points and you would like to be able to break that down in to pieces and analyze the pieces independently. Because then the computer can sort of play out all of the possible moves in that area that make sense and decide who is going to win a fight in a small area. But it turns out you can’t do that because of some non-local effects in the game. So we’ve already seen one of these – ladders. And a ladder can result from a complicated fight. So here in the lower-right corner we have a very simple ladder. But you can imagine a fight where a bunch of different groups of stones are fighting for liberties and trying to capture each other. And then at some point that fight results in some group of stones running away and it gets caught in a ladder. So that ladder can actually go all the way across the board and that fight depends on the results in a totally different part of the board, right. So, the computer may decide that this fight in the lower-right may depends on the ladder, well now any move that white makes in the upper-left corner of the board affects that fights. So you really can’t take pieces of the board and analyze them independently when you have something like that happen.

There’s another problem that makes this even more difficult and that’s the Ko rule. So here we have a black group which is on the edge of being alive. If it’s black’s turn black can fill in this point and then black has two separate eyes. So black is alive. If it’s white’s turn white can capture that single black stone and if white can get another move in this area, white will then capture the three black stones on the right because they are now in atari. Because of the Ko rule, white cannot recapture white’s stone immediately, so black can’t take us back to that previous position. So what black has to do is find a threat somewhere else on the board that white will answer. And then after white answers it, white is allowed to come back here and capture the stone. And so this fight, deciding the life of this group can rely on threats throughout the board. So black needs to find a threat. This capture, this group living or dying is worth maybe 25 points. So black needs to find a threat elsewhere that’s worth 25 points, and then if white answers the threat, black can capture the Ko. So we’ll go back to this position. And now white will try and find a threat worth 25 points somewhere and if black answers that, white will re-capture and so this can go back and forth for quite a long time. Using threats from all over the board. And then finally black will win and connect and make the group alive, or if white wins, white will play another move here and capture the three stones. And then black’s group is dead because it’s only got one eye left. So Ko is another issue. Ko fights can arise and then the computer in order to figure out what’s going to happen with a single group in one particular area has to actually look at the entire board and try and figure out what’s going to actually happen. I’m getting depressed, this is making me not want to write a Go AI…

Progress with Go AI

Alright, so, what is the state of the art. So we started off with Tree search and trying to make the evaluation functions better and prune the tree, basically take the same approach as Chess. That worked ok up to a point, that got computers somewhere around the ten kyu level. Then there was sort of a breakthrough, and Go AI started to use Monte Carlo Tree search. So what we do with Monte Carlo Tree search is we take a move, a candidate move. So these moves near the top here are candidate moves. These leaf nodes that are marked. And what you do is play games out from that candidate move, but you don’t use a smart AI to play the rest of the game because that’s way too slow. What you do is you basically just play the moves out randomly from that point until the end of the game. And then you see who won the game and you do that over and over and then you keep stats back in the candidate move node about how many of those games were wins for black and how many were wins for white. So it sounds kind of counter-intuitive that this would work at all, it seems like it would just be playing lots of stupid moves and it really wouldn’t, the results wouldn’t really be affected much by the move you’re trying to evaluate. But, it tuns out this works really well. So you can actually play a whole bunch of stupid moves and see who wins and then do that thousands of times, and that will actually give you a pretty good idea of how good the move that you’re considering is. So that actually turned out to help quite a bit and then there’s another modifications that has been applied to this recently, that’s called UCT. And what they do there is, rather than evaluate each candidate move evenly, UCT tries to work out which of the moves is better and then which moves are likely to be worse. One of the problems with the original Monte Carlo Tree search is that it would look at moves that it thought was good and spend a lot of time on them, but there might be a move, a kind of surprising move that turns out to be really good and it would not evaluate the move enough to find out that it was good. And so UTC tries to balance that out and give a decent amount of analysis to even moves that initially look bad. And so that was a further refinement, and then just a few weeks a go there was a paper published on a new Neural Network approach to Go, and this looks extremely promising. They took a Neural Network and they trained it on a 160,000 professional games and then the way that this Neural Network works is it just looks at the current board, it doesn’t look at the history of the game or anything like that. It just looks at the current state and predicts where the next move will be and just doing that actually gave it a workable AI. So it beat a version of GnuGo that is about 7 kyu and it beat it 90% of the time. So that’s actually surprisingly good because this approach is not using any of the other work that’s already been done. So once this is merged in to the existing techniques it seems like it will probably be another jump in strength.

Alright, so here’s the history of progress. The Chess programs definitely got off to a better start. Go was pretty bad until about 1990 or so, and then there’s sort of a hockey stick effect over here on the right. That is where the Monte Carlo Tree search started being used. So we have a much steeper slope after that, there’s been much progress in the last 10 or 15 years. So if you look at the list here, in 1998 pros could beat the best programs giving them a 25 stone handicap. That is a truly absurd handicap. I think that none of sitting there that just heard the rules for the first time, I don’t think I could beat any of you with a 25 stone handicap. And if I could I do it one time. And then the second time you would annihilate me. Alright 25 stones is a huge handicap so there was a gigantic gap between the best programs and the pros. Then around 2008 a program beat a pro for the first time. So MoGo running on 256 cores, beat Catalin Taranu on a small board, so that’s on a 9×9 board, which is a much easier task than on the large board. But still, impressive. And then that same year MoGo running with 800 cores beat a 9 dan professional with a 9 stone handicap. So this is on a full-size board. And then in the years since then the handicaps have been getting smaller and smaller. So you can see down here at the bottom in 2013 Crazy Stone beat a 9 dan pro on a full board with only 4 stones. Alright, so that’s an impressive accomplishment. And then this year, Crazy Stone beat another 9 dan pro with 4 stones but won by a wide margin. So it’s looking like maybe the different is 3 ranks now. Between Crazy Stone and some of these top professionals. So it’s actually getting closer, starting to breath down our necks. It looks like we’ll probably see a computer than can compete with the best professionals in the next 10 or 15 years.

Looking for more?

Visit the Archives or subscribe via RSS.