DNA Changes Ecosystems: Our support team changed the company, and now must change itself.

October 25th, 2011 by Rich Armstrong

If you take a look at the Support Engineer job posting for Fog Creek, what you’re actually looking at is a finely crafted trap. It’s a trap to get coders to come talk to our customers. And it’s worked very well. We’ve managed to get five very smart, very capable people to come do a job they probably would never have considered at another company. The difference was in the DNA. They knew we were doing something different because the job posting laid out our core principles.

Years ago, Joel laid out some basic precepts about how he wanted to approach customer service. It’s the first thing I read after getting hired away from Google to grow out Fog Creek’s support team. I was the entire support team for 20 months. (Two guys shared the job before.) I buckled down and I squeezed every improvement I could out of the support workflow, which was easy when it was only me. There was no team communication overhead, and I could tweak the processes without getting buy-in from others.  I also was able to affect the development of the software such that the raw support load decreased, even though the customer base more than doubled. I took Joel’s precepts and built them into the DNA of the support team we ended up hiring, and that allowed us to get a long way.

Eugène Ferdinand Victor Delacroix 055

The one precept I’ve built into our DNA more than any other is “Fix everything two ways”. Everyone says they fix the hard problems, but they don’t. I wanted to make it a reality.  We call it “Fix It Twice” and we use it in a lot of ways.

First, if we think a customer issue can be solved by a deeper change in the software, we always spin that idea off in to a secondary queue. The worst time to think about how to do a deep fix is right after you’ve solved the initial problem. The job demands empathy with the person having the problem, and that leads to a kind of recency bias. You’re always worked-up about the thing that just happened. What you do about that worked-up feeling makes all the difference. (Even recognizing this is helpful for fending off frustration in the support role.) If you just let it go, you end up with the feeling that “stuff never gets fixed.” If, however, you spawn off that potential change into a secondary queue, where it can be evaluated at another time when you’ve regained your distance and objectivity, you get a better sense of where resources might be best dedicated to reduce load overall. This, in itself, makes fix-it-twice worth all the trouble. You’ve taken frustration and turned it into business intelligence.


After that, we realized that the amount of stuff waiting for us in the fix-it-twice queue was a very good proxy for job satisfaction. If the numbers stayed low, that meant we were getting the slack to go and think about how to make our job easier and the software better.  If the numbers stayed low regularly, it meant we could code. That has led to real improvements in the software, like the Do Later plug-in. The support team has stayed happy and productive for a long time…. And then, the roof fell in. We lost one of our support reps to the design team. Well, actually, one of our support reps is a gifted designer and got poached (with our blessing).

When the team sat down to figure out what we wanted from our newest support rep, we realized something pretty momentous: The job had changed. It’s only now that I realize that building it into the DNA of our company has changed the company such that our needs are totally different. A responsive support team and an effective engineering team mean that the easy problems get solved forever and you’re left with mostly tough ones. We knew it’d been getting harder for years, felt it viscerally. The problems just kept getting more complex, less common, harder to diagnose. But now for the first time, that meant that the team had to change because our ecosystem had changed.

If your DNA says that you prey on slow, lumbering, delicious prey, eventually you’re going to find that those walking buffets are getting scarce. After that, you might find that you have to run a little faster to eat. You might also find that the plants your now-scarce prey fed on have changed and been supplanted by others. We find ourselves in a different world, and we had a hand in making it.

So, let’s call this the Kilneolithic epoch. The job posting still lays out our DNA. It’s still who we are. But it’s no longer the finely crafted trap it was before. Now, we’re looking for a different animal. The person we’re looking for is personable and curious and techie, just like always, but we need someone a bit more hardcore. We’re good at troubleshooting .NET and IIS.  But Kiln runs on OpenGrok, Apache, Redis, Mercurial. We need someone who’ll thrive in our new ecosystem, maybe even more than we do.

Recognizing that we’d changed our world was both gratifying and terrifying. We know we’re good at the base job, but it’s taken us to a place where we’re challenged. The team’s up for it, but we need one more body. If you’re up for it, email us at jobs@fogcreek.com. (Yes, this blog post was, itself, a finely crafted trap.)