Consider a cake shop owner in a small town in Siberia. He sells the best cake around. Whenever someone wants a treat after dinner they stroll down to the cake shop and get a nice big slice of cake. But along comes a new treat: ice cream. Our cake shop owner ridicules his neighbor for starting an ice cream stand, saying no one will buy it because it’s cold. This is Siberia, after all. But rather than running an oven all the time, she can just keep her ice cream outside in the freezing cold, so it sells for less. Suddenly a whole bunch of people who could never afford cake are stopping at the ice cream stand after both lunch and dinner. It’s cheap, it tastes good, and the teenagers are having contests to see how much they can eat before passing out from brain freeze. It seems the whole town is there, and some of the cake shop customers start getting ice cream just because they want to hang out with their friends at the ice cream stand. All of a sudden the neighbor is rolling in the dough and she wants to buy out the cake shop since they have a great location.

Now suppose the owner of the cake shop recognizes that lots of his customers will like the ice cream (even in Siberia) because it’s so much cheaper. So he sets up an ice cream stand and hires his neighbor to start selling it as fast as possible. He notices that some of his regular cake shop customers are going to the ice cream stand instead, because it’s cheaper. So cake sales are down a little. Rather than shutting down the ice cream stand though, he brings it into the cake shop, adds a special “cake” flavored ice cream, and becomes the town hero. Not only has he saved his business, but he’s also been able to meet the dessert needs of more of his fellow townspeople. Cake is selling even better than before, ice cream is selling even better than that, and the cake-fanatics and ice cream groupies all get to hang out together.

Clayton Christensen’s talk at this year’s Business of Software was all about how companies disrupt, and are disrupted by, other companies. In building a product, you (or those who came before you) made decisions that are really hard to change after the fact. That’s fine; those “stakes in the ground” are what made the product successful. But it also limits the viable lifetime of the product. At some point disruption happens.

Disruption =

Larger Market + Lower Price + Different Measure

Professor Christensen pointed out that disruption occurs when a company solves a given problem for a larger audience at a lower price point with a different measuring stick for comparing value. Companies that successfully avoid being disrupted are usually able to do so only by disrupting themselves. Few companies are able to pull that trick off, and it typically involves having a different team or business unit set up in order to avoid all the baggage that the last-generation products carry with them.

What does this have to do with Fog Creek? Well, I’ll admit that I was a little skeptical when Fog Creek launched Trello. My worries pretty much lined up with those outlined in this post from a FogBugz customer. Though it’s taken some time, I’m starting to see how important it is for Fog Creek to prepare for the disruption of its flagship product, FogBugz.  By doing so, we can make sure FogBugz will keep solving our customers problems and keep making us money.

Trello fits the model for disrupting Fogbugz. It solves the problem of planning the work on a project for a larger audience than just software companies, at a lower price point than a FogBugz or a Jira, with a different measuring stick: putting Trello and FogBugz next to each other on a feature comparison chart doesn’t make any sense.

Trello’s target market is much larger than that of FogBugz. Where FogBugz targeted software development teams with features like bug tracking, automated crash reporting, and evidence-based scheduling, Trello can provide value to any group of two or more people working together on something that can be broken down into steps. Even though any group of people could use FogBugz to do the same thing, using FogBugz doesn’t really make sense when you’re in HR doing hiring, or in law working through cases, or in a studio vetting country bands. Using Trello does.

Additionally, Trello has a lower price point: free. Everything currently offered by Trello is free, and will remain so going forward. Yes, there will probably be value-added features and services that Fog Creek will charge for at some point. Compared to free, though, FogBugz is expensive at $25 per user per month. When you just want something simple to plan your wedding, that doesn’t make any sense. But if you’re managing software projects for reasonably complex products, then FogBugz easily adds that much value, and our customers make that clear by coming back again and again.

Trello also has a different measuring stick. And that’s the real reason it will eventually overtake FogBugz, even for software projects. Or rather, it will take over some of the roles that FogBugz fills in a software development company. We already use it here at Fog Creek to manage our work at a coarse-grained level. It provides a potentially public view into product development, which is cool to see.

The real key, though, is that Trello, like FogBugz, is opinionated–but it has very different opinions. Rather than seeing work on a project as a large set of small items that need to be tracked individually, it sees project work as a small set of somewhat larger tasks that fit into a bigger whole, a workflow defined by the team. If FogBugz tried to create some kind of dashboard view of your bugs to compete with Trello, you’d be so overwhelmed by minutia that you’d give up and walk away. No one wants that kind of view when they’re dealing with hundreds or thousands of individuals cases. But Trello redefines the way we see our project work and that fundamentally changes the game.

Wait a sec! I’m on the FogBugz team. What am I saying?! Am I just making a Steve Yegge “TMI” mistake by posting this publicly? Am I talking myself out of a job?

TMI?

I don’t think so. And here’s why.

The same legacy that prevents it from winning in the larger Trello market gives it a competitive advantage in the market for software development teams. FogBugz will continue to make money, and it is still growing at a nice pace. It needs investment, but not the kind of investment that would attempt to turn it into Trello. Rather, the kind of investment that will take advantage of disrupting products like Trello while preserving it’s usefulness in the niche of software development teams.

And our customers, for the most part, aren’t going to jump ship for Trello anytime soon. Currently, Trello can replace only a very small part of what FogBugz provides. One customer pointed out that it cannot handle bulk editing, screenshot captures are painful, and categorization and search aren’t designed for the situation where you have lots of items. Additionally, FogBugz supports incoming and outgoing email, automated crash reports, and deep hierarchies of work. You can install it within your network and use it completely internally. It’s also got awesome source control integration with Kiln. These are all things that software development teams care about, often passionately. Trello, and other products like it, may eventually meet some of these needs, or integrate with other tools that do, but that will take time. Time that will allow FogBugz to further differentiate itself.

Besides, learning new ways of working is hard. Anecdote time! We spent some time over the last few months rethinking the UI for FogBugz. As the team lead, I made the call to focus on that, and did what I could to protect the team through the process. Unfortunately, when it was done, Joel pointed out that it wouldn’t fly with our customers. Why? Because we “moved the cheese” in too many ways. It would have required our customers to learn new ways of working, and human nature doesn’t like to do that. Though that does limit what we can do with FogBugz going forward, it also strengthens the ties that our current customers have with the product. They know where to click, which keyboard shortcuts to use, and they know what to expect. That helps them work faster, get in the flow, and keep the boss happy.

Finally, and related to the last point, FogBugz has an ecosystem of supporting technology around it. When our customers start using FogBugz and Kiln, they often integrate with a set of tools and technology. This is one reason it’s harder to innovate on those products, because our customers don’t just rely on not having to learn new ways of working, they also rely on their custom plugins not breaking, on their random Python scripts still working, and on third-party tools continuing to integrate well with FogBugz. Each of those is a barrier to entry – they make it harder for a competing product to win our customers away from us (yep, even if the competitor is made by us as well) because they also add value for the customer.

In short, for almost all of FogBugz existing customers, and most of the larger market of customers using a competing bug tracker, using Trello instead is not the right business decision. And within that market FogBugz can make sure that business decision doesn’t change by doing the right kinds of investment. Investment that will keep FogBugz relevant and growing into the future is core-competency investment, not more features or chasing faster competitors in a race decided by a radically different “finish line”.

And there are tons of things we can do in the core of FogBugz: reduce the complexity of the product both for our users and for our developers; make it faster; phase out old features that aren’t being used by anyone; fix the backlog of bugs; improve our up-time; integrate with Trello and other apps. That last point is probably the most salient. As Trello grows and becomes a better fit for certain kinds of project management work, FogBugz can increasingly integrate with and offload those areas to Trello. At the same time, it can target specific problems that software development teams face, and provide value by solving them well.

In the end our customers will be able to have their cake … and their ice cream too.

Since we are entering the season of giving thanks, I thought I’d drop a little note about those useless little thank-you emails you sometimes get when you help a customer through FogBugz (or, really, any other system).

We get the occasional customer request asking us to implement some sort of auto-ignore feature that won’t re-open a case when the person just responds with “Thanks!”  Now, as the person responsible for customer happiness at Fog Creek, the very idea of something that “auto-ignores” any customer communication sends up all sorts of red flags.

We’ve looked at this from all different angles and have just never come up with an automated method that doesn’t suffer from both false positives and false negatives. In the end, the false positives (closing cases that did need more work) are far more destructive than the small time savings garnered by closing these cases automatically.

The problem really is not the person responding, but your response to them.  If you send a response like this, you’re basically leaving something unconfirmed and basic human decency will prompt some portion of the population to provide you with closure on your interaction.

Hi Bill,

Please reboot your computer.

Rich

Since this is basically a social issue, not a technical one, it calls for a social hack, not a technical hack. Through experimentation, we find that the longer and more formal your closing, the fewer thank-you emails you get in response.

You can use snippets to craft a closing that does not invite response. Here’s a good example that has worked very well for us:

Hi Bill,

This can easily be solved by rebooting your computer.

Let me know if I can be of further assistance, if indeed this was helpful, or if this raises any other questions.

All the best,

Rich

This closing doesn’t have any hubris in it. We don’t assume that we solved your issue. In fact, we don’t assume that we’ve helped at all, or closed the conversation at all. Paradoxically, by inviting dissent or further conversation, we invite only the constructive responses.

Thanks!

Read on!

A Starcraft AI that’s more like a human player: This implementation faces many of the obstacles that human players face, like only having a single viewport, and being able to issue one command at a time. Still, 2000 APM is crazy!

Tuning TCP for more vroom vroom: Adjusting initcwnd for better performance.

Operations efficiency is one of your possible competitive advantages: And Apple, of course, excels at it. “The decision to focus on a few product lines, and to do little in the way of customization, is a huge advantage.”

Crowd sourcing algorithms: Big prizes

Some musings on patents. On Andrew Sullivan’s blog. There is also the always excellent Lewis Hyde, who has a fantastic book on patents and copyright.

 

 

My wife and I are a couple months into our first year of homeschooling, and we’ve been discussing how the kids are doing with it, and the challenges we’ve faced as their teachers and parents. Those discussions have led us to make some important changes, in the same way that a school report card can get parents and children to make changes that lead to more learning and happier kids. In that spirit, I offer here a report card on the build and release management work we’ve been doing on FogBugz and Kiln [1] over the last 12-18 months.

About a year ago, I was in the middle of writing a series of posts about the practices of build and release management. Many of those practices came from looking at what we were doing with FogBugz and Kiln and identifying what worked, and what didn’t. Others came from what I saw and experienced while working within the Microsoft Office organization, and still more came from discussions within our team at the Creek. Though in some areas we were doing things right, most of the posts were a description of what our existing build and release (non-)system was doing utterly wrong and what I hoped we could change to fix it. So the series was often written in a spirit of hope that we would one day live up to the ideal.

In fact, you could take many of the posts I wrote last fall and read them as the opposite of how things existed at Fog Creek in the summer of 2010. So, in the spirit of Jason Cohen’s recent BoS talk about honesty, I’m going to be brutally honest about how we were doing and how we are now doing.

First, how we were doing. Here’s the report card for June of 2010, divided into subjects, of course.

All in all, things were a mess. ”Good grief!”, I hear you cry. “So what have you done about these issues? Anything? What are you planning to do?” I’m glad you asked.

My initial goal in the role of build and release manager was to eliminate the position. I wanted to get things running smoothly, and spread the knowledge and ownership of the problems through the team, so that I could go back to doing product development. As such, a decent amount of the progress described here was done by other great developers here at the Creek. Back in January (when I joined FogBugz as the team lead), I thought I’d gotten about halfway through the work required for that to happen. But, of course, new ideas come up constantly, so maybe it never ends.

Since January, I haven’t been able to do as much. In addition to my small efforts, other great Creekers have picked up some of the slack. Here’s our report card for October 2011.

As you can see, there is still plenty of work to do. The glaring failure in both report cards is our lack of a staging environment. We’re now working on making that a reality, with high hopes that it will help us start catching a new class of bugs before they make it into our customers hands. As part of that work, we’ll also be able to work on:

• Scripts for configuring our servers, by type
• No-downtime upgrades
• Move our internal dogfooding to the staging environment, for better consolidation and testing of deployment scripts

Other possibilities for further work should be pretty clear from the report card, but the staging environment is the key focus when it comes to improving our builds and releases.

One note about the culture for both FogBugz and Kiln. We’ve learned a ton in the last couple years by seeing at close range what can be done in greenfield development on products like Trello, or teams that actively maintain and improve their processes for builds and releases, like StackOverflow. Obviously, when working with a legacy product like FogBugz, the costs of implementing certain practices may vary a lot, and we have to take that into consideration, but the inspiration that comes from seeing what is possible keeps us working to make things better.

When I look at this report card, I feel like my oldest son a little bit. He’s our perfectionist, always expecting to get everything right the first time through, and pretty disappointed when that doesn’t happen. We’re hoping that some of the changes we make in our homeschooling will help him relax a bit so he can learn more freely. And yes, it’s somewhat humbling to look at our build and release report card and admit how far we have to go. But I have high hopes that in another six months we’ll have improved our grades again.

[1] What about Trello?

Trello has the awesome advantage of being greenfield development. As such, they’ve really started from the ground up by doing the right things when it comes to builds and releases. They frequently release midday with no downtime, they have a nice staging environment, their releases take almost no time at all. They have run into some interesting load challenges with updating their clients after an upgrade, but I’ll let the Trello team talk about that in their own sweet time.

If you take a look at the Support Engineer job posting for Fog Creek, what you’re actually looking at is a finely crafted trap. It’s a trap to get coders to come talk to our customers. And it’s worked very well. We’ve managed to get five very smart, very capable people to come do a job they probably would never have considered at another company. The difference was in the DNA. They knew we were doing something different because the job posting laid out our core principles.

Years ago, Joel laid out some basic precepts about how he wanted to approach customer service. It’s the first thing I read after getting hired away from Google to grow out Fog Creek’s support team. I was the entire support team for 20 months. (Two guys shared the job before.) I buckled down and I squeezed every improvement I could out of the support workflow, which was easy when it was only me. There was no team communication overhead, and I could tweak the processes without getting buy-in from others.  I also was able to affect the development of the software such that the raw support load decreased, even though the customer base more than doubled. I took Joel’s precepts and built them into the DNA of the support team we ended up hiring, and that allowed us to get a long way.
 

The one precept I’ve built into our DNA more than any other is “Fix everything two ways”. Everyone says they fix the hard problems, but they don’t. I wanted to make it a reality.  We call it “Fix It Twice” and we use it in a lot of ways.
 

First, if we think a customer issue can be solved by a deeper change in the software, we always spin that idea off in to a secondary queue. The worst time to think about how to do a deep fix is right after you’ve solved the initial problem. The job demands empathy with the person having the problem, and that leads to a kind of recency bias. You’re always worked-up about the thing that just happened. What you do about that worked-up feeling makes all the difference. (Even recognizing this is helpful for fending off frustration in the support role.) If you just let it go, you end up with the feeling that “stuff never gets fixed.” If, however, you spawn off that potential change into a secondary queue, where it can be evaluated at another time when you’ve regained your distance and objectivity, you get a better sense of where resources might be best dedicated to reduce load overall. This, in itself, makes fix-it-twice worth all the trouble. You’ve taken frustration and turned it into business intelligence.

After that, we realized that the amount of stuff waiting for us in the fix-it-twice queue was a very good proxy for job satisfaction. If the numbers stayed low, that meant we were getting the slack to go and think about how to make our job easier and the software better.  If the numbers stayed low regularly, it meant we could code. That has led to real improvements in the software, like the Do Later plug-in. The support team has stayed happy and productive for a long time…. And then, the roof fell in. We lost one of our support reps to the design team. Well, actually, one of our support reps is a gifted designer and got poached (with our blessing).
 

When the team sat down to figure out what we wanted from our newest support rep, we realized something pretty momentous: The job had changed. It’s only now that I realize that building it into the DNA of our company has changed the company such that our needs are totally different. A responsive support team and an effective engineering team mean that the easy problems get solved forever and you’re left with mostly tough ones. We knew it’d been getting harder for years, felt it viscerally. The problems just kept getting more complex, less common, harder to diagnose. But now for the first time, that meant that the team had to change because our ecosystem had changed.
 

If your DNA says that you prey on slow, lumbering, delicious prey, eventually you’re going to find that those walking buffets are getting scarce. After that, you might find that you have to run a little faster to eat. You might also find that the plants your now-scarce prey fed on have changed and been supplanted by others. We find ourselves in a different world, and we had a hand in making it.
 

So, let’s call this the Kilneolithic epoch. The job posting still lays out our DNA. It’s still who we are. But it’s no longer the finely crafted trap it was before. Now, we’re looking for a different animal. The person we’re looking for is personable and curious and techie, just like always, but we need someone a bit more hardcore. We’re good at troubleshooting .NET and IIS.  But Kiln runs on OpenGrok, Apache, Redis, Mercurial. We need someone who’ll thrive in our new ecosystem, maybe even more than we do.
 

Recognizing that we’d changed our world was both gratifying and terrifying. We know we’re good at the base job, but it’s taken us to a place where we’re challenged. The team’s up for it, but we need one more body. If you’re up for it, email us at jobs@fogcreek.com. (Yes, this blog post was, itself, a finely crafted trap.)

The things we have been reading lately

Betrayed by an accelerometer

How to draw a circle

9 Metrics for making good decisions for your startup: From the Kissmetrics blog. This is one of the best blogs on marketing, startups, good web design, and more. It’s worth frequent visits.

A walking man, done in CSS3

Some “leaks” from the forthcoming Steve Jobs biography

How to sort file names: Raymond Chen proves that computer programmers are not normal human beings.

More, more, more: A nice riff on our constant push to do more and be more.

Our latest installment

First there was the Joel test, now there is the Rands test: Go read this now for the sake of your company!

An animated history of the iPhone’s evolution And a lot of other stuff too.

Dennis Ritchie

Paul Rand and Steve Jobs making the logo for NeXT: Look for the link to an interview where Jobs talks about Paul Rand.

A beautiful, interactive exploration of understanding and solving a mathematical problem

Buying lunch in lower Manhattan is not cheap. If you don’t brown-bag, you’re pretty much in for nine bucks, and more if you want to sit down. A meatball sub is $8.50. The Hanover Square Deli does a respectable dduk mandu guk for about $9.50. A meager-looking, but tasty, turkey and guacamole sandwich from British import Pret A Manger will set you back $6.75 (plus tax). Or, if you really want to eke out some savings, you can get a cup of soup and a hunk of bread for about $5.

At Fog Creek, we have our lunch catered every day.  The cost of this to the company is $15.75 per person per day. Not counting drinks and snacks, this amounts to an extra $4,000 per year per employee. So why wouldn’t we just pay people $4,000 a year more? If they want to be frugal, they can buy that $5 soup and pocket the difference.

Well, first off, since lunch is a catered on-site meeting, the cost of that lunch is 100% deductible as a business expense. If you worked here, it’d cost the company $16 to give you $10 to go buy your own mediocre udon noodles.  (Don’t take any taxability or corporate expense stuff here too seriously; I’m not an accountant or lawyer.) Second, Fog Creek’s free lunch is much more because of how it fits with the rest of our workspace and culture.

Free lunch is nothing new. Google and other big tech companies have been giving their employees breakfast and lunch (and sometimes dinner). Free lunch has been around for more than a decade because it works for attracting and retaining top talent. (Or, rather, people think it works.)

Now, the food at Google, and the people who make it, are awesome. The food at Fog Creek is good, but nowhere approaching what Google does. We don’t have Sam, the sushi chef, doing hand-rolls to order. We don’t do miso black cod, lamb shanks, or osso buco. We don’t have a raw vegan station with selections so delicious they attract the most dedicated carnivores.

Here’s what we have at Fog Creek instead: no meetings.

For us, lunch is our only recurring meeting. The only standing interruption in the day/week of a developer here. Everything else is ad hoc or temporary. (One exception is our quarterly meeting to go over financials and to grill the founders with questions.) The default at Fog Creek is no recurring meetings. Once you’ve established that, recurring meetings become the exception, rather than the rule, and tend to wither naturally as their usefulness degrades over time. For example, the FogBugz team is currently doing a stand-up meeting for fifteen minutes every day right before lunch, but this won’t last. The Trello team was doing them before and shortly after launch, but they’ve subsided.

The Kiln team lead assures me that their weekly stand-up is a real recurring meeting, but at some point they’re going to lose interest and go back to coding. It’s what they do. Meetings have a network effect. They need other meetings to legitimize them. If you’re constantly scheduled with meetings, you don’t mind being interrupted one more time. Or, rather, you don’t have the energy to protest. When you proceed from the assumption of no meetings, you have to expend effort to keep a meeting going.

It’s not all a playground, of course. There are downsides. Things, sometimes important things, don’t get communicated to the right people at the right time. More often than not, the worst result is hurt feelings or slight confusion. Sometimes it’s more than that. But we wouldn’t give it up.

For someone who’s used to a standard work environment, it seems silly, cult-like, possibly even daunting, to be “forced” to break bread with your colleagues every day. It seemed odd to me before I came here, but by the end of the first week, nothing seemed more natural. When most of your “socialization” with your colleagues takes the form of mandatory, recurring meetings over a conference table, it’s natural to not want to see them again over the lunch table. People have been making decisions about your time all day; at lunch, you need some time to yourself.

But when most of your time is spent working in a private office, taking breaks according to your natural attention span, having short chats with one or two colleagues, it’s a pleasant prospect to surface for some pleasant conversation about StarCraft or football with nice, intelligent people. And maybe you’ll hash out that new feature, too.

One can make the argument for free lunch based strictly on developer productivity. A free lunch could give you a hundred hours per year from your best people, time saved in driving, waiting in line, etc. But, consider the depressing ease with which such a gain can be wiped out by a few recurring meetings.

So, for our final post in this series, we do have a number: $15.75 per day. $4,000 per employee per year. It’s a lot of money. But without the rest of the company culture, it would be sort of meaningless. A sense of entitlement grows rapidly around any perk you offer, and lunch is no different.

A lot of this series has been based on getting hard data out there so that developers, our main audience, have an easier time talking to management about some of the things that’ll make them more happy, healthy, and productive. For this post, it’s a bit more difficult. You might get your boss to spend $16 a day, but changing the culture of meetings in any workplace is nearly impossible. (During my tenure, Google tried no-meeting Thursdays and formal meeting-reduction task forces, reminiscent of Brazil’s Ministry of Debureaucratization, to no avail.) It would require rebuilding the company culture from bedrock.

Our bedrock is the idea that, once we’ve hired good people, it’s the effort we make to direct their intention, rather than their attention, that creates value. It’s not just our lunch benefit that springs from that, but nearly every other thing we do.

For Fog Creek, our founding principles, and the pains we take to stick by them, are the price of developer happiness, and that can’t be measured in dollars.

Here is our latest installment.

Why code reviews beat testing: We knew we put code reviews in Kiln for a reason.

Geoffrey Moore talks about core vs. context in your business: This is fantastic talk from the 2009 Business of Software conference. If nothing else, it will make you think hard about your business.

The king of code comments?

The recent Node.js controversy

The always fascinating Richard Feynman: Three videos that use Feynman’s voice to talk about beauty, prizes, and curiosity.

Algorithm is not a four-letter word: An awesome overview of maze generation algorithms.

How to generate music with one line of code

Here’s our latest installment of things we’ve been reading.

Harvey Mudd gets more women: The President of Harvey Mudd college has managed to triple the number of female comp sci majors since 2006.

A proposal to fix callback hell in Node.js

Inbound Hiring: I think most startups do this, but here’s a short analysis on early hires from Gabriel Weinberg.

Why blog?: Also from Gabriel Weinberg. What he’s learned over the last year of committing to his blog.

Try 16 different languages in your browser