For most of my programming career, I’ve treated open-source software much like a slaughterhouse: I’m more than happy to use all of the wonderful things that open-source devs produce (Linux, jQuery, and WebKit, to name a few), but I don’t want to know what goes on behind the scenes. I’m sure it’s bloody and not nearly as pretty as what gets tagged and bundled each release.

Working on Kiln the past few years, I’ve become more and more familiar with one open-source project in particular. Kiln is built on top of Mercurial, using both its high-level and low-level interfaces. We’re proud to run one of the largest installations of Mercurial in the world with our Kiln On Demand service. From our users and from our own experience, we’ve come to know Mercurial’s strengths and its weaknesses, and because Mercurial is open source, we can also submit patches to help fix them. Over the past two years, we’ve watched Mercurial release several new versions, each with a patch or two from our team.

Of course, with our engineering resources, we can do more than just submit a patch every now and then. Each year, the Mercurial core team assembles for a code sprint. This past spring, Benjamin asked me if I would like to attend this year’s sprint in Copenhagen, to get my hands dirty on an awesome open-source project. Count me in! Of course, there’s no such thing as a free lunch, even with a generous employer like Fog Creek: Benjamin sent me to actually get stuff done, not just to enjoy the weather.

But what could a lowly coder like me do? I started by getting some patches into Mercurial. The project’s backlog of bugs and small feature requests is seemingly bottomless, giving lots of opportunities to get rid of little problems and paper-cuts. I was able to commit patches to add Git-style parent revision references to Mercurial’s revsets, and to clean up some other gnarly parts of the code base. I also worked on improving Mercurial’s handling of the vagaries of different OSes, especially the differences between Windows and the various unixes. Every OS has its share of reserved filenames, from OS X’s .DS_Store to Windows’s CON and AUX. If you add a file to a repository that’s reserved on a different OS, you won’t be able to check out the repository on that OS—and that’s no good. Additionally, two filenames that differ only in case (like foo and FOO) are allowed on some filesystems, but not others—even within a single OS. Mercurial now helps protect you from yourself, warning you when your changes would cause problems on other computers.

You mean, Etherpad exists in the real world, too?

Now, if you take a bunch of geeks who are used to working alone at their screens, and put them all in a big room to work together, how do you think we’ll collaborate? Alone and at our screens, using Titanpad and email, of course! Traffic on the dev mailing list went way up as patches, feedback, and bug reports bounced around. But the sprint really shined when we started working together to plan and design new features that will make Mercurial more powerful and flexible.

The Mercurial project isn’t as big as some open-source projects, but it has more than six years of history and hundreds of committers. Dozens of them are actively and passionately involved in the project, and most of those were at the sprint, so when it came time to plan the direction of the project as a whole, there were plenty of well-argued opinions and ideas.

The first big feature planning was for filesets—a query language for finding and specifying files in your repository, much like how revsets work for finding changesets. We all collaborated on creating a set of predicates that would make the language useful. Being able to quickly go back and forth about ideas in person made the brainstorming process much smoother than if we were scattered around the globe.

The second big brainstorming question was developing the interface and implementation for a brand-new feature, called “dead changesets”. In a large project, you can end up with changesets, or even whole branches, that are dead ends, ideas that didn’t pan out or features that were cancelled. While having these around for posterity can be helpful, they can also just be clutter in your day-to-day work.

As with any new software feature, there were questions ranging from UI to implementation details to backwards compatibility that needed to be answered. Over a couple hours of aggressive whiteboarding, we worked out a plan that seems to fit the bill: when you mark a set of changesets as “dead”, they’ll still be around (in case you ever need to reference that pie-in-the-sky refactoring), but they won’t be displayed or transferred unless you explicitly request or resurrect them. The feature didn’t make the cut for 1.9, but even so, the team now has a head start on getting this done for another release.

Finally, we discussed bringing one of Kiln’s major features—kbfiles—into Mercurial as an official feature. kbfiles enables you to track the history of large files (like images, libraries, and executables) in your Mercurial repository, without keeping a copy of every version of every file on your computer. We ship kbfiles as part of Kiln, but we believe that it is valuable for every Mercurial user. Over the next few months, we’re going to tie up some loose ends, so that kbfiles works just as well with vanilla Mercurial as it does with Kiln, and make it available as an official extension. You can even follow along in our public Kiln installation as we work on that.

Perhaps most importantly, I had the chance to meet a large part of the Mercurial dev team, including the project lead, Matt Mackall. For a couple months, I had known the team only via IRC and email, so putting names to faces, working side by side, and even simply sharing meals humanized the whole process. Working together is faster, easier, and just plain more fun when you’re all in the same room (even when you’re still interacting virtually), as opposed the usual not-being-on-the-same-continent situation. Thanks to the whole team for being helpful and friendly to this n00b.

Mercurial 1.9 was released earlier this month, with my patches, mpm’s implementation of filesets, and a host of new features, bug fixes, and general awesomeness from the whole team. Mercurial is now faster, easier to use, and more powerful than the last version—an amazing accomplishment for any software product, but especially for one developed by a team as large and diverse as Mercurial’s.

This code sprint was the first time I’d been deeply involved in the inner workings of an open-source project. I’m happy to say that the process wasn’t nearly as bloody as I’d feared. I’m proud to be a part of the project, even if it’s in a small way.

It’s intern season; at least it is here at Fog Creek.

A fresh batch of talented and eager interns started yesterday, and although they missed the fiercely contested Fog Creek/Stack Overflow ping pong tournament, they are already working on important and top secret projects. We’ve got exciting stuff planned, and if all goes well, we’ll be showing off some of it before the leaves start to turn colors, and some of it even sooner than that!

The internship program has been going each summer since the early days of Fog Creek and it’s one of the ways we identify talented new people to come work here. They get a couple of months to see if we’re as great as people say we are, and we get to see them in action, hacking away and showing their chops on real shipping code.

We’ve made the internship program better over time, and Ben Pollack, one of the creators of Kiln, and frequent intern mentor, has written a guide on how to make internships great that covers what he (and we) learned about giving interns a great experience.

Interested in following along or suggesting topics for the conversation?

• Keep an eye on Building Great Software Companies, a collection of excellent articles on running your software company which we add to when we have good new stuff!.

Editor’s Note: Patrick McKenzie has done some work with Fog Creek last October and this May, focusing on improving our marketing.  We invited him to write a guest post about what he has done and where we see ourselves going, so that other software companies can learn from our experiences.

Fog Creek is an enormously successful software company which, for the last ten years, has made a name for itself by producing wonderful software and being a great place to work.  It’s so famous for that that it never had a marketing strategy other than being Fog Creek.  This served it very well for a long time, but it didn’t look like it could continue forever, so they brought me here last year to improve their website, SEO, conversion funnels, and the like.  They’ve generously allowed me to share some of what we have done.

This is the first post in a continuing series, in which the Creekers and I will be examining where we were, what we did, and what sort of results we have achieved on the marketing side of things.  We hope other software companies will be able to learn from our experiences.  To that end, we’re happy to announce Building Great Software Companies, a collection of essays, reports, videos, and tutorials from Fog Creek staff where we share what we’ve learned over the last decade.  Just like sharing open source software enriches the entire community, sharing knowledge of the non-technical aspects of running a software business helps us all.  We’re happy to be in a position where we can give back.

Step 1: Taking A Fresh Look At An Old Site

Fog Creek has a very democratic workplace environment, where changes are largely made by consensus and engineers have responsibility for areas of individual ownership. Someone owns FogBugz’s internal search. Someone owns Kiln’s Mercurial integration. Someone owns getting catered lunches organized every day for 30 ravenous Creekers and the folks working at StackExchange.

No one owned the website.  As long as it kept running, it was no one’s job to monitor, measure, or improve it.  This resulted in the team tending to update the website about once every release cycle — typically, at the yearly launch of the new version of FogBugz, Fog Creek’s flagship software product.

So I spent a few days comprehensively reviewing the website.  We got the whole company together and stepped through the site, one page at a time.  For many, it was the first time seeing some of the pages which were critical to making the rent every month.  And the pages were… substantially less compelling than the software the team at Fog Creek spends so much effort polishing, in a way that was inhibiting users from actually getting to use that software.

Let’s start with the front FogBugz page.  Think quick: you’re here to see if FogBugz meets your software team’s bug tracking needs.  What do you do?

And with some annotations:

Users had to fight the website to get into the 45 day free trial of FogBugz.  Many did not even know FogBugz had a free trial, because you could read pages (and pages, and pages) of text on the site before first encountering an invitation to try it out.

The most clicked on element of the page, by far, was the video.  It didn’t do a great job of convincing people to try the free trial — they ended up abandoned on a page festooned with YouTube branding.  The second most clicked on element, the yellow “FogBugz 7 Now Available” button, probably struck many users as leading to a free trial.  Nope.  It lead to a page of descriptions of new features in FogBugz 7.

How did this happen?

Our Main Product Took Over Our Marketing Even When It Wasn’t Our Main Product Anymore

Fog Creek is a company by engineers, of engineers, for engineers.  Their engineers ship great software.

FogBugz is historically a product which one downloads and hosts on one’s own server.  This method of distribution dominates much of how life at Fog Creek works:

• Why do we have a yearly release cycle? Because after we ship a new version we need it to be installed on customers production systems, and frequent updates make them very unhappy with us, so we make sure it gets done absolutely right the first time.
• Why does our software even have a version number? Because it ships yearly, and you need to know to buy the next version, or we can’t afford lunches for everyone and an office next to the New York Stock Exchange.
• Why do we do one big marketing splash a year? Because marketing happens around the yearly release.  We spend 90% of our energies creating software for that release date — clearly, when that gets exposed to the public, that is the value we are creating and that is when we should do marketing.
• Why does the website only change once a year? Because why would we change it if we didn’t have a big marketing splash?

Some years ago, Fog Creek released a new version of FogBugz: FogBugz On-Demand (hereafter FBOD), which is a hosted software-as-a-service.  This was a huge, huge win for Fog Creek:

• Sales are up, significantly: FBOD opens up new markets for bug tracking among companies which couldn’t or didn’t want to administer their own FogBugz servers.
• Support costs for FBOD are an order of magnitude lower than installable FogBugz, largely because the environment is predictable and under Fog Creek’s control.
• FBOD supports a much faster cycle of iteration than installable FogBugz does.  Changing the downloadable product takes a year, but changing FBOD takes only about a week.

Five years ago, Fog Creek was a successful software company.  Today, Fog Creek is a successful SaaS company with a sideline in selling downloadable software.  This changed the company, but the company didn’t change with it.

For example, like many companies, Fog Creek’s internal deployment practices grew organically, and the scripts for building and deploying the website have gradually grown to do quite a bit more than that.  As of last October, it took 45 minutes to make even the simplest possible website deploy.  Change a word?  Check it into version control, click a button, and that will be live on the Internet in 45 minutes.

This level of friction made changing the website a huge ordeal, so it was only done when it couldn’t possibly be avoided, at the yearly launch.  Had there been anyone who wanted to e.g. A/B test headlines against each other, doing so would have eaten up an entire morning just to see that the change had made it to the live site properly.  As a result, this never got done.

One of the things I did in October was convince an engineer here to take the website deployment process under their wings.  It now takes six minutes, which is about 355 seconds longer than I’d like, but the friction is low enough that more has been done to the website in the last six months than in the last ten years.

Redesigning Focused On Conversion

In December, the team totally overhauled Fog Creek’s web presence, both making the site’s aesthetic design a little more modern and aggressively reworking each page.  The redesign started with mockups where I demonstrated techniques for encouraging conversion, such as giving people large, obvious buttons to click on to start their trials:

As you can see:

• The page now includes a prominent call to action for people to sign up for a FogBugz trial.
• We reworked it to include a little more Internet formatting, like bulleted lists, because nobody reads on the Internet.

The team, for the first time including dedicated designers, iterated a bit and came up with this:

In addition to a few tweaks we made for SEO purposes, which we’ll discuss in detail later in this series, we made it much more obvious how to get into the trial.  That is a frequent subject for A/B testing, a practice the team has adopted in earnest after being shown how fun, easy, and ridiculously profitable it can be.

In Which Stuff Gets Measured

One of the great results of our work together is that the team now actually measures the performance of the website, so that we know whether the improvements we are making actually do anything for Fog Creek. For example, the old website was not well-instrumented and it was not obvious to the team that e.g. placing something low on the page would decrease the number of people who clicked it.

Enter CrazyEgg, which lets you make heat maps of where people interact with a given page.  The hotter the color, the more clicks a particular area is getting.  Here, for example, we have one of the several iterations the site has been through.

You’ll notice that the video gets a lot of clicks.  That video is an hour long explanation of FogBugz by Joel Spolsky, the CEO of Fog Creek.  It ROFLstomps everything else we’ve tried there.  We tried short videos, we tried screencasts, we tried images — in A/B test after A/B test, our users said Give Us More Joel.

One thing which might not be obvious about the video is that, in A/B tests, putting that big honking Play button on top of it doubles the number of people who actually start watching versus more understated Play buttons.

You can also see that the See Pricing link attracts more people than the free trial button, possibly because the free trial isn’t identified as being free.  (And literally right after writing that line I pushed an A/B test to see whether that matters or not.  Anything is possible when your speed of iteration isn’t rate-limited by artificial technical factors.)  Previous tests on that button have played with the color, placement, and wording — it turns out that my favorite shade of orange didn’t really help that much, but adding the word “Now” almost doubled the number of folks who clicked it in one test.

And, sure enough, this trickles straight down to the bottom line.  Compared to the same period last year, FogBugz is getting:

• More trials.
• Better converting trials.
• More seats sold from converting trials.
• More revenue.

This is a wonderful and terrible thing to hear as an engineer.  Historically, Fog Creek has believed to its bones that new features are what moves the company forward.  That worked very well for many years, but recently it hasn’t been the slam dunk it has been in the best.  On the other hand, optimization has made a meaningful difference to their financials already, and the benefits keep compounding every month, as more trials age into paying accounts and as happy, satisfied customers continue paying lots of money for their services.

Pricing Pages That Sell

FogBugz exists to help software developers write great software, but it also exists to pay the rent.  This means that we have to actually consummate transactions on our website.  This was previously delegated to the curiously named “Details” page.  It had been designed primarily to explain to customers the difference between FBOD and the downloadable version of FogBugz.  Sadly, this ended up being quite confusing.

This is a page ostensibly about the user, but it was really about Fog Creek: we need to explain to you, at length, the difference between FBOD and hosted FogBugz because a) we need to know which totally different purchasing funnel to send you into and b) this division is core to how we think of everything.  From the user’s perspective, though, this puts a really hard decision in front of getting the pricing for FogBugz.  Do I want FogBugz, which apparently runs on recent versions of Debian, whatever that means?  (The person making the decision is often a manager and not necessarily the most technical person in the organization.)  Or do I want “professionally hosted FogBugz”?

The page also does a lackluster job of selling FogBugz.  For example, buried in the links on the right rail are customer testimonials.  FogBugz is well-loved by prominent customers and 99.8% of the people visiting this page never learned that because that information is in web-design Siberia.

Plus, the core user question “So how much does this cost me?!” is buried here.  It takes you a few more clicks and quote forms to get to the right answer.  At our meeting last October I asked everyone how many clicks they thought it took to find out how much FogBugz costs, starting from the homepage.  Many thought “two”, there was a plurality for “four”, and nobody guessed the actual answer — eight clicks to fight one’s way to the quote form and I couldn’t figure out how to do it despite being paid to try buying their software.

We’ve iterated a bit on this, and the new version is radically better.

This:

• Defaults the user into selecting FBOD.  If they don’t know they want the server edition, assume they don’t want it.  (We have plenty of data and customer conversations to make this assumption.)  This means we can surface the price immediately.  As a bonus, $25 / user / month sounds cheaper than $4,453.80, which would be a fairly typical license purchase.
• Cross-sells Kiln to users interested in FogBugz.  Kiln is Fog Creek’s other main product to make software developer’s lives better — it makes code reviews a breeze and makes distributed version control (specifically, mercurial) usable for developers who would otherwise be stuck with inferior source control tools.  This page greatly increased the attach rate of Kiln to new FogBugz accounts: why not get a whole new service for just $5 more per month?
• Includes social proof, via icons of some of our prominent, satisfied customers.  The icons whisper “Companies you trust for engineering excellence use FogBugz, because we’re the best option.”
• Includes our money-back guarantee, which further decreases perceived risk in the purchase.
• Answers many common customer questions at the point where they are likely to come up.
• Makes it very easy to buy the software.

There’s Always More To Do

The biggest change from our work together has been getting Fog Creek onboard with a simple proposition: the website is a shipping software product of the company.  It deserves the same level of attention to detail, monitoring, focus, and internal resources that Fog Creek’s other software products get.  It has now moved from a detail of the yearly release focus to a known target for ongoing improvement, and the Creekers who are leading that charge are continuing to make major progress on it.

Revamping the website was only one facet of how we’re pushing forward on marketing.  We’re going to talk later about SEO, email marketing, other enhancements we’ve made to our site and products, and a variety of other topics.  Let us know what would help you.

Interested in following along or suggesting topics for the conversation?

• Comment about this post on your blog — we’ll find you.  (We also have easily available inboxes and are active on your friendly local Twitter @fogbugz and our individual addresses.)
• Sign up for our RSS feed or our developer newsletter, where we’ll announce new articles you might be interested in.
• Keep an eye on Building Great Software Companies, which we intend to continue improving over the coming months.

When we launched Kiln, I knew we were solving a bunch of major problems that users actively had. I knew that because every time that I talked to someone, and walked them through what Kiln was capable of, they got really excited and asked piles of questions of how they could integrate Kiln into their current workflow.

But there was one question that just kept coming up time and time again:

How do you version databases?

I really empathize with this problem, because it impacts every single web application I write. It’s easy to version your code, to version your documentation, to version your images, to version your deployment scripts. But versioning the database schema gets hard. For really big systems, you end up needing piles of migration tools, but for most stuff, that’s overkill. You just need a simple way of tracking and applying changes to your schema. Using version control to store your migration scripts ends up being this weird version controlling your version control.

The great folks at Red Gate actually already came up with a good solution to this problem, called SQL Source Control, which allows you to version your schema, and even optionally your data, right inside your existing source control system. The only problem is that, up to now, it only worked with traditional SCMs.

The key words being: up to now.

Red Gate and Fog Creek are both happy to announce that SQL Source Control 2.1 now includes full support for Mercurial, meaning you can now trivially version your database schema in Kiln. And because SQL Source Control ultimately just works with the SQL statements that generate your database, it’s easy to make sense of the diffs from within Kiln. You’ll finally be able to bring the power of DVCS and Kiln to your database.

If you’d like to try it out, there’s a 28-day free trial available that will let you play with SQL Source Control and see how it integrates into your workflow.

Few people know this, but Kiln started life as nothing but a review system.

Tyler and I both wanted to have a code review system to help us at Fog Creek, and disliked all of the available ones, so we wrote one up that used Mercurial’s branching and merging tools to have a saner workflow for fun as part of a competition. To make that work, we had to give Kiln basic repository hosting abilities. After we brought Kiln in-house, we drastically built up the project hosting capabilities and redefined the product as a complete distributed source control system, but the review system, in more-or-less its original form, stayed with the product as a major feature.

Unfortunately, the review system we came up with was based on FogBugz, and while that system worked great for Fog Creek, it simply didn’t fit a lot of teams very well. Kiln reviews, like FogBugz cases, could be assigned to only one person at a time. This worked great for making sure that a single person was always responsible for a review, but ended up getting in the way of the inherently collaborative nature of improving source code: each developer’s thoughts on the code would end up in its own code review. While they were linked, assembling a coherent picture of all comments on a given piece of code was much harder than it should’ve been.

So we went back to the drawing board, and for Kiln 2.4, came out with an entirely new way to do reviews.

Beginning today, Kiln reviews can have many reviewers, and reviewers can be added or removed at any time. Each reviewer can independently indicate that they approve or reject the changes and comment on the code.

If all reviewers reach a consensus, the review is automatically marked “approved”, and the review opener is notified that the changes have been collectively approved or rejected. If the reviewers cannot reach a consensus, the requester can leave the review open until a consensus can be reached, or can acknowledge the stalemate and close the review with a brand-new status of Abandoned.

This posed a slight issue for us: one of the best things about Kiln is that it integrates extremely tightly with FogBugz, meaning you can see reviews you’re responsible for right alongside your bugs and feature requests. We didn’t want to give up that ability with the new system. So what we did was add a new filter option to FogBugz, “My Code Reviews”, and a new search axis, reviewedby:me, so that you can easily find the reviews that are waiting for your feedback.

We feel that this new system is the best of both worlds:

1. Reviews now have lots of reviewers, so conversation all takes place in a single case–not a collection of related cases.
2. At the same time, everyone is still fully responsible for actually weighing in. While you can abstain from reviewing, that gets logged, and Kiln resists closing reviews before assignees have made a decision and recorded it.
3. And we’ve done this without losing our tight integration with FogBugz.

The new system ships out today to licensed customers, and has been available on Kiln On Demand since late last week. If you haven’t yet had a chance to give Kiln a spin, now’s a great time to sign up for a free Kiln trial so you can play with the new review system first-hand. We think you’ll like what you see.

One of the things that I mentioned when I discussed the importance of having a mission is that the Kiln team’s real purpose in life is to bring the awesomeness of distributed source control to as many people as possible. While we do a lot of that by making Kiln into the best source code management system available, we’re hardly doing it alone: Kiln builds off the amazing base provided by the Mercurial distributed version control system. Mercurial helps make Kiln what it is. For Kiln to really achieve what we want it to achieve, we need to help Mercurial itself shine, too.

This year, at Fog Creek, we’re doing that in two different ways:

1. First, we’re helping sponsor the Mercurial 1.9 code sprint. The code sprint is where the Mercurial developers have a chance to talk face-to-face to solve outstanding issues, agree on how to implement major new features, fix piles of bugs, and so on.
2. Second, we’ll be sending one of our own Kiln developers so that we can contribute some actual manpower to the bug fixing and feature development that goes on at the sprint.

This kind of paying back to open-source software is becoming increasingly common. Even when StackOverflow was brand-new, Jeff and Joel were already donating to support the projects that made StackOverflow possible, and they’ve kept up that tradition as StackOverflow has grown. I think this is a great way to pay back the open-source community, and to help further the creation of more great open-source tools.

If you’re interesting in helping the Mercurial project, the easiest way to help them out is to donate to the project or the sprint.

When we initially added Web Hooks to Kiln, we were excited: they made integrating Kiln with the outside world trivial.  And we’ve seen a lot of uses of Web Hooks: we’ve used them internally to power our continuous integration system, Mortar, and to integrate pushes into HipChat to keep our teams aware of what’s going on in our repositories.  And we’ve heard of other customers getting their own CI systems integrated, and getting integrated with their own chat systems.  And some of our customers who had continuous integration systems or corporate chat environments used Web Hooks to integrate Kiln into…

…er, wait.  Something’s not right here.  I seem to be repeating myself and then saying the same thing again.

While Web Hooks’ flexibility is great, and we’ve used them for some novel things in the past (such as covering a dev’s entire screen with Growl-powered push notifications), most Kiln users just want a quick way to tie Kiln into their existing infrastructure; the flexibility of Web Hooks is overkill for that.

So we’re proud to introduce prefabricated Web Hooks in Kiln for Campfire, HipChat, TeamCity, and Pivotal Tracker.

Team Chat

The first two are very obvious: Kiln can post updates on what’s going on with your source code right into your chat environment, making it really easy to keep your support staff up-to-date on what you’re up to.

Continuous Integration

Next up, a lot of our users wanted integration with a continuous integration system, and by “a continuous integration system”, most of our customers meant “TeamCity”.  So now, Kiln can talk directly to TeamCity: just give it your TeamCity URL, and Kiln will automatically let you choose which build configuration to notify and for which repositories.  Easy as pie.

Pivotal Tracker

Finally, one of the common questions we get is, “Can I use Kiln without FogBugz?”  The answer has always been “yes,” but a lot of people are really asking, “Can I integrate Kiln with other bug trackers?”  As of now, the answer is yes: Kiln now directly supports Pivotal Tracker.

Just tell Kiln your Pivotal API token, then use your normal Pivotal commit syntax, and Kiln will add links to your changesets to the relevant stories.

These are just the first couple baked-in web hooks we’re providing.  We’d love to hear about any others you’d like us to add.  Simply let us know at our StackExchange.

The problem started rather innocently.

Solution

How did we do?

Note: This post has been temporarily removed because we discovered some errors in the survey methodology which don't reflect our usual standards. -Joel Spolsky

If you've reached this page, you're interested in the topic. We'd love you to take our survey on the state of version control so we can check our data against another (admittedly self-selecting) sample. Please start here: https://www.survs.com/survey/JEOWZO8N5W

"Just as with the beta release, it’s helpful to roll out the final release to our hosted customers in stages. We call this “leaking the release,” which sounds kind of gross, but I like to think of it like sugar water leaking out of a bird feeder rather than, …well, you can come up with an appropriately negative metaphor yourself. The birds love it. As before, this gradual process helps us to catch issues before they have a chance of affecting everyone. Now that we’re dealing with 2-3 orders of magnitude more customers, it’s much more likely that we’ll run into performance problems with the new code. One serendipitous benefit of this type of release was that buzz about the new versions progressively increased over the course of the month leading up to the full rollout."

Read more about how FogBugz 8 and Kiln 2 were released to customers.