Archive for the ‘Uncategorized’ Category

Remote Developer Happiness

March 31st, 2014 by Ben McCormack

We’ve written before about a few of the perks of working in the Fog Creek New York office, but what if you work remotely? As it turns out, there are some nice perks to working remotely as well, even cake on your birthday. For your home office, you get everything you would normally get to help you get work done, so you don’t have to spend time thinking, “Gosh, it sure would be nice to have an extra monitor mounted vertically to tail production logs.” Just get it.

My Steelcase Series 7 height-adjustable desk was just installed last week, so I made a video to show it off:

Here’s a list of some of the inventory on, around, and underneath my desk:

I spent a fair amount of time figuring out how to hide cords and arrange things so the desk looks clean. Having extra space on the desk just to have extra space is actually kinda nice. I used to clutter it with trinkets, but those are now on a shelf somewhere. (Want to hide even more cords? You might check out something like this).

Do you want to work remotely for Fog Creek? We’re hiring.

Project Asteroid: Gracefully Dropping Support for Dinosaur Browsers in Trello

March 25th, 2014 by Bobby Grace

According to the leading theory, dinosaurs went extinct roughly 65.5 million years ago after an asteroid landed on Earth. A few million years later you are working on a large scale web application and think to yourself, “Hey. Would it be such a stretch to compare the biodiversity of the Triassic period a.k.a. dinosaur times to that of today’s browser ecosystem?” No, I don’t think it would be. There are a lot of browsers out there and they all look and behave a little bit differently. There are some new browsers that are fast and have all kinds of great features and there are some old, slow dinosaur browsers made for a bygone time.

You are getting more than a little frustrated trying to support all these browsers. You’re looking at all the things you could be doing in these new browsers and it’s becoming clear that those big old dinosaurs are weighing you down. Of course, you could just decide to send people using these old browsers to an “Upgrade your dang browser are you kidding me?!” page one day, but that would be a cataclysmic event that would be sure to upset many, many people. What if there was a friendlier, more graceful way to transition these dinosaurs into the modern age?

Browser Usage on


This is what browser usage on looks like for the last two months or so. We built Trello from the ground up to work on just about any device. The only caveat was that we promised to support the browsers of today and tomorrow. Some of the browsers we supported at launch just can’t be considered modern by today’s standards. I’m talking about Internet Explorer 9 specifically. IE9 was released three years ago in March 2011 and was the most current version of IE at the launch of Trello. But three years ago might as well be 100 million years ago in Internet Time. I mean, three years ago? Doge was just someone’s dog three years ago.

Sure IE9 is old, but more importantly it’s hard to support. New Trello features take longer to develop because we have to bend over backwards to support IE9. Things tend to look wildly different in IE9. Open up the new attachment viewer in IE9 and it’s like a cubist painting gone wrong. IE9 doesn’t have all the new HTML and CSS features like the new browsers. To top it all off, it’s real slow. It takes forever to process JavaScript and make pixels show up. It’s like a kid running and screaming and knocking down all the boxes in the cereal aisle.

We’ve been hammering the site into a passable state for IE9, but it’s still a poor experience for the good-looking Trello crowd who we hold in the highest regard. So we would like to not have to support it, but at Trello-scale, 1.4% of visitors is not a number we can ignore. What do we do?

Enter Project Asteroid

Remember when you read How We Make Trello and your whole worldview was flipped upside down by the mere mention that there are multiple versions of the website out there at any given time? Remember how the website is just a face that chats with the Trello API and that the iOS and Android apps are also just faces and you can make your own face, too? Remember the dangerous potato faces? Remember when I said the API didn’t have any bugs and you thought I was being serious?

Well there’s a special face out there for people using Internet Explorer 9. We call it Asteroid. Get it? Like the thing that killed the dinosaurs. This is what Trello looks like in IE9 today.


This is an old but completely functional version of the site. It doesn’t receive any updates or new features, just the occasional fix. It’s slightly customized. There’s a banner in the header that points you to the platforms page where you can find download links for all the great, modern browsers out there; browsers that are getting all kinds of cool stuff like a new card back and attachment viewer. But you can use Trello in IE9 all the same, perhaps while you are on hold with your work’s IT person trying to convince them to upgrade your browser.

In the end, everyone lives happily ever after. The dinosaur browsers peacefully roam in a land before time, while the modern browsers harness nuclear fusion and terraform Mars millions of years in the future. The dinosaur browsers will fade away eventually, but there’s no need to rid the Earth of them in an instant.

How do we know what face you should be seeing?

Let’s talk about what makes a web app first. A version of the Trello website is just a collection of files: a JavaScript file, a CSS file, some images, and maybe a few others. These files contain all the instructions necessary to make the site look and function in just the right way in your browser. The problem is that these instructions are not understood in the same way by every browser. The newer browsers get the instructions and say “Yeah cool I’m on it” while the dinosaur browsers look at you all confused then go off to find some leaves to munch on. This is annoying and is the reason we got into this mess in the first place.

What happens when we release a new client version? Let’s say the files collectively known as jellydonut-15 are broken in some way. To push a fix, we create a whole new collection of files, jellydonut-16, and you download the latest collection when you refresh or open a new Trello tab. These collections of files never change. jellydonut-15, jellydonut-16 and all the versions of Trello are all stored in a big ol’ file cabinet in the sky called a Content Delivery Network.

When you type in your browser’s address bar and hit enter, you make a request to the Trello server. The server takes the request and figures out who you are and which collection of files you get. The server says to itself, “Ooooh. A new request. Okay, this looks like Bobby. Let’s look him up in the database. Says here that he is on the alpha channel. The alpha channel is using the latest jellydonut version, which is… lemme thumb through this rolodex… jellydonut-22. Okay, I’ll make sure his browser pulls the jellydonut-22 files from that big ol’ file cabinet in the sky called a Content Delivery Network. Hey, when is lunch? *clutches power unit* ‘Ugh, you know what, I’m not hungry. I had some leaky AA batteries last night.’ Hahaha! A computer using AA batteries? That’s a good one. I’m going to tell that joke at the pub when I’m cycled out of production for the night. Oh look, another request…”

Now that you have your face files, you can start chatting up the API which will dutifully get the boards and cards your face asks for. Every face, new and old, talks to the one, single Trello API. I can personally verify that the API does not have a single bug. It’s important to note that the fact that we are cramming dozens of jelly donuts into a file cabinet and the fact that the hungry server knows which face you get are facts that are totally independent of the way the API operates. We don’t need to change the API for every new face. But let’s say jellydonut-17 wants to chat up the API in some new way. We update the API before jellydonut-17 is released so the API can understand what your face is talking about. Usually this new thing is an addition to the API, so it’s backwards compatible. Old faces simply continue to not ask for things they never knew to ask for.

Back to my point. So, in addition to knowing who you are, the server also knows what browser you are using based on the request. If it sees that you are using Internet Explorer 9, it gives you the custom asteroid files, no matter what channel you are on. These files include very simple instructions that won’t upset or confuse the dinosaur browsers and make them go off and eat people hiding in public restrooms. Then you, the person using IE9, see that there are new great things on the horizon if only you would upgraded your browser. Which you do. Hopefully. Right?

So is it working?

The short answer is yes. The asteroid appears to be having a… deep impact.

We launched asteroid on March 6th, 2014. Let’s compare Internet Explorer usage between January 13th–17th, 2014 and February 10th–14th, 2014, a time before asteroid. It goes up across the board for all the versions of IE. IE11 was up 16%, IE10 was up 3%, and IE9 was up 2%.


Now let’s compare February 10th–14th, 2014 to March 10th–14th, 2014, before and after asteroid. IE11 jumps up 23%, IE10 holds steady around 2%, and IE9 drops 8.5%.


For every week we compared before asteroid, IE9 usage was going up. In general, usage goes up for every browser. Now IE9 usage is decreasing. It’s easy to attribute the decreased usage of IE9 to asteroid, but it’s impossible to tell if people are switching to another browser or what. There was an especially big spike in IE11 usage, but we don’t know if that’s directly related. Usage is constantly fluctuating across all browsers.

But it’s good news at any rate. Great features are being developed faster and people can still use Trello on IE9. We can theoretically support this version forever, but we may pull it from production when IE9 usage becomes negligible. At that point, it should be a pretty soft landing. Another nice thing is that we can reuse this strategy when all the Trello developers are shaking their fists at some other browser that is looking pretty old (IE10).

That’s all. Please yell at me on Twitter at @bobbygrace and follow @trello for the latest. Of course, give Trello a shot. It will keep you and your team more organized. Guaranteed.

And please keep your browser up to date.

How We Make Trello

March 13th, 2014 by Bobby Grace

Look around you. Is there something nearby that is firmly bolted to the ground? Okay good. Hold onto it. I’m about to pull back the curtains on everything you thought you knew about how Trello was made. The revelation could result in a rippling shockwave that knocks you off your seat and may have troubling, unpredictable consequences for the time-space continuum. Possibly.

To understand how Trello is made, you have to understand a little about how it works. When you go to in your browser, you are basically running an application–called a client–that talks to the Trello API. The API is a thing that goes to the Trello database and returns with the boards and cards you asked for. You tell the API, “Hello, I’m Bobby. If you don’t mind I would like to see my boards whenever it’s convenient” and in mere milliseconds the API responds with “{user: bobbygrace, boards: ['44321234432', '44321234433']}”. The client takes that and turns it into the beautiful, human-readable information you see on your screen.

You’ve probably downloaded our iOS, Android, Kindle, and/or Windows 8 apps, and are saying to yourself, “These are very well polished apps which I have rated or will rate favorably in their respective app stores. Are they just clients that talk to the Trello API?” Yes. They are just clients. There are clients everywhere. In fact, the Trello API is public and if you know how to speak Trello API, you can make a client, too.

The Trello API is well-written, has no bugs, and is totally rock solid. Or at least it doesn’t change very often. That means we can put out new clients all the time without having to update the API. In fact, we can have multiple versions of the website out at any given time.

“This is pretty interesting and I’m sure it will be relevant later in this post, but I thought this was about how you make Trello.” Okay, okay. I’ll explain our process first and then we’ll come back to this.

Using Trello for Trello

It should come as no surprise that we use Trello to make Trello. We have a board called “Trello Internal” which includes all the stuff related to the API and website that we’re currently working on. Our lists look something like this: “Doing”, “Waiting for test/review”, “Ready for merge”, then a bunch of lists for recent releases, “Incoming Bugs”.


So, let’s say a developer wants to fix a bug. That process goes like this:

  • They pick a card from the “Incoming Bugs” list, assign themselves, and move it to the “Doing” list.
  • They pound on their keyboards like wild primates until it appears to be fixed. (Maybe this is just me.)
  • They create a new branch on their repository, like bobby/oops-no-crash, and push it to Kiln, a top-of-the-line distributed version control and code review system. (Disclaimer: we also make Kiln.) I’m not going to dig too much into how version control works, but basically a branch is a new version based off of a stable, established version that is in production. Everyone who knows how version control works is shrugging and grimacing right now.
  • They put the name of the branch in the card description, like “fixed in bobby/oops-no-crash”, and move it to the “Waiting for test/review” list.
  • Simultaneously, they get their changes reviewed by another developer in Kiln (which you should buy) and notify our intrepid tester Ali T. who checks out that branch and makes sure it’s really fixed.
  • Once it gets four thumbs up, the card is moved to the ”Ready for merge” list.
  • Doug Patti, our release manager, merges the changes into the official Build repo, makes the version with the change the new stable version, cuts a release, and pushes it into the wild. We often joke about how using the Kinect would make this step more exciting. Doug could just point in a different directions and say “Deploy! Deploy! Deploy!” Shamefully we haven’t made this a priority.
  • The fix is out. Bottles of champagne are popped.

Now another developer can pull from the official Build repository, which includes the fix, and merge it into their working branch. Then their changes can go through the above process and everything is real smooth like a good, old-fashioned saxophone solo. We push out fixes many times a day.

I’m sure everything I have said so far sounds reasonable, but bug fixes are small potatoes, right? What about big, new, dangerous potato features that you might want to test out first? Something like the new card back?

How we make and test big, new, dangerous potato features.

Remember how the Build repository contains the stable, established version of Trello? Well actually, there are many stable, established versions of Trello and you might be running any one of them. What you see is not necessarily what everyone else sees. If the walls of reality just crumbled for you, I apologize. Why would we play you like that?

Well lets say we’ve decided to tackle a bigger problem in Trello. We dutifully researched, designed, and developed a whole new version of Trello and it’s completely code reviewed and tested, a long, carefully calculated process that I’ve squashed into one run-on sentence. This version is probably still a confusing, frustrating mess that is riddled with bugs. Why? Because the design is just an educated guess and it hasn’t been subject to eyes on the street. We don’t really know how it works in the real world (a concept which I have fundamentally shaken for you). We ship it to the team and find what really needs fixing, instead of subjecting everyone to not-quite-there version. So we put it out on the “Alpha” channel.


channel switcher

There are three channels of the Trello website: “Stable”, “Beta”, and “Alpha”. When you go to, the server figures out which channel you are on and which version you should see. Everyone is on “Stable”, a handful of folks, maybe 20, have access to the “Beta” channel, and Trello team members and Fog Creekers are on “Alpha”. If you have access to more than one channel, you get a coveted channel switcher.

So while fixes are continuously being pushed out to the stable channel, our big potato versions like the new card back or the new boards page go out to alpha channel and are used by the team with real data. When it hits alpha, the gloves come off. Then there is heated debate. And anger. Glory. Joy. Frustration. Hope. And iteration. Lots of iteration. If only you saw some of the junk that hit the alpha channel… shoot.

Oh and if there are API changes associated with the new client, they are released separately and before the new client hits the alpha channel.

We iterate and refine until it’s ready for the beta channel. The beta version has a prominent “Give us feedback” button and disclaimer saying this is an early “explorer” versions of Trello and says “look out for these new things”. Then we get feedback, iterate, refine… When we’re almost happy, we slowly roll it out to a small, randomly-selected percentage of people on the stable channel. That usually starts at 1% goes up to 15% over a few days. Then we get plenty of feedback. We look for the big things that come up frequently, then iterate and iterate some more.

Now things are great and it goes out to 100% of stable. We draft up a blog post with the biggest, coolest new features, announce it, and then boom a bottle of champagne is popped. We are popping bottles of champagne constantly and if you ask me I think it should stop because it’s getting expensive and champagne gives me the worst hangovers.

Design and Research

I would like to elaborate on the design and research phase since I glossed over it earlier. We use a product planning board that has a list of problem-oriented cards. Cards are titled in the vein of “I can‘t find this feature” or “I want to do this thing but can’t”. These problems come from a few places:

  • Common cases seen in support emails sent in via FogBugz, a world-class bug tracker. (Disclaimer: we also make FogBugz and we think you should buy it.)
  • Frustrations of team members.
  • One-on-one sessions with Trello users conducted by Ben, our customer happiness maker. I don’t know what Ben’s title is.

We use all this feedback as research. We also do numerous internal polls and hallway usability tests. We also look at anonymous usage tracking in Google Analytics to see what features people are actually using. In all, we hope to get a sense of how people are currently using Trello, how they might want something to work, what they aren’t using, and where their pain points are.

On the back of each card, we take this info into account and propose a few solutions. When we settle on a card that really needs fixin’, I’ll take the card, make a few pen and paper sketches based off the proposed solutions, and create a few low-fi mockups and workflows in Sketch. (Disclaimer: we do not make Sketch, but you should buy it.) I try and post workflows and screenshots to the associated Trello card throughout the process so I can get continuous feedback from the team. I know the design is going to change a lot so I don’t waste time with high-fidelity mockups. We can quickly iterate in code anyway. Here’s an example of the new card back mockup. It has notes to point out the important new things. This one is more high-fidelity than others because it covers some visual design aspects.


The next step before developing a prototype is establishing a project codename. This is very important and happens to be lots of fun. The new boards page was codenamed “Borderlands” because it concerned everything outside the main board view. The new card back was codenamed “Jelly Donut” because it was even tastier on the inside (of cards). When we added the ability to have multiple clients, we named it “Hydra” after the mythical multi-headed beast. We’re working on some new search stuff. It’s codenamed “Hayride” which I think is a reference to finding a needle in a haystack? The iOS team has project codenames, too. When they were working on scrolling performance, they codenamed it “Hobbit” after the major motion picture featuring a smooth, lifelike high frame rate. Codenames are important as a point of reference. It’s also what we name our branches. bobby/borderlands is much better than bobby/new-boards-page-starred-boards-other-stuff-too.

Now that it is all codenamed and prototyped up, we ship it off to alpha and start the iteration process. Rollout. Launch. Champagne. Saxophones.

I hope this was informative. Maybe there was something you can take home with you. Our release process and workflow has come a long way, mostly due to the work of Doug and the ability to have multiple clients in production. Maybe Doug will write something more technical about that. Also I should say that the iOS and Android teams have different workflows that are interesting in their own right but not covered here. Ian says the iOS build process is “cmd + B”, but I’m like, come on, you know that’s not what I’m talking about.

That’s all. Please yell at me on Twitter.

We spent a week making Trello boards load extremely fast. Here’s how we did it.

January 22nd, 2014 by Bobby Grace

We made a promise with Trello: you can see your entire project in a single glance. That means we can show you all of your cards so you can easily see things like who is doing what, where a task is in the process, and so forth, just by scrolling.

You all make lots of cards. But when the site went to load all of your hundreds and thousands of cards at once, boards were loading pretty slow. Okay, not just pretty slow, painfully slow. If you had a thousand or so cards, it would take seven to eight seconds to completely render. In that time, the browser was totally locked up. You couldn’t click anything. You couldn’t scroll. You just had to sit there.

With the big redesign, one of our goals was to make switching boards really easy. We like to think that we achieved that goal. But when the browser locked up every time you switched boards, it was an awfully slow experience. Who cared if the experience was easy? We had to make it fast.

So I set out on a mission: using a 906 card board on a 1400×1000 pixel window, I wanted to improve board rendering performance by 10% every day for a week. It was bold. It was crazy. Somebody might have said it was impossible. But I proved that theoretical person wrong. We more than achieved that goal. We got perceived rendering time for our big board down to one second.

Naturally, I kept track of my daily progress and implementation details in Trello. Here’s the log.

Monday (7.2 seconds down to 6.7 seconds. 7% reduction.)

Heavy styles like borders, shadows, and gradients can really slow down a browser. So the first thing we tried was removing things like borders on avatars, card borders, backgrounds and borders on card badges, shadows on lists, and the like. It made a big impact, especially for scrolling. We didn’t set out for a flat design. Our primary objective was to make things faster, but the result was a cleaner, simpler look.

Tuesday (6.7 seconds down to 5.9 seconds. 12% reduction.)

On the client, we use backbone.js to structure our app. With backbone, it’s really convenient to use views. Really, very convenient. For every card, we gave each member its own view. When you clicked on a member on a card, it came up with a mini-profile and a menu with an option to remove them from the card. All those extra views generated a lot of useless crap for the browser and used up a bunch of time.

So instead of using views for members, we now just render the avatars and use a generic click handler that looks for a data-idmem attribute on the element. That’s used to look up the member model to generate the menu view, but only when it’s needed. That made a difference.

I also gutted more CSS.

Wednesday (5.9 seconds… to 5.9 seconds. 0% reduction.)

I tried using the browser’s native innerHtml and getElementByClassName API methods instead of jQuery’s html and append. I thought native APIs might be easier for the browser to optimize and what I read confirmed that. But for whatever reason, it didn’t make much of a difference for Trello.

The rest of the day was a waste. I didn’t make much progress.

Thursday (5.9 seconds down to 960ms)

Thursday was a breakthrough. I tried two major things: preventing layout thrashing and progressive rendering. They both made a huge difference.

Preventing layout thrashing

First, layout thrashing. The browser does two major things when rendering HTML: layouts, which are calculations to determine the dimensions and position of the element, and paints, which make the pixels show up in the right spot with the correct color. Basically. We cut out some of the paints when we removed the heavy styles. There were fewer borders, backgrounds, and other pixels that the browser had to deal with. But we still had an issue with layouts.

Rendering a single card used to work like this. The card basics like the white card frame and card name were inserted into the DOM. Then we inserted the labels, then the members, then the badges, and so on. We did it this way because of another Trello promise: real-time updates. We needed a way to atomically render a section of a card when something changed. For example, when a member was added it triggered the cardView.renderMembers method so that it only rendered the members and didn’t need to re-render the whole card and cause an annoying flash.

Instead of building all the HTML upfront, inserting it into the DOM and triggering a layout just once; we built some HTML, inserted it into the DOM, triggered a layout, built more HTML, inserted it into the DOM, triggered a layout, built more HTML, and so on. Multiple insertions for each card. Times a thousand. That’s a lot of layouts. Now we render those sections before inserting the card into the DOM, which prevents a bunch of layouts and speeds things up.

In the old way, the card view render function looked something like this…

render: ->
  data = model.toJSON()

  @$.innerHTML = templates.fill(
  ) # add stuff to the DOM, layout

  @renderMembers() # add more stuff to the DOM, layout
  @renderLabels() # add even more stuff to the DOM, layout


With the change, the render function looks something like this…

render: ->
  data = model.toJSON()
  data.memberData = []

  for member in members
    memberData.push member.toJSON()

  data.labelData = []
  for labels in labels when label.isActive
    labelData.push label

  partials = 
    "member": templates.member
    "label": templates.label

  @$.innerHTML = templates.fill(
  ) # only add stuff to the DOM once, only one layout


We had more layout problems, though. In the past, the width of the list would adjust to your screen size. So if you had three lists, it would try to fill up as much as the screen as possible. It was a subtle effect. The problem was that when the adjustment happened, the layout of every list and every card would need to be changed, causing major layout thrashing. And it triggered often: when you toggled the sidebar, added a list, resized the window, or whatnot. We tried having lists be a fixed width so we didn’t have to do all the calculations and layouts. It worked well so we kept it. You don’t get the adjustments, but it was a trade-off we were willing to make.

Progressive rendering

Even with all the progress, the browser was still locking up for five seconds. That was unacceptable, even though I technically reached my goal. According to Chrome DevTools’ Timeline, most of the time was being spent in scripts. Trello developer Brett Kiefer had fixed a previous UI lockup by deferring the initialization of jQuery UI droppables until after the board had been painted using the queue method in the async library. In that case, “click … long task … paint” became ”click … paint … long task“.

I wondered if a similar technique could be used for rendering cards progressively. Instead of spending all of the browser’s time generating one huge amount of DOM to insert, we could generate a small amount of DOM, insert it, generate another small amount, insert it, and so forth, so that the browser could free up the UI thread, paint something quickly, and prevent locking up. This really did the trick. Perceived rendering went down to 960ms on my 1,000 card board.

That looks something like this…

Progressive Rendering

Here’s how the code works. Cards in a list are contained in a backbone collection. That collection has its own view. The card collection view render method with the queueing technique looks like this, roughly…

render: ->

   renderQueue = new async.queue (models, next) =>
     @appendSubviews(@subview(CardView, model) for model in models)
     # _.defer, a.k.a. setTimeout(fn, 0), will yield the UI thread 
     # so the browser can paint.
     _.defer next
   , 1

   chunkSize = 30
   models = @getModels()
   modelChunks = []
   while models.length > 0
     modelChunks.push(models.splice(0, chunkSize))

   for models in modelChunks
     # async.queue flattens arrays so lets wrap this array 
     # so it’s an array on the other end...
     renderQueue.push [models]


We could probably just do a for loop with a setTimeout 0 and get the same effect since we know the size of the array. But it worked, so I was happy. There is still some slowness as the cards finish rendering on really big boards, but compared to total browser lock-up, we’ll accept that trade-off.

Trello developer Daniel LeCheminant chipped in by queueing event delegation on cards. Every card has a certain number of events for clicking, dragging, and so forth. It’s more stuff we can put off until later.

We also used the translateZ: 0 hack for a bit of gain. With covers, stickers, and member avatars, cards can have a lot of images. In your CSS, if you apply translateZ: 0 to the image element, you trick the browser into using the GPU to paint it. That frees up the CPU to do one of the many other things it needs to do. This browser behavior could change any day which makes it a hack, but hey, it worked.


I made a lot of bugs that week, so I fixed them on Friday.

That was the whole week. If rendering on your web client is slow, look for excessive paints and layouts. I highly recommend using Chrome DevTool’s Timeline to help you find trouble areas. If you’re in a situation where you need to render a lot of things at once, look into async.queue or some other progressive rendering.

Now that we have starred boards and fast board switching and rendering, it’s easier than ever to using multiple boards for your project. We wrote “Using Multiple Boards for a Super-Flexible Workflow” on the Trello blog to show you how to do it. On the UserVoice blog, there’s a great article about how they structure their workflow into different boards. Check those out.

If you’ve got questions, I’ll try and answer them on Twitter. Go try out the the latest updates on It’s faster, easier, and more beautiful than ever.

Fog Creek is Hiring, and We’re Coming For You!

November 14th, 2013 by Rich Armstrong

Boy are we hiring!

There are currently five different job postings up on our Careers page, and in some of these roles, we’re looking for multiple people.

So, that’s nothing new. But the big news that you may have missed is that, for almost all roles, we’re looking for candidates all over the world, not just in New York city.

That’s right! Fog Creek is coming to you!… Virtually! (No, we will not be literally bringing our espresso machine to you.)

We’re not just looking for drones here. Working remotely is not an impediment to career growth at Fog Creek. Productive teams keep remote employees in mind from the start. We rely heavily on asynchronous communication through products like Trello, and we conduct our (few) meetings online.

Remote developers are equal team members in every way. We even have remote team leads, who manage their teams from afar… well, from various distances, really.

Whether you want to work remotely, or to enjoy our beautiful offices and hanging out with our awesome employees, you’ll find you have a real career here at Fog Creek.

So, you no longer have to move to New York just to get reasonable work hours and awesome colleagues in an ideals-driven culture. You no longer have an excuse. Apply today!

Cage Set Match: Fog Creek Infrastructure Changes

March 1st, 2013 by Mendy Berkowitz

Back in October the New York harbor paid an unwelcome visit to the datacenter that houses our servers. Followers of this space are aware of the heroic efforts that literally kept the lights on. Those events were  inspiring and made us all proud to be part of Fog Creek. But as Sys Admins our job is to view heroic efforts as a failure in planning, preparation and architecture. So as soon as the flood waters receded we set ourselves on a path to improve the continuity and availability of FogBugz and Kiln, with the ultimate goal a second geographically diverse datacenter. As with all lofty goals the path has several large phases and milestones. Now, some of those milestones are upon us and it is time for an update.

The first major step is moving all of our servers from six individual colocation cabinets to seven racks in a cage. The actual move distance is only about 10 meters but represents an upgrade in almost every way. These infrastructure upgrades will enable us to support new releases the FogBugz and Kiln teams are about to ship (teaser alert: we’ve been dog fooding them the last couple of months and they are awesome, contact us if you want to participate in the beta). The cage will provide Fog Creek a solid foundation for the future and the ability to free up enough gear to equip a second datacenter.

We are striving to keep the impact of this move as transparent as possible. However, during March we will have three significant Saturday night maintenance windows during which FogBugz and Kiln will be unavailable. Our goal is to keep these outages a short as possible, as few as possible, and during a time of low usage for a majority of customers. To paraphrase Moltke “No project plan survives first contact with the server room,” so more details on the timing and scheduling will be forthcoming. Watch this space and our status blog for more details. We are committed to the Fog Creek guarantee of not wanting your money if you are not amazingly happy. These upgrades are a huge  part of our commitment to live up to that guarantee. Please let us know if these outages materially impact your business and we will make it right.

Here are some questions you are probably asking and we have asked ourselves over the past months:

Why stay in Peer1?
Why stay in a datacenter located in one of the most expensive pieces of real estate, in a flood zone, in a shared building? All excellent questions, but as we debated the various answers we realized there was nothing wrong with our current datacenter that a second datacenter wouldn’t fix. In a disaster (natural or man made)  situation, a second geographically diverse datacenter with a tested and practiced failover procedure is our best option for providing our customers with continued service. Fog Creek has grown up with Peer1 over the past 10 years and we have a great relationship with Mike and Scott (the real heroes of Sandy) who operate the facility. So we decided to stay and make Peer1 part of our overall strategy.

Why not move to the cloud like Trello?
We aren’t parochial about choosing technology solutions. We want the best technology to solve the problem at hand (just search for cloud vs. dedicated server if you are looking for parochial arguments). Before Sandy arrived we had begun the process of moving Trello to AWS. A technology stack well suited to horizontal scaling and stellar growth made Trello an excellent candidate for cloud hosting. FogBugz and Kiln use different technology stacks and are more I/O intensive than Trello. For these and many other reasons, the cloud isn’t the right solution for FogBugz and Kiln at this time.

We will have before and after pictures and some other fun vignettes into the process. If you want to work on awesome exciting projects, we are looking for experienced unstoppable Sys Admins!

Hurricane Sandy Wrap Up

November 9th, 2012 by Mendy Berkowitz

November 26 Update: Power has been completely restored to the building where our data center is located and the situation is now considered back to normal. More details on the status blog.

For those following our status blog and Twitter feed, the events of the last two week have been hard to miss. We wanted to post a quick summary, current status, and a couple of thoughts.

On Monday, October 29th, we posted that all systems were up and we were actively monitoring the situation. Peer1 was ready to run on backup generator power for days if necessary. That evening, the storm surge from Sandy, assisted by a rising tide and full moon, flooded the basement of our data center, cutting off the fuel supply to the backup generator. The next morning, Peer1 informed us of an impending emergency shutdown of the generator. We executed a protective shutdown to prevent loss or corruption of customer data. Later, when we’d secured confirmation from Peer1 that there was no imminent danger of power loss, we restarted our systems. The total duration of this unplanned, voluntary downtime was 3 hours.

The extraordinary efforts of Fog Creek, Square Space, Stack Exchange and Peer1 over the next three days have been covered by several news outlets. For a blow by blow account, check out the latest Stack Exchange podcast.

Currently, the rooftop generator has a steady supply of fuel. A separate “roll-up” generator (the size of a shipping container) has been parked next to the build to provide a second source of power. Power at the data center has been switched back and forth between these two generators to confirm that continuous, redundant power is available.

The building is still not on city power. Thirty feet of seawater have had to be pumped out of the basement, leaving behind damaged electrical systems that must be repaired and replaced before safely connected to the grid.

We take the trust of our customers seriously. We’ve been gratified by the expressions of support and appreciation. We’ve been inspired by our colleagues’ competence and dedication. However, we believe that extraordinary efforts should not be necessary to maintain smooth operations.

The lessons we’ve taken from this downtime are many and they’re still being processed. The actions we’re taking cannot be summarized in one post-mortem, but the changes will be obvious over the next several months.

We don’t want your money if you’re not amazingly happy. If you were materially affected by this downtime, please email us at and we will make it right.


The Fog Creek Sys Admin Team

Hurricane Sandy

October 30th, 2012 by Ben McCormack

Fog Creek’s services, included FogBugz On Demand, Kiln on Demand, and Trello, have been impacted by the severe flooding from Hurricane Sandy.

Please see for the latest updates.

Fun with BrushBots

September 14th, 2012 by Ben McCormack

This video is only about fun and has nothing to do with any of our products. Enjoy!

A few weeks ago, Betsy Weber of TechSmith (the folks who make SnagIt and Camtasia, among other things) sent us a box full of toys, including sporks, Rubik’s cubes, and kits to make our own BrushBots. What’s a BrushBot?

Toothbrush + Vibrating Pager Motor + Battery = BrushBot

One Friday afternoon, a few of us gathered in the lunch room and had fun putting them together. Check out the video:

Capture pictures for faster and better QA reports

August 6th, 2012 by Sam Vanderpol

I try to keep a finger on the pulse of the testing world. As QA lead at Fog Creek, I need to know what tools may be helpful to us, what technologies people are using, and what methodologies are considered best practice for manual and automated testing.

If we see something interesting we experiment with it, and if it doesn’t make the team better, we drop it, but in the process we’ve picked up a little wisdom.

We do a lot more dropping than we do adopting; after all there are more ways of failing then there are of succeeding. Most products we try don’t make the cut, but recently we came across one that did: qTrace.

The people at QA Symphony asked us to take a look at qTrace, their new QA tool that works a bit differently than anything else we’ve tried.

So what is qTrace? QBert’s distant cousin? No! Even better! qTrace is an image capturing tool on Windows that grabs images of bugs, annotates them, and submits the report to your bug tracker. (It integrates with FogBugz: submit a case with just the click of a few buttons without needing to go through the FogBugz interface.)  Cool? Yeah. Radically different from existing products? Not exactly, but qTrace does one thing that has won us over, and saved the QA team a ton of time.

Automatically capture images

qTrace captures a series of images triggered by page loads and new events occurring on the page. Every single step is automatically captured. So rather than just a screen shot of a bug, you’ve now got every single step that led up to the bug. This also means that instead of the reams of lengthy prose you write to describe what happened to trigger the bug, you can now annotate a series of pictures. You don’t have to assemble these from a screen grab tool. You don’t have to manually describe each step: It’s all recorded in the right order, and you just need to add a few notes to the automatically numbered steps in the recording. This can be sent straight into your bug tracker or made into a pdf or Word document.

That isn’t all though. qTrace also automagically pulls relevant information about the system you are testing on (OS, browser, processor speed, and lots more). Don’t want it to list these specs in your report? No problem, just a click to exclude that data.

The question with every tool we look at is simple: Does it make the team better? Does it make us more efficient? Without a doubt qTrace does. It became part of our process right away because the time saving was immediately obvious.

We love it

It is the best image capturing tool I have used since working at Fog Creek (including our own!), has the most extensive feature list, covers the largest amount of use cases, and is as easy to use as any software I have ever played with.

And it works seamlessly as a FogBugz plugin.

So if you write software that you test, and you use a Bug Tracker (qTrace is nice enough to integrate with a number of different trackers) and you are looking for ways to improve your workflow, communication, bug report quality and efficiency, give qTrace a try, you’ll probably like it!