Archive for the ‘Fog Creek’ Category

Fog Creek and Heartbleed

April 10th, 2014 by Mendy Berkowitz

Along with the rest of the internet, Fog Creek has been reacting to the Heartbleed vulnerability which was discovered this Monday.

TL;DR: FogBugz and Kiln On Demand, Copilot and the Fog Creek website were not vulnerable. Trello was vulnerable and has been remediated.

Fog Creek handles SSL connections at the load balancers in front of the application servers. The load balancers for FogBugz and Kiln On Demand, Copilot and the website are using the 0.9.8 branch of OpenSSL. This version does not have the newer heartbeat extension and is therefore not vulnerable to Heartbleed. None of your data on,, or has been exposed to the Heartbleed vulnerability.

The load balancers for Trello were using the 1.0.1 branch of OpenSSL and were vulnerable. We have upgraded OpenSSL and replaced the Trello certificates. More details and important recommended steps for all users are available in a post on the Trello blog.

We have also reviewed our vendors (for things like this blog) to ensure that our other SSL certificates were not potentially compromised. Fortunately we have not had to replace any other certificates.

If you have configured web hooks in your Kiln On Demand account, we recommend that you verify that the target servers have not been affected. If you use one of the pre-configured types, or your custom web hook uses HTTPS, the data sent to that external service may be at risk. We do not however, send any of your Kiln login credentials, so they are safe.

We take the security of your data seriously. We are committed to protecting it and to maintaining clear and open communication with you. If you have any questions, comments or concerns please contact us.

Fog Creek’s Ad Hoc Remote Work Policy, or, Working From Grandma’s House

September 12th, 2013 by Rich Armstrong

Not grandma.
Not grandma.


Once every few months someone walks into my office and says, “Hey, I’ve got this (reunion|wedding|grandparent). I’d like to figure out how to spend some more time with my family. Can I work remotely from my grandma’s house for a few days?”

I always said yes, and got pretty uneven results. When someone wasn’t productive, it was usually due to unforeseen circumstances that seemed obvious after the fact. So we started a list of things you’ll need for a short stint of working remotely. This list is a “policy” inasmuch as it represents all the things you need to do in order to be “covered” while working remotely for a short stint. If something foreseeable comes up that’s not on this list, then it’s our problem, not yours.

Two important things that this is list is NOT applicable to:

  • Working from home waiting for the cable guy or a mattress delivery. That all happens under your usual flexible work arrangement.
  • Going somewhere you’ve never gone before, like a cabin in Iceland or an apartment in Paris, where you don’t know what the infrastructure will be like. We’ll call this “working from Borneo”. We’re not against Borneo, mind you. It’s just not what we’re talking about here.

One thing this list is kinda applicable to:

  • Permanent remote work. After writing this all down, we realized that almost everything we’d recommend to the short stint people applies to people working remotely. We’re still learning from them, so might have more to say later.


  • A computer to work on. Duh.
  • “Approval” from your team lead. We’re not big on bureaucracy here, but this is just common courtesy.  Most of what you’ll have to do is covered below in “known availability”, but you should start with talking to your team lead.
  • An internet connection capable of easily handling a Google Hangout. Since we’ve opened up to remote workers, Hangouts have taken the place of conversations in the hallway and standups in someone’s office. Downstream is usually not the issue. Most basic internet plans are at least 5 mbps down, which is no problem. Being able to participate in a video conference usually means more that 1 mbps up. A lot of cheap internet service only offers 1 mbps up. How do you know if Google Hangouts will work where you’re going? Do a test Google Hangout with a person at the place you are going to be! (If there’s no one on the other end to do this with, the place you’re going is either vacation or Borneo (see above).)
  • Access to the resources you need to work. Make sure you can:
      • log in to your laptop as a local administrator without connecting to the Fog Creek network. This is especially true for Mac users, as Macs do not cache your credentials
      • connect to the VPN from your new location. Not all Internet connections, or routers, are created equal and not all will allow you to get on the standard VPN
      • RDP into your HQ machine, if you don’t have everything you need on your laptop

    If you’re unsure about any of this, talk to the sysadmins before you discover you can’t work remotely.

  • A headset. The built-in speakers and microphone on your laptop/monitor suck. The added friction of talking to someone without a headset makes your team members not want to invite you to a Hangout, which means you miss communication that you would normally have received.
  • A dedicated room with a door that closes. At Fog Creek, devs have offices with doors that close. People who aren’t devs are surrounded by coworkers who’ve learned to keep from inadvertently distracting each other. Your aunt has not learned this, so it’s probably a good idea to sequester yourself as much as is feasible. You need a place that is as distraction-free as your normal work environment, and maybe a little more.
  • No child care responsibilities during working hours. You can’t take care of kids and work at the same time.
  • Known availability and overlap with your team. Set standard hours and make sure people know them. If you’re not going to be working the hours you normally work, overcommunicate this to your team by posting in chat (ex: “Going to the gym for an hour”). Nothing is more frustrating to team productivity than chat messages like “Has anyone seen X?” “When is X getting online?” “Can someone else take this bug? X isn’t responding.” Try not to pop in and out. Work sustained sessions with only a few significant interruptions (e.g., lunch, dog walking, gym, light-saber fight with your nephew).
  • A dedicated phone that fits into your position’s workflow, where applicable. If you are an employee who spends a reasonable amount of time on the phone, it should be relatively easy for people to get you on the phone and for you to call them. This might require you to contact the sysadmins for phone forwarding. We haven’t done a lot of this yet, so we’re not awesome at it. Maybe you can use your experiences here to make the next person awesome?


Suggestions for Short Stint Remote

When you’re used to working in the office and you leave it, you’re like a snail without a shell. Consider these suggestions a temporary exoskeleton. These suggestions are not because we don’t trust you. We trust you. To get a desired effect, we often accentuate parts of our behavior that don’t come naturally. We might be introverts at a networking event, forcing ourselves to start conversations with strangers. We might be on camera, forcing ourselves to oversmile like idiots. Most of these tips are to encourage unnatural but beneficial behaviors.

  • Don’t be coy. Awareness is largely covered in “known availability” above, so this bullet point covers your mental state. You might be tempted to downplay your absence from HQ. Puritan work ethic runs pretty deep with some.  Fact is you can ship like an animal from places like Hawaii or Cape Cod (where this document was started). But you might be embarrassed to talk about it, or you might be put on the defensive by someone saying “Great beach weather.” Ignore it and ship code. Clamming up or playing down helps no one. It’s the opposite of what we want; it’s the main reason we have a policy. If you’ve met the requirements above, you’re covered. You are not slacking. You are working. You should not be shy about letting people know where you are and what you’re doing because of what they might think.
  • Put a note on your monitor saying where you are and for how long, in case people wander by looking for you.
  • Have boundaries. The people you’re going to be around might not understand that, yes, you will actually be working. Don’t worry. Your in-laws/parents will be impressed by your work ethic!
  • Over-communicate and/or over-deliver. At HQ, you have a few ways of staying involved and available (face time, informal chats, lunch) that are not available to you remotely. You still have stuff like chat, commit messages, code reviews, cases, Trello boards, etc. If you’re a Creeker, chances are good you’re a low-key person who doesn’t crow about their achievements. You might want to flip that bit for this stint. That can be hard, but it’s important to keep involved. The best way to do this is….
  • Have a deliverable. This frees you from having to assert your productivity via chattiness.

That’s it! We put this stuff out there so that you can effectively take advantage of the flexibility offered by your work arrangement and continue to be productive as part of your team. If you learn something while setting up your environment, or while working remotely, that you wish you’d known earlier, please let me know. The list you see above is a work in progress.

Dogfooding Until It Hurts

February 26th, 2013 by Rich Armstrong

Dogfooding. Also called “eating your own dog food.” It’s pretty simple, right? If you work at Uber, maybe take an Uber car ride from time to time. If you work at Khan Academy, you’re probably pretty good at math by now.

In this video from 2009, Joel talks about dogfooding as being more than just using your own product. It’s about using your own product for everything you can imagine, even if that usage is a little uncomfortable.

We dogfood Trello, FogBugz, and Kiln in a ton of different ways.

We’re a software company, so we generate a lot of code. Kiln helps with that. But we even use it to hold things that aren’t “code” per se, such as backups of non-sensitive databases. We don’t have any real crazy use cases, like using code reviews to plan parties or the electric DAG to create subway maps, but that’s just because Kiln has only been around a few years.

FogBugz, on the other hand, we use for tons of off-label stuff. We use FogBugz’s crash reporting ability to read RSS and Twitter feeds and create cases when new items appear. We use its customer email capabilities to receive faxes. And we have a panoply of API scripts automating a bunch of different business processes.

Trello is a natural for dogfooding, of course, because it’s so flexible. We keep kitchen snack requests and lunch menus on boards. We keep track of who’s going to what conferences, and use it to plan their travel. Trello really shines in setting the agenda for and running our bi-weekly company all-hands meeting. Because Trello is so flexible, it invites a lot of different use cases. We’re still figuring out which ones really work, and it’s great to see a fast-growing user base figuring that out alongside us.

So, that’s all well and good, but if you really want to see “dogfooding until it hurts” in action, check out Beeminder, a tool for setting “goals with a sting.” You set up your goals, and if you stray from them, the service fines you an escalating amount of real money until you’re back on track. Beeminder is dogfooding heavily and publicly to keep their development goals on track. These folks are  literally giving away cold hard cash to users as a pre-commitment to do things like delivering user-visible enhancements or blog posts on a regular basis. Amazing. Even better, they have a Trello integration that’ll keep you moving those cards to the Done column regularly (or else).


Let Them Have Cake … And Ice Cream Too!

November 17th, 2011 by Rock Hymas

Consider a cake shop owner in a small town in Siberia. He sells the best cake around. Whenever someone wants a treat after dinner they stroll down to the cake shop and get a nice big slice of cake. But along comes a new treat: ice cream. Our cake shop owner ridicules his neighbor for starting an ice cream stand, saying no one will buy it because it’s cold. This is Siberia, after all. But rather than running an oven all the time, she can just keep her ice cream outside in the freezing cold, so it sells for less. Suddenly a whole bunch of people who could never afford cake are stopping at the ice cream stand after both lunch and dinner. It’s cheap, it tastes good, and the teenagers are having contests to see how much they can eat before passing out from brain freeze. It seems the whole town is there, and some of the cake shop customers start getting ice cream just because they want to hang out with their friends at the ice cream stand. All of a sudden the neighbor is rolling in the dough and she wants to buy out the cake shop since they have a great location.

Now suppose the owner of the cake shop recognizes that lots of his customers will like the ice cream (even in Siberia) because it’s so much cheaper. So he sets up an ice cream stand and hires his neighbor to start selling it as fast as possible. He notices that some of his regular cake shop customers are going to the ice cream stand instead, because it’s cheaper. So cake sales are down a little. Rather than shutting down the ice cream stand though, he brings it into the cake shop, adds a special “cake” flavored ice cream, and becomes the town hero. Not only has he saved his business, but he’s also been able to meet the dessert needs of more of his fellow townspeople. Cake is selling even better than before, ice cream is selling even better than that, and the cake-fanatics and ice cream groupies all get to hang out together.

Clayton Christensen’s talk at this year’s Business of Software was all about how companies disrupt, and are disrupted by, other companies. In building a product, you (or those who came before you) made decisions that are really hard to change after the fact. That’s fine; those “stakes in the ground” are what made the product successful. But it also limits the viable lifetime of the product. At some point disruption happens.

Disruption =

Larger Market + Lower Price + Different Measure


Professor Christensen pointed out that disruption occurs when a company solves a given problem for a larger audience at a lower price point with a different measuring stick for comparing value. Companies that successfully avoid being disrupted are usually able to do so only by disrupting themselves. Few companies are able to pull that trick off, and it typically involves having a different team or business unit set up in order to avoid all the baggage that the last-generation products carry with them.

What does this have to do with Fog Creek? Well, I’ll admit that I was a little skeptical when Fog Creek launched Trello. My worries pretty much lined up with those outlined in this post from a FogBugz customer. Though it’s taken some time, I’m starting to see how important it is for Fog Creek to prepare for the disruption of its flagship product, FogBugz.  By doing so, we can make sure FogBugz will keep solving our customers problems and keep making us money.

Trello fits the model for disrupting Fogbugz. It solves the problem of planning the work on a project for a larger audience than just software companies, at a lower price point than a FogBugz or a Jira, with a different measuring stick: putting Trello and FogBugz next to each other on a feature comparison chart doesn’t make any sense.

Trello’s target market is much larger than that of FogBugz. Where FogBugz targeted software development teams with features like bug tracking, automated crash reporting, and evidence-based scheduling, Trello can provide value to any group of two or more people working together on something that can be broken down into steps. Even though any group of people could use FogBugz to do the same thing, using FogBugz doesn’t really make sense when you’re in HR doing hiring, or in law working through cases, or in a studio vetting country bands. Using Trello does.

Additionally, Trello has a lower price point: free. Everything currently offered by Trello is free, and will remain so going forward. Yes, there will probably be value-added features and services that Fog Creek will charge for at some point. Compared to free, though, FogBugz is expensive at $25 per user per month. When you just want something simple to plan your wedding, that doesn’t make any sense. But if you’re managing software projects for reasonably complex products, then FogBugz easily adds that much value, and our customers make that clear by coming back again and again.

Trello also has a different measuring stick. And that’s the real reason it will eventually overtake FogBugz, even for software projects. Or rather, it will take over some of the roles that FogBugz fills in a software development company. We already use it here at Fog Creek to manage our work at a coarse-grained level. It provides a potentially public view into product development, which is cool to see.

The real key, though, is that Trello, like FogBugz, is opinionated–but it has very different opinions. Rather than seeing work on a project as a large set of small items that need to be tracked individually, it sees project work as a small set of somewhat larger tasks that fit into a bigger whole, a workflow defined by the team. If FogBugz tried to create some kind of dashboard view of your bugs to compete with Trello, you’d be so overwhelmed by minutia that you’d give up and walk away. No one wants that kind of view when they’re dealing with hundreds or thousands of individuals cases. But Trello redefines the way we see our project work and that fundamentally changes the game.

Wait a sec! I’m on the FogBugz team. What am I saying?! Am I just making a Steve Yegge “TMI” mistake by posting this publicly? Am I talking myself out of a job?




I don’t think so. And here’s why.

The same legacy that prevents it from winning in the larger Trello market gives it a competitive advantage in the market for software development teams. FogBugz will continue to make money, and it is still growing at a nice pace. It needs investment, but not the kind of investment that would attempt to turn it into Trello. Rather, the kind of investment that will take advantage of disrupting products like Trello while preserving it’s usefulness in the niche of software development teams.

And our customers, for the most part, aren’t going to jump ship for Trello anytime soon. Currently, Trello can replace only a very small part of what FogBugz provides. One customer pointed out that it cannot handle bulk editing, screenshot captures are painful, and categorization and search aren’t designed for the situation where you have lots of items. Additionally, FogBugz supports incoming and outgoing email, automated crash reports, and deep hierarchies of work. You can install it within your network and use it completely internally. It’s also got awesome source control integration with Kiln. These are all things that software development teams care about, often passionately. Trello, and other products like it, may eventually meet some of these needs, or integrate with other tools that do, but that will take time. Time that will allow FogBugz to further differentiate itself.

Besides, learning new ways of working is hard. Anecdote time! We spent some time over the last few months rethinking the UI for FogBugz. As the team lead, I made the call to focus on that, and did what I could to protect the team through the process. Unfortunately, when it was done, Joel pointed out that it wouldn’t fly with our customers. Why? Because we “moved the cheese” in too many ways. It would have required our customers to learn new ways of working, and human nature doesn’t like to do that. Though that does limit what we can do with FogBugz going forward, it also strengthens the ties that our current customers have with the product. They know where to click, which keyboard shortcuts to use, and they know what to expect. That helps them work faster, get in the flow, and keep the boss happy.

Finally, and related to the last point, FogBugz has an ecosystem of supporting technology around it. When our customers start using FogBugz and Kiln, they often integrate with a set of tools and technology. This is one reason it’s harder to innovate on those products, because our customers don’t just rely on not having to learn new ways of working, they also rely on their custom plugins not breaking, on their random Python scripts still working, and on third-party tools continuing to integrate well with FogBugz. Each of those is a barrier to entry – they make it harder for a competing product to win our customers away from us (yep, even if the competitor is made by us as well) because they also add value for the customer.

In short, for almost all of FogBugz existing customers, and most of the larger market of customers using a competing bug tracker, using Trello instead is not the right business decision. And within that market FogBugz can make sure that business decision doesn’t change by doing the right kinds of investment. Investment that will keep FogBugz relevant and growing into the future is core-competency investment, not more features or chasing faster competitors in a race decided by a radically different “finish line”.

And there are tons of things we can do in the core of FogBugz: reduce the complexity of the product both for our users and for our developers; make it faster; phase out old features that aren’t being used by anyone; fix the backlog of bugs; improve our up-time; integrate with Trello and other apps. That last point is probably the most salient. As Trello grows and becomes a better fit for certain kinds of project management work, FogBugz can increasingly integrate with and offload those areas to Trello. At the same time, it can target specific problems that software development teams face, and provide value by solving them well.

In the end our customers will be able to have their cake … and their ice cream too.

The Price of (Dev) Happiness: Part Three

October 11th, 2011 by Rich Armstrong

Buying lunch in lower Manhattan is not cheap. If you don’t brown-bag, you’re pretty much in for nine bucks, and more if you want to sit down. A meatball sub is $8.50. The Hanover Square Deli does a respectable dduk mandu guk for about $9.50. A meager-looking, but tasty, turkey and guacamole sandwich from British import Pret A Manger will set you back $6.75 (plus tax). Or, if you really want to eke out some savings, you can get a cup of soup and a hunk of bread for about $5.

Post-lunch, Pre-espresso machine

At Fog Creek, we have our lunch catered every day.  The cost of this to the company is $15.75 per person per day. Not counting drinks and snacks, this amounts to an extra $4,000 per year per employee. So why wouldn’t we just pay people $4,000 a year more? If they want to be frugal, they can buy that $5 soup and pocket the difference.

Well, first off, since lunch is a catered on-site meeting, the cost of that lunch is 100% deductible as a business expense. If you worked here, it’d cost the company $16 to give you $10 to go buy your own mediocre udon noodles.  (Don’t take any taxability or corporate expense stuff here too seriously; I’m not an accountant or lawyer.) Second, Fog Creek’s free lunch is much more because of how it fits with the rest of our workspace and culture.

Free lunch is nothing new. Google and other big tech companies have been giving their employees breakfast and lunch (and sometimes dinner). Free lunch has been around for more than a decade because it works for attracting and retaining top talent. (Or, rather, people think it works.)

Now, the food at Google, and the people who make it, are awesome. The food at Fog Creek is good, but nowhere approaching what Google does. We don’t have Sam, the sushi chef, doing hand-rolls to order. We don’t do miso black cod, lamb shanks, or osso buco. We don’t have a raw vegan station with selections so delicious they attract the most dedicated carnivores.

Here’s what we have at Fog Creek instead: no meetings.

For us, lunch is our only recurring meeting. The only standing interruption in the day/week of a developer here. Everything else is ad hoc or temporary. (One exception is our quarterly meeting to go over financials and to grill the founders with questions.) The default at Fog Creek is no recurring meetings. Once you’ve established that, recurring meetings become the exception, rather than the rule, and tend to wither naturally as their usefulness degrades over time. For example, the FogBugz team is currently doing a stand-up meeting for fifteen minutes every day right before lunch, but this won’t last. The Trello team was doing them before and shortly after launch, but they’ve subsided.

The Kiln team lead assures me that their weekly stand-up is a real recurring meeting, but at some point they’re going to lose interest and go back to coding. It’s what they do. Meetings have a network effect. They need other meetings to legitimize them. If you’re constantly scheduled with meetings, you don’t mind being interrupted one more time. Or, rather, you don’t have the energy to protest. When you proceed from the assumption of no meetings, you have to expend effort to keep a meeting going.

It’s not all a playground, of course. There are downsides. Things, sometimes important things, don’t get communicated to the right people at the right time. More often than not, the worst result is hurt feelings or slight confusion. Sometimes it’s more than that. But we wouldn’t give it up.

For someone who’s used to a standard work environment, it seems silly, cult-like, possibly even daunting, to be “forced” to break bread with your colleagues every day. It seemed odd to me before I came here, but by the end of the first week, nothing seemed more natural. When most of your “socialization” with your colleagues takes the form of mandatory, recurring meetings over a conference table, it’s natural to not want to see them again over the lunch table. People have been making decisions about your time all day; at lunch, you need some time to yourself.

But when most of your time is spent working in a private office, taking breaks according to your natural attention span, having short chats with one or two colleagues, it’s a pleasant prospect to surface for some pleasant conversation about StarCraft or football with nice, intelligent people. And maybe you’ll hash out that new feature, too.

One can make the argument for free lunch based strictly on developer productivity. A free lunch could give you a hundred hours per year from your best people, time saved in driving, waiting in line, etc. But, consider the depressing ease with which such a gain can be wiped out by a few recurring meetings.

So, for our final post in this series, we do have a number: $15.75 per day. $4,000 per employee per year. It’s a lot of money. But without the rest of the company culture, it would be sort of meaningless. A sense of entitlement grows rapidly around any perk you offer, and lunch is no different.

A lot of this series has been based on getting hard data out there so that developers, our main audience, have an easier time talking to management about some of the things that’ll make them more happy, healthy, and productive. For this post, it’s a bit more difficult. You might get your boss to spend $16 a day, but changing the culture of meetings in any workplace is nearly impossible. (During my tenure, Google tried no-meeting Thursdays and formal meeting-reduction task forces, reminiscent of Brazil’s Ministry of Debureaucratization, to no avail.) It would require rebuilding the company culture from bedrock.

Our bedrock is the idea that, once we’ve hired good people, it’s the effort we make to direct their intention, rather than their attention, that creates value. It’s not just our lunch benefit that springs from that, but nearly every other thing we do.

For Fog Creek, our founding principles, and the pains we take to stick by them, are the price of developer happiness, and that can’t be measured in dollars.


Experiments with Google Page Speed Service

September 23rd, 2011 by Jude Allred

Our oldest product

Let me tell you a little bit about

  • It’s old; It’s hosted by IIS6 alongside a horde of other websites.
  • It’s the first place people go when deciding whether to use FogBugz or Kiln.  It’s pretty important to us.
  • Recently, we started to treat the website as a product. It has received substantial attention, primarily in the form of A/B tests.

We’re building some fancy new web servers for, but in the meantime we’ve been accepted into the Google Page Speed Service beta. It seemed reasonable that it might provide us with some useful tools, and in the short term it might give us a workaround for some of our IIS6 woes.

There were four things that we hoped Page Speed Service would take care of:

  1. GZIP all of’s static and dynamic content.
  2. Set far-future expires headers on our static content.
  3. Distribute our static content via a CDN.
  4. Fix or remove the broken ETags that IIS 6 Is sending out.

Enter Google Page Speed Service

For this test, we brought Google Page Speed Service to life at This is a totally unused alias of, so we were able to test PSS on an isolated mirror of our production environment.

After a small amount of setup, page speed service was live!  (Setup basically consisted of adding a Google verification .txt record and a CNAME record mapping to

The comparison tool that Page Speed Service provides was inconsistent, but it looks like there might be a lot to love.

  • Its page load times differ from those reported by GTmetrix and local testing
  • The measurements change significantly over the course of running it multiple times

But the results look promising.  Here’s a report on Fog Creek’s home page load time.  On the left is the load time without Google Page Speed and on the right with it.  Google PSS gave us a 10% difference in initial page load time, and a 25% difference in repeat page load time. We got that for very little effort.  Nice!

Here are GTmetrix reports on from 9/23/2011.  First, the results without the Page Speed Service:

And the same tests on, with Page Speed Service enabled:

Overall scoring improvements:

The Fog Creek landing page:

Google PS +1%, YSlow +10%, page weight reduced from 466kb to 365kb

The FogBugz landing page:

Google PS +1%, YSlow +12%, page weight reduced from 608kb to 379kb

The Kiln landing page:

Google PS +0%, YSlow +10%, page weight reduced from 520kb to 411kb

Interestingly, GTmetrix shows page load times (as measured by the onload event) that are consistently slower for than, usually on the order of about 0.7 seconds.

GTmetrix’s tests don’t seem terribly consistent with’s (Google’s promoted testing service), which show consistently faster page loads than GTmetrix. But, it’s hard to draw conclusions since incremental rendering of the page is an important factor which isn’t represented by the load time.

For those of you who wish to dig deeper, take a look at the waterfall diagrams of a given page rendered with or without PSS. You’ll notice that the PSS timelines have their content proxied across 4 of Google’s servers, the overall number of requests has decreased, and the waterfall has rearranged significantly.

For example:, Without Google Page Speed Service:, With Google Page Speed Service:

So how did PSS handle the goals we wanted to meet?

  • PSS has compressed (GZIP) the content that it sends down
  • PSS has far-futured the vast majority (but not all) of our static content
  • PSS has properly configured our ETags
  • PSS is distributing our content via its CDN


1 - PSS’s lossless image compression is lossy.  Our website uses a tiled, textured background image.  Page Speed Service attempted to optimize it, and in the process altered the background images.  For example, this image:

Became this one:

I’ve put together a test case at

As part of the lossy conversion, PSS converted the image from a png into a jpg.  I altered the source image files to be jpgs instead of pngs, and then PSS was able to optimize and bundle them without any difficulty.   I’d also note that the image optimization step (and all of their other optimizations) can be disabled in order to work around problems like this.

2 - PSS supports blacklisting URLS, but not blacklisting HTTPS urls.  I knew from the start that PSS didn’t support HTTPS, and therefore our trial signup forms, e.g.,, would have to be exempt from PSS’s optimizations.

PSS supports blacklisting, which allows you to exempt a specific page from being hooked by PSS’s proxy.  Instead it will go directly to your source server.  Unfortunately you can’t configure HTTPS pages in this manner.  If you have a website with HTTPS content, you’ll have to move all of your HTTPS content to a separate subdomain, e.g. “”, in order to exempt it from PSS.

This was an inconvenient change to make, although not hard. Still, PSS’s support of blacklisting was put to good use.  We use a heavily-modified version of the FairlyCertain A/B testing framework on our properties.  Server-side A/B test frameworks and CDN’s don’t really play well together, but PSS’s blacklisting allows us to blacklist any pages which contain active A/B tests.

3 - For the duration of the beta, once you bring PSS live on a given subdomain (e.g.,, you cannot reconfigure PSS to work with a different subdomain (e.g., without an additional beta acceptance.  I’ve reapplied to include in the PSS beta, but have yet to be approved.  For this (and only this) reason, we’re blocked on moving forward with using Google PSS in production.  If a Googler is reading this and would like to whitelist for the beta, we’d be much obliged.


Pitfalls and onload-event oddities aside, I’m very impressed with Google Page Speed Service.  I’m happy to see that they’re abstracting away the ever-present need to minify, compress, and bundle your static website content, as well as optimize your images.  In our specific case we don’t really care about those  features – we already optimize our images and bundle our scripts, but it is a great feature for most sites.  Our case is probably more edge than common—we’re using PSS primarily to work around IIS6, and it appears to succeed at that.  PSS’s CDN seems promising, but I don’t yet have any data on how it compares to, say, Amazon’s Cloudfront service.  Setting up PSS on our site was an absolute breeze, however, and  I think it may turn out to be a great tool for web developers and site administrators.

The choice of setting up the Page Speed Service on instead of our production subdomain, retrospectively, was a useful mistake.  I appreciate the ease of using tools like GTmetrix to compare the two subdomains directly. I’ve also enjoyed keeping PSS’s configuration changes completely separate from our production environment as well as the ability to share the PSS changes with other people without having to manually configure their browser to proxy traffic through PSS.

The Agonies of Picking a Product Name

September 15th, 2011 by Dan Ostlund

Picking a product name is all agony and no ecstasy. It’s also a giant time-slurping vortex. And in the end, it kind of doesn’t matter.

A product, or a company name, only really needs to achieve one thing: not remind us of some unpleasant bodily function, or the results of a wild and drunken night. But even this seemingly common-sense advice is often and successfully broken. Wii anyone?  No thanks, I went this morning.

Yes the name matters if it’s truly totally weirdly awful, but outside of that it just doesn’t.

Recall that “iPad” was the subject of all kinds of snark and scorn in the first couple of weeks after launch. Who makes those jokes now? No one.

Amazon? I remember lots of people making fun of that name and insisting there would be endless confusion, but it worked out fine, and at least it wasn’t Cadabra, one of the early names Bezos considered, and something sure to upset the Steve Miller Band.

At least a dozen people I know wondered why you would name an e-reader Kindle, something that reminded them of fire. Fire. Books. Fire + Books, bad. Anyone think that anymore? Nope. Everyone just buys them by the millions.

If Google wasn’t such a familiar and ubiquitous name, most people would reject it as a company name on the sensible grounds that it is sillier than a bald cat. But we’re all used to it, and we don’t care. And that’s the point.

And of course there is no shortage of bad-seeming names on really successful products. Dare I mention FogBugz in this context?

We just recently went through the naming agony since we just launched Trello, a tool for project and task and process management, and…well, go check it for yourself. 27,000 users have signed up in the last 24 hours, and they’ve been saying really nice things on the Twitters.

Trello was code named Trellis when it was in development. It got that name because one of Fog Creek’s co-founders, Michael, suggested it as a code name in an early meeting. It was fine. It stuck.

Eventually we decided that we needed An Actual Product Name. Which, of course, we did, but we spent way more time on this than we needed to. Two of the team members took some time to come up with some names. Actually, a couple of hundred names. For one reason or another, these were all rejected or shelved, mainly because all possible domains have been plucked up by an automatic domain-registering spaceship.

Feeling morose over this, they then hired a professional naming person. He came up with about 200 names, some of them very good, and one of them “lasagna”.

Not knowing about the previous naming efforts, I jumped in one day with about 125 name ideas. Then several of us sat down together and reviewed our name ideas, and came up with about 200 more. We tried everything: animal names, plays on various aspects of what Trello does (board, card, list, task), Japanese words, and every combination in between. We threw them all at the wall, from the practical to the nutty. Kardboard, Hippolist, 5 Camels, Listly, Idealist—all were suggestions at one point or another.

Late in the process we thought, maybe we could just use the code name, Trellis. Why did it have to change? But, we couldn’t buy They weren’t selling. We tried to buy, but it was more than we wanted to spend. We really thought it was important to get a .com domain anyway, so we weren’t thrilled with this option.

The wheels continued to spin and every couple of weeks the product name came up and we would lurch off on another round of fumbling around for a new name.

Work continued on the product code named Trellis and for a couple of months we didn’t have much time to think about the name until the mid-September launch forced the issue: nothing like a deadline to provide some clarity.

So Joel organized a company-wide brain storming session and we got out our markers and some giant pieces of paper, tacked those up on the wall, and came up with about another 150 names. Some really good names came out of that session. It was raucous, and it was fun.

We argued, we lobbied, we pointed out when a possible name rhymed with wee, and then we voted. Of our choices, about none had available .com domains. The whole thing was getting depressing. And we were out of time.

Trellis was still one of the contenders, so, after all of these sessions, and all of this brain-storming, and the many hundreds of suggestions, Michael just decided to start playing with variations on Trellis. He came up with a number of them, and then looked for domains he could buy. happened to be one of the domains he could get for a reasonable amount of money and we decided it sounded good. We had made a grand wandering journey all the way from Trellis to Trello. That’s like leaving New Jersey and getting to New York by way of Kinshasa. But here we are.

After all of that it’s clear that the name just isn’t that important.

Gold Bond Medicated Powder. What’s a Gold Bond, actually? And Medicated Powder isn’t a name, it’s a thing. Might as well call your product Lined Paper With Three Holes. But Gold Bond Medicated Powder works, doesn’t it? No one ever thinks about it, they just get some if they have itchy feet.

We ended up with a name very similar to the one we started with, one that all of us like just fine, and one which is just going to be what people call our new, supremely free, project management app. I think everyone would have gotten used to Hippolist too, but it doesn’t matter, because it’s called Trello now.

In the end we should have just had Michael do the whole name picking in thirty minutes and been done with it.

Fog Creek Launches Trello

September 13th, 2011 by Dan Ostlund

Today we’re happy to announce a new product. It’s called Trello and it’s used for all kinds of project and task management. It’s based on a simple board and card metaphor, and we hope you’ll find it as useful as we do. Oh, and it’s free. But wouldn’t you rather hear all the details about Trello from Joel?

You can follow the Trello twitter here. Or you can check out the Trello blog here. There will be several posts there over the next couple of days.


The Price of (Dev) Happiness: Part Two

September 1st, 2011 by Rich Armstrong

Last time, we took a look at the exact cost of the contents of our private dev offices. For this post, we promised to take a look at the price of the offices themselves.

Since the beginning, Fog Creek’s promise has always been that every developer gets a private office with a door that closes. Don’t want a private office? You get one anyway. If you want camaraderie, you can walk down the hall, put your witticisms on company chat, or store them all up and let fly at lunch.

In the previous post, the underlying message was, “Give your people the workstations they need to be comfortable and healthy.” The message in this post is not, “Give your people private offices.” We’re satisfied enough with the benefits that we’re going to continue it, but we’re not here to proselytize that policy.

The point this time is this: your office space and how you design it is an expression of your priorities as a company. It speaks to everyone who comes through the space, every day. Do not be shy about spending money on it.

Let’s get the number out of the way. We spend a bit over 6% of revenue on office space. Compared to other companies we’ve surveyed confidentially, that’s on the high side, but not by much. And we’re growing fast these days because of the stunning success of Kiln, so we won’t be above 6% for long. We’re based in Manhattan and, as you might imagine, office space is very expensive. But that’s what the company was founded on: a good job for awesome coders in New York City. Without making light of the achievements of our neighbors, we didn’t want to be the best place for devs to work in Hackensack. So it’s time to pony up.

Of course, a lot of companies who are competing for the same talent are actually venture-funded startups. They have different needs, revenue/reward models, and external expectations than we do. I’m going to go out on a limb and say Fred Wilson might take issue if you buy the $10,000 espresso machine before showing a profit. We’ve been profitable since inception and so can work a little differently.

Espresso machine


Also, the price tag on our office space is not only because we have private offices. We need a lunch room and kitchen to accommodate everyone having lunch at the same time, away from their desks. Our summer head count can grow by ten because of our summer internship program. Okay, so we don’t actually need the saltwater tropical fish tank or the marble shower or the library.  But we’re also expressing our company culture by how we structure our office, and if that can keep us happy and motivated, plus attract more smart people who share our values, that’s very much worth the extra money.

Here’s a few good snapshots of our office, if you want to see more.

Next, the cost of lunch, and its primary benefit: no standing meetings.


The End of an Internship

August 24th, 2011 by Benjamin Pollack

One thing that really made a huge difference to me when I was an intern is that I knew that what I was working on was actually going to ship to actual customers.  Not at some distant point in the future. Now.  Copilot started going live piecemeal about eight weeks after we started, which meant that we actually got to see customers react to what we’d made while we were still in our internship. I’ve mentioned before that this is one of the reasons that I got completely addicted to Fog Creek. But for the past two summers, we haven’t really done that with our interns. Yes, they wrote major code, and yes, it shipped to customers…but only months after the interns had already gone home.

Interns party at the beach on their last day

This year, on the Kiln team, we changed that: three times this summer—as part of Kiln 2.5.119 and Kiln 2.5.166 for Kiln On Demand, and in Kiln 2.5.177 for customers with on-site Kiln installations—we released intern code. Every last intern on the Kiln team has gotten to see their code actually running in the production Kiln environment before they’ve gone home. Some interns even shipped new code with all three releases, getting feedback from real, live, breathing customers each time.

But these are interns, I hear you say. So what? We’re talking dinky features here, right?

You tell me. They made Kiln show you images in diffs and when browsing repositories. They got Kiln running the newest version of Mercurial from the top of the stack to the bottom, including landing piles of patches in TortoiseHg and a few in Mercurial itself. They enhanced Kiln’s code search to allow filtering on files by globs and regexes, allowed greater flexibility in subscribing to Kiln’s activity feed, and dramatically improved Kiln’s ability to work with continuous integration systems. Not content with that, they added lots of UI refinement in changeset/bug integration, and gave back to the Mercurial community by taking our proprietary kbfiles extension, which makes it trivial to work with large files in a DVCS, and turning it into the generic largefiles extension for Mercurial that will hopefully be included in an upcoming Mercurial release.

These are huge changes. Changes that make a daily impact in our customers’ lives.

This is one of the things that I absolutely love about Fog Creek’s internship program. We treat interns like full-fledged employees, and our interns continually reward us by producing amazing things.

This obviously works well for us here at Fog Creek, but I also think it works out really well for our interns. One of our interns, Andrew Pritchard, noted:

Working at Fog Creek has already expanded my horizons more than any of my previous experiences—and not only because I am working for the first time on code destined to land in paying customers’ eager hands!… As it turns out, complete immersion, combined with the intrinsic and extrinsic motivation (thanks Joel!) that come with a paying job, can make learning an entirely new set of technologies surprisingly easy. And complete immersion is exactly what it has been—from the very first day, I was adding ASP.NET MVC actions from scratch, writing jQuery selectors, refactoring CSS into idiomatic Less, and creating markup for stylized buttons for the oft-requested case-changeset dissociation feature, which made it into the first Hosted release of the summer.

When I finished working on Copilot for my summer, I had learned a lot about writing real software. Not stuff for class; stuff to ship. And it has served me amazingly well, not just at Fog Creek, but on every semester of school I’ve had since then, and on every hobby project I’ve taken on.

I’m sad to see this summer’s internship come to a close, but I’m very happy at what we were able to get done this summer—and I think all of our Kiln users will be, too.

And don’t forget: if you’re a student, and you want to be part of an internship or co-op like this, apply for our next cycle. I’d love to have you on my team.