Archive for the ‘General’ Category

7 Ways to Get More Out of Beta Testing

October 23rd, 2014 by Gareth Wilson

The weird and wonderful bugs that get thrown up when real users first start using your code never ceases to amaze. There’s always some odd edge case that had been overlooked, despite you having thought about little else for several weeks. We’ve been through this many times whilst developing FogBugz, Kiln and Trello. Beta testing is an important stage in the testing process. To learn a little about them check out the following video, then read on for our key tips:



Here’s 7 things you can do to get the most out of our your beta tests.

1. Ask for a commitment to provide feedback
Response rates will be higher if you ask your beta testers upfront to commit to providing feedback. This doesn’t have to be formal, just part of an application form. But having agreed to it, people are more likely to follow through.

2. Do not release with known bugs
Most beta testers will only provide feedback once. So you don’t want to burn any tester to just hear about known issues.

3. Allow enough time
Use the following as a rough guide. For a major development effort, say about a year’s work, then you’ll want 10-12 weeks for beta testing. Decrease as necessary – so if it took a month to develop, then around a week will suffice.

4. Be feature complete
Only beta test when you’re feature complete. Adding in things as you go sets you back to the start. Otherwise, it just means the new code and its impact on existing functionality isn’t as well tested as the rest. Something you’ll regret later.

5. Make it easy to get in touch
You want to make it as easy as possible for your beta testers to provide feedback, so give them direct emails and offer to jump on a Hangout/Skype if they’d prefer.

6. Remind, but don’t annoy
Whilst your product might be front and center for you, it’s not going to be that way for your beta testers. You’ll want to remind them along the way. But don’t over do it, they’re helping you out so you don’t want to annoy them with too many emails.

7. Don’t forget to provide feedback
Make sure during and after the test that you send them updates about how you are putting their feedback to use. People like to hear that their time wasn’t wasted. And don’t be tight with the swag – a free t-shirt can do wonders!

To learn more about beta testing be sure to check out the video above.

Fog Creek and Heartbleed

April 10th, 2014 by Mendy Berkowitz

Along with the rest of the internet, Fog Creek has been reacting to the Heartbleed vulnerability which was discovered this Monday.

TL;DR: FogBugz and Kiln On Demand, Copilot and the Fog Creek website were not vulnerable. Trello was vulnerable and has been remediated.

Fog Creek handles SSL connections at the load balancers in front of the application servers. The load balancers for FogBugz and Kiln On Demand, Copilot and the website are using the 0.9.8 branch of OpenSSL. This version does not have the newer heartbeat extension and is therefore not vulnerable to Heartbleed. None of your data on,, or has been exposed to the Heartbleed vulnerability.

The load balancers for Trello were using the 1.0.1 branch of OpenSSL and were vulnerable. We have upgraded OpenSSL and replaced the Trello certificates. More details and important recommended steps for all users are available in a post on the Trello blog.

We have also reviewed our vendors (for things like this blog) to ensure that our other SSL certificates were not potentially compromised. Fortunately we have not had to replace any other certificates.

If you have configured web hooks in your Kiln On Demand account, we recommend that you verify that the target servers have not been affected. If you use one of the pre-configured types, or your custom web hook uses HTTPS, the data sent to that external service may be at risk. We do not however, send any of your Kiln login credentials, so they are safe.

We take the security of your data seriously. We are committed to protecting it and to maintaining clear and open communication with you. If you have any questions, comments or concerns please contact us.

Fog Creek’s Ad Hoc Remote Work Policy, or, Working From Grandma’s House

September 12th, 2013 by Rich Armstrong

Not grandma.
Not grandma.


Once every few months someone walks into my office and says, “Hey, I’ve got this (reunion|wedding|grandparent). I’d like to figure out how to spend some more time with my family. Can I work remotely from my grandma’s house for a few days?”

I always said yes, and got pretty uneven results. When someone wasn’t productive, it was usually due to unforeseen circumstances that seemed obvious after the fact. So we started a list of things you’ll need for a short stint of working remotely. This list is a “policy” inasmuch as it represents all the things you need to do in order to be “covered” while working remotely for a short stint. If something foreseeable comes up that’s not on this list, then it’s our problem, not yours.

Two important things that this is list is NOT applicable to:

  • Working from home waiting for the cable guy or a mattress delivery. That all happens under your usual flexible work arrangement.
  • Going somewhere you’ve never gone before, like a cabin in Iceland or an apartment in Paris, where you don’t know what the infrastructure will be like. We’ll call this “working from Borneo”. We’re not against Borneo, mind you. It’s just not what we’re talking about here.

One thing this list is kinda applicable to:

  • Permanent remote work. After writing this all down, we realized that almost everything we’d recommend to the short stint people applies to people working remotely. We’re still learning from them, so might have more to say later.


  • A computer to work on. Duh.
  • “Approval” from your team lead. We’re not big on bureaucracy here, but this is just common courtesy.  Most of what you’ll have to do is covered below in “known availability”, but you should start with talking to your team lead.
  • An internet connection capable of easily handling a Google Hangout. Since we’ve opened up to remote workers, Hangouts have taken the place of conversations in the hallway and standups in someone’s office. Downstream is usually not the issue. Most basic internet plans are at least 5 mbps down, which is no problem. Being able to participate in a video conference usually means more that 1 mbps up. A lot of cheap internet service only offers 1 mbps up. How do you know if Google Hangouts will work where you’re going? Do a test Google Hangout with a person at the place you are going to be! (If there’s no one on the other end to do this with, the place you’re going is either vacation or Borneo (see above).)
  • Access to the resources you need to work. Make sure you can:
      • log in to your laptop as a local administrator without connecting to the Fog Creek network. This is especially true for Mac users, as Macs do not cache your credentials
      • connect to the VPN from your new location. Not all Internet connections, or routers, are created equal and not all will allow you to get on the standard VPN
      • RDP into your HQ machine, if you don’t have everything you need on your laptop

    If you’re unsure about any of this, talk to the sysadmins before you discover you can’t work remotely.

  • A headset. The built-in speakers and microphone on your laptop/monitor suck. The added friction of talking to someone without a headset makes your team members not want to invite you to a Hangout, which means you miss communication that you would normally have received.
  • A dedicated room with a door that closes. At Fog Creek, devs have offices with doors that close. People who aren’t devs are surrounded by coworkers who’ve learned to keep from inadvertently distracting each other. Your aunt has not learned this, so it’s probably a good idea to sequester yourself as much as is feasible. You need a place that is as distraction-free as your normal work environment, and maybe a little more.
  • No child care responsibilities during working hours. You can’t take care of kids and work at the same time.
  • Known availability and overlap with your team. Set standard hours and make sure people know them. If you’re not going to be working the hours you normally work, overcommunicate this to your team by posting in chat (ex: “Going to the gym for an hour”). Nothing is more frustrating to team productivity than chat messages like “Has anyone seen X?” “When is X getting online?” “Can someone else take this bug? X isn’t responding.” Try not to pop in and out. Work sustained sessions with only a few significant interruptions (e.g., lunch, dog walking, gym, light-saber fight with your nephew).
  • A dedicated phone that fits into your position’s workflow, where applicable. If you are an employee who spends a reasonable amount of time on the phone, it should be relatively easy for people to get you on the phone and for you to call them. This might require you to contact the sysadmins for phone forwarding. We haven’t done a lot of this yet, so we’re not awesome at it. Maybe you can use your experiences here to make the next person awesome?


Suggestions for Short Stint Remote

When you’re used to working in the office and you leave it, you’re like a snail without a shell. Consider these suggestions a temporary exoskeleton. These suggestions are not because we don’t trust you. We trust you. To get a desired effect, we often accentuate parts of our behavior that don’t come naturally. We might be introverts at a networking event, forcing ourselves to start conversations with strangers. We might be on camera, forcing ourselves to oversmile like idiots. Most of these tips are to encourage unnatural but beneficial behaviors.

  • Don’t be coy. Awareness is largely covered in “known availability” above, so this bullet point covers your mental state. You might be tempted to downplay your absence from HQ. Puritan work ethic runs pretty deep with some.  Fact is you can ship like an animal from places like Hawaii or Cape Cod (where this document was started). But you might be embarrassed to talk about it, or you might be put on the defensive by someone saying “Great beach weather.” Ignore it and ship code. Clamming up or playing down helps no one. It’s the opposite of what we want; it’s the main reason we have a policy. If you’ve met the requirements above, you’re covered. You are not slacking. You are working. You should not be shy about letting people know where you are and what you’re doing because of what they might think.
  • Put a note on your monitor saying where you are and for how long, in case people wander by looking for you.
  • Have boundaries. The people you’re going to be around might not understand that, yes, you will actually be working. Don’t worry. Your in-laws/parents will be impressed by your work ethic!
  • Over-communicate and/or over-deliver. At HQ, you have a few ways of staying involved and available (face time, informal chats, lunch) that are not available to you remotely. You still have stuff like chat, commit messages, code reviews, cases, Trello boards, etc. If you’re a Creeker, chances are good you’re a low-key person who doesn’t crow about their achievements. You might want to flip that bit for this stint. That can be hard, but it’s important to keep involved. The best way to do this is….
  • Have a deliverable. This frees you from having to assert your productivity via chattiness.

That’s it! We put this stuff out there so that you can effectively take advantage of the flexibility offered by your work arrangement and continue to be productive as part of your team. If you learn something while setting up your environment, or while working remotely, that you wish you’d known earlier, please let me know. The list you see above is a work in progress.

Get Started with Your Own Solari Board

August 8th, 2013 by Will Thompson

Some time ago we introduced our Big Board as part of our “How Fog Creek does Customer Service” series. We explained how the Big Board is useful to us because of the high density of information, but we also left you with an open ended question: “Want your own big board?” The Solari board, one of the widgets on our Big Board, is so important to our workflow that we felt that everyone else should be able to use it with anything: so we open sourced it on GitHub.

IRL Solari boards are manufactured by the firm of Solari di Udine in Italy. You’ll find them all over the world in train stations. They’re not just a great example of mechanical displays. The sound they make is functional. When the board updates, the clattering of the flipping letters alerts everyone in the waiting room to the fact that new information is coming in. It’s a beautiful solution for public displays of information. To capture this feature, we have our Solari board play a clattering sound recorded from the real Solari board in Union Station in New Haven, Connecticut.

This javascript widget is a UI metaphor. When a new tech call comes in, the board makes the clattering sound, and everyone is gently alerted to the fact that a new call has been scheduled. This might sound bothersome, but it actually helps a lot. If you look up and see that the new call is assigned to someone else, but the email address is someone you’re working with, you can quickly swipe the case. We use the “Track” column to display the extension of the assigned rep on the tech call. It looks like a real train track number, and we all know each other’s extensions.

The source code comes with an example Python script that gathers a set of cases from a FogBugz installation via the FogBugzPy Python wrapper and the FogBugz XML API.

And should your webpage ever lose connection to the POST endpoint (where the Solari board retrieves its data), the fail whale is sure to let everyone know.



Now get out there and condense that data into a nice, aesthetically pleasing, train-inspired Solari board!


The Solari board was originally created by Rich Armstrong as a way to display tech calls, then rewritten and refined by Rob Sobers

Friday Linkparty

October 21st, 2011 by Dan Ostlund

The things we have been reading lately

Betrayed by an accelerometer

How to draw a circle

9 Metrics for making good decisions for your startup: From the Kissmetrics blog. This is one of the best blogs on marketing, startups, good web design, and more. It’s worth frequent visits.

A walking man, done in CSS3

Some “leaks” from the forthcoming Steve Jobs biography

How to sort file names: Raymond Chen proves that computer programmers are not normal human beings.

More, more, more: A nice riff on our constant push to do more and be more.

Friday Linkparty

October 14th, 2011 by Dan Ostlund

Our latest installment

First there was the Joel test, now there is the Rands test: Go read this now for the sake of your company!

An animated history of the iPhone’s evolution And a lot of other stuff too.

Dennis Ritchie

Paul Rand and Steve Jobs making the logo for NeXT: Look for the link to an interview where Jobs talks about Paul Rand.

A beautiful, interactive exploration of understanding and solving a mathematical problem

The Price of (Dev) Happiness: Part Three

October 11th, 2011 by Rich Armstrong

Buying lunch in lower Manhattan is not cheap. If you don’t brown-bag, you’re pretty much in for nine bucks, and more if you want to sit down. A meatball sub is $8.50. The Hanover Square Deli does a respectable dduk mandu guk for about $9.50. A meager-looking, but tasty, turkey and guacamole sandwich from British import Pret A Manger will set you back $6.75 (plus tax). Or, if you really want to eke out some savings, you can get a cup of soup and a hunk of bread for about $5.

Post-lunch, Pre-espresso machine

At Fog Creek, we have our lunch catered every day.  The cost of this to the company is $15.75 per person per day. Not counting drinks and snacks, this amounts to an extra $4,000 per year per employee. So why wouldn’t we just pay people $4,000 a year more? If they want to be frugal, they can buy that $5 soup and pocket the difference.

Well, first off, since lunch is a catered on-site meeting, the cost of that lunch is 100% deductible as a business expense. If you worked here, it’d cost the company $16 to give you $10 to go buy your own mediocre udon noodles.  (Don’t take any taxability or corporate expense stuff here too seriously; I’m not an accountant or lawyer.) Second, Fog Creek’s free lunch is much more because of how it fits with the rest of our workspace and culture.

Free lunch is nothing new. Google and other big tech companies have been giving their employees breakfast and lunch (and sometimes dinner). Free lunch has been around for more than a decade because it works for attracting and retaining top talent. (Or, rather, people think it works.)

Now, the food at Google, and the people who make it, are awesome. The food at Fog Creek is good, but nowhere approaching what Google does. We don’t have Sam, the sushi chef, doing hand-rolls to order. We don’t do miso black cod, lamb shanks, or osso buco. We don’t have a raw vegan station with selections so delicious they attract the most dedicated carnivores.

Here’s what we have at Fog Creek instead: no meetings.

For us, lunch is our only recurring meeting. The only standing interruption in the day/week of a developer here. Everything else is ad hoc or temporary. (One exception is our quarterly meeting to go over financials and to grill the founders with questions.) The default at Fog Creek is no recurring meetings. Once you’ve established that, recurring meetings become the exception, rather than the rule, and tend to wither naturally as their usefulness degrades over time. For example, the FogBugz team is currently doing a stand-up meeting for fifteen minutes every day right before lunch, but this won’t last. The Trello team was doing them before and shortly after launch, but they’ve subsided.

The Kiln team lead assures me that their weekly stand-up is a real recurring meeting, but at some point they’re going to lose interest and go back to coding. It’s what they do. Meetings have a network effect. They need other meetings to legitimize them. If you’re constantly scheduled with meetings, you don’t mind being interrupted one more time. Or, rather, you don’t have the energy to protest. When you proceed from the assumption of no meetings, you have to expend effort to keep a meeting going.

It’s not all a playground, of course. There are downsides. Things, sometimes important things, don’t get communicated to the right people at the right time. More often than not, the worst result is hurt feelings or slight confusion. Sometimes it’s more than that. But we wouldn’t give it up.

For someone who’s used to a standard work environment, it seems silly, cult-like, possibly even daunting, to be “forced” to break bread with your colleagues every day. It seemed odd to me before I came here, but by the end of the first week, nothing seemed more natural. When most of your “socialization” with your colleagues takes the form of mandatory, recurring meetings over a conference table, it’s natural to not want to see them again over the lunch table. People have been making decisions about your time all day; at lunch, you need some time to yourself.

But when most of your time is spent working in a private office, taking breaks according to your natural attention span, having short chats with one or two colleagues, it’s a pleasant prospect to surface for some pleasant conversation about StarCraft or football with nice, intelligent people. And maybe you’ll hash out that new feature, too.

One can make the argument for free lunch based strictly on developer productivity. A free lunch could give you a hundred hours per year from your best people, time saved in driving, waiting in line, etc. But, consider the depressing ease with which such a gain can be wiped out by a few recurring meetings.

So, for our final post in this series, we do have a number: $15.75 per day. $4,000 per employee per year. It’s a lot of money. But without the rest of the company culture, it would be sort of meaningless. A sense of entitlement grows rapidly around any perk you offer, and lunch is no different.

A lot of this series has been based on getting hard data out there so that developers, our main audience, have an easier time talking to management about some of the things that’ll make them more happy, healthy, and productive. For this post, it’s a bit more difficult. You might get your boss to spend $16 a day, but changing the culture of meetings in any workplace is nearly impossible. (During my tenure, Google tried no-meeting Thursdays and formal meeting-reduction task forces, reminiscent of Brazil’s Ministry of Debureaucratization, to no avail.) It would require rebuilding the company culture from bedrock.

Our bedrock is the idea that, once we’ve hired good people, it’s the effort we make to direct their intention, rather than their attention, that creates value. It’s not just our lunch benefit that springs from that, but nearly every other thing we do.

For Fog Creek, our founding principles, and the pains we take to stick by them, are the price of developer happiness, and that can’t be measured in dollars.


Friday Linkparty

October 7th, 2011 by Dan Ostlund

Here is our latest installment.

Why code reviews beat testing: We knew we put code reviews in Kiln for a reason.

Geoffrey Moore talks about core vs. context in your business: This is fantastic talk from the 2009 Business of Software conference. If nothing else, it will make you think hard about your business.

The king of code comments?

The recent Node.js controversy

The always fascinating Richard Feynman: Three videos that use Feynman’s voice to talk about beauty, prizes, and curiosity.

Algorithm is not a four-letter word: An awesome overview of maze generation algorithms.

How to generate music with one line of code

Friday Linkparty

September 30th, 2011 by Dan Ostlund

Here’s our latest installment of things we’ve been reading.

Harvey Mudd gets more women: The President of Harvey Mudd college has managed to triple the number of female comp sci majors since 2006.

A proposal to fix callback hell in Node.js

Inbound Hiring: I think most startups do this, but here’s a short analysis on early hires from Gabriel Weinberg.

Why blog?: Also from Gabriel Weinberg. What he’s learned over the last year of committing to his blog.

Try 16 different languages in your browser

Experiments with Google Page Speed Service

September 23rd, 2011 by Jude Allred

Our oldest product

Let me tell you a little bit about

  • It’s old; It’s hosted by IIS6 alongside a horde of other websites.
  • It’s the first place people go when deciding whether to use FogBugz or Kiln.  It’s pretty important to us.
  • Recently, we started to treat the website as a product. It has received substantial attention, primarily in the form of A/B tests.

We’re building some fancy new web servers for, but in the meantime we’ve been accepted into the Google Page Speed Service beta. It seemed reasonable that it might provide us with some useful tools, and in the short term it might give us a workaround for some of our IIS6 woes.

There were four things that we hoped Page Speed Service would take care of:

  1. GZIP all of’s static and dynamic content.
  2. Set far-future expires headers on our static content.
  3. Distribute our static content via a CDN.
  4. Fix or remove the broken ETags that IIS 6 Is sending out.

Enter Google Page Speed Service

For this test, we brought Google Page Speed Service to life at This is a totally unused alias of, so we were able to test PSS on an isolated mirror of our production environment.

After a small amount of setup, page speed service was live!  (Setup basically consisted of adding a Google verification .txt record and a CNAME record mapping to

The comparison tool that Page Speed Service provides was inconsistent, but it looks like there might be a lot to love.

  • Its page load times differ from those reported by GTmetrix and local testing
  • The measurements change significantly over the course of running it multiple times

But the results look promising.  Here’s a report on Fog Creek’s home page load time.  On the left is the load time without Google Page Speed and on the right with it.  Google PSS gave us a 10% difference in initial page load time, and a 25% difference in repeat page load time. We got that for very little effort.  Nice!

Here are GTmetrix reports on from 9/23/2011.  First, the results without the Page Speed Service:

And the same tests on, with Page Speed Service enabled:

Overall scoring improvements:

The Fog Creek landing page:

Google PS +1%, YSlow +10%, page weight reduced from 466kb to 365kb

The FogBugz landing page:

Google PS +1%, YSlow +12%, page weight reduced from 608kb to 379kb

The Kiln landing page:

Google PS +0%, YSlow +10%, page weight reduced from 520kb to 411kb

Interestingly, GTmetrix shows page load times (as measured by the onload event) that are consistently slower for than, usually on the order of about 0.7 seconds.

GTmetrix’s tests don’t seem terribly consistent with’s (Google’s promoted testing service), which show consistently faster page loads than GTmetrix. But, it’s hard to draw conclusions since incremental rendering of the page is an important factor which isn’t represented by the load time.

For those of you who wish to dig deeper, take a look at the waterfall diagrams of a given page rendered with or without PSS. You’ll notice that the PSS timelines have their content proxied across 4 of Google’s servers, the overall number of requests has decreased, and the waterfall has rearranged significantly.

For example:, Without Google Page Speed Service:, With Google Page Speed Service:

So how did PSS handle the goals we wanted to meet?

  • PSS has compressed (GZIP) the content that it sends down
  • PSS has far-futured the vast majority (but not all) of our static content
  • PSS has properly configured our ETags
  • PSS is distributing our content via its CDN


1 - PSS’s lossless image compression is lossy.  Our website uses a tiled, textured background image.  Page Speed Service attempted to optimize it, and in the process altered the background images.  For example, this image:

Became this one:

I’ve put together a test case at

As part of the lossy conversion, PSS converted the image from a png into a jpg.  I altered the source image files to be jpgs instead of pngs, and then PSS was able to optimize and bundle them without any difficulty.   I’d also note that the image optimization step (and all of their other optimizations) can be disabled in order to work around problems like this.

2 - PSS supports blacklisting URLS, but not blacklisting HTTPS urls.  I knew from the start that PSS didn’t support HTTPS, and therefore our trial signup forms, e.g.,, would have to be exempt from PSS’s optimizations.

PSS supports blacklisting, which allows you to exempt a specific page from being hooked by PSS’s proxy.  Instead it will go directly to your source server.  Unfortunately you can’t configure HTTPS pages in this manner.  If you have a website with HTTPS content, you’ll have to move all of your HTTPS content to a separate subdomain, e.g. “”, in order to exempt it from PSS.

This was an inconvenient change to make, although not hard. Still, PSS’s support of blacklisting was put to good use.  We use a heavily-modified version of the FairlyCertain A/B testing framework on our properties.  Server-side A/B test frameworks and CDN’s don’t really play well together, but PSS’s blacklisting allows us to blacklist any pages which contain active A/B tests.

3 - For the duration of the beta, once you bring PSS live on a given subdomain (e.g.,, you cannot reconfigure PSS to work with a different subdomain (e.g., without an additional beta acceptance.  I’ve reapplied to include in the PSS beta, but have yet to be approved.  For this (and only this) reason, we’re blocked on moving forward with using Google PSS in production.  If a Googler is reading this and would like to whitelist for the beta, we’d be much obliged.


Pitfalls and onload-event oddities aside, I’m very impressed with Google Page Speed Service.  I’m happy to see that they’re abstracting away the ever-present need to minify, compress, and bundle your static website content, as well as optimize your images.  In our specific case we don’t really care about those  features – we already optimize our images and bundle our scripts, but it is a great feature for most sites.  Our case is probably more edge than common—we’re using PSS primarily to work around IIS6, and it appears to succeed at that.  PSS’s CDN seems promising, but I don’t yet have any data on how it compares to, say, Amazon’s Cloudfront service.  Setting up PSS on our site was an absolute breeze, however, and  I think it may turn out to be a great tool for web developers and site administrators.

The choice of setting up the Page Speed Service on instead of our production subdomain, retrospectively, was a useful mistake.  I appreciate the ease of using tools like GTmetrix to compare the two subdomains directly. I’ve also enjoyed keeping PSS’s configuration changes completely separate from our production environment as well as the ability to share the PSS changes with other people without having to manually configure their browser to proxy traffic through PSS.

Looking for more?

Visit the Archives or subscribe via RSS.