One of the first features we added to Kiln was diff viewing: click a changeset, see what was changed. In terms of raw page views, viewing diffs is one of the most popular features in Kiln.

But viewing a single diff isn’t always everything you want. Often, you’ll want to see all the changes made between two changesets that are farther apart—perhaps the difference between two tagged releases, or the sum of the changes made in a feature branch. With Kiln Harmony, now you can!

Simply load any changeset, and you’ll see the diff from the changeset’s first parent. Click the “Diff from another changeset…” link to search for another changeset (you can search by commit hash, tag, and branch names, or phrases in the commit message, or even filter by author and date—it’s the full power of Kiln’s search engine). Click the results to view the diff from that changeset!

You can see some examples on our demo site:

• What changed in Mercurial 2.6.1? https://mirrors.kilnhg.com/Code/Mirrors/Tools/Mercurial/History/2.6/2.6.1
• What’s new since the last Git release? https://mirrors.kilnhg.com/Code/Mirrors/Tools/Git/History/master/pu

This feature is available in Kiln 3.0.33 and higher. Sign up for a free trial and try it out!

Do you use Sprint.ly for project management? Kiln now has a web hook that you’ll like! You can manage your Sprint.ly items using special comments in your commit messages.

Simply enable the web hook in your Kiln account, then reference Sprint.ly items using Sprint.ly’s commands. As you push to Kiln, you’ll see your changesets linked from your Sprint.ly items.

This web hook is available in Kiln 3.0.28 and higher. Sign up for a free trial and try it out!

We really liked seeing this post on how WooThemes uses Trello. We love it when people show their Trello boards. However, the screenshot has the text on the cards blurred out individually. That seems like a huge hassle. There had to be a better way to share screenshots without giving away the particulars of their board.

So we poked around on the internet and found this technique for blurring text with CSS. (Neat!) We tailored it for Trello and created a bookmarklet that we call “Blur My Board”.

Check out the bookmarklet here!

Need an example? Here’s an unaltered screen shot of the Trello dev board:

Click the “Blur My Board” bookmarklet and here’s what it looks like afterward:

Now snap a screenshot. Refresh the page and everything goes back to normal.

If only there were some aphorism we could use that would capture the concept that images convey more information than text. Oh well, we hope you enjoy the bookmarklet at least.

If you want to show us your board, get your bookmarklet here, and after you’ve blurred it, put it online somewhere and tweet us about it @trello!

Back in October the New York harbor paid an unwelcome visit to the datacenter that houses our servers. Followers of this space are aware of the heroic efforts that literally kept the lights on. Those events were  inspiring and made us all proud to be part of Fog Creek. But as Sys Admins our job is to view heroic efforts as a failure in planning, preparation and architecture. So as soon as the flood waters receded we set ourselves on a path to improve the continuity and availability of FogBugz and Kiln, with the ultimate goal a second geographically diverse datacenter. As with all lofty goals the path has several large phases and milestones. Now, some of those milestones are upon us and it is time for an update.

The first major step is moving all of our servers from six individual colocation cabinets to seven racks in a cage. The actual move distance is only about 10 meters but represents an upgrade in almost every way. These infrastructure upgrades will enable us to support new releases the FogBugz and Kiln teams are about to ship (teaser alert: we’ve been dog fooding them the last couple of months and they are awesome, contact us if you want to participate in the beta). The cage will provide Fog Creek a solid foundation for the future and the ability to free up enough gear to equip a second datacenter.

We are striving to keep the impact of this move as transparent as possible. However, during March we will have three significant Saturday night maintenance windows during which FogBugz and Kiln will be unavailable. Our goal is to keep these outages a short as possible, as few as possible, and during a time of low usage for a majority of customers. To paraphrase Moltke “No project plan survives first contact with the server room,” so more details on the timing and scheduling will be forthcoming. Watch this space and our status blog for more details. We are committed to the Fog Creek guarantee of not wanting your money if you are not amazingly happy. These upgrades are a huge  part of our commitment to live up to that guarantee. Please let us know if these outages materially impact your business and we will make it right.

Here are some questions you are probably asking and we have asked ourselves over the past months:

Why stay in Peer1?
Why stay in a datacenter located in one of the most expensive pieces of real estate, in a flood zone, in a shared building? All excellent questions, but as we debated the various answers we realized there was nothing wrong with our current datacenter that a second datacenter wouldn’t fix. In a disaster (natural or man made)  situation, a second geographically diverse datacenter with a tested and practiced failover procedure is our best option for providing our customers with continued service. Fog Creek has grown up with Peer1 over the past 10 years and we have a great relationship with Mike and Scott (the real heroes of Sandy) who operate the facility. So we decided to stay and make Peer1 part of our overall strategy.

Why not move to the cloud like Trello?
We aren’t parochial about choosing technology solutions. We want the best technology to solve the problem at hand (just search for cloud vs. dedicated server if you are looking for parochial arguments). Before Sandy arrived we had begun the process of moving Trello to AWS. A technology stack well suited to horizontal scaling and stellar growth made Trello an excellent candidate for cloud hosting. FogBugz and Kiln use different technology stacks and are more I/O intensive than Trello. For these and many other reasons, the cloud isn’t the right solution for FogBugz and Kiln at this time.

We will have before and after pictures and some other fun vignettes into the process. If you want to work on awesome exciting projects, we are looking for experienced unstoppable Sys Admins!

With Ben M firing off useful, pithy blog posts faster than Rick Moranis traveling in space, there sure is a lot to keep track of on the Fog Creek blog these days. To make sure our Sales team stays abreast of new posts, I threw together a simple IFTTT mashup that activates reminder cases within their normal FogBugz workflow any time a new post comes out.

While IFTTT doesn’t currently have a FogBugz channel, it was easy to set up a recipe that monitors our blog’s RSS feed and emails a FogBugz mailbox after any feed activity. Here’s the cool part: when sending an email into FogBugz, you can seamlessly route it to a specific case by including the case number in the email’s subject line. So, all I had to do was create an IFTTT recipe and corresponding FogBugz case for each team member, include the respective case number in the IFTTT action’s subject line, and voilà! Now each Sales member is notified within FogBugz any time a new blog post comes out, and they’re free to close the case, set a Do Later task, forward it to their weird uncle, or do anything else they see fit.

A big part of our series on how Fog Creek does customer service has been avoiding having cases—which represent communication with our customers—fall through the cracks. Filters are probably the broadest solution, but here at Fog Creek, we’ve also installed a 46” television (with embedded Windows XP—rock on), mounted it vertically, and run a full screen web page in Google Chrome. We call it Big Board.

The goal with the Big Board is to give the support team useful information in higher density. We found out early that when you put a number on the board, it will move, up or down, for a short amount of time, usually no more than a month. Then it’ll get “stuck.” The novelty of a moving number attracts interest, which moves the number more quickly. But then people move on to other things, and you have a number that doesn’t really mean much to anyone.

The solution here is not to remove numbers from the big board (though we have largely stopped putting numbers up there). The real issue here is to make sure from the beginning that you can change out non-moving metrics for ones that are consistently interesting, and interestingly displayed.

Solari Board for Scheduled Calls

Borrowing a metaphor from railway stations, the Solari board shows us scheduled calls (for more details, read Schedule Calls, Protect Your Support Team). The email address tells us which customer we’ll be calling and the track number tells us which support rep will be taking the call. The data is backed by FogBugz and will automatically update when a case is added or removed.

The skeumorphic Solari board has an important feature of its real-life counterpart: it makes noise when new information comes in. We use a recording of a real-life Solari board. When we hear the clattering, we know there’s a new tech call.

Office Calendar

Our company’s shared Google vacation calendar shows who’s out over the next couple of days. Is someone at the doctor, on vacation, or at a conference? That shows up here.

FogBugz and Kiln Details

We have an internal build system that we call “Mortar,” which integrates nicely with Kiln and has a local web front-end. Early in its creation, the engineers started putting useful information for the support team on mortar’s home page. We decided to take the most useful information, like current version numbers for Licensed and On Demand, and put those on the Big Board. Below that, we see the current primary contacts for the two teams. The support team channels interruptions through the primary contact, which minimizes noise for the rest of the team.

Support Team Distribution

The radar chart uses a poorly documented Google Chart API. There was a lot of fumbling around to get it to work correctly, but it’s actually a very information-dense part of the display. It gives us “feel” metrics that are also occasionally actionable. It changes quickly enough to be useful, and it’s gratifying to see the size of the yellow polygon shrink over the course of the day, finally collapsing down to green and letting us move on to more strategic tasks.

Bonus Calendar! Support Team

The grid below the radar chart shows us which support team members are out of the queue for the next two weeks. If a team member is on vacation, sick, or on a “dev day,” their avatar will show up here. The blank avatars you see represent available dev days, which are days that a member of the team can spend out of the support queue to work on strategic objectives. The first generation of the Big Board was developed by a member of our support team through a series of dev days.

Want your own big board? Well, you can use something like Geckoboard, or you can write one yourself. In your initial design, it’ll be difficult to pick the perfect metrics. You’re far better off spending your time making sure you can change out tiles quickly. Since you get to control which browser you use, you don’t have to worry about making sure it looks good on other browsers, which is nice. Play around with it and let us know via Twitter if you come up with anything cool!

When a case is assigned to you, FogBugz is going to notify you about everything that happens to that case: changing projects, email replies, useless banter, EVERYTHING. Well, almost everything. The one exception is that if the edit involves changing the assignment to someone else, FogBugz isn’t going to let you know. This of course is by design; why send you a notification when someone else is going to take care of the work?

If you’re the type of person that wants to know when someone steals a case from you, check out the Case Swiped Notification Plugin. Once it’s installed and you enable it in My Settings > Options, you’ll get an email notification every time someone swipes one of your cases. Now you can tell the offender to keep their grubby paws off your cases (or better yet, thank them for helping you out).

Here at Fog Creek, Kiln is the backbone of our development workflow, helping us write cleaner code and manage our release cycles. We also want to hear from YOU about how Kiln is benefiting your team or organization.

We will be putting together a virtual focus group to better understand how our customers use Kiln and what they would like to see in future versions. If you are successfully using Kiln within your organization, we would love for you to be a part of this group. If you consider yourself fanatically in love with Kiln, we REALLY want you in this group (that’s why this message is in a blog post and not, say, in an email blast to our customer list).

To be considered for the focus group, please fill out a brief survey. Participants for the focus group will be selected from those who have completed the survey. The discussion itself will take less than an hour and participants will receive a $50 Amazon gift card as an honorarium.

Take the survey here: Tell us about your experience with Kiln

November 26 Update: Power has been completely restored to the building where our data center is located and the situation is now considered back to normal. More details on the status blog.

For those following our status blog and Twitter feed, the events of the last two week have been hard to miss. We wanted to post a quick summary, current status, and a couple of thoughts.

On Monday, October 29th, we posted that all systems were up and we were actively monitoring the situation. Peer1 was ready to run on backup generator power for days if necessary. That evening, the storm surge from Sandy, assisted by a rising tide and full moon, flooded the basement of our data center, cutting off the fuel supply to the backup generator. The next morning, Peer1 informed us of an impending emergency shutdown of the generator. We executed a protective shutdown to prevent loss or corruption of customer data. Later, when we’d secured confirmation from Peer1 that there was no imminent danger of power loss, we restarted our systems. The total duration of this unplanned, voluntary downtime was 3 hours.

The extraordinary efforts of Fog Creek, Square Space, Stack Exchange and Peer1 over the next three days have been covered by several news outlets. For a blow by blow account, check out the latest Stack Exchange podcast.

Currently, the rooftop generator has a steady supply of fuel. A separate “roll-up” generator (the size of a shipping container) has been parked next to the build to provide a second source of power. Power at the data center has been switched back and forth between these two generators to confirm that continuous, redundant power is available.

The building is still not on city power. Thirty feet of seawater have had to be pumped out of the basement, leaving behind damaged electrical systems that must be repaired and replaced before safely connected to the grid.

We take the trust of our customers seriously. We’ve been gratified by the expressions of support and appreciation. We’ve been inspired by our colleagues’ competence and dedication. However, we believe that extraordinary efforts should not be necessary to maintain smooth operations.

The lessons we’ve taken from this downtime are many and they’re still being processed. The actions we’re taking cannot be summarized in one post-mortem, but the changes will be obvious over the next several months.

We don’t want your money if you’re not amazingly happy. If you were materially affected by this downtime, please email us at customer-service@fogcreek.com and we will make it right.

The Fog Creek Sys Admin Team

Fog Creek’s services, included FogBugz On Demand, Kiln on Demand, and Trello, have been impacted by the severe flooding from Hurricane Sandy.

Please see http://fogcreekstatus.typepad.com/ for the latest updates.