Archive for the ‘FogBugz’ Category

Production Debugging: A story about “Exception code: 0xe053534f”

March 20th, 2014 by Aaron Maenpaa

Imagine for a moment that you’ve been running an ASP.NET app for years, and on one otherwise sunny Saturday morning, you get a call saying that the app has started crashing randomly. What do you do?

Step 0: Check the event log

It took me way too long to get used to checking the Event Log (Start → Event Viewer). I don’t know if I expected it to be loggier, or what, but there’s good stuff in there. You should check it out.

You might even find an error like this one (Windows Logs → Application) :

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17651, time stamp: 0x4e21213c
Exception code: 0xe053534f
Fault offset: 0x000000000000cacd
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

This report is a little terse but it can be the gateway to figuring out what’s wrong. “Faulting” sounds bad: It means “crashing”. Translating to English, It’s saying “the IIS worker process (w3wp.exe) crashed”. Since this is exactly what you’re looking into, it’s a start. Exception code 0xe053534f is a stack overflow (I wish I had advice for how to map exception codes to what the problem is, but as far as I can tell, Google and hope are your best bet).

Ok, so what now?

Step 1: Googling

Looking around, amongst the “I’m getting 0xe053534f, how do I fix it?” posts on forums you’ll realize that what you want is a crash dump, and you’ll find some advice on how to obtain one, like this, or this.

You probably don’t want to use AdPlus

While AdPlus is indeed super cool, and indeed has a mode that will allow you monitor all of the IIS worker processes (a command which I will not include here because I don’t want to encourage you to use it), in my experience the overhead was way too high: The throughput on the monitored servers dropped to the point where 1. They basically weren’t doing any work, 2. They stopped crashing (Don’t worry… subset of servers during a time of reduced throughput requirements, negligible customer impact).

You probably don’t want to try and set up an Unhandled Exception Module

At least not while you’re under the gun. I never actually got it to work, and it wouldn’t actually help us here since it probably wouldn’t get called for a stack overflow anyway.

Step 2: Check the event log

Did you know that Windows Error Reporting has your back? If you look at the Event Log, the event right after the crash should be from Windows Error Reporting and hopefully it has some more information like:

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0
Problem signature:
P1: w3wp.exe
P2: 7.5.7601.17514
P3: 4ce7afa2
P5: 6.1.7601.17651
P6: 4e21213c
P7: e053534f
P8: 000000000000cacd
Attached files:
These files may be available here:
Analysis symbol:
Rechecking for solution: 0
Report Id: 3bfe0e87-a15d-11e3-a83d-14feb5d64ad7
Report Status: 4

… okay, so that’s not super informative, but there’s a line in there that should get you excited, C:\ProgramData\Microsoft\Windows\WER\ReportQueue is a folder that might already contain crash dumps… no collection required! You probably want to look there and see if there are dumps that look relevant to resolving your problem.

What if there aren’t any crash dumps?

Check all affected servers, you only need one. I never got around to trying it, but SysInternals ProcDump looks like it might do the job, hopefully without taking the toll that AdPlus did.

DebugDiag also came up repeatedly while I was looking around, maybe that would do the trick. Looks more complicated to get going than ProcDump though.

Step 3: Windbg

Okay, you’ve got a crash dump, open it up in windbg, and look around (there are tons of windbg cheat sheets around like this one, this one, or this one). We’re going to:

  1. Load the CLR extensions (.loadby sos mscorwks).

  2. Load the symbols (.symfix).

  3. Inspect the stack (!clrstack).

  4. Inspect the exception (!printexception).

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1cbc.3d34): Unknown exception - code e053534f (first/second chance not available)
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for KERNELBASE.dll -
00000000`76ed135a c3              ret
0:078> .loadby sos mscorwks
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
0:078> !clrstack
* Symbols can not be loaded because symbol path is not initialized. *
*                                                                   *
* The Symbol Path can be set by:                                    *
*   using the _NT_SYMBOL_PATH environment variable.                 *
*   using the -y <symbol_path> argument when starting the debugger. *
*   using .sympath and .sympath+                                    *
PDB symbol for mscorwks.dll not loaded
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
c0000005 Exception in C:\Windows\Microsoft.NET\Framework64\v2.0.50727\sos.clrstack debugger extension.
PC: 00000642`ffcbf8e4  VA: 00000000`00000000  R/W: 0  Parameter: 00000000`00000000
0:078> .symfix
0:078> !clrstack
OS Thread Id: 0x3d34 (78)
(!clrstack processes a max of 1000 stack frames)
Child-SP         RetAddr          Call Site
000000000e0e6c70 000007ff01a617e9 HtmlAgilityPack.HtmlNode.CloseNode(HtmlAgilityPack.HtmlNode)
000000000e0e6cf0 000007ff01a617e9 HtmlAgilityPack.HtmlNode.CloseNode(HtmlAgilityPack.HtmlNode)
000000000e0e6d70 000007ff01a617e9
<many, many, many more lines of that>
0:078> !printexception
Exception object: 0000000270090138
Exception type: System.StackOverflowException
Message: <none>
InnerException: <none>
StackTrace (generated):
StackTraceString: <none>
HResult: 800703e9

Here’s what we’ve learned so far:

  1. We’ve confirmed a stack overflow as the cause of our problem.

  2. We’re seeing the error in HtmlAgilityPack, but we can’t actually see the base of the stack because !clrstack gives up too soon.

Delve Deeper

!clrstack won’t go deep enough to show us the base of the stack, but we can get the base of the native stack via k 1000

0:078> k 1000
Child-SP          RetAddr           Call Site
<the native error handling frames>
00000000`0e0e6c70 000007ff`01a617e9 0x000007ff`01a617bc
00000000`0e0e6cf0 000007ff`01a617e9 0x000007ff`01a617e9
<lots of stack frames>
00000000`0e13ad70 000007ff`01a5d442 0x000007ff`01a617e9
00000000`0e13adf0 000007ff`01a5cf28 0x000007ff`01a5d442
00000000`0e13ae80 000007ff`01a5c39a 0x000007ff`01a5cf28
00000000`0e13aed0 000007ff`01a5bd9c 0x000007ff`01a5c39a
00000000`0e13af20 000007ff`01a5bbaa 0x000007ff`01a5bd9c
00000000`0e13afe0 000007ff`01a5b224 0x000007ff`01a5bbaa
00000000`0e13b020 000007ff`01a5aff3 0x000007ff`01a5b224
00000000`0e13b060 000007ff`01a5af05 0x000007ff`01a5aff3
00000000`0e13b0b0 000007ff`01a6f836 0x000007ff`01a5af05
00000000`0e13b0f0 000007ff`01a582e1 0x000007ff`01a6f836
00000000`0e13b4e0 000007ff`01a50fd0 0x000007ff`01a582e1
00000000`0e13b7f0 000007ff`0221f79c 0x000007ff`01a50fd0
00000000`0e13bdf0 000007ff`019513ef 0x000007ff`0221f79c
00000000`0e13be70 000007ff`0194c972 0x000007ff`019513ef
00000000`0e13bf90 000007ff`01949ff0 0x000007ff`0194c972
00000000`0e13c2f0 000007ff`01341aa2 0x000007ff`01949ff0
00000000`0e13c6a0 000007ff`013409c6 0x000007ff`01341aa2
00000000`0e13c900 000007ff`00ffc48e 0x000007ff`013409c6
00000000`0e13c9d0 000007ff`00f1708c 0x000007ff`00ffc48e
00000000`0e13ccd0 000007ff`008f5ec4 0x000007ff`00f1708c
00000000`0e13cdc0 000007ff`00d23e05 0x000007ff`008f5ec4
00000000`0e13d600 000007ff`00c7e416 0x000007ff`00d23e05
00000000`0e13d6d0 000007ff`00c7ad5e 0x000007ff`00c7e416
00000000`0e13d7d0 000007ff`00c78d85 0x000007ff`00c7ad5e
00000000`0e13d960 000007ff`00c6ccb9 0x000007ff`00c78d85
00000000`0e13d9b0 000007ff`00c69be1 0x000007ff`00c6ccb9
00000000`0e13dad0 000007ff`00c691ab 0x000007ff`00c69be1
00000000`0e13dc50 000007ff`00c684c4 0x000007ff`00c691ab
00000000`0e13dcb0 000007fe`f14ee0ba 0x000007ff`00c684c4
00000000`0e13dcf0 000007fe`f47451c7 mscorwks!UMThunkStubAMD64+0x7a
<more native frames>

We’ve got the base of our stack, we just can’t read it :). Fortunately, the !IP2MD command will turn the last pointer into a description of the method:

0:078> !IP2MD 0x000007ff01a617e9
MethodDesc: 000007ff0237bed8
Method Name: HtmlAgilityPack.HtmlNode.CloseNode(HtmlAgilityPack.HtmlNode)
Class: 000007ff01d1da00
MethodTable: 000007ff0237c3d8
mdToken: 0600000b
Module: 000007ff0237abc0
IsJitted: yes
CodeAddr: 000007ff01a61700

Putting together some copy/paste, some Vim, some !IP2MD and some grep you can turn that list of pointers into the base of your stack:

HtmlAgilityPack.HtmlDocument.PushNodeEnd(Int32, Boolean)
FogCreek.HtmlParser.HtmlParser.ExtractSafeHtml(System.String, FogCreek.HtmlParser.IHtmlTransform[])
FogCreek.FogBugz.__Global.GetMessageTextByMsg(Wasabi.Runtime.Mail.MailMessage, Boolean, Boolean, System.Nullable`1<Boolean>)
FogCreek.FogBugz.CFullTextIndexer.IndexBugEvent(FogCreek.Search.Interfaces.IDocument, System.DateTime ByRef, Int32 ByRef, FogCreek.FogBugz.CBugEvent, System.Nullable`1<Boolean>)
FogCreek.FogBugz.CFullTextIndexer.IndexBug(Int32, Int32, Int32 ByRef)
FogCreek.FogBugz.CFullTextIndexer.MemorySafeIndexBug(Int32, Int32, Int32)
FogCreek.FogBugz.CFullTextIndexer.IndexItem(System.String, Int32, Int32, Int32 ByRef)
FogCreek.FogBugz.CFullTextIndexer.UpdateByType(System.String, Int32)
FogCreek.FogBugz.CFullTextIndexer.HeartbeatUpdateIndex(FogCreek.FogBugz.CNotification, System.String, Int32)
FogCreek.FogBugz.__Global.SingleHeartbeatFogBugz(Int32, Int32, Int32, System.DateTime)
FogCreek.FogBugz.__Global.SingleHeartbeatPlatform(Int32, Int32, Int32, System.DateTime)
FogCreek.FogBugz.__Global.DoWork(Int32, FogCreek.FogBugz.CTrialManager)
System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)

Step 4: Fix the Bug

What we’ve learned:

  1. We’ve got a crash caused by recursion in HtmlAgilityPack.

  2. Our code is calling into HtmlAgilityPack while processing email for the search index.

You can fish around in the dump a little more to try and find stuff like parameters that will help you track down the requests causing problems, but that was enough to point me in the right direction.

Not coincidentally, we’ve shared our fork of HtmlAgilityPack on GitHub (and a patch on Codeplex).

Bonus: Even More Debugging

While investigating a separate crash I found use for a different command: !analyze -v, which helpfully extracted the relevant exception and call stack all on it’s own:

0:054> !analyze -v
<lots of stuff>
EXCEPTION_OBJECT: !pe ff5ce5e0
Exception object: 00000000ff5ce5e0
Exception type: System.InvalidOperationException
Message: The ConnectionString property has not been initialized.
InnerException: <none>
StackTrace (generated):
StackTraceString: <none>
HResult: 80131509
MANAGED_OBJECT: !dumpobj ff5d0210
Name: System.String
MethodTable: 000007ff001d0d90
EEClass: 000007ff001ccdf0
Size: 2226(0x8b2) bytes
String:    at System.Data.SqlClient.SqlConnection.PermissionDemand()
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at FogCreek.Database.Connection.Open() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Database\FogUtil.Database\Connection.cs:line 79
at FogCreek.Database.Command.ExecuteNonQuery() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Database\FogUtil.Database\Command.cs:line 164
at FogCreek.Database.IndexFile.Unlock() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Database\FogUtil.Database\IndexFile.cs:line 149
at FogCreek.Search.Store.Lock.Release() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Search\Store\Lock.cs:line 26
at Lucene.Net.Index.IndexWriter.Finalize() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Search\Lucene.Net\Index\IndexWriter.cs:line 562
<even more stuff>

Oh look, a finalizer causing a problem. I’ve never seen that before</sarcasm> (Seriously though, if anyone has ever told you that you probably shouldn’t use finalizers, this is why: they’re hard to test, they can interfere with the operation of the garbage collector, and they can straight up crash your app if they generate exceptions).


I put this together because while the internet is rife with WindDbg cheat sheets, it doesn’t seem to have quite as many walk-throughs of how the various and sundry commands might apply to debugging ASP.NET crashes, which commands are useful and what kind of output you might expect. Hopefully this helps you. I know it will help remind me of what to do the next time I need to resort to WinDbg.

PS: If debugging production apps makes you feel a tingling sense of accomplishment, you might like to know that we’re hiring.

FogBugz Visits the Head(er) Shrinker

February 7th, 2014 by Adam Wishneusky

The look and feel of FogBugz On Demand hasn’t changed much in two years. We thought we’d update it, starting with the part that shows up on every page: the header. We re-designed the FogBugz On Demand header to make it more modern, compact and organized.

R.I.P. old header…

old header

Long live new header!

new header

Not everyone has the luxury of a 4K display. When you’re working on a small laptop screen, space is limited. Menu bars, docks and taskbars gobble up vertical space. To make more room for your actual work, we made the new header 15% shorter. We did this by consolidating the two rows into one. The FogBugz and Kiln links from above moved into a hover menu:

new - product dropdown

To fit the menus from the top-right into the new design, we made smarter use of the width of the bar. Our customers told us that they work mostly with cases, so we grouped all of the case-related controls at the left. We used symbols for the right-hand menus and removed the List Cases button. You now get to your filters by hovering over Cases.

The first thing we learned when we rolled the new design out to 10% of accounts was that you click List Cases a LOT. To give you faster access than hovering then clicking inside the new menu, we made Cases clickable. Hover over it to get to a specific filter, or just click on it to show your current case list.

new - cases dropdown

Hover over the four right-hand menus and you’ll see that Admin is now “gear,” Extras is now “toolbox,” Help is now “?” and My Settings is now the avatar menu. Log Out has moved into the avatar menu. If you have a larger screen, you have no doubt noticed that the old search box didn’t do you any favors. Whatever length text you entered, it stayed tiny. The new box knows that sometimes you use a lot of search axes in order to narrow down and speed up your search. While it starts out compact,

new - search box 1

When you click into it and type, it grows so you can see what you’re doing:

new - search box 2

We’d love to know what you think of it so drop us a line. Not using FogBugz On Demand? Sign up for a free trial today!

Bee: A Desktop FogBugz Client for Mac

November 20th, 2013 by Adam Wishneusky

Bee is a Mac client for FogBugz. It makes tracking your cases simpler and faster with its beautiful design with fluid animations. It is deeply integrated with Mac OS X so you can receive notifications from your teammates and use QuickLook to view case attachments. You can time cases from your menubar and Bee also intelligently tells you what you should be working on next. In addition to FogBugz, Bee also works with plain text files for your project notes and supports syncing tasks with GitHub and JIRA.

Bee requires Mac OS X 10.8+ and is available from the Mac App Store. A free trial and further information is available from the site.

Case Event Merging – One Case to Rule Them All

November 8th, 2013 by Stephen Asbury

If you’ve been using FogBugz for issue tracking or customer support for a while, you’ve probably encountered a situation like this. One of your customers sends you an email asking for a new feature, and FogBugz creates a case for you to track the request.


Five minutes later, the same customer sends another email with brilliant new insights as to how the feature should work. Here comes another case.


So now you have two cases about the same request. What do you do? You don’t want to reply to the same customer request in two places. You don’t want to copy and paste information from one into the other. You or someone else at your company might miss the duplicates list and the other email chain. It would be much better if you could see both emails in a single case.

Now you can!

Introducing Case Event Merging

Well, maybe I should say re-introducing. We introduced case merging back in August of 2012. At that time we received an excellent solution from one of our customers, for which we are very grateful. We liked the idea so much that we have built it into the FogBugz On Demand Performance Upgrade, hopefully with a few improvements.

Now, when a case is marked as a duplicate, FogBugz does two things. First, it places a message at the top of the duplicate case telling you that its events are now also merged into another case.


Second, FogBugz provides you with controls on the main case, located in the “Duplicates” area of the side bar, to show or hide the merged events.


Three controls indicate the types of events you can merge from duplicate cases. From left to right these are:

Email events are shown by default so that you can see the full conversation with your customer. Comments and BugzScouts are initially hidden. Click on each control to toggle the visibility of the merged events. The events are inserted into the normal event list and highlighted in purple, as shown below. They will also let you know which case the event comes from.


We have found that most times we want to see events from a duplicate case, it’s the email events we care about. Sometimes you do need the comments, so you can click the control to show them. When looking at crash report cases created by BugzScout, there are often so many events that showing them all can be overwhelming, so they’re hidden by default. Click the control to show them if you need to see every report of the error.

No more clicking around to get all of the information you need! Case event merge consolidates information from multiple cases, allowing you to see everything you need – in one case.  If you’d like to use native Case Event Merge, sign up for the Performance Upgrade to FogBugz On Demand (if you haven’t already). Get started by sending us your site URL.

Update: We have improved case merging! See the docs for details.

Responsive HTML Emails: a Different Strategy

November 1st, 2013 by Tina Ye

Email has become quite the funny thing. It follows us everywhere. To the grocery line. To the dinner table. To the… shower? (Well… why not.) So when we set out to rethink the FogBugz email notifications from the ground up, we knew we had to ensure a great in-shower email experience—and that meant optimizing for mobile.

There are a wealth of resources on this topic. Campaign Monitor has an  exhaustive, start-to-finish guide. Smashing Magazine posted about various mobile email design tricks. MailChimp has written definitively on the topic. And if you don’t want to read so many words, Litmus recently provided a juicy summary of their own.

However, there is one small problem with all of the above: they all advocate the use of media queries.

A Media Query Quandary

Media query support can pretty much be  taken for granted in desktop browsers, but it remains unknown to many email clients today. One of the most popular mobile email clients is the Gmail app for Android and iPhone, and it remains a staunch non-attendee of the media query party. Without media queries to detect device size and reconfigure the layout accordingly, users could be stuck squinting at something like this:


Pinch me!

A mobile design strategy that doesn’t address one of the most popular mobile email clients isn’t a very good strategy. So we thought… what if we took a different approach?

One Layout to Rule Them All

Most mobile email design articles still want you to think “multiple layouts for multiple screen sizes.” For instance, design a two-column layout for desktop, and then adapt it into a single-column layout for mobile. This strategy relies heavily on media queries to detect the device width and then activate the appropriate layout by reflowing content, shrinking headers, or hiding entire sections. You can see why this would be a problem when media query support is still so spotty…

So rather than depend on media queries to modify the layout, we thought, “Why not just design one layout that looks good on all screen sizes?”

Impossible? Nay, it’s possible!

Mobile First

I was inspired by Luke Wroblewski’s  mobile first approach. Here, one considers the use case with the most limitations first (mobile), and then designs upwards from there. After all, it is easier to fit a small kitten into a big jar than a big kitten into a small jar (They were having a sale at the metaphor factory, okay?). Aaaanyway, it’s a great design philosophy, and it made all the difference in how we arrived at our final design.

We began by asking, what layouts work well on mobile? The answer for us was to think “single column.”

Screen Shot 2013-10-14 at 4.03.36 PM-1

By single column, I mean that the primary flow of the content is downward rather than both downward and side-by-side. Side-by-side content usually requires reflowing to fit on a smaller screen. In contrast, a linear downward flow means content simply scales horizontally to fit the available space. Voila, one layout for any screen size!

Screen Shot 2013-10-15 at 1.43.06 PM

That said, there are times when side-by-side content will not need to be reflowed to fit on smaller screens. Sometimes, the contents of a column can work just as nicely when narrowed down on a smaller screen. Of course, I would not advocate compressing paragraphs of text down to 160px-wide corridors, but it is okay to maintain the side-by-side position of things like labels, asides, notes, and icons. You’ll see lots of examples of this in the final FogBugz notification email design.

Finally, lest you think a single-column approach is too aesthetically restrictive, here are a couple of very nice-looking designs using a predominantly single-column approach. Check out this lushly illustrated piece from Code School. Or this beautifully typeset promo email from Tinkering Monkey.*


These are the “desktop versions” of the emails, but because they are single-column, you can easily imagine how they’d scale down to mobile. Screenshots courtesy of Email Wizardry.

As for the design of the Fogbugz notification emails, our goal is to communicate the status of a case quickly and effortlessly. We don’t need a whole lot of stylized typography or eye-catching illustration to do this—that would be distracting and counter-productive. On the other hand, we do need to convey a lot of information, so we relied on strong typographical hierarchy, a flexible grid, and subtle color to organize the design. Here’s a peek at the final design:


Because of the diversity of information present, it’s quite a bit more complex than a strictly single-column layout. But it acts like one: it scales nicely horizontally, and does not need to be re-stacked or rearranged to fit on a smaller screen. One layout, multiple screens.

Now that I’ve hopefully convinced you to free yourself from the shackles of designing multiple layouts, let’s see what this looks like at the implementation level…

Writing the Codez

On mobile, you have limited real estate, so it makes sense to use as much of the screen as possible. A fluid width: 100% layout does the trick. We simply apply that inline to the largest container element, and get something like this:


But on desktop, this same code would create a readability nightmare:


This is easy to fix. Let’s put in some breathing room on the sides, with a simple little max-width added (also inline) to the main container element:


Ah… much better. Now we’re not violating any typographic rules about sane line lengths.

And that’s basically it! You now have your completely responsive, mobile-and-desktop-friendly, superhero-quality email layout, all without using a single media query.

And you thought this section was going to be long.

Wait, What About Outlook?

Darn, we can’t go party just yet! As it turns out, even an old CSS2 property like max-width isn’t recognized in all email clients. Without it, our emails will just go back to being super-wide and hard-to-read. What can we do to solve this?

If you can say with confidence that none of your users will ever view an important email in Lotus Notes, Outlook, or even Apple Mail, then you can skip all this nonsense and head down to the club. For the rest of us, we’ll catch up with you later. It won’t take long.

Lotus Notes 8 & Outlook 2000-2003

Since Lotus Notes 8 and Outlook are exclusively desktop clients, we can get away with forcing them to always show a fixed-width desktop-optimized view. We do that by targeting them with a conditional: Lotus Notes and older versions of Outlook use Internet Explorer as their rendering engine, so we check for IE, then insert a fixed-width constraining element around the entirety of the email’s content.

<!--[if IE]>
  <table width="540" align="center" cellpadding="0" cellspacing="0" border="0">
<!--[if IE]>

Note that the constraining element is a table. This is because, as we discovered, divs cannot be reliably centered in all email clients.

Outlook 2007+

Moving on, we can use the exact same strategy in Outlook 2007 and newer. The only difference is, Outlook 2007+ uses Microsoft Office as its rendering engine. Don’t ask why; just add gte mso 9 to the conditional. That bit of magic incantation there stands for “greater than or equal to Microsoft Office version 9,” which covers Office 2000 or newer:

<!--[if (gte mso 9)|(IE)]>
  <table width="540" align="center" cellpadding="0" cellspacing="0" border="0">
<!--[if (gte mso 9)|(IE)]>

Apple Mail

Unexpectedly, our testing revealed that Apple Mail also fails to respect max-width! Fortunately, it does understand media queries, so we can add a fallback. Put this in a linked stylesheet or between <style> tags in the head:

@media only screen and (min-device-width: 541px) {
  .content {
    width: 540px !important;

This just says, “If my device isn’t an itty-bitty phone, then make the whole thing fixed width.”

Done! Here is everything all together in one file.

Finally, remember to test test test. We test on  Litmus, our coworkers, and a small percentage of our real live users. Test until you can test no more, and then release it into the world!

Now we can go party!


* I should note that these two examples do use media queries for small refinements like tweaking the spacing and size of the graphics. However, if you view them on a mobile device that doesn’t understand media queries, the experience would still be pretty good. These layouts may use media queries, but they do not depend on them.

FogBugz On Demand Performance Upgrade

October 22nd, 2013 by Dane Bertram

Some months ago we launched a beta for an upcoming infrastructure upgrade of FogBugz On Demand. The response we received was great! Hundreds of you—our awesome customers—signed up to kick the tires on a shiny, new, kiwi-riding-an-ocelot fast FogBugz On Demand service.

When we first launched the beta there were a number of core features that hadn’t been finished just yet: custom fields, the working on menu, quick case add on the list page, etc. We can happily report that all of those features (and more!) have now been implemented! As a result, the beta is now closed and we’ve started rolling out the FogBugz On Demand Performance Upgrade (née “Project Ocelot”) to large swaths of our customers over the past few weeks. Thousands of you in fact.

How can I tell if I have the new hotness?

If you created your FogBugz On Demand account after August 5th, 2013 you’ve been using it the whole time! If not, have you seen this cheery little “Welcome to fast” dialog after logging into your FogBugz On Demand account?

"Welcome to Fast!" dialog from the FogBugz On Demand Performance Upgrade

If you haven’t, your account hasn’t been enrolled in the performance upgrade yet. We’re currently selectively enrolling customers that are an ideal fit given the features that have currently been completed in the new interface. Want to sneak a peek? Let us know your FogBugz On Demand account info and we can enable “the switcher” for you—a toggle for enabling and disabling the Performance Upgrade that lives on your user options page within FogBugz.

If the beta is closed, why haven’t you enrolled everyone?

Just because we’ve completed the beta, doesn’t mean the FogBugz team is off on a sandy beach somewhere sipping Kiwi Koolattas. We’re busy adding more features, porting top-used plugins, and fixing the bugs you’ve been reporting to us about the new, faster FogBugz On Demand and releasing juicy updates every week. Keep the feedback coming!

There are still plenty of things left for us to implement in the new interface. Currently only the list, case, and search pages have been super-charged and there are a number of existing features that haven’t been updated to work with the new interface just yet. In the coming months as we knock more items off our super-fast FogBugz-powered to-do list, we’ll be enrolling more and more accounts.

What makes the Performance Upgrade so fancy? Why’d it take so long? What’s the secret sauce?

Easy there, Captain Questions! We—the FogBugz team—are also planning to write up a bunch of juicy blog posts telling not only the history of “Project Ocelot,” but also describing various parts of the underlying tech that power it. Hybrid modes, generated code, escape hatches, client-side MVC, elasticsearch, JSON APIs, and more!

A huge “Thank you!” again, to all of our faithful beta testers and a warm “Welcome to Fast!” for those of you whose accounts have already been enrolled in the FogBugz On Demand Performance Upgrade. We hope you like it!

FogBugz and Raygun Integration

August 7th, 2013 by Adam Wishneusky

If you’re reading this, there is a good chance you track your bugs in FogBugz. Maintaining a bug database is important if you want to produce high quality software, but are you also collecting automatic crash reports from your application? FogBugz has a super-simple built-in API called BugzScout to help you do it, but it’s purposefully quite basic. If shiny graphs are more your style, take a look at Raygun, which just launched FogBugz integration!

Raygun supports most popular languages and frameworks, including Ruby/Rails, .NET, Java, Node.js, PHP, Python, JavaScript, Xamarin, and many more. When you receive a crash report in Raygun, you can cross-link it with your FogBugz case to make sure it gets fixed. For more details, head over to Raygun’s blog.

Wait, you’re not tracking your bugs? Sign up for a free trial of FogBugz today!

Introducing the FogBugz Service for Zapier

July 15th, 2013 by Sonny Kim

Zapier is a service that lets you automate tasks between different online services. Although FogBugz already integrates really well with Kiln and email, it doesn’t integrate with most of the other online services out there. For example, there’s no simple way to automatically create cases in FogBugz when there are tweets mentioning our products, or to automate card creation in Trello when new cases are created in FogBugz. But now that we added the FogBugz Service to Zapier, you can easily trigger actions between FogBugz and any of the 200+ services that Zapier supports.

One way we can use the FogBugz Service on Zapier is by creating a *zap* between Twitter and FogBugz. A *zap* consists of two things: a trigger and an action. The simple idea behind Zapier is that one service triggers an action by another service. For this particular *zap*, a tweet that mentions one of our products on Twitter will trigger a case to be created automatically in FogBugz so that we can assign the tweet to someone on the support team to respond. This *zap* can also help maintain a history of tweets within our FogBugz instance.

Automatically turn the tweets you care about into FogBugz cases!

Another use case is to integrate FogBugz and Trello. A formal FogBugz-Trello integration has been highly desired for some time now, but since there are many different ways in which development teams can use FogBugz in conjunction with Trello, just implementing a rigid case-to-card and project-to-board relationship doesn’t work for most teams. But with Zapier, you’re free to create your own relationships between the entities in your custom FogBugz-to-Trello workflow. Here are some pre-baked *zap*s between FogBugz and Trello:

A custom integration with FogBugz used to require getting your hands dirty while digging into another service’s API. Well, not anymore. With the FogBugz Service on Zapier, you won’t be bothered to learn the intricacies of another service’s API. So go check it out and start zapping it up!

Sign Up For the FogBugz Beta (Closed)!

March 26th, 2013 by Aaron Maenpaa

Update: The beta is now closed. A big “Thank You!” to everyone who signed up.


Want to see what’s coming in the next version of FogBugz? Now you can! The FogBugz team has been hard at work on a project we’ve code-named “Ocelot” for the past several months. We’ve rebuilt the core pages of FogBugz to use a completely new architecture that’s significantly faster. Like, seriously fast. And now we’re ready to start sharing it with you!


What’s shiny and new?

The new version of FogBugz is a single page app when it comes to the list page and the case page. What’s that mean exactly?

  • a super fast list page for listing and filtering your cases
  • a wicked fast case page for viewing and editing individual cases
  • blazing fast search for finding the cases you’re interested in
  • and blink-of-an-eye fast transitions between these pages

We’ve been dogfooding the new UI internally for a couple of months now and it has really improved our own experience of using FogBugz. We hope you’ll like it!

Riding the Ocelot


A Word of Caution

This is a beta, so there will be bugs. If you find one, please contact our customer service team and let them know. Furthermore, since we’re trying to get this into your hands as soon as is humanly possible, there are some caveats to the beta:

  • The beta is for FogBugz On Demand accounts only.
  • The beta UI has only been tested in Firefox and Chrome. Support for IE10 and Safari are on our TODO list and they should mostly work, but they might not and you will probably run into cosmetic issues.
  • Customizations from the BugMonkey Plugin will not apply to the beta UI.
  • With a few exceptions, existing plugins will not interact with the beta UI.
  • A number of features have not yet been implemented including:
    • Custom fields
    • Quick case add on the list page
    • Graceful handling of concurrent case editing
    • Similar speed improvements and single-page-appification to Wikis, Discussion Groups, Reporting, Admin pages, etc.
    • The Working On menu
    • Searching Wiki Pages and Discussion Groups from the main search box
  • With the exception of some specific text used in outgoing emails, the beta UI is not localized


Share your Feedback

We’re looking for feedback on: How the application feels, your general impressions, any missing features that impact your day-to-day workflow, and of course any bugs you find. Once you’ve started using the beta, please contact our customer service team and let them know what you think!

Once your account is enabled for the FogBugz Beta, you can enable and disable the beta UI on a per user basis from a link in the top right corner. If you later decide that you’d like your entire account taken off of the Beta, just let us know and we’ll be happy to take care of that for you.

Sign Up!

Dogfooding Until It Hurts

February 26th, 2013 by Rich Armstrong

Dogfooding. Also called “eating your own dog food.” It’s pretty simple, right? If you work at Uber, maybe take an Uber car ride from time to time. If you work at Khan Academy, you’re probably pretty good at math by now.

In this video from 2009, Joel talks about dogfooding as being more than just using your own product. It’s about using your own product for everything you can imagine, even if that usage is a little uncomfortable.

We dogfood Trello, FogBugz, and Kiln in a ton of different ways.

We’re a software company, so we generate a lot of code. Kiln helps with that. But we even use it to hold things that aren’t “code” per se, such as backups of non-sensitive databases. We don’t have any real crazy use cases, like using code reviews to plan parties or the electric DAG to create subway maps, but that’s just because Kiln has only been around a few years.

FogBugz, on the other hand, we use for tons of off-label stuff. We use FogBugz’s crash reporting ability to read RSS and Twitter feeds and create cases when new items appear. We use its customer email capabilities to receive faxes. And we have a panoply of API scripts automating a bunch of different business processes.

Trello is a natural for dogfooding, of course, because it’s so flexible. We keep kitchen snack requests and lunch menus on boards. We keep track of who’s going to what conferences, and use it to plan their travel. Trello really shines in setting the agenda for and running our bi-weekly company all-hands meeting. Because Trello is so flexible, it invites a lot of different use cases. We’re still figuring out which ones really work, and it’s great to see a fast-growing user base figuring that out alongside us.

So, that’s all well and good, but if you really want to see “dogfooding until it hurts” in action, check out Beeminder, a tool for setting “goals with a sting.” You set up your goals, and if you stray from them, the service fines you an escalating amount of real money until you’re back on track. Beeminder is dogfooding heavily and publicly to keep their development goals on track. These folks are  literally giving away cold hard cash to users as a pre-commitment to do things like delivering user-visible enhancements or blog posts on a regular basis. Amazing. Even better, they have a Trello integration that’ll keep you moving those cards to the Done column regularly (or else).


Looking for more?

Visit the Archives or subscribe via RSS.