Archive for the ‘FogBugz’ Category

Juggle Your Caseload With ‘Postponed Cases’ in FogBugz

November 24th, 2014 by Tim Kington

Have you ever had a case that you couldn’t work on right away?  Maybe it was a support case and you needed to check in with the customer in a few weeks, or maybe it was a bug that you couldn’t work on until something else was fixed.  In the past, you would leave the case open and get used to ignoring it.  After a few days of that, the case becomes invisible to you, and you forget to follow up when you should.  With the new Postponed Cases feature in FogBugz, your cases reactivate themselves when it’s most useful to you.

blog-deferred-profit

Reactivate Cases By Time or Case

Postponed Cases adds two options to the case page when you resolve a case as “Postponed” (or “Waiting for info” for Inquiry cases): Reactivate Time and Reactivate On Cases. When should you use them? Consider the following:

  • Need to check in with a customer in a few days? Resolve the case as Postponed and set the Reactivate Time. The case will be reactivated (and you will be notified) at that time.
  • Waiting on a developer to fix something before you can work on your case? Resolve as Postponed and enter the other case in Reactivate On Cases.  When the other case is resolved, your case will be reactivated.
  • Worried that the developer might let the issue slide?  Set your case to reactivate when theirs is resolved, but set a Reactivate Time too.  The case will be reactivated when the other case is resolved or the Reactivate Time arrives, whichever comes first.

blog-deferred-wakeup

Available Now

Gone are the days of dropping things on the floor because you’ve gotten used to ignoring them in the list page.  Use the Postponed Cases feature, and your customers and coworkers will be amazed at your case-juggling skills. It’s available now, so try it out in FogBugz straight away. Documentation is on our help site.

Project Jackalope – Notifications, Personal Activity Feeds, and Periodic Email Notifications

November 13th, 2014 by Brett Huff

Image credit: http://hangitontheline.blogspot.ca/2012/04/history-of-lepus-cornutus.html

Project Jackalope

Fog Creek has had a strong tradition of holding a spectacular internship program.  Each year our summer interns work on real code that drives significant features of existing products, or sometimes whole new products.  This year we had 5 interns at Fog Creek (plus 3 at Trello), and 3 of them worked on FogBugz. Continuing our tradition of using animal names to refer to our intern class projects, they worked on Project Jackalope.

Beyond “My Cases”

In a highly interactive environment, FogBugz was showing weakness.  You would only know that someone had assigned you a new case by checking your “My Cases” filter, or by checking your email.  If a case is important you would have to leave that case open on a tab, or watch your email for updates.  Neither of these options is ideal for a dynamic work environment.  So we decided that FogBugz is sufficiently mature that we can now re-invent email.

Notifications and Activity Feeds

Notifications, Personal Activity Feeds, and Periodic Emails are designed to stop the email madness.  You now get notifications in FogBugz when people comment on or change cases you’re interested in.  You can see all of your activity across cases, instead of in just one case.  And when your emails are less time critical (because you see them change within FogBugz), you can now slow those emails down to one every 3 hours, or turn them off entirely.  So without further ado, I’ll let the interns each present their features:

Personal Activity Feeds: or the “What did I get done yesterday again?” page

November 13th, 2014 by Matthew Hayes

It seems like pretty much every web-app these days has added a page to show you what you’ve been doing lately. These types of pages are useful for a number of reasons. You can review what you’ve done and what others have been doing that involves you in a nice, compact story. Since the most-recent events are at the top of the feed, you can quickly find the relevant information you’re looking for, even if you don’t quite know how to search for it. You can also skim through other users’ feeds to get a quick impression of what they’ve been up to and how you might coordinate with them.

As a project management and issue tracking tool, FogBugz is quite different from “social network” type apps, but these activity feeds have become so prevalent that we want and expect a similarly useful page whenever we see a clickable username on the web.  In the past, whenever you clicked on your co-worker’s name within FogBugz you got a page that showed you their email address, current working schedule, and the projects they spend their time on. While certainly useful to know, this information is generally pretty static, and not all that useful day-to-day after the first time you’ve seen it.

Introducing: FogBugz Personal Activity Feeds


Welcome to the FogBugz personal activity feed.  You now have that place you can point your boss to and say “Of course you should give me a raise… just look at all the stuff I’ve been doing!”  Or when you come back from that long vacation, instead of trying to come up with a complex axis query/search/incantation to figure out what everyone has been up to (or worse, asking them to take time out of their day to catch you up in-person), you can just click on their name in that case they sent you and see how they’re way too busy to get to it themselves.

My Activity

When you’re trying to reflect or summarize all the cases you’ve opened, resolved, or at all been involved with this week, there’s now a shiny new page that organizes all that information for you. To view your own activity feed, just click the new “My Activity” link in the drop down menu under your account avatar:
Activity Menu

When you do, you’ll notice two tabs on the left to switch between your personal activity feed and your (also shiny and new) notifications page.

Activity Feed and Working Schedule

The new activity feed page still has the user’s email address for easy access, as well as convenient links to all the cases currently assigned to that user, and even a link to the old user info page (now more accurately labeled as their ‘Working Schedule’) in case you need any of that information. The activity itself is grouped by day, so it’s easy to find what you’re looking for.  Fear clicking names no more!

Periodic Email Notifications: For Inbox Zero Bliss

November 13th, 2014 by David Patrzeba

That inner OCD sometimes gets the better of all of us.  You’re happily working away when the familiar ring comes over your computer speakers: “You’ve got mail!”  Quickly, you drop what you’re doing to go look at the latest chain letter that your co-worker is emailing you (because, what could be more important).  Then you’re disappointed to find out that it’s just an email from FogBugz telling you that your indecisive manager has changed the priority on that typo bug, again.  After regaining your inbox zero bliss, you go back to your normal work and struggle to regain The Zone.  After 6.28 minutes, Mail interrupts you again.  That typo bug is now an inquiry instead of a bug, and The Zone continues to elude you.

Introducing: Periodic Email Notifications

06_digest-email01
From your user options page within FogBugz you can now choose to get periodic emails instead of instant emails.  The periodic emails are sent every 3 hours, and cover all of the activity that’s happened since your last periodic email was sent.  You can now get the upper hand for 3 solid hours, and not make your bliss depend on your co-workers respecting your do-not-disturb hour.

The Power of Periodic Emails

Periodic emails also reduce the time you spend in your inbox by only showing you the things going on within FogBugz that require your attention and that you’ve missed.  If you’re like us and spend most of your time in FogBugz, then you’re probably using the personal notification menu to keep on top of the cases that come your way.  Your next periodic email is then only a list of things that you haven’t already seen; gently reminding you to not lose cases in the cracks.  If you spend little of your time in FogBugz, instead using your time to write great code, then the periodic email is a reminder that there is a wider world out there, and someone may need your help on those pesky cases that keep getting in the way of your “real work.”

Configure your email settings to control the frequency of email notifications from FogBugz

Really, No More Email

As noted in the Notifications blog post, we also now respect the “no, seriously; don’t send me any email” option.  Now that you can get all of your notifications from within FogBugz itself, there’s no reason to force some types of email notifications into your inbox (formerly, wiki modifications and “Notify More Users” would trump the “Don’t send me email, kthxbai” option).

 

We’re as interested in your productivity as you are.  Honestly. We use FogBugz too, so we’re also interested in our own productivity. Keeping more of our day out of our email and focusing on tasks has helped us significantly.  We hope it helps you too!

Notifications: Love FogBugz? Now you never have to leave!

November 13th, 2014 by Graham Carling

Since the dawn of time, people have been using FogBugz to effectively keep track of their projects. Armed with the power of cases and wiki pages, developers have set out to build some of the greatest software the world has ever known.  Of course, projects change, and each developer needed a system for keeping track of what was going on with the work that was relevant to them. For a long time, this was accomplished by using a system of email notifications that would alert each user whenever something important happened. Unfortunately, in order to see any meaningful information about what happened one would have to open the email, read the summary, and then click through to FogBugz; an unnecessarily clunky process. However, times they are a-changin’, and a new era of case-up-to-date-ness is being ushered in. What if FogBugz could tell you what’s happening in FogBugz from within FogBugz, instead of forcing you to switch to your email?

Introducing: FogBugz Notifications

Notifications Icon
Notifications are an addition to FogBugz that lets you keep up-to-date with what’s going on with your cases and wiki pages without ever having to leave FogBugz. There are two main components to notifications:

  • The notifications popup – this lives at the top of the page in the header, nestled between the tools menu and the user menu.
  • The full-blown “let me see them all” notifications page.

Notification Pop-ups

The notifications popup gives you concise information on each case or wiki page that you’ve been notified about, using icons to display the reason why you’re being notified. Included with these icons are the title of the case/wiki page and a short description of why you are being notified. For more information you can mouse over the case notification to see FogBugz’ existing case hover popup, or you can click through to view the full case.

All Notifications Page

The full “See all notifications” page gives you a more detailed description of each of your notifications and also gives you the ability to load additional notifications that happened further in the past. This page now gives you a single go-to place to get caught up on anything happening within FogBugz that might require your attention. You can load as many notifications as you want, giving you one page for all of your notification needs.

How are Notifications Useful?

Notifications do a great job of reducing pain when team members make changes on cases or wiki pages that you care about. Its purpose is not to tell you every detail of what’s happened. It’s to give you a short summary of why your attention might be needed and to make it extremely easy to then see the full story by navigating to the case or wiki page you’re being notified about. It also has a number of useful features for helping you keep up-to-date and manage your notifications:

    • Mark individual notifications as read/unread without navigating to the case/wiki page you’re being notified about

Mark Notifications as read

    • Mark all notifications as read at once
    • If a notification for a case has been marked as read and a new event happens on that case, instead of creating a new notification the old one bubbles up with the new information. This helps keep your notifications organized and easier to manage, especially for active cases.
    • Navigating to a case from a notification will bring you to the specific events within that case that caused you to be notified

Case Event information

  • Informative icons to help quickly determine what’s going on and why you’re being notified on a given case or wiki page

Notification Icons display the reason you receive this notification
Notifications works hand-in-hand with our newly added personal activity feeds and periodic email notifications to give you more control over your FogBugz workflow. We hope you’ll really enjoy these new features… we know we have enjoyed creating them!

Search Axis Auto-Complete: Helping you find what you’re looking for (even faster)

July 23rd, 2014 by Aaron Maenpaa

In April, I talked about how we’re trying to make our axis search more discoverable, and ultimately, more widely used. Our first step was to introduce the search guide, which brought our axis documentation directly into FogBugz where it’s more relevant and contextual. The next step (which I hinted at), was to more actively help you create axis queries via auto-complete:

Auto-complete in action

… wait, what? What are axis queries?

In addition to supporting run-of-the-mill “full-text” search (e.g., “bob widget bug”), FogBugz also supports axis queries which let you filter cases based on criteria that apply to specific case fields. You probably know that you can specify exact values for fields in a filter, but it can be slow to click around for a one-time need to find a case. Instead of choosing between specific but slow and fast but vague, try an axis query. Compared to full-text search, axes can greatly reduce the number of matching results you need to scan over (or worse yet click and view). If you know a case was about a crash and there was an email in the case mentioning “I tried reinstalling”, you could simply search for those words, but if you know that “crash” was in the case title, using the title axis could make finding the case much faster:

On my FogBugz installation, that’s 239 cases down to 4! If you also know the case is in your “Support” project and had a tag “regression”, add project:Support tag:regression to your query. Avoiding scanning over a long result list just saved enough time to watch at least 29 cat gifs, and that’s just scratching the surface! So far, axis auto-complete has been really well received in our internal testing. On the second day of our Summer 2014 internship, I observed one of our interns doing an axis search using the new auto-complete feature… which is something I would have never expected even a few months ago. Axis search has gone from a largely hidden power-user feature to a productivity booster that’s discoverable enough for brand new users to find it.

… so what does this auto-complete…uh…complete?

When you start typing any axis name, you will see a list of potential matches. Typing a value for an axis will auto-complete possibilities (for example, your list of project names after the project: axis). Auto-complete also knows how you can sort your cases, suggesting sortable column options when you start to type them after “orderBy:” Finally, it lists past queries from your search history in case you’re just trying to make the same search you made the other day.

assigned

Axes: Including assignedTo, editedBy, opened and many more.

me

Objects: Including the names of people, projects, milestones in FogBugz, as well as special names like me, today, tomorrow.

auto-complete historical searches

Historical searches: Helping you run your favorite searches again.

Our favorite searches:

With all this talk about axis searches, it seems like we should share some of the searches we run frequently to help get you started:

 Michael: “What was that case again?”

 Justin: “Cases I’ve assigned out.”

 Adam: “I put this on everything.”

 Aaron: “Who’s doing what?”

Documentation is available on our help portal for auto-complete, the search guide and our complete list of axes. Like the look of it, but don’t have FogBugz? You can try FogBugz for free today!

Eight Jackalopes Walk Into An Office

July 9th, 2014 by Elizabeth Hall

Intern Group Final

As always, our interns are working on real features for our real products and have already made tremendous contributions to the Trello and FogBugz teams. We think they’re pretty special – not only did they stand out amongst the 776 applications we received but during these first few weeks they’ve had the chance to teach us a thing or two. Whether it’s coming up with new strategies for dealing with the Tanner during Werewolf or showing off cool video games they’ve made at hackathons, the class of 2014 is constantly impressing us.

Before we blink and it’s all over we want to take a moment to show off this year’s amazing talent. Here they are our eight wonderful interns (lovingly dubbed jackalopes*) in their own words.

Alex Lew
20140627-DSC_0021

Hi! My name’s Alex, and I’m an intern on the Platform team this summer. Obligatory fun fact: I grew up in North Carolina, and my high school’s library had the largest collection of books and music in and about Esperanto of any library in the United States (excepting the Library of Congress).

Laura Watiker
20140627-DSC_0035

My name’s Laura and I’m working on the Trello team this summer. During the school year, I go to Oberlin College in Ohio, studying CS and Econ. I grew up in the New York City suburbs, but I’m happily living in the Brooklyn housing this summer. Aside from pushing lots of computer keys, I do work with Oberlin’s co-ops, play a bunch of instruments, and theorize 2048 strategy.

Graham Carling
20140627-DSC_0039

My name is Graham, I just finished my Sophomore year at Brown. I am originally from Manhattan and a graduate of Stuyvesant high school, a short walk from the Fog Creek offices. I am majoring in Computer Science (I know, shocking) and for the last couple semesters I have been working as a TA for some of Brown’s CS classes. Outside of work I like going to concerts, mainly of the electronic variety, along with seeing my friends from high school and working on a side project of mine. So far my summer has been packed full of fun events, and when I’m not going fishing or shooting arrows I’ve been working on the new notification center feature for FogBugz.

Jonathan Pevarnek
20140627-DSC_0040
Hello, I’m Jonathan, a masters student in Computer Science at the University of Michigan. I’m from a small island called Grosse Ile in south-east Michigan. When I’m not working, I like to read books, go on bike rides, and work with a campus community service organization (Circle K). This summer, I’ve mainly been working with Doug to extend Trello email integration.

Matthew Hayes
20140627-DSC_0055

I am Matthew Hayes, from Syosset, Long Island, NY. I just finished my junior year as a CS major at Cornell University upstate in Ithaca, and will be working on a FogBugz activity feed this summer. Fun fact: I caught the biggest fish on this year’s fishing trip (evidence below).

Peter Johnson
20140627-DSC_0025

My name’s Peter, and I’m working on adding saved searches and list sorting to Trello this summer. I just finished my junior year at William & Mary in Virginia, and in my free time I enjoy running long distances and playing Dots!

Steven Lyubomirsky
20140627-DSC_0052

If you’re inclined to take me at my word, my name is Steven Lyubomirsky, I’m a rising junior at Princeton, I study computer science, and am incredibly handsome. If you’re not, at least three of those can be independently verified. I live under the watchful guard of my blue parakeet in Fort Lee, NJ, a safe distance from the incredibly radioactive center of the universe (NYC, of course), where I like to demarcate my domain with my nightly walks. This summer I’m working on the Platform team.

David Patrzeba
20140627-DSC_0065

My name is David Patrzeba and I am an intern on the FogBugz team (Project Jackalope). I started on June 6, 2014 because the newest addition to my family decided to arrive on my original start date of June 2, 2014. I am a NJ native and will be traveling home every weekend to be with them, but during the week I will be staying at the Clark St. residence in Brooklyn. I am a rising senior at Rutgers University (The State University of New Jersey) where I am studying Electrical & Computer Engineering, Computer Science, and Economics. Fun Fact: I served in a Special Operations unit in the Army for 7 years, and I may be Fog Creek’s oldest intern ever.

0611142012-MOTION
Matt wasn’t lying. He really did catch the biggest fish.

*Each year the intern class is named after the next letter in the alphabet. This year is “J”. Do you have a fantastic “K” animal name recommendation for next year? Talk to me!

The Search Guide: Helping you find what you’re looking for

April 11th, 2014 by Aaron Maenpaa

As you may already know, FogBugz offers an impressive array of search axes to help you slice and dice your data. These axes allow you to build searches to find particular cases or any cases matching pretty much arbitrary criteria. While all of these axes are documented on our help site, we kept finding that while axis search is really valuable to us when we use FogBugz, we kept talking to customers who just didn’t know about it.

We’re trying to change that.

The first thing we’re doing is adding a prompt to the search box:

search guide button

When you click it, it expands to show an axis guide:

search axis guide

We’re hoping that the prompt stays out of the way during normal use, but helps make the axes more discoverable, presents the axis documentation in a more useful format, and ultimately helps you find the cases you’re looking for, when you’re looking for them.

… but wait, there’s more

Like I said, this is the first thing we’re doing. We’re working on some other stuff we think you’ll like. Here’s a sneak peak (provided you haven’t already reached your daily recommended intake of Comic Sans ;-):

search auto-complete

Like the look of it, but don’t have FogBugz? You can sign up for a free FogBugz trial today!

Fog Creek and Heartbleed

April 10th, 2014 by Mendy Berkowitz

Along with the rest of the internet, Fog Creek has been reacting to the Heartbleed vulnerability which was discovered this Monday.

TL;DR: FogBugz and Kiln On Demand, Copilot and the Fog Creek website were not vulnerable. Trello was vulnerable and has been remediated.

Fog Creek handles SSL connections at the load balancers in front of the application servers. The load balancers for FogBugz and Kiln On Demand, Copilot and the fogcreek.com website are using the 0.9.8 branch of OpenSSL. This version does not have the newer heartbeat extension and is therefore not vulnerable to Heartbleed. None of your data on fogbugz.com, kilnhg.com, copilot.com or fogcreek.com has been exposed to the Heartbleed vulnerability.

The load balancers for Trello were using the 1.0.1 branch of OpenSSL and were vulnerable. We have upgraded OpenSSL and replaced the Trello certificates. More details and important recommended steps for all users are available in a post on the Trello blog.

We have also reviewed our vendors (for things like this blog) to ensure that our other SSL certificates were not potentially compromised. Fortunately we have not had to replace any other certificates.

If you have configured web hooks in your Kiln On Demand account, we recommend that you verify that the target servers have not been affected. If you use one of the pre-configured types, or your custom web hook uses HTTPS, the data sent to that external service may be at risk. We do not however, send any of your Kiln login credentials, so they are safe.

We take the security of your data seriously. We are committed to protecting it and to maintaining clear and open communication with you. If you have any questions, comments or concerns please contact us.

Production Debugging: A story about “Exception code: 0xe053534f”

March 20th, 2014 by Aaron Maenpaa

Imagine for a moment that you’ve been running an ASP.NET app for years, and on one otherwise sunny Saturday morning, you get a call saying that the app has started crashing randomly. What do you do?

Step 0: Check the event log

It took me way too long to get used to checking the Event Log (Start → Event Viewer). I don’t know if I expected it to be loggier, or what, but there’s good stuff in there. You should check it out.

You might even find an error like this one (Windows Logs → Application) :

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17651, time stamp: 0x4e21213c
Exception code: 0xe053534f
Fault offset: 0x000000000000cacd
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

This report is a little terse but it can be the gateway to figuring out what’s wrong. “Faulting” sounds bad: It means “crashing”. Translating to English, It’s saying “the IIS worker process (w3wp.exe) crashed”. Since this is exactly what you’re looking into, it’s a start. Exception code 0xe053534f is a stack overflow (I wish I had advice for how to map exception codes to what the problem is, but as far as I can tell, Google and hope are your best bet).

Ok, so what now?

Step 1: Googling

Looking around, amongst the “I’m getting 0xe053534f, how do I fix it?” posts on forums you’ll realize that what you want is a crash dump, and you’ll find some advice on how to obtain one, like this, or this.

You probably don’t want to use AdPlus

While AdPlus is indeed super cool, and indeed has a mode that will allow you monitor all of the IIS worker processes (a command which I will not include here because I don’t want to encourage you to use it), in my experience the overhead was way too high: The throughput on the monitored servers dropped to the point where 1. They basically weren’t doing any work, 2. They stopped crashing (Don’t worry… subset of servers during a time of reduced throughput requirements, negligible customer impact).

You probably don’t want to try and set up an Unhandled Exception Module

At least not while you’re under the gun. I never actually got it to work, and it wouldn’t actually help us here since it probably wouldn’t get called for a stack overflow anyway.

Step 2: Check the event log

Did you know that Windows Error Reporting has your back? If you look at the Event Log, the event right after the crash should be from Windows Error Reporting and hopefully it has some more information like:

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0
 
Problem signature:
P1: w3wp.exe
P2: 7.5.7601.17514
P3: 4ce7afa2
P4: KERNELBASE.dll
P5: 6.1.7601.17651
P6: 4e21213c
P7: e053534f
P8: 000000000000cacd
P9:
P10:
 
Attached files:
 
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_16c01b4e25db8aa7e52fdd921dbac8b8dba70ff_e37855c7
 
Analysis symbol:
Rechecking for solution: 0
Report Id: 3bfe0e87-a15d-11e3-a83d-14feb5d64ad7
Report Status: 4

… okay, so that’s not super informative, but there’s a line in there that should get you excited, C:\ProgramData\Microsoft\Windows\WER\ReportQueue is a folder that might already contain crash dumps… no collection required! You probably want to look there and see if there are dumps that look relevant to resolving your problem.

What if there aren’t any crash dumps?

Check all affected servers, you only need one. I never got around to trying it, but SysInternals ProcDump looks like it might do the job, hopefully without taking the toll that AdPlus did.

DebugDiag also came up repeatedly while I was looking around, maybe that would do the trick. Looks more complicated to get going than ProcDump though.

Step 3: Windbg

Okay, you’ve got a crash dump, open it up in windbg, and look around (there are tons of windbg cheat sheets around like this one, this one, or this one). We’re going to:

  1. Load the CLR extensions (.loadby sos mscorwks).

  2. Load the symbols (.symfix).

  3. Inspect the stack (!clrstack).

  4. Inspect the exception (!printexception).

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1cbc.3d34): Unknown exception - code e053534f (first/second chance not available)
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for KERNELBASE.dll -
ntdll!NtWaitForSingleObject+0xa:
00000000`76ed135a c3              ret
0:078> .loadby sos mscorwks
------------------------------------------------------------
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
------------------------------------------------------------
0:078> !clrstack
*********************************************************************
* Symbols can not be loaded because symbol path is not initialized. *
*                                                                   *
* The Symbol Path can be set by:                                    *
*   using the _NT_SYMBOL_PATH environment variable.                 *
*   using the -y <symbol_path> argument when starting the debugger. *
*   using .sympath and .sympath+                                    *
*********************************************************************
PDB symbol for mscorwks.dll not loaded
------------------------------------------------------------
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
------------------------------------------------------------
c0000005 Exception in C:\Windows\Microsoft.NET\Framework64\v2.0.50727\sos.clrstack debugger extension.
PC: 00000642`ffcbf8e4  VA: 00000000`00000000  R/W: 0  Parameter: 00000000`00000000
0:078> .symfix
0:078> !clrstack
OS Thread Id: 0x3d34 (78)
(!clrstack processes a max of 1000 stack frames)
Child-SP         RetAddr          Call Site
000000000e0e6c70 000007ff01a617e9 HtmlAgilityPack.HtmlNode.CloseNode(HtmlAgilityPack.HtmlNode)
000000000e0e6cf0 000007ff01a617e9 HtmlAgilityPack.HtmlNode.CloseNode(HtmlAgilityPack.HtmlNode)
000000000e0e6d70 000007ff01a617e9
<many, many, many more lines of that>
0:078> !printexception
Exception object: 0000000270090138
Exception type: System.StackOverflowException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 800703e9

Here’s what we’ve learned so far:

  1. We’ve confirmed a stack overflow as the cause of our problem.

  2. We’re seeing the error in HtmlAgilityPack, but we can’t actually see the base of the stack because !clrstack gives up too soon.

Delve Deeper

!clrstack won’t go deep enough to show us the base of the stack, but we can get the base of the native stack via k 1000

0:078> k 1000
Child-SP          RetAddr           Call Site
<the native error handling frames>
00000000`0e0e6c70 000007ff`01a617e9 0x000007ff`01a617bc
00000000`0e0e6cf0 000007ff`01a617e9 0x000007ff`01a617e9
<lots of stack frames>
00000000`0e13ad70 000007ff`01a5d442 0x000007ff`01a617e9
00000000`0e13adf0 000007ff`01a5cf28 0x000007ff`01a5d442
00000000`0e13ae80 000007ff`01a5c39a 0x000007ff`01a5cf28
00000000`0e13aed0 000007ff`01a5bd9c 0x000007ff`01a5c39a
00000000`0e13af20 000007ff`01a5bbaa 0x000007ff`01a5bd9c
00000000`0e13afe0 000007ff`01a5b224 0x000007ff`01a5bbaa
00000000`0e13b020 000007ff`01a5aff3 0x000007ff`01a5b224
00000000`0e13b060 000007ff`01a5af05 0x000007ff`01a5aff3
00000000`0e13b0b0 000007ff`01a6f836 0x000007ff`01a5af05
00000000`0e13b0f0 000007ff`01a582e1 0x000007ff`01a6f836
00000000`0e13b4e0 000007ff`01a50fd0 0x000007ff`01a582e1
00000000`0e13b7f0 000007ff`0221f79c 0x000007ff`01a50fd0
00000000`0e13bdf0 000007ff`019513ef 0x000007ff`0221f79c
00000000`0e13be70 000007ff`0194c972 0x000007ff`019513ef
00000000`0e13bf90 000007ff`01949ff0 0x000007ff`0194c972
00000000`0e13c2f0 000007ff`01341aa2 0x000007ff`01949ff0
00000000`0e13c6a0 000007ff`013409c6 0x000007ff`01341aa2
00000000`0e13c900 000007ff`00ffc48e 0x000007ff`013409c6
00000000`0e13c9d0 000007ff`00f1708c 0x000007ff`00ffc48e
00000000`0e13ccd0 000007ff`008f5ec4 0x000007ff`00f1708c
00000000`0e13cdc0 000007ff`00d23e05 0x000007ff`008f5ec4
00000000`0e13d600 000007ff`00c7e416 0x000007ff`00d23e05
00000000`0e13d6d0 000007ff`00c7ad5e 0x000007ff`00c7e416
00000000`0e13d7d0 000007ff`00c78d85 0x000007ff`00c7ad5e
00000000`0e13d960 000007ff`00c6ccb9 0x000007ff`00c78d85
00000000`0e13d9b0 000007ff`00c69be1 0x000007ff`00c6ccb9
00000000`0e13dad0 000007ff`00c691ab 0x000007ff`00c69be1
00000000`0e13dc50 000007ff`00c684c4 0x000007ff`00c691ab
00000000`0e13dcb0 000007fe`f14ee0ba 0x000007ff`00c684c4
00000000`0e13dcf0 000007fe`f47451c7 mscorwks!UMThunkStubAMD64+0x7a
<more native frames>

We’ve got the base of our stack, we just can’t read it :). Fortunately, the !IP2MD command will turn the last pointer into a description of the method:


0:078> !IP2MD 0x000007ff01a617e9
MethodDesc: 000007ff0237bed8
Method Name: HtmlAgilityPack.HtmlNode.CloseNode(HtmlAgilityPack.HtmlNode)
Class: 000007ff01d1da00
MethodTable: 000007ff0237c3d8
mdToken: 0600000b
Module: 000007ff0237abc0
IsJitted: yes
CodeAddr: 000007ff01a61700

Putting together some copy/paste, some Vim, some !IP2MD and some grep you can turn that list of pointers into the base of your stack:

HtmlAgilityPack.HtmlNode.CloseNode(HtmlAgilityPack.HtmlNode)
HtmlAgilityPack.HtmlNode.CloseNode(HtmlAgilityPack.HtmlNode)
HtmlAgilityPack.HtmlDocument.CloseCurrentNode()
HtmlAgilityPack.HtmlDocument.PushNodeEnd(Int32, Boolean)
HtmlAgilityPack.HtmlDocument.Parse()
HtmlAgilityPack.HtmlDocument.Load(System.IO.TextReader)
HtmlAgilityPack.HtmlDocument.LoadHtml(System.String)
FogCreek.HtmlParser.HtmlParser.Initialize(System.String)
FogCreek.HtmlParser.HtmlParser.ExtractSafeHtml(System.String, FogCreek.HtmlParser.IHtmlTransform[])
FogCreek.HtmlParser.HtmlParser.ExtractText(System.String)
FogCreek.FogBugz.__Global.GetMessageTextByMsg(Wasabi.Runtime.Mail.MailMessage, Boolean, Boolean, System.Nullable`1<Boolean>)
FogCreek.FogBugz.CFullTextIndexer.IndexBugEvent(FogCreek.Search.Interfaces.IDocument, System.DateTime ByRef, Int32 ByRef, FogCreek.FogBugz.CBugEvent, System.Nullable`1<Boolean>)
FogCreek.FogBugz.CFullTextIndexer.IndexBug(Int32, Int32, Int32 ByRef)
FogCreek.FogBugz.CFullTextIndexer.MemorySafeIndexBug(Int32, Int32, Int32)
FogCreek.FogBugz.CFullTextIndexer.IndexItem(System.String, Int32, Int32, Int32 ByRef)
FogCreek.FogBugz.CFullTextIndexer.UpdateByType(System.String, Int32)
FogCreek.FogBugz.CFullTextIndexer.HeartbeatUpdateIndex(FogCreek.FogBugz.CNotification, System.String, Int32)
FogCreek.FogBugz.__Global.SingleHeartbeatFogBugz(Int32, Int32, Int32, System.DateTime)
FogCreek.FogBugz.__Global.SingleHeartbeatPlatform(Int32, Int32, Int32, System.DateTime)
FogCreek.FogBugz.__Global.DoWork(Int32, FogCreek.FogBugz.CTrialManager)
FogCreek.FogBugz.__Global.StartMultiHeartbeat()
FogCreek.FogBugz.HttpHandler.ProcessRequest(System.Web.HttpContext)
System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)

Step 4: Fix the Bug

What we’ve learned:

  1. We’ve got a crash caused by recursion in HtmlAgilityPack.

  2. Our code is calling into HtmlAgilityPack while processing email for the search index.

You can fish around in the dump a little more to try and find stuff like parameters that will help you track down the requests causing problems, but that was enough to point me in the right direction.

Not coincidentally, we’ve shared our fork of HtmlAgilityPack on GitHub (and a patch on Codeplex).

Bonus: Even More Debugging

While investigating a separate crash I found use for a different command: !analyze -v, which helpfully extracted the relevant exception and call stack all on it’s own:

0:054> !analyze -v
<lots of stuff>
EXCEPTION_OBJECT: !pe ff5ce5e0
Exception object: 00000000ff5ce5e0
Exception type: System.InvalidOperationException
Message: The ConnectionString property has not been initialized.
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131509
 
MANAGED_OBJECT: !dumpobj ff5d0210
Name: System.String
MethodTable: 000007ff001d0d90
EEClass: 000007ff001ccdf0
Size: 2226(0x8b2) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:    at System.Data.SqlClient.SqlConnection.PermissionDemand()
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at FogCreek.Database.Connection.Open() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Database\FogUtil.Database\Connection.cs:line 79
at FogCreek.Database.Command.ExecuteNonQuery() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Database\FogUtil.Database\Command.cs:line 164
at FogCreek.Database.IndexFile.Unlock() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Database\FogUtil.Database\IndexFile.cs:line 149
at FogCreek.Search.Store.Lock.Release() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Search\Store\Lock.cs:line 26
at Lucene.Net.Index.IndexWriter.Finalize() in d:\code\hosted\build\FB\8.9.106H\fogbugz\FogBugzUtils\FogCreek.Search\FogUtil.Search\Lucene.Net\Index\IndexWriter.cs:line 562
<even more stuff>

Oh look, a finalizer causing a problem. I’ve never seen that before</sarcasm> (Seriously though, if anyone has ever told you that you probably shouldn’t use finalizers, this is why: they’re hard to test, they can interfere with the operation of the garbage collector, and they can straight up crash your app if they generate exceptions).

Conclusion

I put this together because while the internet is rife with WindDbg cheat sheets, it doesn’t seem to have quite as many walk-throughs of how the various and sundry commands might apply to debugging ASP.NET crashes, which commands are useful and what kind of output you might expect. Hopefully this helps you. I know it will help remind me of what to do the next time I need to resort to WinDbg.

PS: If debugging production apps makes you feel a tingling sense of accomplishment, you might like to know that we’re hiring.


Looking for more?

Visit the Archives or subscribe via RSS.