Post-Mortem Debugging Your Application with Minidumps and Visual Studio .NET

Contents

What Is a Minidump?
Creating a Minidump
Build Issues
Writing a Minidump with MiniDumpWriteDump
Reading a Minidump with Visual Studio .NET
How Microsoft Uses Minidumps
Further Improvements
Conclusion

What Is a Minidump?

A minidump is a file containing the most important parts of a crashed application. It is written on the user’s machine and then the customer can submit it to the developer. The developer can load the dump to help determine the cause of the crash and develop a fix.

Since the early days of Windows NT, the Dr. Watson program has been able to generate crash dump files, denoted with the .dmp extension. However, they were not as useful as they should have been because of two problems:

1. They were huge. The dump of an application included every byte of the entire process space, so a crash in something simple like Notepad would be several megabytes in size, and a crash in something like Word can be many hundreds of megabytes. The files were just too big to send by e-mail or FTP.
2. Their content was not necessarily useful. Dr. Watson was, in fact, a just-in-time (JIT) debugger, and it is difficult for a debugger to get the full path to a loaded module. Full debuggers such as the Visual Studio debugger complete a number of steps to get the paths, but Dr. Watson did not. This usually resulted in unhelpful module names such as MOD0000 and so on.

Minidumps were designed to fix these problems in a number of ways:

• Instead of saving the entire process space, only certain sections are saved. There is no point in saving copies of modules such as Kernel32.dll; if the Version number is included, it is easy to get a copy from a Windows CD. The actual memory heap of the application is by default not saved in a minidump; it is not required to debug a surprisingly high percentage of crashes. You can save the heap if you need to, however.
• The minidump save code works to get accurate and full information for modules, including their names, paths, Version information, and internal timestamps.
• The minidump save code also gets the list of threads, their contexts (that is, register sets), and the memory behind their stacks.
• The whole file is compressed, further reducing its size. A minidump for Notepad is around 6K on Windows XP, almost 300 times smaller than the previous crash dump of the same process.

Note   Kernel-mode minidumps, which are generated by Windows XP after a computer stops responding, also exist, but this article discusses more common user-mode minidumps.

Creating a Minidump

There are three ways to create a minidump:

• Add code to your own application to write a minidump if the application has an unhandled exception.
• In the integrated development environment of Visual Studio .NET, click Save Dump on the Debug menu when debugging the application.
• Do nothing.

The first option is discussed in more detail in the following sections.

The second option works only on a workstation with the debugger already configured, which is likely useful only within an organization (for example, with another developer or tester). If you debug a crash with Visual Studio .NET, you can then click Save Dump As on the Debug menu. You can save as a Minidump or Minidump With Heap. You do not need any symbols or PDBs configured to save a dump file; you will need them later, however, to read it back.

The third option works only in Windows XP, which automatically creates a minidump if an application has an unhandled exception and no JIT debugger has been set up. Also, the minidump is submitted directly to Microsoft, so you will not have the opportunity to determine why.

Build Issues

To configure your application to create dumps when it stops responding, you must configure your builds to generate full debugging information, especially for retail builds. After generating a PDB, you also must archive every binary that ships to customers and its matching PDB; you will need the PDB later to debug any minidumps that your customers submit.

Also, ensure that you set the Version information correctly on your binaries. Every release of every component that you ship should have a different Version, so you can match them to the minidump. The Version field assists you in matching the bits. The debugger itself does not use the Version information, however; it matches binaries based on the internal timestamp in the PE header.

In terms of output, generating debug information for release builds has a small effect. A PDB, taking up some space on a build machine, is generated, and the binary will be a few hundred bytes larger, to record the name of the PDB in the debug directory within the PE file. You should not provide customers with the PDB; this could allow customers to more easily reverse-engineer your application.

Writing a Minidump with MiniDumpWriteDump

The key API to saving a minidump is MiniDumpWriteDump, which is exported from Dbghelp.dll, a redistributable DLL that comes with the Platform SDK. Make sure that you are using the Windows XP Version 5.1.2600; earlier beta and release candidate versions had problems with the API, and versions 5.0.x are from Windows 2000 and do not export the function. If you have a Version earlier than 5.0, then it comes from the Systems Debugger package (including WinDbg and so on) and is not redistributable.

To call the API, you need to catch the crash by setting an unhandled exception handler with the SetUnhandledExceptionFilter API. This allows the filter function to be called at almost any time an unhandled exception occurs in an application. In certain unhandled exceptions, such as a double stack fault, the operating system will immediately terminate the application without calling the filter or a JIT debugger.

In your filter function, you must load Dbghelp.dll. This is not as simple as calling LoadLibrary; as with Windows 2000, you will access the one from the System32 directory, which does not have the right export. The sample code in the attached file tries to load from the same location as the EXE. Install the correct Version of Dbghelp.dll into the same directory as the EXE; if this does not work, then the code reverts to a plain LoadLibrary, which will only work if the application is running on Windows XP.

After loading the DLL, it then checks for the named export; if correct, it then creates a file with an appropriate name, such as saving to the Temp directory and using the application name with the .dmp extension. This handle is then passed to the API itself, with some additional information such as the process ID and the type of dump file. The sample uses MiniDumpNormal. You may want to or-in the flag value of MiniDumpWithDataSegs, which is the equivalent to the Minidump With Heap option in the Visual Studio debugger. This does result in substantially larger dump files.

Once the .dmp file is created, the application informs the user where it is stored. The user can then e-mail it or use FTP to get you the file to investigate.

To use the sample code provided, add the mdump.h file and declare one MiniDumper object at global scope. Its constructor takes an argument, which should be the basename of the minidump file. Add mdump.cpp to your project. You need to have the correct Dbghelp.dll file in the same directory as the EXE to run successfully.

You cannot debug the code that writes the minidump with a debugger (in the sample code, Minidumper::TopLevelFilter). If a debugger is attached to a process, then the unhandled exception filter will never be called. If you encounter problems, you will need to use MessageBox debugging.

Reading a Minidump with Visual Studio .NET

This section uses an example of a manually created minidump from Notepad in Windows 2000 and debugging in Windows XP.

Start Visual Studio .NET and, on the File menu, click Open Solution. Change the Files of type drop-down menu to Dump Files (*.dmp; *.mdmp), navigate to the minidump, and create a default project by clicking Open.

To launch the dump with the debugger, press F5. This will provide you with information to get started. The debugger creates a fake process; in the Output window, various module load messages are displayed. The debugger is recreated the crashed processes state only. After displaying a warning that the EXE contains no debug information, the debugger stops with the user's crash, such as an access violation. If you then examine the Call Stack window, you will notice the lack of symbols and a lack of useful information.

Figure 1. Initial Stack with No Symbols

To read a minidump, you typically need copies of the binaries involved. To find the right binaries, open the Modules window.

Figure 2. Initial Modules with No Binaries

Figure 2 shows the Notepad example and demonstrates two things. First, the paths to the binaries are marked with asterisks, which denotes the path on the user's workstation but the binary could not be found at that location on your machine. Second, the messages read “No matching binary found” in the Information field. The key to finding matching binaries is to look at the Version field and the file name. In this example, the Version of most system files is 2195, which indicates Windows 2000. It does not immediately indicate the exact service pack (SP) or quality fix engineering (QFE), however. For more information, see the DLL Help database at http://support.microsoft.com/servicedesks/fileversion/dllinfo.asp.

At this point, you will need to find a Windows operating system CD or a user's workstation with the correct versions and copy the correct versions into a directory. It is not normally necessary to find every binary that was in the process, but it is important to find each binary that features on every relevant call stack. This will often involve both operating system binaries (such as Kernel32.dll) and your application binaries (in this example, Notepad.exe).

When you have found the bits and copied them to a local directory, on the Debug menu, click Stop Debugging. In Solution Explorer, right-click the project icon and click Properties on the shortcut menu. This will take you to the Debugging page. Fill in the Command Arguments to be MODPATH, followed by an equal sign, and followed by a semicolon-delimited list of places to look for binaries. In this example, it is:

MODPATH=m:/sysbits

After setting the path, press F5 to reload the minidump. MODPATH reuses the command argument to get the value to the debugger; a future Version of Visual Studio .NET may have a better way of setting this, perhaps as an option in the Properties dialog box.

Although finding the binaries is unlikely to improve the call stack, it should resolve some issues in the Modules window, as shown in Figure 3.

Figure 3. Modules with Binaries

Instead of saying “No matching binary found”, it now says a combination of “Cannot find or open a required DBG file” or “No symbols loaded”. The first message occurs with system DLLs that use DBGs for their debugging information. The second message occurs for DLLs that use PDBs. Getting matching binaries will not necessarily improve your call stack; you also need debug information that matches them.

Method A: Symbols the Hard Way

To completely analyze a minidump, you should find debug information for everything, but to save time, you can find only the information you need. The example stack lists User32.dll and Kernel32.dll, so they need matching debug information.

Matching Debug Information

The best place to locate system symbols is at http://www.microsoft.com/ddk/debugging. You can also locate system symbols on the Support compact disc that ships with Windows NT Server and Windows 2000 Server operating systems. In this example, they were copied to the location of the binaries. In real instances, you will have non-Microsoft binaries listed, so you will need to have PDBs for them. Also in this example, the DBG and PDB for Notepad were also copied, because it was the sample application used.

After you click Stop Debugging on the Debug menu, pressing F5 will list a call stack, as shown in Figure 4. You may find that, as you add new binaries and debug information, your call stack will change. This is to be expected; a call stack can only be walked accurately with debug information, so as you add more information, your stack will get more accurate, which will often expose additional frames that were not in the original.

In this example, there was no crash. In real instances, you would have sufficient information to determine the cause of around 70 percent of your user crashes. In addition, this stack was created with the stripped symbols that Microsoft ships for system component, so there was no line number information. For your own binaries with full PDBs, you will get an even richer stack.

Figure 4. Call Stack with Symbols and Binaries

Symbol Servers

If you are dealing with many minidumps and performing general debugging, storing and accessing all the binary and PDB/DBG files can be difficult. Windows NT developed a technology known as a symbol server, which was originally conceived as storage for symbols but was extended to support finding binary files. The Windows NT debugger was the first tool to support this, but Visual Studio .NET also does so as an undocumented feature. For more information on symbol server, see http://www.microsoft.com/ddk/debugging/symbols.asp.

You can also retrieve symbols from the Microsoft symbol server. These symbols will be cached and indexed locally for you.

Method B: Symbols The Easy Way: Using Symbol Server

First, go to http://www.microsoft.com/ddk/debugging and download the debugging tools. You need Symsrv.dll, which needs to be on the path. You can either copy it next to devenv.exe or into your System32 directory, to allow Visual Studio .NET access to it. Once you have copied Symsrv.dll, you can safely uninstall the debugging tools. You will also need to make a local directory. For this example, create a local directory as C:/localstore.

In the Project Properties dialog box, set the Symbol Path on the Debugging page to:

SRV*c:/localstore*http://msdl.microsoft.com/download/symbols

This string tells the debugger to use the symbol server to get symbols and create a local symbol server, where they will be copied. Now, when you press F5 on the minidump, the symbols are copied from the Microsoft Web site, copied to the local store, and then used by the debugger. After the first time you do this, your performance will be quicker, because they will be retrieved from the local store rather than from the Web.

When debugging a non-Microsoft application, you should use a combination of methods A and B. Use A to get the system components and add paths to your bits and symbols by separating them with semicolons, such as:

c:/drop/build/myapp;SRV*c:/localstore*http://msdl.microsoft.com/download/symbols

Because the symbol server is an undocumented feature of Visual Studio .NET, there is no error reporting. If the syntax is incorrect or Symsrv.dll is not on the path, the symbols will fail to load with the error "No symbols loaded". You can also use symbol servers to store and retrieve binaries, but the MODPATH syntax needs to use symsrv*symsrv.dll* instead of SRV*.

Note   The Microsoft symbol server does not contain binary files, but any symbol servers that you create can.

Symbol servers work for "live" debugging, as well, not only for minidumps. To do this, you will need to set the Symbol Path on the Debugging page properly.

How Microsoft Uses Minidumps

Microsoft has been using minidumps to improve the quality of its applications for more than a year. Microsoft Internet Explorer 5.5 and Microsoft Office XP were the first products to ship with the new Dr. Watson, a utility that catches when an application stops responding, creates a minidump, and asks the user if he or she wants to submit the information to Microsoft.

If the user clicks Send Error Report, then Dr. Watson creates and submits the minidump to a server on the Microsoft Web site. The utility performs the user prompting and minidump saving in a different process, which is spawned from the crashing process. This places less demand on the crashed application, increasing the chances of getting a valid minidump from the user.

Further Improvements

On the server side, the minidumps are analyzed into areas of similar crashes, based on where the crash occurred and in what component. This gives product teams statistics of how often the application crashed and how often a given crash occurs. The teams also receive the actual minidumps of the crashes for further investigation. In some cases where the crash is fully understood, users are automatically directed to a Web page that contains information about a known workaround or contains a fix to correct the problem. Since the release of Internet Explorer 5.5 and Office XP, many other product teams use similar technology to gather crash information. It is also a standard part of Windows XP.

The sample discussed in this article provides a basis for understanding minidumps. The sample does not discuss retrieving the dump file from the user. In the simplest case, the user can be prompted to send the minidump by e-mail. Also, potential privacy issues can occur, because user data may be present on the stack and is guaranteed to be present in a minidump that includes the full heap. This should be made clear to users. The Microsoft data collection policy, which also contains additional information about the exact contents of minidumps, is at http://watson.microsoft.com/dw/1033/dcp.asp.

In addition, the creation of minidumps from a Windows service that has an unhandled exception poses additional challenges, to do with desktop access (for example, if no one is using the console, then you cannot prompt them) and security contexts.

Conclusion

Minidumps are a new technology to port-mortem debug crashes on user workstations. It is easy to add code to existing applications to automatically create them when an unhandled exception occurs. Visual Studio .NET can easily read them back to recreate the crash and allow the developer to debug the problem. Symbol server can be used to make it much easier to find system symbols to aid in this analysis.