Home > IIS Stories > .NET Debugging #1/3 – What are in a memory dump

.NET Debugging #1/3 – What are in a memory dump

 

When I attended debugging training first time, The title of debugging training was ‘Win32 User Mode debugging’, and most training materials, were surely difficult & most boring, but the concepts inside, & troubleshooting techniques were very intereting. After 1 week of training, I still had no idea to start debugging professionally, and anything was not easy to debug.

Basically, debugging tasks using WinDBG, is not such a simple way, but I couldn’t give up learning, because  it’s so powerful to troubleshoot any issues, such as poor performance, application crash, and memory leak problems.

I’d like to cover easy & practical information of debugging for IT professionals who experienced simple debugging skills, or troubleshotting using memory dumps.

‘What are in a memory dump’ is one thing. This article is not desinged to show you how to debug, and I’m not the right persion to spend a lot of time to cover all details, and background knowledge that I also don’t know clearly.

So, let me summarize terms to be used in my blog posts as follows, it’s a good idea to search & review actual concepts of them, to help your understanding on this topic.

Tems to be used,

Memory dump
Process, Thread, Private Bytes
Windows Debugger, WinDBG
Call-stack, Base Frame Pointer
Programming – Local Variable, Parameters, Functions
Crash, Access Violation

 

Collecting a Memory Dump

 

There are multiple ways to collect memory dumps, and I think the following way is the easiest way, to use the task manager.

For example, if you want to debug W3WP.EXE – IIS worker process, by checking its memory dump file using debugger,

image

. Open the task manager, and select the target process to collect a memory dump,

. Right-click, select ‘Create dump file’, and check the location to save the dump

. Now, you have w3wp.dmp – memory dump – which is a sort of snapshot of w3wp process memory.

Another common way is to attach the debugger(WinDBG) to a target process, and this will be discussed later.

 

Reading a Memory Dump

 

FYI, it can be a good idea to try to read & follow this section to get overall ideas of debugging tasks, as this is not intended to teach how to debug.

To understand how to read a memory dump using WinDBG, it will be easier to see C program code(DebugDemo.exe) first as follows,

int myFunction1(int a, int b);
int myFunction2(int c);

int _tmain(int argc, _TCHAR* argv[])
{
    int iVar1 = 1;
    int iVar2 = 2;
    int iReturn = 0;

    iReturn = myFunction1(iVar1, iVar2);

    return 0;
}

int myFunction1(int a, int b)
{
    int i = 0;
    int j = 0;

    i = a + b;
    j = myFunction2(i);

    printf("The return value from myFunction2 is %d", j);

    getchar();
    return i;
}

int myFunction2(int c)
{
    int i = 5;
    i = i * c;   

    return i;

}

It’s not necessary to understand all the details of the above codes, we only need to see that myFunction1 has two parameters (iVar1 = 1, iVar2 = 2), and there’re also 2 local variables – i, j, after calling myFunction2, the value of ‘i’ is 3,  ‘j’ is 15.

This program will print “The return vale from myFunction2 is 15”, and will wait for user key input, because of getchar(), as shown below.

image

At this moment, in the task manager, we can see DebugDemo.exe, which is running the above code, and we can collect a memory dump for the running process, using the task manager, as mentioned above.

To read memory dump, we can open the memory dump in WinDBG, or attach the debugger to the live process as shown below,

image

And, select the target process(DebugDemo.exe) to attach the debugger,

image

To read a memory dump more exactly, it’s required to set symbol path(a location for *.pdb files) in the file menu, as shown below,

image

Now, we’re ready to read call stacks using WinDBG, ‘k’ series commands help, the below picture is just for your information about an output of k command.

image

Again, the topic of this article is ‘what are in a memory dump’, so, to help us to see things in a memory dump, I’m going to run debugger commands to check local variables, and parameters in a function(myFunction1) that has a known source code. To move forward, let’s review the structure of call-stack briefly,

Call stack layout.svg

In the above, from Frame Pointer, local variables are located above of it, and parameters are located at below of it. Using WinDBG, it’s possible to read local variables and parameters using the address of frame pointer.

image

For example, as shown above, two parameters of myFunction1(iVar1 = 1, iVar2 = 2) are found at 0068f9f4+8, and 0068f9f4+C.
In the debugger, dd(display DWORD) command with a given address(0068f9f4 – EBP, extended base frame pointer) can find & show you values using relative locations of EBP. And the local variable(i=a+b, 3) is also shown at 0068f9f4-8.

To make this simply, kn & dv commands can do the same thing,

image

Now, you’re seeing local variables & parameters from the memory(or memory dump) using WinDBG. When you review this call stack from the bottom to top, it’s also possible a function that is running now, callers, & callees. The call stack from this memory is exact situation that is in the memory, and executing functions by CPU.

By checking local variables, and parameters, when they’re indicating addresses of heap, it’s also possible to check heap memory with its data. As you can see, using debugger commands to analyze memory dump is not easy to learn for most people. Many great experts developed this tool, and a set of commands to read memory dump, has been scripted to automate debugging tasks, such as checking process heap, stack objects, process status. the set of scripted commands is a debugger extension, that will be used later. SOS, PSSCOR are very useful debugger extensions, will be introduced in a later blog post about .NET debugging.

 

What are in a memory dump

 

A process memory dump is a snapshot of a running process, can be written into a file(a dump file, *.dmp). So, basically it includes all the data of process memory, such as call-stack for each thread, local variables, parameters, a function – currently running, functions with callers & callees, process heaps – allocated at that time. WinDBG is the tool to analyze memory dumps to find something mainly for troubleshooting.

In summary, a memory dump includes,

. Call-stacks for each thread with the information of CPU registers
. A call-stack that displays functions of caller & callee
. A currently running function(or last function) at the top of call-stack.
. local variables and parameters for each function
. Data from allocated process heap, although it’s very difficult to find & read
. Processor Time-spent for each thread, good to check high-cpu problem in multi-threads

By setting a trigger to collect a memory dump exactly when a crash (Access Violation, Null reference exception, unmanaged, or unhandled exceptions) happens, it’s possible to check the exception details, and last function that threw the exception, or caller function that caused the exception, or a caller function that passed a wrong parameter that caused exceptions. They are in a call-stack that threw the exception, as long as the memory dump was collected at a perfect timing.

By collecting a memory dump file when process became unresponsive status(process, or service hang issue), usually, the last function at the top of stack frame can give us clues to check what it has been waitting for, or why processors should keep running something.

 

Side effects of collecting memory dump files

 

On any server, running mission critical services, collecting a memory dump can be a concern because the target process can be frozen during the process memory is being written into a dump file.

How long does it take to create a memory dump

It takes time to write the target process memory into a file in a disk. For example, if a target process memory is around 300MB, creating dump file takes a bit more than time to write 300MB file into a disk. In some cases, such as collecting memory dump files of multiple w3wp.exe processes those are using huge size of process memory, it will take longer. In addition, some processes, such as SQLServer.exe can consume many gigabytes of memory, so, it is not a good idea to collect a dump for the really BIG process on mission critical services. In general, it takes a few seconds to collect a memory dump for a single process.

FYI, the size of process memory dump can be guessed by using a performance counter, which is Process\(PROCESS_NAME)Private Bytes in PerfMon. By checking this, you can make sure the disk free space is enough to write dump files before collecting memory dumps.

In summary,

. The target process will be frozen during creating its memory dump file.
  : This can cause downtime of your service process.

. Free disk space for dump files to be created.
  : lack of free disk space on the system drive(C:) can be more critical than any other issues.
  : You should be very careful when it’s configured to collect memory dump everytime when any exceptions happen in the process. Exceptions can happen a lot for just several seconds.

On the other hand, it’s also required to consider that the purpose of collecting memory dump files. If you need to collect a memory dump, because of performance issues, such as no response from web servers, then, it can be more important to prevent further disasters in your service, and the service already has poor performance, the side-effect of collecting a memory dump can be ignored.

 

References

Call Stack – http://en.wikipedia.org/wiki/Call_stack

Advertisements
Categories: IIS Stories
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: