How to find .NET exceptions in the dump file
- Posted in:
- .NET
Our application is written in 3 languages and very often it is hard to get a full call stack. We are constantly improving it but, in some cases, it is really hard to find the source of the problem.
Very often I had to investigate crash dump without any visible .NET stack in it. Usually in this case, I want to see if there are any .NET exceptions and their call stack. For this, I use the following command in WinDbg:
!DumpHeap -type Exception
Typically it will print any object that has the word “Exception” in their name. Something like this:
7ffcbebbc7f8 1 80 System.Collections.Generic.Dictionary<NLog.Config…
7ffcbfeeed68 2 80 System.Lazy<System.Collections.Generic.IEqualityComparer<System.Exception>>
7ffcbeb71088 1 112 NLog.LayoutRenderers.ExceptionLayoutRenderer
7ffcb918f328 1 128 System.OutOfMemoryException
7ffcb918f428 1 128 System.StackOverflowException
7ffcb918f528 1 128 System.ExecutionEngineException
7ffcb92be9c0 4 256 System.Windows.Threading.DispatcherUnhandledExceptionEventHandler
7ffcb91ce6f8 4 512 System.IO.IOException
Typically list is much longer and this is just a small part of it.
Some are exceptions that are pre-created by runtime. For example, System.OutOfMemoryException. Obviously, if there is out of memory situation, there is no memory to create a new System.OutOfMemoryException object and as a result, runtime pre-created it to be able to throw at any time.
The next one is the System.StackOverflowException. To create a new object, you need to call some function and that call needs a stack. But we are out of the stack, so we cannot create it. So again, the runtime just pre-creates it.
The last one is the System.ExecutionEngineException. Typically this exception is thrown when the internal state of runtime is corrupted and obviously, it is not a good idea to create any new objects.
Some of these are real exceptions like System.IO.IOException. You can click on the left side on 7ffcb91ce6f8 and then you will see something like this:
0:000> !dumpheap -mt 7ffcb91ce6f8 Address MT Size 00022efcd658 7ffcb91ce6f8 128 00022efed2d8 7ffcb91ce6f8 128 00022f0117c8 7ffcb91ce6f8 128 00022f01f230 7ffcb91ce6f8 128
Then you can click on the address, for example on 00022efcd658 and you will something like this:
0:000> !dumpobj /d 22efcd658 Name: System.IO.IOException MethodTable: 00007ffcb91ce6f8 EEClass: 00007ffcb91a6ed8 Tracked Type: false Size: 128(0x80) bytes File: C:\Program Files\Altium\AD20\System\DotNet\Runtime\shared\Microsoft.NETCore.App\6.0.30\System.Private.CoreLib.dll Fields: MT Field Offset Type VT Attr Value Name 00007ffcb91de470 4000236 8 ...ection.MethodBase 0 instance 0000000000000000 _exceptionMethod 00007ffcb918d708 4000237 10 System.String 0 instance 000000022efcd6d8 _message 00007ffcb9191d20 4000238 18 ...tions.IDictionary 0 instance 000000022f00c4f8 _data 00007ffcb918f0b8 4000239 20 System.Exception 0 instance 000000022efcd4f8 _innerException 00007ffcb918d708 400023a 28 System.String 0 instance 0000000000000000 _helpURL 00007ffcb9392ef8 400023b 30 System.Byte[] 0 instance 000000022f00fc58 _stackTrace 00007ffcb9392ef8 400023c 38 System.Byte[] 0 instance 0000000000000000 _watsonBuckets 00007ffcb918d708 400023d 40 System.String 0 instance 0000000000000000 _stackTraceString 00007ffcb918d708 400023e 48 System.String 0 instance 0000000000000000 _remoteStackTraceString 00007ffcb90db588 400023f 50 System.Object[] 0 instance 0000000000000000 _dynamicMethods 00007ffcb918d708 4000240 58 System.String 0 instance 000000022f00c5f0 _source 00007ffcb91851b8 4000241 60 System.UIntPtr 1 instance 00007FFCC2AAAB49 _ipForWatsonBuckets 00007ffcb9184298 4000242 68 System.IntPtr 1 instance 0000000000000000 _xptrs 00007ffcb91794b0 4000243 70 System.Int32 1 instance -532462766 _xcode 00007ffcb91794b0 4000244 74 System.Int32 1 instance -2146232800 _HResult You can use the following command to see it better:
!PrintException 22efcd658
And then you will see something like this:
0:000> !PrintException 22efcd658 Exception object: 000000022efcd658 Exception type: System.IO.IOException Message: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. InnerException: System.Net.Sockets.SocketException, Use !PrintException 000000022EFCD4F8 to see more. StackTrace (generated): SP IP Function 000000025F46F6A0 00007FFCC2AAAB49 System_Net_Sockets!System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException… 000000025F46F6D0 00007FFCC1310C7B System_Net_Sockets!System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sou… 000000025F46F720 00007FFCC137AF52 System_Net_Security!System.Net.Security.SslStream+<ReadAsyncInternal>d__188… 000000025F46A650 00007FFCC2AA7AB0 System_Private_CoreLib!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x20 000000025F46A680 00007FFCC2A95E5C System_Private_CoreLib!System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSucces… 000000025F46A6C0 00007FFCC18A9A5A System_Private_CoreLib!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotific… StackTraceString: <none> HResult: 80131620
As you can see this way it is possible to see all exceptions that are still live. This is nice and helpful, but it takes a lot of time. I had to search exceptions manually and filter out non-exceptions because that something just had the word “Exception” in it. Then copy/paste the exception address and type!PrintException. Is there a better way? And it turns out that there is a better way.
You can download the Microsoft MEX debugging extension here. Then type the following command
.load pathToMex.dll
and then this command:
!dae
which is “Dump all exceptions” and it will print something like this:
4 exceptions: 0x000000022efcd658 0x000000022efed2d8 0x000000022f0117c8 0x000000022f01f230 In Generation: 0 from .NET v6.0.3024.21525 HResult: 0x80131620 Type: System.IO.IOException Message: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. Inner Exception: 0x000000022efcd4f8 Stack Trace: SP IP Function 000000025f46f6a0 00007ffcc2aaab49 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException… 000000025f46f6d0 00007ffcc1310c7b System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.System.Th…. 000000025f46f720 00007ffcc137af52 System.Net.Security.SslStream+<ReadAsyncInternal>d__188`1[[System… 000000025f46a650 00007ffcc2aa7ab0 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 000000025f46a680 00007ffcc2a95e5c System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Sys… 000000025f46a6c0 00007ffcc18a9a5a System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebu… 000000025f46a6f0 00007ffcc1e7d22f System.Net.Http.HttpConnection+<<CheckUsabilityOnScavenge>g__ReadA… This command will find all exceptions and display their call stack which helps a lot when you have many different exceptions.
I hope it helps someone.