Case of frozen PeekMessage

September 15. 2020 0 Comments

Posted in:
Windows

Recently I was investigating why our application freezes during certain operations. After collecting few dump files, I found that every single time application freezes in quite old code that was written about 20 years ago. Loading documents could be quite long, and some person decided that it would be good idea if user can press Escape and cancel loading. To detect key press, that code calls something like this:
PeekMessage(Msg, CurrentFormHandle, WM_KeyFirst, WM_KeyLast, PM_Remove Or PM_NoYield)

Callstack looks like this:

win32u!NtUserPeekMessage+0x14
user32!_PeekMessage+0x43
user32!PeekMessageW+0x143

It looks quite innocent and I do not see any reason why it should hang. After some thinking I came with two theories:

CurrentFormHandle returns invalid window handle that got reused by some thread that is sleeping
Code that calls PeekMessage has infinite loop and PeekMessage is just took more time and appears in crash dumps.

Item #2 was quickly dismissed as analyzing code reveals that only one place has for-loop and I was able to find index value and max value on stack and they were 0 and 2. Clearly not infinite loop.

Then I tried to reproduce problem on my PC and after some time I was able to freeze our application. After that #1 was dismissed as I found that handle is correct value and it is windows that suppose to be. Then I start thinking that something could be wrong with that window, but there are too many possibilities. As result I decided to dive deeper and check what happens in kernel side.

To create kernel dump you need https://docs.microsoft.com/en-us/sysinternals/downloads/livekd and a lot of free space on disk as dump file will have the same size as amount of RAM you have. I went to directory where WinDbg is installed and ran following command:

livekd64.exe -o filename

After that it created 32 Gb file and you can open it in WinDbg the same way as any dump file.

First command you need is to find address of your process:
!process 0 0 MyApp.exe

You will see something like this: PROCESS ffff988f5ae03080

Then you set current process using this command:
!process ffff988f5ae03080

It will dump all threads for that process. Typically, thread looks like this:

THREAD ffff988f4c60d080 Cid 8a50.69f0 Teb: 000000000028b000 Win32Thread: ffff988f6c4b2400 WAIT: (WrUserRequest) UserMode Non-Alertable ffff988f616130c0 QueueObject

Then you need to set current thread using this command:
.thread /p/r ffff988f4c60d080

After you can use typical call stack commands like k or kv. In my case callstack looked like this:

nt!KiSwapContext+0x76
nt!KiSwapThread+0x500
nt!KiCommitThreadWait+0x14f
nt!KeWaitForSingleObject+0x233
nt!KeWaitForMultipleObjects+0x45b
win32kfull!xxxRealSleepThread+0x362
win32kfull!xxxInterSendMsgEx+0xdd9
win32kfull!xxxSendTransformableMessageTimeout+0x3ea
win32kfull!xxxCalcValidRects+0x32d
win32kfull!xxxEndDeferWindowPosEx+0x1ac
win32kfull!xxxSetWindowPosAndBand+0xc3
win32kfull!xxxSetWindowPos+0x79
win32kfull!xxxSetForegroundWindow2+0x772
win32kfull!xxxProcessActivationEvent+0xa6770
win32kfull!xxxProcessEventMessage+0x343
win32kfull!xxxScanSysQueue+0x83c
win32kfull!xxxRealInternalGetMessage+0xee6
win32kfull!NtUserPeekMessage+0x158
win32k!NtUserPeekMessage+0x2a
nt!KiSystemServiceCopyEnd+0x25
win32u!NtUserPeekMessage+0x14
user32!_PeekMessage+0x43
user32!PeekMessageW+0x143

It looks like PeekMessage is attempting send message to different thread and that thread does not respond. After checking special place, I confirmed that xxxInterSendMsgEx is for sending inter-thread messages. But how can I find which thread is it? Likely to me, special places told me that first parameter of xxxInterSendMsgEx is PWND and just pointer to regular HWND. As I ran kv command:

ffff8808`47efca80 00000000`00000000 00000000`00000000 ffff8808`49ff2310 : win32kfull!xxxInterSendMsgEx+0xdd9

and then:
? poi(ffff8808`47efca80)

And found window handle that I was looking for. Then using Spy++ from Visual Studio I found thread and found why application hangs.

It turns out that we have progress window that supposed to show that something is happens during long operation. That window is created in background thread. As result main window has pending deactivation and new foreground window should be progress window. And PeekMessage decided to send message to that window because it is in different thread. But activity in main thread and background thread reacts on the same WPF event and activity in main thread took lock in System.Windows.FrameworkTemplate.LoadContent and thread in background attempting to take the same lock and as result we have a deadlock.

Mystery solved. I hope it helps someone.