PInvoke crash that happened in x86 only

Recently I was asked to check why one of our applications crashes when it calls CreateNamedPipe function. Everything works fine in the x64 version, but the x86 version crashes. There are no differences in the code between x86 and x64 and they compiled the same.

Let’s see how CreateNamedPipe defined in our code:

[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
static extern SafePipeHandle CreateNamedPipe(string lpName, uint dwOpenMode,
    uint dwPipeMode, uint nMaxInstances, uint nOutBufferSize, uint nInBufferSize,
    uint nDefaultTimeOut, SECURITY_ATTRIBUTES securityAttributes);

and here is how it called:

SafePipeHandle safeHandle = CreateNamedPipe(
                $@"\\.\pipe\{pipeName}",
                cPipeAccessDuplex |
                cFileFlagOverlapped,
                cPipeTypeMessage |
                cPipeWait,
                cPipeUnlimitedInstances,
                nOutBufferSize: 0,
                nInBufferSize: 0,
                cWaitForever,
                securityAttr);

Everything looks correct. I checked flags, compared them with documentation and everything looks fine. Checked securityAttr and it contains valid data.

Maybe declaration of function is incorrect? Let’s check it here: https://www.pinvoke.net/default.aspx/kernel32/CreateNamedPipe.html

And everything the same as on this site. Checked SECURITY_ATTRIBUTES on the same site and in the documentation. Still, everything is correct.

Well, it is time to check where crash happened and perhaps it will give us some information. If I run this application from WinDbg it will crash here:

KERNELBASE!CreateNamedPipeW+0x72:
76150102 8b4804          mov     ecx,dword ptr [eax+4] ds:002b:00000010=????????

Effectively, CPU is trying to read from address 0x10, and this is definitely not valid address. Let’s us check where is address come from:

mov     eax,dword ptr [ebp+24h]
…
test    eax,eax
je      KERNELBASE!CreateNamedPipeW+0x23bfb (76173c8b)
mov     ecx,dword ptr [eax+4]

Usually, when you see ebp+constant then it is parameter and if you see edp – constant, then it means local or temporary variable. In our case there is plus, so it must be some parameter. Then code checks if it’s zero and if it is not zero then it attempts to read at this address plus 4. So, it is must be some kind of pointer. There are only 2 pointers passed: lpName and securityAttributes.

Register eax has value 0xC. 0xC + 4 will be 0x10. And because it is parameter, simplest explanation would be that invalid data passed for this parameter. Then I checked all parameters, but I don’t see 0xC there. But sometimes tells me that I saw it before. And then I move mouse over securityAttr and bingo. nLength has 0xC:

I don’t know why Visual Studio showing parameters sorted by name, but SECURITY_ATTRIBUTES is declared like this:

    [StructLayout(LayoutKind.Sequential)]
    public struct SECURITY_ATTRIBUTES
    {
        public int nLength;
        public IntPtr lpSecurityDescriptor;
        public int bInheritHandle;
    }

Now, we have full picture: C# suppose to pass address of structure, but instead of address there is nLength  - first member of that structure. And that tells us that C# passes whole structure on stack instead of passing pointer to structure. As a result, application will crash. For x64 rules are clear:

“Structs and unions of size 8, 16, 32, or 64 bits, and __m64 types, are passed as if they were integers of the same size. Structs or unions of other sizes are passed as a pointer to memory allocated by the caller.”

This structure is clearly bigger than 64 bits, so it passed by reference but for x86, it passed by value. To be honest, x86 has so many different rules and conventions that easy to get lost. Different compilers have different rules in this case.

Anyway, how should we fix it? Very simple, you have to tell compiler that you would like to pass struct by reference:

[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
static extern SafePipeHandle CreateNamedPipe(string lpName, uint dwOpenMode,
    uint dwPipeMode, uint nMaxInstances, uint nOutBufferSize, uint nInBufferSize,
    uint nDefaultTimeOut, ref SECURITY_ATTRIBUTES securityAttributes);

As you can see, I added ref for securityAttributes parameter.

I hope it helps someone.