Our application is written in 3 languages and very often it is hard to get a full call stack. We are constantly improving it but, in some cases, it is really hard to find the source of the problem.
Very often I had to investigate crash dump without any visible .NET stack in it. Usually in this case, I want to see if there are any .NET exceptions and their call stack. For this, I use the following command in WinDbg:
!DumpHeap -type Exception
Typically it will print any object that has the word “Exception” in their name. Something like this:
7ffcbebbc7f8 1 80 System.Collections.Generic.Dictionary<NLog.Config…
7ffcbfeeed68 2 80 System.Lazy<System.Collections.Generic.IEqualityComparer<System.Exception>>
7ffcbeb71088 1 112 NLog.LayoutRenderers.ExceptionLayoutRenderer
7ffcb918f328 1 128 System.OutOfMemoryException
7ffcb918f428 1 128 System.StackOverflowException
7ffcb918f528 1 128 System.ExecutionEngineException
7ffcb92be9c0 4 256 System.Windows.Threading.DispatcherUnhandledExceptionEventHandler
[...Read More]
A few weeks after we converted our application to .NET I received an email that states that there are a lot of network exceptions thrown by our application during normal work. None of them are visible to the customer and it just a lot of exceptions.
Obviously, I started my investigation and after some time I found a way to reproduce them. All I needed was to enable all Exceptions in Visual Studio and tell our application to establish a network connection to any server. And after some execution stopped with System.IO.IOException and message: “Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..”
[...Read More]
I was porting our application to .NET 8 and I found that some 3rd party library throwing InvalidProgramException with the message “Common Language Runtime detected an invalid program”. It is not something that you see often and I started investigating.
The code looks like this:
…
ilgenerator.Emit(OpCodes.Brtrue_S, (sbyte)(OpCodes.Newobj.Size + 4 + OpCodes.Throw.Size));
ilgenerator.ThrowException(typeof(ArgumentOutOfRangeException));
…
That 3rd party library trying to generate IL code that checks a boolean condition and if it is false then throw an exception and otherwise skip that block and continue execution. I reviewed the code it tries to generate and I didn’t find anything suspicious.
At that moment, I had an assumption that the IL code was not valid and newer version
[...Read More]
The previous part is here.
And finally, we are in the most important part which explains what works differently between the .NET Framework and .NET 6. Just in case I will talk about behavioral changes and about obvious parts like some methods that do not exist in .NET 6 but I will
- The .NET Framework will suppress all exceptions when it releases a reference to a COM object but .NET 6 will crash. One example of such a problem is when some object is freed despite having a reference count of more than zero. It is a bug and needs to be fixed. I just was surprised by how often it happened in our application.
- Accessing .NET from DllMain will lead to deadlocks. We had these in the .NET Framework, but they happened much more often in the .NET 6. And obviously, it needs to be fixed. It will take some time because it works most of the time and deadlock happens quite seldom.
- string.GetHashCode will return different values every time you start the application. Basically during startup runtime will generate a random seed and add it to the hash code. If you want hash code to persist then write a function that hashes string differently.
- When the application converts a floating point number to a string, the result will be slightly different. Here blog post that explains the changes. For example, when the .NET Framework outputs 3.1415, .NET may output as 3.141500001 or 3.141499999.
- Encoding.Default will be always UTF8 in .NET 6. If you want to return old behavior then you need to create Encoding from the current Windows code page.
- System.Diagnostics.ProcessStartInfo.UseShellExecute is true for .NET Framework but false for .NET As a result, previously it was possible to open the http link without changing properties but now UseShellExecute must be set specifically to true.
- Previously it was possible to bind to read-only property in WPF. Now it throws an exception and you need to add “, Mode=OneWay”.
- SHA512.Create("System.Security.Cryptography.SHA512CryptoServiceProvider") will return null. We replaced it with new SHA512CryptoServiceProvider().
- There is no SHA512Cng class. We replaced it with SHA512CryptoServiceProvider class that uses SHA512Cng internally when the application is running for Windows.
- Setting SecurityProtocolType.Ssl3 will throw an exception.
- Some controls were removed from WinForms. For example ContextMenu. They were replaced with a set of different controls quite some time ago but our code used an old version for some reason.
- In the .NET Framework, your application can load assembly A 2.0.0.0 from the application directory. Later it can load assembly K located in a different directory and assembly K can load assembly A version 1.0.0.0 from the same directory that contains assembly K and it will work. This will fail in the .NET 6 because another version with a higher version is already loaded. To fix this you will need to use AssemblyLoadContext.
- If you are using different assembly versions in WPF then it will work properly. You will have to rename assemblies to include the version in the assembly name. I spent quite a bit of time on this and I didn’t find any better way.
- If you are enumerating over HashSet sometimes it will contain elements in a different order in .NET 6. We found that during tests. But HashSet does not guarantee the order so it was an issue on our side.
- string.Compare("C-O", "CA") will return 1 for the .NET Framework and -1 for .NET 6. It is because Microsoft changed how comparing works to have the same result for all platforms. Previously it used Windows API for this.
[...Read More]
And just for perspective, we have 824 C# projects and when someone thinks about converting, they may believe that they need to convert all these projects to support .NET X (where X >= 5), change the build system, etc.
It turns out that there is an easy way to do and it allows us to do proof of concept very quickly and then test compatibility. And to do this you need to do nothing. Yep, you heard me right. Our application is a native application that loads .NET Framework runtime and then loads .NET Framework assemblies.
It turns out that all we need to do is to change the .NET Framework loader to load .NET 6 runtime. It is described
[...Read More]
Previous part is here. This post will explain, how to restart your web site at specific intervals without returning errors to clients. You can scroll to solution, if you are no interested in my thoughts and to see what I try.
Thoughts and research
As I mentioned in
this post, I decided to restart my web site periodically to avoid out of memory issue. And I would like to mention that it wasn’t easy to do. Obviously, restarting web site is super simple and all you need is to restart its service. But that means that your web site will be unavailable for some time. Depending on size of the web site, we are talking about seconds or
[...Read More]
Some time ago, I wrote series of posts on how to run .NET Core app on AWS Lightsail Linux instance. Everything worked nice but sometimes, about once per month my web site stopped responding. And I cannot connect to my instance at all to diagnose that issue. All I can do is just restart my AWS instance. At the beginning I thought it could be AWS issue, or perhaps some issues in .NET. I updated everything I can, but problem persists. And when it happened last time, I decided to check kernel logs and I found this:
Feb 10 07:31:18 ip-1-2-3-4 kernel: [1512878.216567] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/www.example.com.service,task=dotnet,pid=511,uid=1001
Feb 10 07:31:18 ip-1-2-3-4 kernel: [1512878.216597] Out of memory: Killed process 511 (dotnet) total-vm:3007440kB, anon-rss:105260kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:712kB oom_score_adj:0
Feb 10 07:31:18 ip-1-2-3-4 kernel: [1512878.241642] oom_reaper: reaped process 511 (dotnet), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[...Read More]
Recently I was asked to check why one of our applications crashes when it calls CreateNamedPipe function. Everything works fine in the x64 version, but the x86 version crashes. There are no differences in the code between x86 and x64 and they compiled the same.
Let’s see how CreateNamedPipe defined in our code:
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
static extern SafePipeHandle CreateNamedPipe(string lpName, uint dwOpenMode,
uint dwPipeMode, uint nMaxInstances, uint nOutBufferSize, uint nInBufferSize,
uint nDefaultTimeOut, SECURITY_ATTRIBUTES securityAttributes);
and here is how it called:
SafePipeHandle safeHandle = CreateNamedPipe(
$@"\\.\pipe\{pipeName}",
cPipeAccessDuplex |
cFileFlagOverlapped,
cPipeTypeMessage |
cPipeWait,
cPipeUnlimitedInstances,
nOutBufferSize: 0,
nInBufferSize: 0,
cWaitForever,
securityAttr);
Everything looks correct. I checked flags, compared them with
[...Read More]
Previous part is here.
I found that I was missed one critical part that I did in my application but forgot to explain here. Let me explain little bit. Effectively secure communication via HTTPS protocol happens between browser and nginx that is web server. Nginx communicates with .NET application via HTTP protocol. And effectively .NET application believes that it is communicating via HTTP. It called SSL termination.
And as result .NET application can return something that is not compatible with HTTPS protocol. In my case it returns link to profile from Gravatar service via HTTP. This in turn leads to complain from web browser that there is mixed content: HTTP and HTTPS. And that one only minor problem but
[...Read More]