Avoid Crowdstrike

In a strange twist of fate, our company got Crowdstrike on all our computers. I will not explain why due to privacy reasons, but we got it. And oh boy, there is a lot to talk about. Let’s begin.

The first thing everyone notices immediately is that our computers are slower. I have my home computer that was on the low-end side when I built it new in 2018. And I have a top-of-the-line computer at work that is a couple of years younger. Many tasks my home computer does faster. Like opening Task Manager or the Open dialog in any application. You just feel that the computer has become slower.

One may say that it is pure psychology. “You expect it to be slower, and it is slower. Your bias confirmed”. But the problem here is that I didn’t know that Crowdstrike was installed on my computer for about a month or so. I just noticed that the computer became slower, but I didn’t have time to investigate. Until somebody just told me that we have it now, and then it clicked.

But one may say that feeling slower is 100% subjective, and it is true. But I measured it. One of the modules of our applications is written in Delphi. I got a new notebook, installed a clean version of Windows 11 and Delphi on it. Cloned a git repository, compiled one of the modules in it, and started our application under the debugger. It took less than 35 seconds to start hot. Just to clarify, hot means the second run that is immediately after the first or second run. So Windows should keep everything in a file cache.

The Delphi debugger is not very efficient because the version of Delphi we use is a 32-bit process, and it must start a 64-bit debugger to debug a 64-bit process. As a result, the debugger and IDE must use some interprocess communication, and the protocol is very chatty. This is why it took 35 seconds to start, while without a Delphi debugger, it starts in 4 - 6 seconds.

Ok, 35 seconds is slow, but then I joined that notebook to our organization's infrastructure. Crowdstrike installed, and boom, doing the same takes 1 minute 25 seconds. More than double. And not only does our application start slower. It stays quite slow and hard to work with. For example, the standard Windows “Open” dialog opens from 10 to 15 seconds. It is hard to explain how bad it is, but working with our application under a debugger feels like running in the water. It simply destroys all productivity. Without Crowdstrike, it was just slower than without it, but it didn’t kill productivity.

To be honest, it is not only Crowdstrike contributed to the slowdown, because later I found that all disks were encrypted with BitLocker. But I couldn’t retest the impact because that notebook was already assigned to a different person, and that model became obsolete, and it is not possible to buy exactly the same one. But according to many studies, slowdown is around 0%-5% for real workloads. Also, that notebook had 64 GB of memory, so BitLocker may affect only the first run, and after that, all files will already be in the file cache because all running applications combined consume less than 10 gigabytes.

I also tested the same scenario in VM. My main computer was running Crowdstrike, and disks were encrypted with BitLocker. Then, I installed Windows 11 in a virtual machine on my computer, installed Delphi, compiled, and ran the same module from the same branch as a notebook. Then I compared with the case when I do the same directly from my computer. The virtual machine was 10%-20% faster than doing the same on the host computer! Normally virtual computer should never be faster than the host computer.

Somebody may say that I can work on the virtual computer, and that problem is solved. But obviously, I cannot work on the virtual machine because it effectively bypasses antivirus and CrowdStrike and would be clearly against corporate policy.

At this moment, some people may say that a small slowdown in performance is an acceptable sacrifice for security. In the case of Delphi, it takes 2 times longer, and it is not small by any means. Moreover, I had a quite fast notebook with fresh Windows 11 on it. Most developers have a slower notebook with Windows that is several years old. So for most of them it takes from 2 to 2 minutes and 30 seconds.

Moreover, this slowdown is more than the typical attention threshold. For 35 seconds, most developers will wait until the application starts. But nobody will wait 2 minutes. People will do something else. They will read email, review code, read a blog, or do something else. As a result, they typically will spend more than 2 minutes before they remember that they are waiting for the application to start. So the typical start time will take close to 3 or even 4 minutes instead of 35 seconds.

But it is not only the start time that will suffer. The developer may debug a complex problem and keep everything in their head. What they already tried and what they want to try. When nothing distracts the developer, the whole process is quite smooth. Now, imagine while waiting, this developer decided to review a completely different piece of code. 5 minutes later, when the application starts, the developer will simply forget what they did and what they didn’t. It's called losing context.

So, slowdown not only affects how many times per day you can start the application, but it also considerably slows down refactoring and debugging because the developer will switch attention to something else.

Moreover, developers will also make mistakes they would normally never make, because they forget that they must also do something else. Let’s say I refactored class A, and I remember that I also need to change class B because they are related. But because I constantly switch to something else, I can simply forget about that. Class B stays unchanged, and this is a big problem. I am constantly losing context, and it is a bad thing.

I have another example. I have a .NET application that reads around 100Mb of serialized data from the disk. It is binary data that is written by the same application. It is a custom binary serialization that I wrote that reads integers and strings from a .NET BinaryStream.

Total reading time is 1.2 seconds. 0.8 seconds of them the application spent in antivirus and CrowdStrike. Almost 80% of these 0.8 seconds is CrowdStrike. So, again, CrowdStrike doubled that time. Just in case, I measured reading from that file many times without writing to it.

I’m sure that even by this moment, there are still some people who can agree to make these sacrifices in the name of security. But the problem here is that Crowdstrike was written quite badly, and I’m not even sure that it can catch anything serious.

We all know that they have failed to do basic things like checking arguments or deploying in stages. But it is far worse than this. Let me provide a concrete example. One day, I was starting our application from the Delphi debugger. And while waiting for 2 minutes for it to start, I decided to review my changes and stage them.

I was using Visual Studio as a Git client. I selected one file, reviewed it, right-clicked, and selected “Stage”. Boom, Windows crashed to the blue screen of death. Well, Windows does this sometimes. Restarted PC and tried to stage files again. But when I clicked on the “Stage” menu item in Visual Studio, nothing happened.

Then I ran the “git status” command, which told me that the git index is corrupted. I never seen this error before, so I spent 30 minutes investigating how to repair this index. Then I ran a command that looked most promising. It took 40 minutes to execute. The index wasn’t repaired, and “git status” produced a different error.

I decided that it would be faster to clone the repository again. While it takes around 40 minutes, it will definitely solve this problem once and for all. Plus, I never know if I can trust the rebuilt index. After 40 minutes of cloning, I spent another 20 minutes copying changed files from the corrupted repo to the new repo. Remember, I couldn’t see which files were changed. I must use only my memory and tools that compares directory.

After I finished all of that, I decided to check if the application actually works. I started the application from the Delphi debugger. And remember, it takes 2 minutes to start, so I decided to review and stage files. And as soon as I clicked on the “Stage” command, Windows crashed again.

After the restart, I checked the Windows mini-dump, and csagent.sys is clearly in the call stack. And as some of you probably guessed, the git index was corrupted again, and I must clone the repository again and manually copy changes again.

So 30 minutes, plus 40, plus 40, plus 20, plus 40, plus 20, total 3 hours and 10 minutes I spent trying to stage a single file. Effectivelly I spent half of the day because of Crowdstrike. I say half I had because I need to add time for 2 restarts, waiting for Windows to load everything, making coffee, and talking while waiting, losing context, etc.

Just in case it wasn’t a single bad day. Before Crowdstrike, my computer typically crashed to a bluescreen once per year or maybe even 2. It was quite rare. Now, with Crowdstrike, my computer typically crashes 1 - 2 times per month. Sometimes 1 – 2 times per week. And as you see above, once it crashed 2 times in a single day. And only God knows what was corrupted during these crashes.

As you can see, I have real damage that happened because I have Crowdstrike on my computer. I would never have these issues if I didn’t have a Crowdstrike on my computer. Will it protect my computer from bad actors? I don’t know, but right to me, CrowdStrike itself is a bad actor. It is damaging files on my computer, and it slows down everything. And I make mistakes because of it.

But let me ask you an important question: “Did anybody ever hear about a resonating case when CrowdStrike saved the day and prevented major cybercrime?” I personally didn’t hear anything like this. But everyone heard about millions of computers turned to bricks and billions of dollars lost because of what looks to me like low-quality software called  CrowdStrike.

So my verdict would be quite simple – avoid CrowdStrike at all costs if you can. The fact that it is quite popular but looks like low quality forces me to suspect that something is quite dirty here.

I hope it was helpful.