What can we learn from the xz utils exploit?

I’m sure that many of you heard about a backdoor in xz util that allows specific people to get root access if Linux is configured to allow incoming SSH connections. There are a lot of systems like that because it is a typical configuration for pretty much every Linux machine. Fortunately, it was caught quickly, and as far as we know there were no real hacks made via that backdoor.

But it reveals a lot of problems and most of them are typical for open-source development.  I would like to review them and to explore if these problems apply to the closed-source world.

Let’s start with the obvious problem. You rarely get paid for developing open-source projects. Most of the work is done by hobbyists who do all the work in their free time. That time by definition is very limited and it means that they have to heavily reuse somebody else work. As a result, project B relies on Project C, and Project A relies on Project B. People who developing Project A may not even realize what Project B is using.

As I found in many videos Project C could be supported by some guy for many years and nobody even knows who is that guy but everybody relies on his work. There could be really strict security reviews for all projects above Project C, but Project C itself could be an easy target to insert malicious code.

This person could be overloaded with work and real-life problems and pay less attention to source code review. And somebody bad can push him to his limits and speed up the process of accepting a malicious person as a maintainer. And that is exactly what happened.

The next thing was that source code became extremely complex and nobody could keep the whole picture in their head. And as a result, it became much simpler to push malicious code. “These are just tests, we don’t need to review them”. “This is a build script, looks good to me”. As a result attack was split into 3 different parts and nobody could see the full picture. Just because it is too complex and each of these parts looks innocent and becomes malicious only when combined.

And somebody raised a correct question: “How many similar things already exist in open-source code?” This attack was relatively simple and was easy to trace back. But instead of clear 3 stages, somebody could push 100 really obfuscated stages.

Later somebody raised a valid question: “But this can happen to closed-source projects as well, right?” But unfortunately, the answer would be no. Or more correctly, this is very hard to do for closed-source projects. Let me explain why.

Firstly, in open-source projects, you are selecting the area you want to work on. Sure maintainer can reject your patch, but why they would do it if that patch contains useful code that helps their project? Nobody can tell you what you must work on. It is the nature of open source. You do what you want because you do it for free.

For closed-source projects, you do what you are told. They pay you money and they put your time where they see fit. You can discuss it to a certain degree but most of the time you have to do what you are told. If you don’t like it, all you can do is leave this company.

Secondly, during the code review process, the driving force and motivation for the person who does the code review is love for the project. But eventually, that person can simply burn out after many years of “thankless work”. With time people change and their priorities will change.

For example, a student can start helping to maintain a certain project. That student has a lot of free time and is willing to help. It happened for many years but eventually, that person got a family and he could not afford to spend that much time on the project anymore. That person is a responsible person and he tries his best but simply because he cannot spend as much time as before, something bad can easily pass his guard simply because he is overloaded with this project and real-life obligations.

Again in closed-source projects, review is part of your job. It is what you paid for. Moreover, if you do this job badly, you can be fired and lose your source of income. I agree that it does not guarantee that a person will do proper code review or that that person will do it at all, but if the process is built correctly it will have quite a high chance that review will be done properly.

Obviously, before any project (open source or not) will allow you to commit to sensitive areas you need to gain their trust. It means that it could be years before you will get this privilege. Also obviously even after that, you will be closely monitored for some time after you get this privilege.

And lastly, in open-source projects, you have full anonymity. All information that other people know is your email and fake name. They don’t need anything else to contribute to an open-source project. Right now nobody knows who was the person who inserted the backdoor in xz utils and there is no way to find it.

In closed-source projects that person must reveal their identity. They need to go to the interview, they need to provide their details to receive their salary and many other details. Sure, some governments can create a cover for that person to hide their real identity but the person’s face will be still visible.

Also, imagine that somebody can recognize malicious code is being committed. This could lead to the arrest of that person. What open-source community can do about this? You're right – nothing. As I said, all they have is an email and a fake name.

So yes, while theoretically, it is still possible to do it in closed-source, it is much less likely and it will cost way more to execute it right. Also, there is a much greater risk for the person who executing it.

In conclusion, I would like to show that the problem is much deeper than this particular backdoor and it is possible because of the current nature of open-source development. Also, I don’t want to say that closed-source is better. All I want to say is that open-source development has its own disadvantages as well as closed-source.

But it is not all grim. There are plenty of very smart people in the open-source community and I’m sure that they will find ways to combat this problem and to improve their processes to accommodate new challenges.