Lies and statistics

There is a phrase that is often attributed to British Prime Minister Benjamin Disraeli, and it goes: "There are three kinds of lies: lies, damned lies, and statistics". This phrase perfectly describes how Elon Musk talks about the safety of full self-driving.

Basically, he stated that according to their statistics self-driving crashes way less than when a human is controlling a vehicle. Thus, everybody should use self-driving because it is simply safer.

Let’s assume that statistics is correct. In this case, it sounds like good news for self-driving and everybody should use it, right? Absolutely not, because it is just a different form of lies. Let me explain what I mean.

At the beginning, I must state that in many cases when you are using the self-driving feature and it detects a potential crash, it will disconnect and demand that the driver takes control. If the driver didn’t expect it, then there will be a crash but technically it is not self-driving crashing your car because it was disengaged. Even if the driver was paying attention but self-driving does something stupid, the driver simply may not have enough time to rectify and the crash would happen again and it will still count as a human error.

This will eliminate a lot of blame from self-driving software and it will look much better on paper because it takes only a few milliseconds to decide that we are about to crash, disengage, and demand driver attention even if it is way too late. But it is not a full story.

Now let’s think about when a typical driver will engage in autopilot or self-driving. I would assume that any sane person will do it when it is relatively safe. After all, the driver will be fully responsible if a car kills someone when it drives itself. Just imagine there is a bunch of kids crossing in front of your car. Will you test autopilot in this situation or just press break?

I would assume that any reasonable person would press a brake pedal just in case. I don’t want to check if the software will react properly because the cost of a mistake is extremely high.

For example, just recently I drove around 900 miles. Around 90%-95% of the time and more than 95% of the distance I was using autopilot because I was driving on highways and it works quite well there.

I disengaged it only when I wasn’t sure that it would make the right decision. For example when concrete blocks from the construction site are too close to the left side of my car. Or when I overtake a truck. Or when I wasn’t sure that it would stop in time because there was a traffic jam in front of me and everybody stopped. You got the idea.

So when it is safe I engage autopilot. When it is not, I take control and drive by myself. In other words, I use autopilot when the chance of a mistake and its cost is low, and drive by myself in complex situations.

Statistics simply compare the number of hours or miles driven by a driver and by a self-driving computer. And then comparing the number of crashes and self-driving looks better in this case. But only because they didn’t take into account the conditions it was happening and this is where lies are.

Imagine me driving 100 000 miles with a teenager who just got their driving license. I will drive in the city and in every complex situation. And the teenager will drive on highways where you mostly stay in your lane and keep. There is a really high chance that by that kind of statistic, that teenager will have fewer crashes than me.

But it didn’t make the teenager a better driver. It's just situations are much easier for the teenager and there is a really low chance of making any kind of mistake. And with autopilot, I can intervene at any time and take control. As a result, in many situations where autopilot would crash, I simply didn’t let it happen.

In conclusion, this kind of statistic is just pure lies because they didn’t compare apples to apples. Plus you need to take into account disengaging when self-driving is about to crash and cases when the driver prevents software from crashing your car. After that, you will understand that we actually have no idea how safe is it.

And here is advice – never blindly trust statistics. Statistics works only when conditions are exactly the same. Otherwise, it is meaningless stuff that people massage to their advantage like in this example.