Today's news that a Tesla driver was killed in early May while driving his Model S with Autopilot engaged will inevitably trigger a series of tough questions — many of which have already been posed — about whether semi-autonomous driving features are ready for prime time, and whether automakers should be putting safety-critical "beta" software into real customers' hands. For that matter, what does "beta" even mean in this context? Will a self-driving car ever be completely incapable of crashing? (Not likely, but if it is, it's still decades away, and may require that human drivers stop driving.)
But many will be quick to remind that Tesla's Autopilot is not a fully self-driving system anyway — it's generally considered Level 2 on NHTSA's 0-4 scale of autonomy. At Level 2, there's still a lot that the driver is still responsible for, which is really laid bare in an unusually long series of warnings printed in the Model S owner's manual:
Warning: Do not depend on Traffic-Aware Cruise Control to adequately and appropriately slow down Model S. Always watch the road in front of you and stay prepared to brake at all times. Traffic-Aware Cruise Control does not eliminate the need to apply the brakes as needed, even at slow speeds.
Warning: Traffic-Aware Cruise Control can not detect all objects and may not detect a stationary vehicle or other object in the lane of travel. There may be situations in which Traffic-Aware Cruise Control does not detect a vehicle, bicycle, or pedestrian. Depending on Traffic-Aware Cruise Control to avoid a collision can result in serious injury or death.
Warning: Traffic-Aware Cruise Control may react to vehicles or objects that either do not exist or are not in the lane of travel, causing Model S to slow down unnecessarily or inappropriately.
Warning: Traffic-Aware Cruise Control may misjudge the distance from a vehicle ahead. Always watch the road in front of you. It is the driver's responsibility to maintain a safe distance from a vehicle ahead of you.
Warning: When you enable Traffic-Aware Cruise Control in a situation where you are closely following the vehicle in front of you, Model S may apply the brakes to maintain the selected distance.
Warning: Traffic-Aware Cruise Control has limited deceleration ability and may be unable to apply enough braking to avoid a collision if a vehicle in front slows suddenly, or if a vehicle enters your driving lane in front of you. Never depend on Traffic-Aware Cruise Control to slow down the vehicle enough to prevent a collision. Always keep your eyes on the road when driving and be prepared to take corrective action as needed. Depending on Traffic-Aware Cruise Control to slow the vehicle down enough to prevent a collision can result in serious injury or death.
Warning: Driving downhill can increase driving speed, causing Model S to exceed your set speed. Hills can also make it more difficult for Model S to slow down enough to maintain the chosen following distance from the vehicle ahead.
Warning: Traffic-Aware Cruise Control may occasionally brake Model S when not required based on the distance from a vehicle ahead. This can be caused by vehicles in adjacent lanes (especially on curves), or by stationary objects.
Several of these warnings apply directly to the situation apparently faced by the driver in this crash. "There may be situations in which Traffic-Aware Cruise Control does not detect a vehicle, bicycle, or pedestrian. Depending on Traffic-Aware Cruise Control to avoid a collision can result in serious injury or death," it reads. In the incident, Tesla says that Autopilot failed to see the "white side of the tractor trailer against a brightly lit sky," as did the driver.
Later, more warnings in the manual caution that "bright light" can cause problems, and that "Traffic-Aware Cruise Control may not slow down or may brake or accelerate Model S inappropriately":
Traffic-Aware Cruise Control is particularly unlikely to operate as intended in the following types of situations:
- The road has sharp curves.
- Visibility is poor (due to heavy rain, snow, fog, etc.).
- Bright light (oncoming headlights or direct sunlight) is interfering with the camera's view.
- The radar sensor in the center of the front grill is obstructed (dirty, covered, etc.).
- The windshield area in the camera's field of view is obstructed (fogged over, dirty, covered by a sticker, etc.).
Caution: If your Model S is equipped with Traffic-Aware Cruise Control, you must take your vehicle to Tesla Service if a windshield replacement is needed. Failure to do so can cause Traffic-Aware Cruise Control to malfunction.
Warning: Many unforeseen circumstances can impair the operation of Traffic-Aware Cruise Control. Always keep this in mind and remember that as a result, Traffic-Aware Cruise Control may not slow down or may brake or accelerate Model S inappropriately. Always drive attentively and be prepared to take immediate action.
Warning: Traffic-aware cruise control may not brake/decelerate for stationary vehicles, especially in situations when you are driving over 50 mph (80 km/h) and a vehicle you are following moves out of your driving path and a stationary vehicle or object is in front of you instead. Always pay attention to the road ahead and stay prepared to take immediate corrective action. Depending on Traffic-Aware Cruise Control to avoid a collision can result in serious injury or death.
Again, many of these seem to apply to the crash at hand. The problem, of course, is that these are just words in an owner's manual — drivers will, on occasion, fail to heed them or may never read the owner's manual in the first place. Even prompts in an in-car UI, like the one that Tesla uses to tell drivers to pay attention, are easily ignored and bypassed.
Tesla says that Autopilot has driven 130 million miles in owners' vehicles, now with one fatality; that compares to a US average of one vehicular fatality every 94 million miles. So yes, it is statistically doing better than average, but there's an expectation — however fair or unfair — that the computers can and should be perfect. Every incident is going to be scrutinized at length. (Think back to Google's first at-fault crash, which involved only minor damage and no injuries.)
The question isn't whether these warnings are fair; by and large, they're totally reasonable. The question is whether Tesla — and the rest of the auto industry — will find a way to render the warnings obsolete.