CW-Net Uses Explainable Deep Learning to Help Humans Predict Self-Driving Car Mistakes

Explainable deep learning improves human mental models of self-driving cars

Many systems involving human-robot interaction require real-time explanations, including AI wingmen, drone navigation systems and robotic surgeons. Similarly to AVs, many of these applications increasingly rely on deep learning, with a long tail of potentially catastrophic failure cases. Many regulatory bodies have already made explainable AI a core component of their legislation, with AVs likely to follow suit as they are widely deployed with various users11. As such, the success of CW-Net suggests that similar algorithms may prove essential for meeting the regulatory standards for deploying AVs, while building appropriate trust in the technology. In future work, it would be prudent to extend CW-Net to a larger set of concepts -- perhaps in an unsupervised manner to overcome the challenges of labelling -- and better cover the vast array of concepts relevant to AV settings. The black-box ML planner uses a modified version of the DriveIRL architecture (Fig. 1b). For the trajectory generator G, we use a heuristic generator that produces 143 jerk-optimal trajectories to anchor waypoints along the route. For the scene encoder H, we use the hierarchical vector transformer (HiVT) pretrained for multi-agent motion prediction. Apart from the scene embedding h, this produces an additional 3 trajectories for the AV, for a total of k = 146 candidate trajectories. In the scene-trajectory encoder E, trajectories are encoded using a recurrent neural network and then fed jointly with the scene embedding into a transformer layer, which produces the scene-trajectory embeddings z. The reward model R is a multilayer perceptron (MLP). In CW-Net (Fig. 1c), the classifier C and the new reward model are MLPs. We avoided testing other methods apart from CW-Net because, at the time of writing, we are unaware of other works that are capable of modelling interpretable-by-design IRL systems. Moreover, we broadly seek a general comparison of concept-based explanations (that is, CW-Net) against no explanation (that is, our control) to help generalizability of the results across the myriad concept-based explainability techniques in the literature. CW-Net training We used two datasets: * Dataset 1: 500,000 scenarios and 8 concept labels (Supplementary Table 1) * Dataset 2: 3,000,000 scenarios and 10 concept labels (Supplementary Table 2) For a full list of the concept labels and their meanings, see section 'Concept details'. Each scenario was associated with 146 trajectories, thus giving between 73-438 million training data points for the concept classifier, each with multiple concept labels. Our algorithm assumes CW-Net training has access to the original dataset used to train the black-box ML planner, along with annotated human-understandable concept labels for each of these data points. The annotations can be multi-label, meaning that one datum can be associated with as many concepts as desired or useful. The experiments with CLOSE and ASV concepts used models trained on dataset 1. The experiments with BIKE and PEDESTRIAN concepts used models trained on dataset 2. During training, the parameters of the trajectory generator G, the scene encoder H and the scene-trajectory encoder E are frozen, and only the concept classifier C and the new reward model R' are trainable. Two separate losses are optimized jointly. First, a concept classification loss is used to train C to predict the correct concept label(s). In our setting, this loss combines cross-entropy with binary cross-entropy for different concepts, depending on the semantics of the corresponding scenario types. For example, in dataset 1, we use cross-entropy to model the steering concepts of the car (LEFT, RIGHT and STRAIGHT), and the speed concepts (STOP and SLOW), while also using binary cross-entropy to predict the presence of other concepts such as ASV, INTERSECTION and CLOSE. These losses are then averaged into one: where M is the number of concepts modelled using categorical cross-entropy (for example, steering and speed), and M is the number of concepts modelled using binary cross-entropy (for example, presence of features such as ASV, INTERSECTION and CLOSE). c and represent the true and predicted labels for the jth concept under CCE for the ith data point, whereas c and represent the true and predicted labels for the lth concept under BCE for the ith data point. On dataset 2, we take a different approach and model everything, including the speed concepts (STOP, SLOW and FAST), with binary cross-entropy. In general, these parameters can be tuned to fit the task at hand. Second, a cross-entropy loss is used to train the network to predict the correct trajectory, which we define as the original trajectory chosen by the black-box planner. Both losses are averaged: A focal loss is applied to counteract data imbalances, just as in the original DriveIRL planner. Computationally, our networks were trained on a large distributed setup using PyTorch Lightning. Concept separation When adding interpretability modules post hoc, as we have, there is the possibility that the network will not have learned to separate the concepts of interest, and thus fail to be able to predict them accurately. We observed this in the experimental prototype we tested (Supplementary Table 2), when certain concepts such as CLOSE and PEDESTRIAN had poor precision and high recall, relatively speaking. There are two important points to note here. First, the better trained and more sophisticated an architecture is, the more it naturally learns to separate an impressive number of concepts in an unsupervised manner, so this is unlikely to be an issue for most companies with the flagship models in the future. Second, even if CW-Net has not learned to separate certain concepts (for example, red traffic lights compared with green ones), this could highlight the reason why the AV fails to act appropriately -- such as stop compared with go -- in a given situation (for example, because there are insufficient traffic light scenarios in the training data). Hence, from an explainability perspective, concepts with low accuracy are often particularly useful, as we demonstrate in the paper. Alternative architecture Alongside our primary causal architecture shown in Fig. 1, we also developed an alternative that gave post hoc justifications for AV behaviour (Extended Data Fig. 1). Specifically, we froze the weights of the pretrained black-box ML planner and trained a concept classifier head C to work in parallel to the reward layer R. Similarly to the causal architecture (Fig. 1c), C used the scene-trajectory embeddings z to classify the concepts. This approach is beneficial because of its relative simplicity and accessibility, although the drawback is that it may be less faithful to the model's reasoning process, as the concept classifications are not directly used by the model to rank state-trajectory pairs. However, there is ample evidence that these explanations are often capable, so we include both as an option and demonstrate their utility. Specifically, this parallel architecture was used in the real-world scenarios with the CLOSE concept (section 'Unexpected stopping for nearby vehicles'). Concept details Dataset 1 concepts were as follows: * LEFT, RIGHT, STRAIGHT: Classification of driving direction concepts, trained with cross-entropy loss. For example, the concept LEFT represents training scenarios in which the car was turning left. * STOP, SLOW: Classification of car speed concepts, trained with cross-entropy loss. The concept of, for example, STOP represents training scenarios in which the car was stopped. * ASV (approaching stopped vehicle): Scenarios in which the car was approaching a stopped vehicle. Trained with binary cross-entropy. * INTERSECTION: Scenarios in which the car was at an intersection. Trained with binary cross-entropy. * CLOSE: Scenarios in which the car was within 3 m of another vehicle. Trained with binary cross-entropy. Dataset 2 had the following concepts: * SLOW: Scenarios in which the car was driving at 1-2 m s. * STOP: Scenarios in which the car was stationary. * FAST: Scenarios in which the car was driving faster than 2 m s. * STOP SIGN: Scenarios in which the car was close to a stop sign. * TRAFFIC LIGHT: Scenarios in which the car was close to a traffic light. * INTERSECTION: Scenarios in which the car was at an intersection. * PEDESTRIAN: Scenarios in which the car was close to a pedestrian. * FOLLOWING: Scenarios in which the car was following another vehicle. * BIKE: Scenarios in which the car was close to a cyclist. * PUDO (pedestrian pickup/drop-off): Scenarios in which the car was in a pedestrian pickup-drop-off zone. All concepts in dataset 2 were trained with a binary cross-entropy loss. Although the AV was trained on 8-10 concepts (depending on the dataset), we focus our study on a subset of concepts (CLOSE, ASV, BIKE and PEDESTRIAN) because they are the only ones that relate to other road users and are easy to test counterfactually. The concept FOLLOWING involves other road users too, but no notable occurrences happened during real-world deployment. Simulation results We tested our CW-Net model across the entire nuPlan validation dataset to see how its driving performance compares with the original black-box ML planner it was trained from. The dataset is a large-scale planning benchmark for autonomous driving and measures how close a trained AV is to a human expert in L distance, progress along the route and safety (no collisions). In the black-box model, when following the lane or decelerating from high speed, the planner was able to make progress along the route (>93% of human driving distance), while avoiding collisions (>90% collision-free) and staying close to the ground-truth human expert trajectory (<1 m displacement at 5 s). Performance was worse when starting from a stop, with less progress (74% of human driving distance), more collisions (81% collision-free) and greater deviation from the human expert (1.2 m displacement at 5 s). Overall, the results showed our variation of the AV architecture had less than 0.01 L difference to the original black-box agent on average across all measurements, and not meaningfully different, showing that it is possible to train our more interpretable model in Fig. 1 without sacrificing performance. The full results are in Extended Data Table 1. For concept accuracy verification, we used 5% holdout data from our training datasets; the results are given in Supplementary Tables 1 and 2. Across both datasets, the mean accuracy was 0.54, precision 0.23, recall 0.77 and F1 score 0.31. Overall, the results indicated that CW-Net did not separate all concepts equally well, which suits our purposes as the explanations will highlight when and how this happens, and how it relates to driving performance, thus helping with mental model refinement (see section 'Concept separation'). Notable results include an F1 score of 0.82 for detecting the SLOW concept, and close to zero for detecting the BIKE concept, showing the latter is perhaps not well encoded or understood by the car. Mental model elicitation studies These studies were designed to replicate the driver's experiences in the private track tests (section 'Mental model improvement in deployment') with a larger cohort, to help demonstrate the robustness of our findings regarding the drivers' mental models (Extended Data Fig. 2). The same study was run separately with experts (other drivers and test engineers from Motional) and non-experts (randomly sampled users from Prolific.com). Specifically, we were interested in testing the following hypotheses: * Hypothesis 1: The participants' responses before observing the explanation (in the video replay) would be more consistent with the belief of the safety driver before observing the explanation (in the car). * Hypothesis 2: The participants' responses after observing the explanation would be more consistent with the belief of the safety driver after observing the explanation. * Hypothesis 3: The participants' responses after observing the explanation would be more consistent with the ground-truth reason for AV behaviour. * Hypothesis 4: Counterfactual prediction ability would correlate with user mental model goodness, the latter as measured by the nearest-neighbour tasks and the free-form text response in ref. . * Hypothesis 5: The two measures of mental model goodness would correlate with each other and have similar distributions across groups. Design and materials We focused on the same three surprising events observed during the private track tests. The experiment was designed to measure mental models through a combination of nearest-neighbour and prediction tasks, accompanied by confidence scores and free-form text responses, in which we could probe the participants' mental models (Extended Data Fig. 2). First, users saw the respective video and were asked to rate two possible reasons for AV behaviour, along with confidence scores (that is, the nearest-neighbour mental model elicitation task). Then, users were asked to make the same counterfactual prediction as the driver (that is, the prediction task), along with a confidence score, and a free-form text rationale explaining their reasoning. Then, users saw the same video with the explanation and repeated the questions. This within-participant design mirrors the experience of the safety drivers during the on-road tests. Participants For the expert group, we recruited nine safety drivers, test engineers and test specialists from Motional (aged between 18 years and 80 years, 8 male and 1 female). All participants volunteered to participate and were not paid. For non-experts, we sampled users sourced from Prolific.com (www.prolific.com), which is known for its high-quality user base. Thirty users were randomly sampled US citizens aged between 18 years and 80 years, native English speakers and an even number of male and female. To ensure high-quality text responses, we paid users above the average rate with USD15 per hour and stressed that they should take the study only if they were certain they understood the instructions. The study received MIT IRB approval from the Committee on the Use of Humans as Experimental Subjects, exempt ID: E-5903, start date 1 July 2024, end date 31 August 2026. Metrics We used the nearest-neighbour task and the free-form text rationale as direct measures of mental models. For the nearest-neighbour task, we measured mental model goodness as a combination of choice accuracy and confidence. Specifically, we formalized mental model improvement as shifting from an incorrect to a correct belief (with respect to the ground-truth reasons for AV behaviour), or increasing confidence in the correct belief, or decreasing confidence in the incorrect belief (Extended Data Fig. 3). For the free-form text rationale, we used an LLM-as-a-judge (GPT-5) to determine which text response (before or after observing the CW-Net explanation) is closer to the ground-truth reason for AV behaviour (see Supplementary Methods for details). The LLM prompts were tuned over three iterations on the expert responses (Extended Data Fig. 4a) and then evaluated once on the non-expert responses (Extended Data Fig. 4b). Mental model improvement was formalized as instances where the free-form response after observing the CW-Net explanation is closer to the ground truth, compared with the free-form response before observing the explanation. We used the prediction task as a measure of downstream performance that relies on mental models, thus measuring them indirectly. We measured prediction improvement as a combination of choice accuracy and confidence in the same way as for the nearest-neighbour task (Extended Data Fig. 5). To analyse the relationship between direct (nearest neighbour, free-form text rationale) and indirect (prediction) measures of mental models, we used a linear mixed-effects model (LME) to evaluate the effect of the mental model change (improve or worsen) on the prediction change variable (that is, the delta in confidence change in the prediction), while controlling for individual participant variation as a random effect (Extended Data Fig. 5). To collapse accuracy and confidence on a single scale for computing the confidence deltas, we simply flipped the sign of confidence values for inaccurate beliefs to negative. All code for this analysis will be available on publication (see section 'Code availability' below). Study conclusions We found evidence favouring all of our hypotheses. * Hypothesis 1: The participants' initial belief (before observing the explanation) was often more similar to the safety driver's initial belief (P < 10, exact binomial test). * Hypothesis 2: The participants' final belief (after observing the explanation) was often times more similar to the safety driver's final belief (P < 0.0002, exact binomial test). * Hypothesis 3: The participants' final belief (after observing the explanation) was often times more similar to the ground truth (P < 10, exact binomial test). * Hypothesis 4: There was a clear relationship between mental model category and predictive ability of users, with only free-form text response categories failing to reach significance with experts (Extended Data Fig. 5). * Hypothesis 5: Interaction analysis showed no significant difference in the relationship between nearest-neighbour score improvement and text rationale improvement across the two groups (P = 0.842; Extended Data Fig. 4c), indicating a consistent underlying mechanism for both experts and non-experts. Non-experts demonstrated a significant positive correlation (β = 0.27, P < 0.001), suggesting that improved scores were strong predictors of improved mental model rationale in this group. Although experts exhibited a nearly identical positive coefficient (β = 0.23), the relationship did not reach statistical significance, probably because of a smaller sample size (P = 0.243). Public roads evaluation using SAGAT We deployed CW-Net in manual mode on Las Vegas public roads to collect complex, naturalistic scenarios analogous to those already discovered during the private track tests (Extended Data Fig. 6). We used these as materials for an online SAGAT study (n = 100). This setup allows us to validate the robustness of the CW-Net explanations using a well-established rigorous framework (Extended Data Fig. 2). We used a between-participant design in which we compared CW-Net explanations (experimental group) against baseline explanations describing speed and steering (control group) (Extended Data Fig. 7). Gathering materials We collected data for ASV (01:02:55), CLOSE (00:50:42), and BIKE ( > 3 hours). We labelled sequences where human driving mimicked the ML planner in surprising scenarios that resembled those already discovered in the private track tests (for example, CLOSE: stuck beside vehicles; ASV: braking for hallucinations; Extended Data Fig. 6). PEDESTRIAN replaced BIKE because there were no naturally occurring cyclists in the new video data. We sampled two surprising events per concept (6 total) and six corresponding 'unsurprising' events to ensure explanations did not degrade situational awareness in regular driving (Supplementary Table 5). Activation thresholds were 0.5 for ASV/PEDESTRIAN and 0.94 for CLOSE, derived from prior private track data (Fig. 3). Note the distribution of concept activations did not change significantly between the private-track tests and the public-road tests (Supplementary Table 7), despite the tests being performed more than a year apart, on different AVs, with different software stacks and under completely different conditions. This demonstrates the robustness and reliability of the algorithm. Study design We used a between-participants design (n = 100) assessing SAGAT perception, comprehension, and projection. Participants were split into experimental (CW-Net explanations) and control (speed/steering placeholder) groups. Attention checks based on material content and viewing times reduced the pool to 99 participants, 51 in the experimental and 48 in the control. Materials Stimuli included 13 videos (6 surprising, 6 unsurprising and 1 attention check). The experimental group saw concept activations, whereas the control saw speed and steering data (Extended Data Fig. 8). Following each video, a 'blackout' screen presented six binary questions: four related to perception, one to comprehension and one to projection. Participants We recruited gender-balanced US residents (18+ years, native English speakers) by Prolific.com. Participants were paid $12 per hour. The study received MIT IRB approval from Committee on the Use of Humans as Experimental Subjects, exempt ID: E-5903, start date 1 July 2024, end date 31 August 2026. Metrics We analysed the average of each participant on each question type using two-tailed t-tests, splitting data by surprising and unsurprising events for each situational awareness dimension. Study conclusions After collection and attention check filtering, we collected 99 responses out of the target 100. The results are shown in Extended Data Fig. 7. In surprising events, explanations significantly improved situational awareness after Bonferroni correction, showing large effect sizes for perception (Cohen's d = 1.290) and comprehension (d = 0.996), alongside a medium effect for projection (d = 0.606). In unsurprising events, no significant differences occurred (perception d = 0.085, projection d = -0.142, comprehension d = -0.514), confirming that explanations provided benefit in anomalous situations without adversely affecting situational awareness during routine operations. Furthermore, we confirmed feature robustness by comparing concept distributions to previous private track tests using Wasserstein distance (Supplementary Table 7), finding no meaningful changes. Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

System helps humans predict when self-driving cars will make mistakes

The CW-Net technique explains the behavior of an autonomous vehicle using understandable concepts. "Instead of just wondering why the car stopped, having real-time data provides feedback that lets you test the system during deployment," says Eoin Kenny. Self-driving cars are often controlled by deep learning models that sometimes fail in unexpected situations. For instance, the car might inexplicably brake and block the path of an oncoming emergency vehicle. A human driver or passenger may need to react rapidly to prevent a collision. To help humans better anticipate a vehicle's mistakes, researchers from MIT and autonomous vehicle technology company Motional developed a new method that provides clear explanations of the underlying model's decisions. Usually, the internal reasoning process of a deep learning model is opaque and difficult to understand. But the new method, called the Concept-Wrapper Network (CW-Net), translates that reasoning process into concepts that faithfully describe the autonomous vehicle's decisions without altering its driving performance. CW-Net explains the decisions of machine learning-based planners using understandable concepts, like "approaching stopped vehicle" or "close to cyclist." These explanations can correct misconceptions drivers and passengers have about vehicle behavior and improve their situational awareness. In road tests on a private track, CW-Net explanations helped safety drivers more accurately predict vehicle behavior; a larger simulation study with nonexpert users yielded similar results. These experiments show how CW-Net can provide important feedback for engineers as they troubleshoot in-vehicle artificial intelligence systems. In the longer term, this technique could boost the safety and transparency of autonomous vehicles, while building appropriate trust in drivers and passengers. "This work shows how explanations are supportive to the human's mental model and understanding of the behavior of a system, and how it could be used in engineering and development to improve the technology," says Julie Shah, an MIT professor of aeronautics and astronautics, director of the Interactive Robotics Group in the Computer Science and Artificial Intelligence Laboratory (CSAIL), and co-senior author of the paper on CW-Net. "Unless we are building these technologies in a way that we can rely on and predict their behavior, then it is a shaky and unsafe foundation for their use." She is joined on the paper by lead author Eoin Kenny, a former MIT postdoc who is now a senior AI researcher at J.P. Morgan Chase; co-senior author Momchil Tomov, a staff research scientist at Motional; as well as Motional team members Akshay Dharmavaram, Sang Uk Lee, Tung Phan-Minh, Shreyas Rajesh, Yunqing Hu, and Laura Major, president and CEO of Motional. The research appears today in Nature. Faithful explanations Machine-learning-based planners act as the "brain" of a self-driving car. These powerful deep-learning architectures process data from the vehicle's cameras and lidar sensors, generate a high-level summary of the vehicle's environment, decide what the car should do next, and output a trajectory for it to follow. The planners are usually black-box models, which means their internal decision-making process is so complex it is difficult to understand. This can leave scientists and safety drivers in the dark about why an autonomous vehicle made an unexpected decision, like phantom braking. The researchers designed CW-Net to explain a vehicle's decisions using understandable concepts, while ensuring those explanations accurately reflect the true reasons behind its behavior. "Especially in high-stakes settings like self-driving cars, it's important that the explanations are not potentially misleading. Because CW-Net is causally faithful in how it makes decisions, that provides certain guarantees around the explanations," Kenny says. CW-Net is a "concept classifier," an AI algorithm that has been trained to predict the high-level concepts that exist within input data. The researchers plug the CW-Net module into the middle of an autonomous vehicle's existing machine-learning planner architecture. It translates the model's internal reasoning process into understandable concepts, like "approaching stopped vehicle" or "close to cyclist." Then it forces the final piece of the planning model architecture to use those concepts when it decides what the vehicle should do next. In this way, CW-Net ensures the concepts faithfully explain the vehicle's actions. At the same time, CW-Net uses the concepts it classified to generate clear explanations that are output along with the vehicle trajectory, in real-time. "Instead of just wondering why the car stopped, having real-time data provides feedback that lets you test the system during deployment. You could also give that data to an engineer to potentially improve the system," Kenny says. The researchers trained CW-Net to predict concepts using a dataset of 130 million examples of scenes from self-driving cars, with multiple labeled concepts in each scene. Using such a large, labeled dataset enables it to identify concepts accurately in a wide range of settings. They also designed CW-Net to mimic the driving decisions of machine-learning-based planners, so the module would not negatively impact vehicle performance. In the end, CW-Net generates accurate, understandable explanations without altering the original deep learning model. Improving situational awareness To test CW-Net, the researchers deployed the module on a real autonomous driving test vehicle (a Motional robotaxi) on a private track with a safety driver. They found that CW-Net helped the safety driver better predict how the vehicle would behave in surprising situations. For instance, the vehicle consistently stopped when it approached a cyclist, and the safety driver assumed it did so because it detected that cyclist. But CW-Net explanations revealed that the model wasn't properly configured to detect the cyclist and chose a trajectory that would have caused a collision. Instead, it stopped because its emergency braking procedure kicked in when it got too close. Armed with this information about the model's mistake, the safety driver could reduce speed or engage manual driving mode sooner in similar situations. This could also help engineers fix the model to avoid this failure in the future. In larger online simulation studies using real driving situations captured on the roads of Las Vegas, the researchers saw similar results. CW-Net explanations significantly improved participants' abilities to predict how an autonomous vehicle will behave. In the future, the researchers could extend CW-Net so the module can cover more concepts and explore different training and design techniques that could boost performance and improve interpretability. "Our study shows how crucial interpretability can be to these high-stakes environments, and how it should be on the mind of people as they are making AI in the future, for self-driving cars or other safety-critical environments," Kenny says.