Causal versus Marginal Shapley Values for Robotic Lever Manipulation Controlled using Deep Reinforcement Learning

Abstract

We investigate the effect of including application knowledge about a robotic system states’ causal relations when generating explanations of deep neural network policies. To this end, we compare two methods from explainable artificial intelligence, KernelSHAP, and causal SHAP, on a deep neural network trained using deep reinforcement learning on the task of controlling a lever using a robotic manipulator. A primary disadvantage of KernelSHAP is that its explanations represent only the features’ direct effects on a model’s output, not considering the indirect effects a feature can have on the output by affecting other features. Causal SHAP uses a partial causal ordering to alter KernelSHAP’s sampling procedure to incorporate these indirect effects. This partial causal ordering defines the causal relations between the features, and we specify this using application knowledge about the lever control task. We show that enabling an explanation method to account for indirect effects and incorporating some application knowledge can lead to explanations that better agree with human intuition. This is especially favorable for a real-world robotics task, where there is considerable causality at play, and in addition, the required application knowledge is often handily available.