Evaluating Artificial Neural Network Robustness for Safety-Critical Systems

Abstract

With the power to perform more complex tasks than humans, artificial neural networks (ANNs) have been applied to execute tasks in safety-critical systems (SCSs), such as object detection, image recognition, and navigation. An ANN should provide consistent performance when input deviates from the training data. This corresponds to the attribute of robustness in the ANN.

The obstacles to developing robust ANN-based safety-critical systems (ANN-SCSs) encompass four interrelated aspects: 1) the inherent complexity and nonlinearity of ANNs that call for innovative testing and verification (T&V) techniques; 2) the need to establish a well-defined connection between robustness and safety by considering various factors; 3) the vital nature of addressing the immaturity of robustness evaluation and measurement to ensure the seamless integration of ANNs in safety-critical applications in operation; and 4) the development of precise and practical robustness measurement in operation without labeled data. It is vital to have methods to accommodate the ever-changing nature of real-world data and the diversity of ANN architectures and use cases. Consequently, addressing these four challenges holistically is essential to facilitate a safe and reliable transition toward incorporating ANNs in SCSs.

This thesis provides knowledge on ANN robustness evaluation in the context of SCSs. It develops new knowledge, methods, and guidance, combining traditional risk analysis concepts with convolutional neural network theory and robustness studies. Four main research papers have been published and submitted as a result of the work in this thesis. These papers together provide scientific contributions to 1) the systematization of knowledge and understanding for T&V of ANN-SCSs; 2) a new method for analyzing the influence of ANN robustness on the safety of autonomous vehicles; 3) a systematic summary of methods and metrics to measure ANN-SCS robustness in operation; and 4) empirical results that demonstrate the applicability of distance metrics in selecting more robust ANN models from several alternatives using unlabeled data in operation. The systematization of knowledge, the method to evaluate ANN robustness, and insights on the advantages and disadvantages of the corresponding metrics pave the way for a future where the robustness and safety of ANN-SCSs can be quantified and enhanced, ensuring improved operational safety and effectiveness in real-world scenarios.