Classification of Failures in the Perception of Conversational Agents (CAs) and Their Implications on Patient Safety
Peer reviewed, Journal article
Published version
Date
2021Metadata
Show full item recordCollections
- Institutt for IKT og realfag [555]
- Publikasjoner fra CRIStin - NTNU [37247]
Original version
Studies in Health Technology and Informatics. 2021, 281 659-663. 10.3233/SHTI210253Abstract
The use of Conversational agents (CAs) in healthcare is an emerging field. These CAs seem to be effective in accomplishing administrative tasks, e.g. providing locations of care facilities and scheduling appointments. Modern CAs use machine learning (ML) to recognize, understand and generate a response. Given the criticality of many healthcare settings, ML and other component errors may result in CA failures and may cause adverse effects on patients. Therefore, in-depth assurance is required before the deployment of ML in critical clinical applications, e.g. management of medication dose or medical diagnosis. CA safety issues could arise due to diverse causes, e.g. related to user interactions, environmental factors and ML errors. In this paper, we classify failures of perception (recognition and understanding) of CAs and their sources. We also present a case study of a CA used for calculating insulin dose for gestational diabetes mellitus (GDM) patients. We then correlate identified perception failures of CAs to potential scenarios that might compromise patient safety.