A comprehensive survey on deep facial expression recognition: challenges, applications, and future guidelines

Abstract

Facial expression recognition (FER) is an emerging and multifaceted research topic. Applications of FER in healthcare, security, safe driving, and so forth have contributed to the credibility of these methods and their adoption in human-computer interaction for intelligent outcomes. Computational FER mimics human facial expression coding skills and conveys important cues that complement speech to assist listeners. Similarly, FER methods based on deep learning and artificial intelligence (AI) techniques have been developed with edge modules to ensure efficiency and real-time processing. To this end, numerous studies have explored different aspects of FER. Surveys of FER have focused on the literature on hand-crafted techniques, with a focus on general methods for local servers but largely neglecting edge vision-inspired deep learning and AI-based FER technologies. To consider these missing aspects, in this study, the existing literature on FER is thoroughly analyzed and surveyed, and the working flow of FER methods, their integral and intermediate steps, and pattern structures are highlighted. Further, the limitations in existing FER surveys are discussed. Next, FER datasets are investigated in depth, and the associated challenges and problems are discussed. In contrast to existing surveys, FER methods are considered for edge vision (on e.g., smartphone or Raspberry Pi, devices, etc.), and different measures to evaluate the performance of FER methods are comprehensively discussed. Finally, recommendations and some avenues for future research are suggested to facilitate further development and implementation of FER technologies.