A Cross-Modal Integrated Sensor Fusion System for Fatigue and Awareness Assessment in Demanding Marine Operations
Abstract
The maritime domain is characterized by demanding operations. These operations can be especially complex and dangerous when they require coordination between different maritime vessels and several maritime operators. Although the use of modern technology and automation have greatly increased in the maritime domain in recent decades, human operators are still an indispensable part of maritime operations and, therefore, have a direct impact in the quality and safety of such complex operations. This situation is not likely to change in the near future; the operational loop will likely include humans for a long time to come. Although human operators will increasingly work in onshore control centers instead of on vessels, their impact will remain significant.
Performance failure in a critical moment can lead to disastrous results, including near misses, economic and environmental loses and fatalities. Several human factors can lead to poor performance, including failing completely or partially to follow procedure, lack of situational awareness, and physical or mental fatigue (MF). Among these issues, MF is responsible for reducing cognitive capabilities, situational awareness, and decision-making skills.
The maritime industry has extensive regulations regarding control and management of fatigue in maritime operators. Although these regulations offer valuable information regarding best practices to reduce and mitigate the effects of fatigue, they failed in providing objective way to measure the development of fatigue in maritime operators. Early detection and assessment of MF can be used to reduce human error, to the benefit of crew members, ship owners, and the maritime environment. However, objective assessments of MF in real-time in maritime operations are difficult. Subjective methods such as questionnaires and surveys have limited value as they are usually biased and do not provide real-time tools to approach the problem. This context calls for more objective methods. It likewise calls for real-time data, as it is the only way to prevent accidents. Assessing maritime operators' MF levels before accidents happen is crucial. Monitor the decrementing of operator performance in demanding maritime situations is the best way to increase safety.
In this context, the main goal of this thesis is to investigate how MF can be objectively measured during demanding maritime operations. The best approach to quantify MF is through the use of physiological sensors. Different sensors such as electroencephalogram (EEG), electrocardiogram, electromyogram, temperature sensor, and eye tracker can be applied, individually or in conjunction, in order to collect relevant data that can be mapped to an MF scale. In the work presented in this thesis, realistic vessel simulators were used as a platform for experimenting with different operational scenarios and sensor setups.
This thesis bridge the gap between relevant sensor data and a quantifiable MF level using both data-driven and model-based approaches. Data-driven methods are in fast development, fuelled by the popularization of neural networks (NNs). This work investigates the use of different NNs combined for the MF assessment (MFA) task. Among the different architectures tested, Convolutional Neural Networks (CNN) showed the best performance when dealing with multiple physiological data channels. Optimization was used to improve the performance of CNN in the cross-subject MFA task. Testing different combinations of physiological sensors indicated a setup consisting of EEG sensor only was the best option, due to the trade-off between assessment precision and sensor framework complexity. These two factors are of great importance when considering an MFA system that could be implemented in real-life scenarios.
A model-based approach was also investigated in order to implement an MFA algorithm capable of dealing with not only cross-subject analysis but also with cross-task scenarios. This is another strong requirement for an MFA system to be feasible for real-life applications. The thesis is concluded with a theoretical discussion about the required building blocks for an MF prediction system, which would be a natural next step regarding the use of MF information to increase the safety of maritime operations.