Real-Time Detection of Robot Failures by Monitoring Operator’s Brain Activity with EEG-based Brain-Computer Interface

Abstract

The rapidly growing field of Brain-Computer Interfaces (BCIs) provides an innovative method for human-robot interaction, enabling machines to be controlled directly via brain signals. This study investigates the use of BCI technology to monitor the operator’s brain activity in real-time and detect robot malfunctions through the identification of Error-Related Potentials (ErrPs). In this thesis, a real-time system was developed to monitor the brain activity of robot operators continuously using an OpenBCI consumer-grade EEG device. Machine learning algorithms were implemented to analyze brain activity, specifically targeting ErrPs, which indicate that the operator is noticing something unexpected. By recognizing these brain signals, our system can identify potential robot malfunctions based on the operator’s cognitive response and trigger an immediate halt to the operation. This system integrates several key components: signal processing techniques such as resampling, filtering, and normalization to prepare the EEG data for analysis; machine learning classifiers to identify ErrPs associated with robot malfunctions; a robotic simulation that generates realistic scenarios to elicit ErrPs in participants’ brain activity for safe system testing; and real-time brain signal acquisition allowing immediate detection of ErrPs and response to faults. All these components are packed into one coherent system that detects robot malfunctions and triggers intervention. The final system underwent live testing with 10 participants, demonstrating its capability to detect ErrPs associated with robot malfunctions effectively. These tests showed that the system achieves an average sensitivity of 0.53 while maintaining a specificity of 0.98, suggesting that the system rarely reacts without a valid reason while being able to correctly detect more than half of the events of interest. For some users, sensitivity reached as high as 0.7 or more. These findings demonstrate the potential of consumer-grade EEG devices for practical applications of BCI-based robot fault detection.