Augmenting monitoring performance during multi-UAV supervisory control with adaptive displays

Unmanned aerial systems have become a main sector to explore and further push the boundaries of automation. In contrast to conventional, manned aircraft, unmanned aerial vehicles (UAV) are controlled by human operators from dislocated control stations. Possible scenarios for using UAV instead of manned aircraft are of both, civil and military nature, including surveillance, reconnaissance, agricultural imaging, border patrol, construction inspection, and firefighting. The main advantage is the reduction of risk for human personnel, which results from operating in hazardous terrain or conflict areas. Today, a single aircraft-size UAV typically requires control through several human operators via dedicated human-machine interfaces. However, research on future architectures targets on inverting the human-UAV ratio, allowing one human operator to control many UAV simultaneously. Because human cognitive and physical resources are limited by nature, the key technology allowing one human operator to safely control many UAV is automation. The human operator mainly plays the role of a supervisor. As a supervisor, the human operator is the “ultima ratio” in abnormal situations that cannot be handled by the automation. The underlying human-machine interaction paradigm is commonly known as human supervisory control. Within this paradigm building and maintaining sufficient situation awareness is very important. However, human factors research revealed that insufficient situation awareness is the main cause of human error in context of human supervisory control. The key enabler of good situation awareness is adequate monitoring behavior. Therefore, it is a safety-critical requirement that displays are designed in a way that supports human operators to selectively perceive the information they need. However, human factors research again showed that monitoring behavior is very often inadequate, e. g., because of distraction, perseveration or fatigue. In this work, we present an assistant system to improve the monitoring behavior of a human operator in charge of supervisory control of multiple highlyautomated unmanned aerial vehicles. The underlying concept is based on the real-time assessment and augmentation of monitoring performance. The realtime assessment of monitoring performance is based on an eye tracking-based tool that allows to assess the demand for attention of information conveyed by a user interface for supervisory control of multiple highly-automated UAV. The real-time augmentation of monitoring performance is based on two strategies to adapt the characteristics of the user interface used by a human operator. The first strategy invokes visual cues on a situation awareness display to support visual search for information demanding attention. The second strategy invokes visual cues on a peripheral display to support the human operator’s awareness for monitoring behavior adequacy. Our results indicate that assisting the monitoring task using visual cue invocation improves monitoring performance and situation awareness.