Enhancing driver experience in SAE level 3 automated vehicles through multimodal and emotion-aware in-vehicle agents

Zeng, X., Alam, M. S., Bazilinskyy, P.

Submitted for publication.

ABSTRACT In-vehicle agents (IVAs) are emerging as transformative innovations in intelligent transportation systems, particularly in automated driving contexts. This paper integrates two complementary studies on the development and evaluation of robot-like IVAs equipped with multimodal interaction and emotional feedback for SAE Level 3 automated vehicles. The first study introduced a robot-like IVA capable of communicating through gestures and facial expressions. An experiment with 12 participants showed that both modalities reduced workload (NASA TLX mean scores: baseline = 33%, facial expressions = 23%, gestures = 18%) and enhanced perceived usefulness and satisfaction. Seven participants preferred gestures for practicality and anticipatory cues, while five preferred facial expressions for their emotional and aesthetic qualities. The second study (N=12) developed a prototype with emotional feedback via facial emotion detection. The results revealed that emotional feedback and working status did not significantly affect overall workload or acceptance, although feedback influenced physical and temporal demands, and its interaction with working status significantly affected workload. Voice communication remained the primary way of interaction, while challenges included the accuracy of emotion detection and accounting for physical conditions such as fatigue and stress.