Beyond Text: Enhancing human-computer interaction through multimodal embodied conversational agents

• Autores: Novelli, Emiliano; Alvarado, Yoselie; Guerrero, Roberto A.
• Año de publicación: 2025
• Idioma: inglés
• Tipo de recurso: documento de conferencia
• Materia: Ciencias Informáticas; Embodied conversational agents; Visemes; Mobile applications; Multimodal interaction; Facial animation; Affective computing; Natural user interfaces
• Institución: Universidad Nacional de La Plata

This paper presents the development of a conversational agent that enriches human-computer interaction beyond traditional textbased interfaces by adopting a multimodal approach integrating speech, facial gesture dynamics, and emotional expressions on mobile platforms. The agent features real-time speech synthesis synchronized with visemes and is capable of displaying basic emotional states through animated facial expressions. Inspired by the principles of Embodied Conversational Agents and Natural User Interfaces, the application leverages vector graphics, animation engines, and several technologies for multiplatform support. The design emphasizes natural interaction, emotional perception, and usability, while seeking to circumvent the Uncanny Valley phenomenon by investigating varied strategies of visual representation. Evaluation results demonstrate that the system performs well in terms of small-screen interface legibility, computational performance, and user affective experience. This work contributes to the field of multimodal interfaces by demonstrating the feasibility and advantages of incorporating emotional and gestural cues into mobile conversational systems.
Red de Universidades con Carreras en Informática