MERCI/PERCY: Real-Robot Multimodal Affective Dialogue Dataset and Benchmark

PERCY is a real-time conversational system deployed on the ARI social robot, and MERCI is the multimodal dataset collected from live human-robot interactions using this system. The project investigates how visual and textual affect signals behave in real embodied dialogue and supports affect-aware multimedia indexing and benchmark evaluation.

Date: 2024 - 2026

Persons participating in the project:

• PIs: Dr. Francisco Cruz, Dr. Imran Razzak
• Associates: Zhijin Meng, Mohammed Althubyani
• Corresponding contact: zhijin.meng@unsw.edu.au

Research areas:

• Human-Robot Interaction
• Affective Computing
• Multimodal Dialogue
• Emotion Recognition
• Cross-Modal Consistency
• Social Robotics
• Multimedia Indexing
• Conversational AI
• Dataset and Benchmark

Description:
PERCY provides the deployed robot dialogue stack on ARI (ASR/TTS, turn management, multimodal affect logging). MERCI is the resulting corpus from real sessions (30 participants; ~12.5 hours aligned audio-video; transcripts; FER and sentiment annotations; persona metadata). The MTAP extension adds turn-level cross-modal affect consistency analysis and benchmark protocols.

Useful Links:

• MERCI dataset
• PERCY codebase
• MERCI analysis repo (forthcoming)

Media:
Additional images/video

Selected Publications	Web
Meng, Z., Althubyani, M., Xie, S., Razzak, I., Sandoval, E. B., Bamdad, M., & Cruz, F. (2025, November). PERCY: Personal emotional robotic conversational system. In Australasian Joint Conference on Artificial Intelligence, (pp. 466-478). Singapore: Springer Nature Singapore.
Althubyani, M., Meng, Z., Xie, S., Cruz, F., Razzak, I., Prasad, M., ... & Kocaballi, B. (2025, October). MERCI: A Multimodal Dataset for Personalised and Emotionally-Aware Dialogues. In International Conference on Content-Based Multimedia Indexing (CBMI), (pp. 1-7). IEEE.