A Parallel Multimodal Integration Framework and Application for Cake Shopping

Hui Fang, Dongdong Weng*, Zeyu Tian

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Featured Application: This paper introduces a flexible parallel multimodal integration framework that outlines a development paradigm and encourages multimodal application across various scenarios. Multimodal interaction systems can provide users with natural and compelling interactive experiences. Despite the availability of various sensing devices, only some commercial multimodal applications are available. One reason may be the need for a more efficient framework for fusing heterogeneous data and addressing resource pressure. This paper presents a parallel multimodal integration framework that ensures that the errors and external damages of integrated devices remain uncorrelated. The proposed relative weighted fusion method and modality delay strategy process the heterogeneous data at the decision level. The parallel modality operation flow allows each device to operate across multiple terminals, reducing resource demands on a single computer. The universal fusion methods and independent devices further remove constraints on the integrated modality number, providing the framework with extensibility. Based on the framework, we develop a multimodal virtual shopping system, integrating five input modalities and three output modalities. The objective experiments show that the system can accurately fuse heterogeneous data and understand interaction intent. User studies indicate the immersive and entertaining of multimodal shopping. Our framework proposes a development paradigm for multimodal systems, fostering multimodal applications across various domains.

Original languageEnglish
Article number299
JournalApplied Sciences (Switzerland)
Volume14
Issue number1
DOIs
Publication statusPublished - Jan 2024

Keywords

  • multimodal integration
  • multimodal interaction
  • virtual agents
  • virtual shopping
  • wearable devices

Cite this