The effects of crossmodal semantic reliability for audiovisual immersion experience of virtual reality

Previous studies have reported that immersion experience can be improved by pairing it with auditory-visual stimuli; however, whether the semantic relationship of auditory-visual stimuli can also modulate visual virtual experience remains unclear. By using the psychophysics method, this study investigates the category performance under three different crossmodal semantic reliability conditions: semantically reliable, semantically unreliable and semantically uncertain. The results revealed a faster category speed for the crossmodal semantically reliable condition regardless of the category, indicating that crossmodal semantic reliability led to sufficient multisensory integration. In particular, under crossmodal semantically unreliable conditions, category speed was faster for non-living stimuli, indicating robust representation for non-living objects. These results indicate that adopting semantically reliable visual and auditory stimuli as multisensory inputs can efficiently improve the multisensory immersion experience.