Intermodal Plasticity in Artificial Synapses via Mechano-Optical Gating for Enhanced Visual Recognition

Traditional artificial intelligence (AI) faces significant challenges in achieving efficient and versatile multisensory integration applications. Inspired by multisensory integration in biological systems, intermodal perception platforms have emerged to enhance efficiency and broaden applicability. Here, we present a novel intermodal perception system based on an array of artificial mechano-optical synaptic devices. The mechanoluminescent (ML) material ZnS: Cu enables direct conversion of mechanical stimuli into optical signals, thereby facilitating efficient modulation of the optoelectronic synaptic devices array. The integration of a PDMS-ZnS: Cu mechanoluminescent layer with the artificial optoelectronic synaptic devices array enables hardware implementation of individual and cooperative intermodal plasticity in response to mechanical and optical stimuli. Furthermore, we successfully demonstrate a dynamic “learning-forgetting-relearning-forgetting” associative memory process for letter images. We investigate intermodally fused synaptic plasticity driven by spatiotemporally varied mechanical-optical pulse synergies. Additionally, an intermodal artificial neural network (ANN) implemented on this platform achieves an accuracy of 92.7% in handwritten digit recognition tasks. Unlike conventional systems requiring separate sensors and signal converters, our integrated device provides a pivotal hardware-integrated technological foundation for developing adaptive intermodal intelligent perception systems capable of operating in complex environments.