Multisensory Interconnected Networks
Energy efficiency, parallel information processing, and unsupervised learning make the human brain a model computing system for unstructured data handling. Different types of programmable resistors (memristors) can act as artificial synapse or neuron. The integration of these elements into an artificial neural network allows for the emulation of essential brain functions. In our previous work, we focused on artificial synapses made of ferroelectric tunnel junctions. Using both inorganic [1,2] and organic [3-5] ferroelectric tunnel junctions we have demonstrated synaptic behavior such as programmable synaptic weight, long and short-term potentiation and depression, paired-pulse facilitation/depression, and Hebbian and anti-Hebbian learning through spike shape and timing-dependent plasticity.
More recently, we worked on the development of multisensory interconnected networks for smart sensing applications with in-sensor processing and memory. Our systems use multiple sensors for the detection of vision, hearing, touch, smell, and taste information. The output voltages of the sensors are encoded into optical spikes using ring oscillators and edge detectors coupled to LEDs. The conversion to optical spikes avoids voltage degradation and parasitic resistance issues in sensory data communication, and allows accurate encoding with various spike coding principles, including rate coding, temporal coding, or a combination of both. Spike coding is more robust than voltage amplitude coding and it can carry larger data volumes and distinguish multiple inputs with a single detector. In our multisensory interconnected networks, photomemristor arrays integrate the optical spikes and decode the sensory information. Each photomemristor works as an artificial optoelectronic synapse that receives signals from a sensory nerve and produces a post-synaptic current at the optical spiking rate. Sensory inputs change the spiking rate and post-synaptic current of the photomemristors at run-time through a persistent photoconductivity effect, providing built-in memory of sensory information.
As a first smart sensor, we demonstrated an optoelectronic spiking afferent nerve with neural coding, perceptual learning and memorizing capabilities to mimic tactile sensing and processing [7]. The sensory system uses sensors made of 2D material MXene to detect hand-written pressure information. The pressure inputs are converted to light pulses for subsequent processing by a photomemristor array. Importantly, each photomemristor in the array can process multiple sensory inputs via optical spikes, emulating the integration of multiple action potentials from various axon terminals of pre-neurons to dendrites of post-neurons via synapses. With neural coding, our bio-inspired spiking afferent nerve does not only detect simultaneous pressure inputs, but also recognizes Morse code, braille, and object movement. Furthermore, with dimensionality-reduced feature extraction and learning, it recognizes and memorizes handwritten alphabets and words, providing a promising approach towards e-skin, neurorobotics and human-machine interaction technologies.
As the next breakthrough, we realized the first bioinspired multisensory interconnected network that integrates the five primary senses (vision, touch, hearing, smell, taste) through crossmodal learning. In this smart sensing system, the physical stimuli are sensed and converted to potential changes through various detectors, the potential changes are encoded into optical spikes for communication using spike encoders, and the multimodal environmental information is decoded, filtered, and memorized by a photomemristor array. Finally, the crossmodal signals are integrated by an artificial neural network using associative learning. The hierarchical and cognitive multisensory interconnected network is capable of not only sensing, encoding, transmitting, decoding, filtering, memorizing, and recognizing multimodal information, but it also allows for crossmodal recognition and imagination through crossmodal learning for robotic sensing and processing. As examples, we demonstrated the visualization of alphabet letters upon handwritten input, recognition of multimodal visual/smell/taste information and the imagination of never-seen pictures when hearing their description. Our multisensory interconnected network provides a promising approach towards robotic sensing and perception.
We are currently working on the realization of dynamic machine vision using the build-in memory of photomemristors for motion recognition and prediction. As another ambitious goal, we are developing new multisensory interconnected networks to demonstrate three forms of robotic sensory-thinking: multisensory language understanding for human-machine communication, multisensory abstract thinking for machine numerosity and arithmetic perceptions, and multisensory inspirational thinking for machine imagination and creativity. The realization of these advanced sensory-thinking capabilities will greatly enhance the quality of human-machine interactions, offering smarter solutions for well-being (braille recognition, robotic prosthetics, health monitoring, personal assistance robot), the internet of things, lifelong learning systems, and autonomous environmental-friendly transportation.
Publications
1. Q. Qin, L. Äkäslompolo, N. Tuomisto, L. Yao, S. Majumdar, J. Vijayakumar, A. Casiraghi, S. Inkinen, B. Chen, A. Zugarramurdi, M. Puska, S. van Dijken. Resistive switching in all-oxide ferroelectric tunnel junctions with ionic interfaces. Advanced Materials 28, 6852 (2016).
2. N. Tuomisto, S. van Dijken, M. Puska. Tsu-Esaki modeling of tunneling currents in ferroelectric tunnel junctions. Journal of Applied Physics 122, 234301 (2017).
3. S. Majumdar, B. Chen, Q. Qin, H.S. Majumdar, S. van Dijken. Electrode dependence of tunneling electroresistance and switching stability in organic ferroelectric P(VDF-TrFE)-based tunnel junctions. Advanced Functional Materials 28, 1703273 (2018).
4. S. Majumdar, H.W. Tan, Q. Qin, S. van Dijken. Energy-efficient organic ferroelectric tunnel junction memristors for neuromorphic computing. Advanced Electronic Materials 5, 1800795 (2019).
5. S. Majumdar, H. Tan, I. Pande, S. van Dijken. Crossover from synaptic to neuronal functionalities through carrier concentration control in Nb-doped SrTiO3-based organic ferroelectric tunnel junctions. APL Materials 1, 091114 (2019).
6. H. Tan, S. Majumdar, Q. Qin, J. Lahtinen, S. van Dijken. Mimicking neurotransmitter release and long-term plasticity by oxygen vacancy migration in a tunnel junction memristor. Advanced Intelligent Systems 1, 1900036 (2019).
7. H. Tan, Q. Tao, I. Pande, S. Majumdar, F. Liu, Y. Zhou, P.O.A. Persson, J. Rosen, S. van Dijken. Tactile sensory coding and learning with bio-inspired optoelectronic spiking afferent nerves. Nature Communications 11, 1369 (2020).
8. H. Tan, Y. Zhou, Q. Tao, J. Rosen, S. van Dijken. Bioinspired multisensory neural network with crossmodal integration and recognition. Nature Communications 12, 1120 (2021).
9. H. Tan and S. van Dijken. Dynamic machine vision with retinomorphic photomemristor-reservoir computing. Nature Communications 14, 2169 (2023).