URL: http://www.multisense.org

Principal investigators: Andreas K. Engel (Hamburg) and Peter König (Osnabrück)

This project has been funded by an Advanced Grant of the European Research Council (ERC) between 2011 and 2016.

The integration of information across different sensory channels is a fundamental and complex problem that we have to solve in every moment of our daily life. Despite intense research in the past decades, we do not yet have a comprehensive view on the nature of unified multisensory experience and its underlying cognitive and physiological mechanisms.

This project has studied four key aspects of multisensory processing that are currently unresolved: (1) the role of dynamic binding in multisensory experience; (2) the role of action and sensorimotor contingencies in multisensory processing; (3) the modulation of multisensory experience by top-down factors; (4) the adaptivity of these three mechanisms in conditions of altered multisensory experience. To address these issues, we have applied psychophysiological, behavioral (e.g., eye tracking, sensory augmentation) and physiological (e.g., MEG, EEG, tACS) approaches in studies on human subjects and combined these with advanced signal analysis approaches.

Using these approaches, we have obtained neurophysiological evidence demonstrating that oscillatory brain signals in multiple frequency ranges are modulated by multisensory stimulation and are functionally relevant for implementing interactions across sensory systems. Importantly, we could also demonstrate that this involves changes in functional connectivity, mediated by phase coupling of neural oscillations. Our results show that factors influencing crossmodal interactions are manifold and operate in a stimulus-driven, bottom-up fashion, as well as via top-down control. Our data on sensorimotor interactions suggest that multisensory interaction and integration can be best understood within an embodied perspective, tying together sensory processing and behaviour in a unified framework. The results demonstrate a tight integration of sensory, cognitive and motor systems and the necessity to investigation of whole system in a ecologically valid context. In a set of studies on sensory augmentation we could demonstrate that newly acquired senses are integrated with the classic modalities. Furthermore, our investigations on adaptivity of multisensory processing show that developmental changes strongly shape the functional dynamics of the cortical networks. In blind participants, we could show that, due to the altered postnatal experience, functional connectivity within occipital cortex and between different cortical systems is profoundly altered.

During the implementation of the project, we advanced existing approaches and developed novel methodologies in several domains, including multisensory stimulus paradigms, sensory augmentation techniques, methods for neurostimulation and for removal of artefacts, as well as methods for analysis of BCI signals and of neural connectivity in EEG or MEG data. Furthermore, our work contributes to internet-based study methodology and to open-source data repositories. Dissemination of the results of the project was highly successful with a large number of publications, including original papers, review articles, talks and popular science presentations. Furthermore, the project has triggered the creating of two spin-off companies.

Taken together, this project has pursued a highly innovative and systematic approach, the results of which substantially augment our understanding of the nature and mechanisms of unified multisensory experience. Our project makes strong contributions to advancing a dynamic view on multisensory processing that includes contextual top-down modulation of multisensory processing, emphasizses the relation to action, and considers dynamic functional connectivity as a key mechanism for multi sensory integration.

URL: http://www.multisense.org

Principal investigator: Andreas K. Engel (Hamburg)

This project is funded by an Advanced Grant of the European Research Council (ERC) from 2023 until 2028.

Functional coupling is a hallmark of brain networks, and there is a wealth of studies that have observed a relation between functional connectivity and cognition or sensorimotor processing. However, the vast majority of studies on this topic are correlative in nature, and causal evidence for the role of functional coupling is almost completely lacking. The cICMs project aims to fill this gap. We focus on two different types of intrinsic coupling modes (ICMs): phase ICMs, arising from phase coupling of oscillatory signals, and envelope ICMs, resulting from coupled fluctuations of signal envelopes. We manipulate, analyze and model the different types of ICMs with the aim to establish an integrated view on their functions and interrelations which currently is still lacking. The neurophysiological experiments are complemented by information-theoretic analyses and computational modeling. This research program aims to develop an integrated multiscale framework for ICMs that is likely to have implications for a better understanding of the mechanisms underlying cognition and the complexity of the human mind.