Neurocomputation of Language
We combine neurocomputational modeling with empirical experimentation to investigate the neural basis of language. Grounded in principles of neural information processing, and informed by empirical insights on the nature and time course of comprehension, our models aim to offer mechanistic explanations of how an interpretation is constructed in real time as spoken or written linguistic signal unfolds on a, more or less, word-by-word basis.
-
dr. Harm Brouwer
Principal Investigator
Themes that are central to our research are:
The nature and time course of comprehension: Using high-temporal resolution neurophysiological recordings, we aim to identify what processes are involved in mapping from signal to meaning, and how these processes are temporally organized. Through direct linking hypotheses to online behavioral (reading times) and neurophysiological (event-related potentials) processing indices, our neurocomputational models allow us to empirically validate fine-grained mechanistic explanations.
The word-by-word comprehension models that we develop instantiate the Retrieval-Integration (RI) theory of the N400 and the P600, the two most salient components of the Event-Related brain Potential (ERP) signal for language comprehension. In these models, the N400 reflects the contextualized retrieval of the meaning of an incoming word from long-term memory, and the P600 the integration of this retrieved meaning with the unfolding utterance representation. Reading times, in turn, are estimated as meaning-driven surprisal. We show that our models capture signature N400 and P600 patterns, as well as behavioral reading times, and we empirically validate predictions from the RI model using novel neurobehavioral experiments.
Further reading:
Brouwer, H., Fitz, H., and Hoeks, J. C. J. (2012). Getting real about Semantic Illusions: Rethinking the functional role of the P600 in language comprehension. Brain Research, 1446, pp. 127-143. doi: 10.1016/j.brainres.2012.01.055
Brouwer, H., Crocker M. W., Venhuizen, N. J., and Hoeks, J. C. J. (2017). A Neurocomputational Model of the N400 and the P600 in Language Processing. Cognitive Science, 41(S6), pp. 1318-1352. doi: 10.1111/cogs.12461
Brouwer, H., Delogu, F., Venhuizen, N. J., and Crocker, M. W. (2021). Neurobehavioral Correlates of Surprisal in Language Comprehension: A Neurocomputational Model. Frontiers in Psychology 12:110. doi: 10.3389/fpsyg.2021.615538
Neural representation and computation of meaning: Neurocomputational models require us to be explicit about the representations computed during comprehension. Informed by neurophysiology and linguistic theory, our models recover neural representations of word-level and utterance-level meaning. Crucially, beyond capturing literal meaning, we develop utterance representations that capture inferences driven by world knowledge, as well as information theoretic notions of expectancy.
In our Distributional Formal Semantics (DFS) framework, propositional and sub-propositional meaning is defined as a vector relative to a set of formal models that capture different states of the world, such that each unit represents a probabilistic truth value relative to an individual model. DFS offers probabilistic, distributed meaning representations that are inherently compositional, and that naturally capture fundamental semantic notions such as quantification and entailment. Furthermore, the probabilistic nature of these representations allows for probabilistic inference, and inherently provides an information-theoretic notion of 'information', in terms of 'comprehension-centric' Surprisal and Entropy.
Further reading:
Venhuizen, N. J., Crocker, M. W., and Brouwer, H. (2019). Expectation-based Comprehension: Modeling the interaction of world knowledge and linguistic experience. Discourse Processes, 56:3, pp. 229-255. doi: 10.1080/0163853X.2018.1448677
Venhuizen, N. J., Crocker, M. W., and Brouwer, H. (2019). Semantic Entropy in Language Comprehension. Entropy, 21(12), 1159. doi: 10.3390/e21121159
Venhuizen, N. J., Hendriks, P., Crocker, M. W., and Brouwer, H. (2022). Distributional Formal Semantics. Information and Computation, 287: 104763. doi: 10.1016/j.ic.2021.104763
The cortical organization of the language network: By aligning our neurocomputational models with insights on the cortical organization of language, as informed by spatial neuroimaging and lesion studies, we derive functional-anatomic mappings that allow for spatiotemporal predictions about language processing in the brain. This cortical grounding is critical to understanding how processes dynamically interact, and how this affects neurophysiological signals.
We have developed a functional-anatomical mapping of the RI theory of the N400 and the P600 to a minimal cortical network for language comprehension. In this mapping, the left posterior part of the Middle Temporal Gyrus (BA 21) serves as an epicenter (or hub) in a neurocognitive network for the retrieval of word meaning, the ease of which is reflected in N400 amplitude. The left Inferior Frontal Gyrus (BA 44/45/47), in turn, serves a network epicenter for the integration of this retrieved meaning with the unfolding utterance representation; these semantic and pragmatic integrative processes are reflected in P600 amplitude. This mapping has been critical to our understanding of the spatiotemporal dynamics of the N400 and the P600, and has fundamentally changed our approach to ERP data analysis.
Further reading:
Brouwer, H. and Hoeks J. C. J. (2013). A Time and Place for Language Comprehension: Mapping the N400 and the P600 to a Minimal Cortical Network. Frontiers in Human Neuroscience 7:758. doi: 10.3389/fnhum.2013.00758
Brouwer, H. and Crocker, M. W. (2017). On the proper treatment of the N400 and P600 in language comprehension. Frontiers in Psychology 8:1327. doi: 10.3389/fpsyg.2017.01327
Brouwer, H., Delogu, F., and Crocker, M. W. (2021). Splitting Event-Related Potentials: Modeling Latent Components using Regression-based Waveform Estimation. European Journal of Neuroscience, 53, pp. 974-995. doi: 10.1111/ejn.14961
People:
Dr. Harm Brouwer (PI) – http://www.hbrouwer.eu/
Dr. Paula Roncaglia-Denissen - https://www.tilburguniversity.edu/staff/m-p-roncaglia
Dr. Noortje Venhuizen - https://njvenhuizen.github.io/
Partners
Saarland University