PhD student at Aalto Media Department / Jelena Rosic
Guest lecture / Dr Janne Kauttonen
Moderated by Professor Lily Díaz-Kommonen
Abstracting and modeling experience – The view from a neurophenomenological isomorphism
In the 2020 seminar, I will look into epistemological and methodological convergence in research on media (artefacts) that combines empiricism of natural sciences with computational approaches. The line of research linking media and neuroscience has been prone to overinterpretation and misconceptions often due to the levels of abstraction and complexity regarding computational tools in neuroimaging. I will present my research that draws on the neurophenomenological programme (methodological bridging of subjective/objective gap) with an aim to underline levels of formalism between multisource data for the meaningful interdisciplinary study paradigms. These sources in my work include 1st/2nd person data (empirical phenomenology) with neuroimaging paradigms and models to characterize different approaches to brain connectivity. I have been using this conceptual and methodological framework with the specialized micro-phenomenology method to research augmented sense-making in interactive media, frameworks for affective computing and more extensively neurocinematics studies (cognitive neuroscience using films as stimuli) in order to model experience in a rigorous manner. The recent experimental work done as an Early-Stage researcher at Enactive Virtuality Lab at Tallinn University builds on previous work of understanding how film and its formal aspects constitute ecologically valid paradigm considered to approximatereal-life dynamic situations. In these terms, the perspective of expert annotation methods of stimuli, the so-called ‘stimulus error’ between the content of stimuli, experience and their brain mapping with the narrative context are addressed.
Finally, methodological aspects of this interdisciplinary approach are discussed with Dr Janne Kauttonen through his interdisciplinary expertise and research in human brain functions during naturalistic conditions using fMRI.
Introducing the guest
In 2019 review of naturalistic neuroscience and its stimuli (Sonkusare et al.Trends in CogSci), authors point out to Kauttonen et al. (2018) study: “A recent study deserves particular mention as it took advantage of cinematic style of the nonlinear narrative structure of ‘Memento’ (C. Nolan), a film that depicts successive scenes in reverse chronological order… (…). Analysis using MVPA identified patterns of activation in higher order cortical networks (precuneus, angular gyrus, cingulate gyrus, frontal poles) during presentation of key-frame scenes, which revealed contextual information about prior events.”
“Now I know what this goes with!” – recall and narrative reconstruction in Memento
In this study we looked into how the human brain recalls and connects relevant memories with unfolding events. We hypothesized that repeating key-frames in Memento serve as immediate recall cues and would facilitate reconstruction of the story piece-by-piece. The chronological version of Memento, shown in a separate experiment for another group of subjects, served as a control condition. Using fMRI BOLD measurements and multivariate event-related pattern analysis method and representational similarity analysis, focal fingerprint patterns of hemodynamic activity were found to emerge during presentation of key-frame scenes. Despite the highly differential sensory-feature content of the key-frames, focal fingerprint patterns of BOLD activity were shared across the second presentation of the key-frame scenes (i.e., when they served as immediate recall cues) in various higher-order cortical regions. The distributed patterns of brain activity within these regions appear to underlie ability to recall relevant memories and connect them with ongoing events to facilitate understanding of what goes with what in a complex narrated story. Interestingly, there also seems to be anticipatory component in the higher-order sensory areas with the effects preceding onset of the key-frames.