New Media Doctoral Seminar

Programme:

PhD student at Aalto Media Department / Jelena Rosic

Guest lecture / Dr Janne Kauttonen

Moderated by Professor Lily Díaz-Kommonen

Abstracting and modeling experience – The view from a neurophenomenological isomorphism

In the 2020 seminar, I will look into epistemological and methodological convergence in research on media (artefacts) that combines empiricism of natural sciences with computational approaches. The line of research linking media and neuroscience has been prone to overinterpretation and misconceptions often due to the levels of abstraction and complexity regarding computational tools in neuroimaging. I will present my research that draws on the neurophenomenological programme (methodological bridging of subjective/objective gap) with an aim to underline levels of formalism between multisource data for the meaningful interdisciplinary study paradigms. These sources in my work include 1st/2nd person data (empirical phenomenology) with neuroimaging paradigms and models to characterize different approaches to brain connectivity. I have been using this conceptual and methodological framework with the specialized micro-phenomenology method to research augmented sense-making in interactive media, frameworks for affective computing and more extensively neurocinematics studies (cognitive neuroscience using films as stimuli) in order to model experience in a rigorous manner. The recent experimental work done as an Early-Stage researcher at Enactive Virtuality Lab at Tallinn University builds on previous work of understanding how film and its formal aspects constitute ecologically valid paradigm considered to approximatereal-life dynamic situations. In these terms, the perspective of expert annotation methods of stimuli, the so-called ‘stimulus error’ between the content of stimuli, experience and their brain mapping with the narrative context are addressed.

Finally, methodological aspects of this interdisciplinary approach are discussed with Dr Janne Kauttonen through his interdisciplinary expertise and research in human brain functions during naturalistic conditions using fMRI.

Introducing the guest

In 2019 review of naturalistic neuroscience and its stimuli (Sonkusare et al.Trends in CogSci), authors point out to Kauttonen et al. (2018) study: “A recent study deserves particular mention as it took advantage of cinematic style of the nonlinear narrative structure of ‘Memento’ (C. Nolan), a film that depicts successive scenes in reverse chronological order… (…). Analysis using MVPA identified patterns of activation in higher order cortical networks (precuneus, angular gyrus, cingulate gyrus, frontal poles) during presentation of key-frame scenes, which revealed contextual information about prior events.” 

“Now I know what this goes with!” – recall and narrative reconstruction in Memento

In this study we looked into how the human brain recalls and connects relevant memories with unfolding events. We hypothesized that repeating key-frames in Memento serve as immediate recall cues and would facilitate reconstruction of the story piece-by-piece. The chronological version of Memento, shown in a separate experiment for another group of subjects, served as a control condition. Using fMRI BOLD measurements and multivariate event-related pattern analysis method and representational similarity analysis, focal fingerprint patterns of hemodynamic activity were found to emerge during presentation of key-frame scenes. Despite the highly differential sensory-feature content of the key-frames, focal fingerprint patterns of BOLD activity were shared across the second presentation of the key-frame scenes (i.e., when they served as immediate recall cues) in various higher-order cortical regions. The distributed patterns of brain activity within these regions appear to underlie ability to recall relevant memories and connect them with ongoing events to facilitate understanding of what goes with what in a complex narrated story. Interestingly, there also seems to be anticipatory component in the higher-order sensory areas with the effects preceding onset of the key-frames.

BIO

Dr Janne Kauttonen

Dr Janne Kauttonen works as a staff researcher at Haaga-Helia University of Applied Sciences. His research background is strongly inter-disciplinary, including aspects from statistical physics, computational sciences, neuroscience and behavioral sciences. His particular areas of expertise are dynamical systems, analysis of large datasets, statistics, machine-learning and mathematical modelling. After obtaining his PhD in physics from University of Jyväskylä in 2012 in the field of statistical physics, he has worked as a researcher at Aalto University, Carnegie Mellon University and Laurea University of Applied Sciences. At Aalto and Carnegie Mellon, he developed computational tools and did research in human brain functions during naturalistic conditions (fMRI data) and visual system of rodents at neuronal level (2-photon imaging data). At Laurea University of Applied Sciences he studied cognitive biases of humans and voting behavior using text analytics and statistical methods. His current research at Haaga-Helia involves studying human interaction with other humans and artificial agents using physiological and behavioral measurements.

BIO

Jelena Rosic, MA

Jelena Rosic is a PhD student at Aalto Media Department and Early Stage researcher at Enactive Virtuality Lab, Tallinn University. She has a background in professional film editing (&dramatic arts studies), cognitive film studies with embodied perspective and neuroaesthetics of film. Her interdisciplinary research combines empirical phenomenology and enactive cognitive sciences to study embodiment in the spirit of neurophenomenology (Varela 1996). She has specialized in the second-person methods, in particular, the micro-phenomenological interviewing and analyses method (trained with Prof Claire Petitmengin,) to explore usually unrecognized or inaccessible dimensions of lived experiences. Her micro-phenomenological research in the context of augmented sense-making has focused on identifying the conceptual and empirical groundings of meta-perceptual modalities that are reported to emerge in lived experiences of technologically enhanced embodiment. As a member of the NeuroCine research group led by Professor Pia Tikka at Aalto University, she gained several years of experience in conducting interdisciplinary research within neurocinematics, a line of research that applies cognitive neuroimaging methods to study intersubjectively shared narrative experiences of film viewers. In the NeuroCine she participated in developing systematic annotation methods of film stimuli for optimising the linking of cinematic features to fMRI data. As a PhD candidate at the Crucible Studio, a research group at Aalto University that investigated interactive media combining theoretical and empirical methods, she applied her neurophenomenological approach to the study of enactive narratives. Jelena’s current research at Enactive Virtuality Lab applies the micro-phenomenological method to provide understanding on the lived experience of narrative sense-making in the context of movie viewing and engaging immersive environments.