TCN lab research projects
Naturalistic Neuroscience
The goal of naturalistic neuroscience research is to improve the understanding of how the human brain processes complex real-world stimuli. We have done pioneering work to study the neural underpinnings of neurodevelopmental disorders using naturalistic neuroscience methods, especially film viewing during functional magnetic resonance imaging.
In this study we demonstrated that the brains of adults with autism spectrum disorder do not 'tick together' with others during viewing of a narrative movie portraying social interaction. The figure shows brain areas where inter-subject correlations were larger in neurotypical participants than in adults with autism spectrum disorder. SMG, supramarginal gyrus; LOC, lateral occipital cortex; PCC, posterior cingulate cortex; PC, precuneus; ACC, anterior cingulate cortex; Cau, caudate nucleus; SMC, supplementary motor cortex.
This study showed that functional connectivity in large scale brain networks in adults with autism spectrum disorder was altered as compared with neurotypical adults during film viewing. Reconfiguration of functional brain networks in autism spectrum disorder. The figure illustrates functional subnetwork similarities and differences between neurotypical adults and adults with autism spectrum disorder.
Here we demonstrated that brain activity in adults with ADHD was desynchronized in comparison to neurotypical adults when the participants were watching a film depicting multi-talker conversation embedded with distractors (speech or music).
More recently, we have started to study how the difference between participants’ own spontaneous behaviour and observed behaviour reflects to brain activity during video viewing. Furthermore, we use virtual reality game (EPELI) also to study the brain basis of neurodevelopmental disorders. For example, Merzon et al. (2023) showed how manifestation of the symptoms during complex goal-directed behaviours reveals large-scale rewiring of the functional connectome in ADHD children. We also have ongoing projects where we collect mobile electroencephalography (EEG, see A) and magnetoencephalography (MEG, see B) in adults with ADHD and neurotypical controls.
We are also heavily involved in ongoing research using naturalistic neuroscience methods in studying the neural basis of narcissistic traits. See https://www.helsinki.fi/en/researchgroups/narcissism-face-and-social-interaction for a collaborative project.
REFERENCES
Merzon, L., Tauriainen, S., Triana, A., Nurmi, T., Huhdanpää, H., Mannerkoski, M., Aronen, E., Kantonistov, M., Henriksson, L., Macaluso, E., & Salmi, J. (preprint). Real-world goal-directed behavior reveals aberrant functional connectivity in children with ADHD. Psyarxiv.
Glerean, E., Pan, R. K., Salmi, J., Kujala, R., Lahnakoski, J. M., Roine, U., Nummenmaa, L., Leppämäki, S., Nieminen-Von Wendt, T., Tani, P., Saramäki, J., Sams, M., & Jääskeläinen, I. P. (2016). Reorganization of functionally connected brain subnetworks in high‐functioning autism. Human brain mapping, 37(3), 1066-1079.
Salmi, J., Metwaly, M., Tohka, J., Alho, K., Leppämäki, S., Tani, P., Koski, A., Vanderwal, T., & Laine, M. (2020). ADHD desynchronizes brain activity during watching a distracted multi-talker conversation. NeuroImage, 216, 116352.
Salmi, J., Roine, U., Glerean, E., Lahnakoski, J., Nieminen-von Wendt, T., Tani, P., Leppämäki, S., Nummenmaa, L., Jääskeläinen, I.P., Carlson, S., Rintahaka, P., & Sams, M. (2013). The brains of high functioning autistic individuals do not synchronize with those of others. NeuroImage: Clinical, 3, 489-497.
Virtual reality in neuropsychological assessment and rehabilitation
Virtual reality (VR) and game technology have strong underexploited potential to advance neuropsychological assessment, monitoring, and rehabilitation. VR allows emulating situations resembling those where the problems (i.e., symptoms) are manifested in everyday life, and simultaneous measurement of rich behavioral and neural data. See the video below for an EPELI game that we are using for clinical research related to attentional executive functions. The game is implemented by Peili Vision LTD (www.peilivision.fi). A few examples of our recent EPELI studies below.
Seesjärvi and colleagues (2022) extracted various behavioural metrics, indexing memory, attention, planning, activity level and impulsivity, from rich VR data acquired when the participants performed demanding everyday executive tasks in a virtual home. These measures were associated with parent-reported everyday life problems and showed excellent performance in discriminating between children with ADHD and typically developing children.
Merzon and colleagues (2022) demonstrated that eyes can tell if the child has attention deficits. A machine learning classifier trained with eye movement data obtained when the participants were doing naturalistic tasks in an open ended environment predicted at 92% accuracy whether a child has ADHD or not.
The development of a rehabilitation version of EPELI for children started in 2022 and is currently in progress. This project relates to our collaboration with Oulu University (Profs. Johanna Uusimaa and Mirjami Mäntymaa, and co-supervised PhD student Merja Nikula).
Besides ADHD, we also use VR for researching attentional executive functions in other clinical groups, such as autism and stroke patients. EPELI has also been in clinical use at the Helsinki University Hospital since 2021 and it is now being used also in the private healthcare sector. Besides EPELI, our team is also participating to the development of other clinical VR methods (see Uimonen et al. 2024).
REFERENCES
Uimonen, J., Villarreal, S., Laari, S., Arola, A., Ijäs, P., Salmi, J., & Hietanen, M. (2024). Virtual reality tasks with eye tracking for mild spatial neglect assessment: a pilot study with acute stroke patients. Frontiers in Psychology, 15, 1319944.
Salmi, J., Merzon, L., Eräste, T., Seesjärvi, E., Huhdanpää, H., Aronen, E. T., Mannerkoski, M., MacInnes, W. J., & Laine, M. (2023). Fluctuations of Attention During Self-paced Naturalistic Goal-Directed Behavior in Attention-Deficit/Hyperactivity Disorder. JAACAP Open.
Merzon, L., Pettersson, K., Aronen, E. T., Huhdanpää, H., Seesjärvi, E., Henriksson, L., MacInnes, W. J., Mannerkoski, M., Macaluso, E., & Salmi, J. (2022). Eye movement behavior in a real-world virtual reality task reveals ADHD in children. Scientific Reports, 12(1), 20308.
Seesjärvi, E., Puhakka, J., Aronen, E. T., Lipsanen, J., Mannerkoski, M., Hering, A., Zuber, S., Kliegel, M., Laine, M., & Salmi, J. (2022). Quantifying ADHD symptoms in open-ended everyday life contexts with a new virtual reality task. Journal of Attention Disorders, 26(11), 1394-1411.
Online cognitive testing and rehabilitation
During the past decade there have been considerable advances in the development of online cognitive testing platforms. Together with Prof. Matti Laine (Åbo Akademi University), we have spearheaded these developments. SOILE online cognitive testing platform originally developed at Åbo Akademi University that has been successfully used in various studies (see www.braintrain.fi) is now being updated at Aalto University to be hopefully soon released as an open source tool. With SOILE it is possible to create both conventional neuropsychological tests as well as video game based tasks. It can also be used to administer questionnaires and for recording web camera and audio. Examples where SOILE has been used or is currently being used below.
Our first clinical study with SOILE was conducted in individuals with Parkinson’s disease. We demonstrated that patients with mean age of 65 years were able to successfully complete eight week randomised controlled trial with pre- and post-test as well as five week training program with three weekly sessions.
Besides improvements in the trained cognitive tasks and another task that taps the same cognitive domain, depression scores in the training group were decreased compared to the controls.
More recently, we have used SOILE in online cognitive testing in ADHD adults (Jylkkä et al., 2023), integrating it with browser-based version of EPELI (see also Laine et al., 2023, Seesjärvi et al., 2023). Moreover, we are involved in a project run by our collaborators at the University of Helsinki (Prof. Anssi Peräkylä), where web camera videos collected with SOILE are analysed with advanced machine learning algorithms to retrieve information reflecting participants socio-emotional signals during conversation. Such fully automated online testing and rehabilitation methods have considerable potential to improve future healthcare, being user friendly, accessible, scalable, and cost-efficient.
REFERENCES
Fellman D., Salmi J., Ritakallio L., Ellfolk U., Rinne JO., Laine M. (2018). Training working memory updating at home may have positive effects on memory and mood in Parkinson’s disease: A randomized controlled trial. Neuropsychological Rehabilitation,3, 1-36.
Jylkkä J., Ritakallio L., Merzon L., Kangas S., Kliegel M., Zuber S., Hering A., Laine M., Salmi J. (2023). Assessment of goal-directed behavior and prospective memory in adult ADHD with an online 3D videogame simulating everyday tasks. An article preprint.
Laine M., Jylkkä J., Ritakallio L., Eräste T., Kangas S., Hering A., Zuber S., Kliegel M., Fellman D., Salmi J. (2023). Spontaneous memory strategies in a videogame simulating everyday prospective memory tasks. An article preprint.
Seesjärvi E., Laine M., Kasteenpohja K., & Salmi J. (2023). Assessing goal-oriented behavior in virtual reality: children prefer head-mounted displays but flat screens provide a viable performance measure for remote testing. An article preprint.
Extended reality and real-world measurements
One of our key long-term interests is to develop quantitative measures to assess human goal-directed behaviour in the actual real-world situations. We are currently studying means to use hand tracking, eye tracking, and motion capture to quantify human-environment interaction while the participants are moving around in a real-room and interacting with objects. In the video below demonstrating our ongoing tech developments, the participant performs a list of instructed everyday household chores similar to those in EPELI (see above, Seesjärvi et al. 2022, Merzon et al. 2022) by moving around in a virtual room and interacting with objects using her own hands. Although the participant sees a fully virtual room, the furniture and objects can be real (towards the end of the video) when they are embedded with object trackers. Using a pass-through camera it is possible do similar recordings also when the participant is performing the tasks in a fully realistic room. The participant is equipped with high resolution XR goggles, full body motion capture suit, mobile EEG device, and biosensors attached to several parts of the body.
In real-world measurements, we use smart suits to quantify child body motion during the school classes (measurement of hyperactivity and attention). The suit has been developed by our colleagues at BABA Center (https://www.babacenter.fi/). Prof. Sampsa Vanhatalo’s group has previously used these methods in studying the development of movement abilities in babies (see Airaksinen et al. 2020).
We also use other technologies, such as eye glasses with gaze tracking to quantify human behaviour in real-world situations. For instance, in an ongoing study children with autism and typically developing children take part to two sessions. In one of the sessions they view a 360 video of a group conversation with head-mounted VR display (Varjo XR-3) while their eye movements and physiological signals (Noraxon Ultium EMG/ECG) are being recorded. In the other session, they participate to a similar four person group discussion situation themselves, where we measure their eye movements with eye tracking glasses (Pupil Labs Invisible) and physiological signals with a similar device than in 360-VR session. For both data, gaze is mapped to face and body using machine learning algorithms (e.g., RetinaFace and OpenPose, see video).
REFERENCES
Airaksinen, M., Räsänen, O., Ilén, E., Häyrinen, T., Kivi, A., Marchi, V., Gallen, A., Blom, S., Varhe, A., Kaartinen, N., Haataja, L., & Vanhatalo, S. (2020). Automatic posture and movement tracking of infants with wearable movement sensors. Scientific reports, 10(1), 1-13.
Merzon, L., Pettersson, K., Aronen, E. T., Huhdanpää, H., Seesjärvi, E., Henriksson, L., MacInnes, W. J., Mannerkoski, M., Macaluso, E., & Salmi, J. (2022). Eye movement behavior in a real-world virtual reality task reveals ADHD in children. Scientific Reports, 12(1), 20308.
Seesjärvi, E., Puhakka, J., Aronen, E. T., Lipsanen, J., Mannerkoski, M., Hering, A., Zuber, S., Kliegel, M., Laine, M., & Salmi, J. (2022). Quantifying ADHD symptoms in open-ended everyday life contexts with a new virtual reality task. Journal of Attention Disorders, 26(11), 1394-1411.
Neural basis of developmental disorders
Besides developing new experimental designs for behavioral and brain research, we have also several other projects related to investigating the neural basis of developmental disorders. For instance, we have recently demonstrated that adult ADHD is characterized by weaker structural connectivity, as well as weaker working memory related functional connectivity, compared with neurotypical adults. The networks differentiating adults with and without ADHD encompassed many areas related to attention, working memory and other executive functions, and sensory processing. The knowledge of neural basis of ADHD in adults is still rather limited and it would be important to advance research in this domain since the prevalence of ADHD in adults is raising heavily.
We also investigate structural and functional alterations in neural processing related to other developmental disorders such as autism and psychopathy (see Noppari et al. 2022).
REFERENCES
Noppari, T., Sun, L., Lukkarinen, L., Putkinen, V., Tani, P., Lindberg, N., Saure, E., Lauerma, H., Tiihonen, J., Venetjoki, N., Salomaa, M., Rautio, P., Hirvonen, J., Salmi, J., & Nummenmaa, L. (2022). Brain structural alterations in autism and criminal psychopathy. NeuroImage: Clinical, 35, 103116.
Tolonen, T., Roine, T., Alho, K., Leppämäki, S., Tani, P., Koski, A., Laine, M., & Salmi, J. (2023a). Abnormal wiring of the structural connectome in adults with ADHD. Network Neuroscience, 7(4), 1302–1325.
Tolonen, T., Leppämäki, S., Alho, K., Tani, P., Koski, A., Laine, M., & Salmi, J. (2023b). Aberrant functional connectivity during working memory tasks in adults with ADHD. Poster presented at the Brain and Mind Symposium, Espoo, Finland. 19-20.10.2023.
Augmented Reality Glasses: Supporting Older Adults with Daily Routines
AR4U-project studies how augmented reality (AR) glasses can assist older adults in managing their daily routines. In our vision, there will be AR spectacles, similar in size to regular eyeglasses, that can be worn throughout the day—from morning until bedtime. These glasses will provide reminders, activation, and assistance for essential tasks like meals, medication, hygiene, and housekeeping. In addition to assisting with daily activities, the glasses will also feature a virtual companion to alleviate feelings of loneliness.
The need for such solutions is immense. As the global population ages rapidly, there is a growing shortage of caregivers and financial resources to meet the demand. By helping older adults maintain independence and manage more tasks themselves, we can improve their well-being and extend their active years at home.
In this project, we are developing platform-independent software and testing it on augmented and mixed-reality goggles. Our main goal is to explore how AR can support older adults in living longer at home. We are collecting user data to identify the most valuable use cases and features for older adults. Our application is designed to work across various hardware platforms, ensuring we are ready to enter the market as the hardware matures.
The AR4U Research-to-Business project aims to transform our research into viable business opportunities. This 18-month project is funded by Business Finland.
Project team:
Gautam Vishwanath, Technical Lead
[email protected]
Timo Niskanen, Commercial Lead
[email protected]
Juha Salmitaival, Primary Investigator
[email protected]
To know more about our research you can also take a look at our publications
Back to the main page.
- Published:
- Updated: