Sebastian J. Schlecht: Creating the perfect illusion in virtual reality

Imagine you have a set of headphones on. They let through the sounds of the world around you. In the far corner of your room there are large speakers. You hear music but you can’t differentiate whether the sounds you hear are from the headphones or from the speakers in the corner. This is the goal of Professor Sebastian J. Schlecht and his team at Aalto University – to create a perfect auditory illusion which can be virtually overlayed on the real world.

According to Professor Schlecht, they aren’t far from this immersion he calls transfer-plausibility. What you need is simple: a set of headphones and a tracking device, which tells the location and movement of a person’s head.

‘If you think the visual domain, it’s still a very difficult challenge. It is still very easy to tell the difference between a real photograph and one rendered on a computer. High-end Hollywood productions might be the exception, but they show how difficult that is. To do that in real time is a different challenge. But for the audio side, I think we’re now close to creating a perfect illusion’, he says.

In an interesting and quickly developing field, the work of Schlecht and his team in Aalto University’s Acoustics and Media Labs occupy a special niche.

‘Working with virtual reality (VR) and sound is always a much smaller group than the whole VR community. Graphics department is ten times bigger than audio’, he explains.

Enhanced reality has an amazing potential

Now virtual reality is mainly used as a gaming platform. Commercial VR headsets are widely on the market. According to Schlecht, it’s been discussed if VR will also rise as a social medium. It may become the new Facebook. VR is already being used in training doctors, law enforcement and social workers, for example.

In the next five years, we’ll see how augmented reality (AR) will enter our lives. In virtual reality you sink into the world that is entirely created by a computer and in augmented reality the virtual gets mixed with the real.

Schlecht is excited to see how enhanced reality will develop artistically.

‘As a media it’s really powerful. It has an amazing potential, because you can change the laws of physics and give people different identities. Artistically it’s an amazing medium. Of course, everything has a problematic version of it and there are some ethical considerations’, he says.

The journey from all the tiny details to the immersive experience of VR is complicated and full of variables. The complex connections are a challenge that takes a lot of time to solve.

Ultimately, the scientist hopes that it would be possible to create a natural-like telepresense. Instead of Zoom meetings, there would be a feeling of shared space with real sized hologrammes.

Crossing the boundaries of disciplines at Aalto

To use VR or AR as effectively as possible, there is a pressing need for expertees in different fields. If Schlecht could choose, he would gather experts from computer science, game design, 3D animation, psychology, cognitive science, interaction design and immersive storytelling, to name a few.

In Schlecht’s mind, VR covers all the fields of science, since everything is a part of reality VR is trying to depict. Academically it’s a huge challenge and it’s necessary to cross borders.

‘In comparison to other academic places Aalto is already doing a great job at trying to facilitate cross-disciplinary work and giving opportunities for this’, he says.

‘But of course, these boxes, the disciplines, are a long tradition. Doing courses from another school means more work. Yet, I would strive for a more mixed group. Here in Aalto we already have the possibility of working with artists and more technical people, which is a really good start.’

Schlecht became a scientist by following the topics he likes to spend time with – music and mathematics. What brought him to Aalto University was the unusual position as Professor of Practice for Sound in Virtual Reality. With the cross-school position he became a full time academic, for the first time.

He was a half time musician before taking the job as a fixed time professor. After a couple of years at Aalto, he says he has enjoyed the time there.

‘There’s an amazing support around me. Most of my time here has been impacted by the pandemic. Regardless of that there’s a great sense of community in doing this together and supporting each other, which I’m very grateful for. And it extends to the research – we collaborate a lot.’

Schlecht also gives praise to the education Aalto gives to new lecturers. The teaching side of his job gives more opportunities to meet like-minded people from different fields and make friends.

Helsinki - safe and easy to reach

Schlecht lives in central Helsinki with his partner and daughter. He says he couldn’t think of a more relaxed place to live with a family. To him, Helsinki is a very calm and organized place that’s also safe and welcoming.

‘Helsinki is a city with a great balance of being very international and cosmopolitan city and being relatively small, at least compared to Berlin where I lived before. I’m using my bike most of the time. You can go anywhere in max half an hour. There’s culture and museums but you can also go to the seaside or the forest. Having all this at your disposal is of course amazing’, Schlecht praises.

His daughter is now in second grade. Currently they have a common project, since the father is learning Finnish.

‘My daughter went to eskari (preschool) when we came. She is having the whole Finnish experience. I’m trying to read with her. The second grade texts are still a challenge for me, but sometimes they make sense. Having a kid’s perspective to this has made learning the language more fun’, he smiles.

The scientist hasn’t got a lot of time for hobbies, but he greatly enjoys biking from Kruununhaka to Otaniemi for work through the islands. It’s clear to a musician what he misses from the pre-pandemic time – playing and hearing live music.

Sebastian J. Schlecht in short

• Sebastian J. Schlecht (1986, Germany) is Professor of Practice for Sound in Virtual Reality in Aalto University, Helsinki, Finland.
• Before he joined Aalto, he was a post-doctoral researcher in semantic audio processing and perception-based spatial audio signal processing at Friedrich-Alexander University Erlangen-Nürnberg, Germany.
• He teaches multiple courses in Aalto University, including “Coding virtual worlds” and “Virtual Acoustics”.
• Before his position at Aalto, he was a half-time musician. He started playing violin at the age of 5 and piano when he was 8.

Text: Aino Soutsalmi / Medita
Photos and video: Mika Vartiainen / Aalto Studios