Matias Harju

Sound Designer, Musician

Narrative techniques of room-scale Audio Augmented Reality (AAR)

In my master's thesis project for Aalto University MediaLab I'm exploring some of the characteristic narrative techniques applicable for room-scale Audio Augmented Reality (AAR). As a narrative art form AAR is still very marginal and little used, even though it carries very interesting possibilities of bringing an acoustic level of storytelling to museums, art exhibitions, shopping centres, etc.

The technologies of AAR have been researched relatively much since 1990's, Aalto University Acoustic Lab being one of the leading institutions in this field. However, the artistic and narrative possibilites of AAR are still an under-explored territory: to my knowledge there are or have been only occasional experiments or exhibitions around the world using AAR with a strong storytelling component, audio walks being perhaps a happy exception.

My goal with this thesis is to get a grip of the medium, and with the help of practical narrative demos start forming a categorisation/taxonomy for characteristic narrative techniques. Hopefully, in consequence, this would be useful for me and other content-creators when starting to use AAR for immersive storytelling and narrative illusions.

In my project I'm focusing in room-scale AAR, meaning an experience where the user can freely move around in a space while the virtual audio material is rendered binaurally to her ears using headphones with accurate position and orientation tracking. This "six-degrees-of-freedom" (6DoF) approach potentially creates an immersive illusion of augmented audio objects positioned in the space around the listener. The tracking also allows user interaction with the virtual world, opening possibilites for dynamic narrative.

Headphones with a pozyx tag and a Unity scene.jpg

Since commercial 6DoF AAR systems open for content creators are not yet readily available, I decided to construct my own system to test how the medium would work and what kind of different narrative techniques work with it. My prototype uses UWB (Ultra Wideband) indoor positioning system by pozyx allowing about 10 cm accuracy and multiple simultaneous users. Integrated with the system is an IMU (Inertial Measurement Unit) in each headphone set tracking the head-orientation. The virtual audio world is running in Unity, while the binaural spatial rendering is handled by dearVR plugin.

In addition to my own experiments I have tried to find other narrative room-scale AAR experiences and analyse them. So far I have personally encountered with only two, Sound of Things, an installation by Holger Förterer and Sounds of Silence, an exhibition at the Bern Museum of Communication in Switzerland. The first one used IR cameras for tracking two sets of headphones, and was extremely accurate and very impeessive. The Bern exhibition, which is still running when writing this text (until 7 July 2019), is realised with similar UWB technology as my prototype: the usomo system is a commercial product developed over several years by a Berlin-based company, and it seems a very viable option for room-scale AAR projects.

Currently I'm running technical experiments and building a set of miniature demo stories to test a variety of different narrative techniques I have chosen to explore. I will post updates on coming public exhibitions (hopefully in the beginning of August), and as soon as I have something academic to report, I will of course let you know!