Keywords: 360 film, virtual reality, interactive music, 3D audio, film sound
Roles: sound editor, sound designer, composer
Émotive VR is a French experimental 360º film project where spectator’s emotions control musical and visual elements in real-time.
I have worked in the project as the sound editor, sound designer and composer between January-July 2018 through an internship programme from Aalto University, Helsinki.
In this post I will share my experiences on the project and explain the workflows I’ve been using during the post-production of the Émotive VR. I’m going to be quite detailed, so I have divided the post into several chapters. You’ll find the navigation links at the end of this first text.
Neuro-interactive 360 film
In 2014 a French filmmaker Marie-Laure Cazin realised an interactive film project Cinéma émotif where the storyline of her film Madamoiselle Paradis was changing according to spectators’ emotions. A couple of selected audience members were wearing EEG headsets, and their valence (positive–negative) and excitement levels (high–low) were measured as they were watching the film. This data was then used to change the course of the film in real-time together with some audio effects on the soundtrack.
Émotive VR is a similar project, however this time the spectator will be inside a 360º film called Freud – La dernière hypnose. The film takes the audience into Sigmund Freud’s last hypnosis session with his young patient Karl. The story is based on Jean-Paul Sartre’s text Scénario Freud.
Unlike in the Cinéma émotif now the emotions will not change the storyline, but they will affect music and some visual elements in certain parts of the film. The spectator is placed in either Freud’s or Karl’s subjective (first-person) position, although there are cuts to objective (third-person) vantage points, too.
The core components of Émotive VR are two 10-11 minute cinematic 360-degree (spherical) monoscopic videos with actors and dialogue. These linear film sequences are complemented with spatialised sound design and interactive music. The videos run in Unity game engine which receives control parameters from the EEG system. Audio in 2nd order ambisonics and head-locked stereo will run separately but (hopefully) in sync in Wwise sound engine. The VR platform is HTC Vive / Steam VR.
Emotiv VR is a joint research project by the École superieur des beaux-arts TALM (ESBA-TALM), Le Laboratoire des Sciences du Numérique (LS2N) of the University of Nantes / Polytech Nantes, RFI Ouest Industries Creatives, Le Crabe Fantôme production company in Nantes, and DVgroup VR production company in Paris.
My physical location during the internship was in Le Mans, at the ESBA-TALM art school. I worked in a close relationship with the other intern Christophe Rey whose main job was in video editing, video stitching and 3D modelling. From time to time we travelled to Nantes to meet with the students at the University of Nantes working on the EEG algorithms and Unity under the supervision by professor Toinon Vigier.
The topics of hypnosis, psychoanalysis, self-knowledge, and psychology in general are extremely interesting in the context of VR, and exploring their relationships in terms of sound design would have been an intriguing challenge to me. Unfortunately it turned out that due to the massive workload with practical realisation of the sounds and music I did not have any extra time to research or explore around the topic.
Structure of the experience
The Émotive VR experience starts with the spectator sitting down on a swivel chair. An EEG headset (by Emotiv) is installed on the spectator’s head after which an HTC Vive and headphones are put on. Software related to the EEG measurements is already running, and the first scene of the Émotive VR Unity project is ready to start.
A 360 image of a web of neural connections surround the viewer in all directions. Short, high-pitched “sounds of neurones” whistle around. A steady synth pad grounds the soundscape in intention to create a relaxed feeling. After a few seconds a female voice welcomes the spectator to the experience and explains about the EEG measurements and the calibration that will take place next.
Audio/music workflow in a nutshell: - VO recorded into Pro Tools - Cleaned with iZotope RX, edited, slightly EQ'd and compressed, exported to Wwise - Neurone sounds played with theremin, recorded into Pro Tools, edited, exported to Wwise - Played back from a random container with randomised pitch and randomised positions in 3D space - Synthpad played with Pro Tools' Xpand!2, two separate tracks exported to Wwise - Spatialised in 3D - Routed through 2nd order Abisonic bus in Wwise with Auro-Headphones spatialiser
A series of IAPS (International Affective Picture System) images are shown to the spectator in order to calibrate the EEG sensory data for measuring valence (positive–negative). There is no audio in this sequence.
Similar to the first introduction, however this time the voice explains more about when and how the EEG measurements are used during the experience. She also welcomes the spectator to the 1938 Vienna where the frame-story (the prologue) is set.
Freud, la dernière hypnose starts with credits appearing on blackness. Music with theremin, piano and strings is playing.
Audio/music workflow in a nutshell: - Piano and theremin theme ("Freud's theme") played with Pro Tools' MiniGrand and a real theremin - String theme ("Karl's theme") played in Pro Tools with Spitfire Chamber Strings sample library - Exported to Wwise as a ready-made mix, using a headlocked stereo-bus
This is an audio-only prologue scene where Sigmund Freud reflects his experiences with hypnosis in the early days of his career. The spectator can hear street sounds coming from a window, a clock ticking in the room, Freud lighting a cigar, walking around on a wooden floor and talking. Visually there’s only a computer-generated trace of smoke following Freud’s movements.
Audio workflow in a nutshell: - Monologue recorded on location in Nantes with a boom and an Ambisonic microphone - After many tests I decide to use only the boom mic - Edited and spatialised in Reaper using IEM room encoder and 2nd order Ambisonics - Foleys for footsteps and clothes recorded in Le Mans, spatialised in Reaper - Street sounds constructed from my own field recordings - Played back from Wwise
The spectator finds herself in a late 19th century room with paintings on the wall, fireplace, bed, books on a shelf… We’re now inside a 360 film. There are two characters in the room: Karl von Schroeh, the patient, is sitting and making dyskinetic movements with his hands. Freud is standing, now younger and in the early years of his career.
The spectator is encouraged to make a choice between the two characters by gazing them for a while. When gazing at Karl the spectator hears his hand foleys and fragmented breathing. When gazing at Freud the spectator hears him making little noises with his mouth, coughs, sighs, etc. A visual timer indicates when the selection will be done. In the background we hear a short loop of music with theremin and piano.
Audio/music workflow in a nutshell: - Foleys and sounds edited from the wild location recordings, synced to the picture in Reaper - Foleys integrated and spatialised as game objects using Unitys' own audio engine - Music created in Pro Tools with theremin and MiniGrand software synth, exported to Wwise
The 360 films: Freud and Karl versions
If the spectator chooses Freud, an 11 minute long 360-film about Freud’s last hypnosis session with Karl starts. The story is told from Freud’s perspective, although the camera keeps jumping between first-person and third-person views. If the spectator chooses Karl the same story is shown from his perspective.
There’s no music in the films until about halfway when both Freud and Karl start hallucinating due to Freud’s hypnosis. That is when the interactive part starts: the valence level of the spectator affects the arrangement and feel of the music: when the measured and calculated valence level goes towards positive side more instrument layers and rhythmic content is added; when the level gets negative a re-harmonising effect and low-tones are used.
There will also be a graphical meter inserted somewhere inside the VR world providing the spectator with visual feedback of the valence level.
Audio/music workflow in a nutshell: - Dialogue recorded on location with wireless lavaliers - SoundField Ambisonic microphone running simultaneously to capture the room - Dialogue edited and cleaned using Reaper and iZotope RX - Foleys recorded and edited in Reaper - Sound effects created with theremin and other sound sources with several plugins in Reaper - Spatialisation done and synced with the 360-video using the FB360 toolkit - After several picture re-edits syncing audio with Vordio - Ambisonic 2nd order files and headlocked stereo files exported to Wwise - Videos running in Unity video player, synced with audio - Music created in Pro Tools using theremin and software synths and sample libraries, - Stems exported to Wwise for building the interactive music
After one of the films has finished the spectator is taken back to the choice sequence, where she can choose to watch the other version (or the same again) or exit to the epilogue. When gazing at the exit sign the spectator hears a distant sound of soldiers marching.
In the epilogue the spectator is taken back to the old Freud walking and talking in his 1938 Vienna apartment. This time we hear German soldiers marching pass the window, reminding of the due exile for this Jewish neurologist and his family.
At the time of writing this I don’t know how the end credits will be realised, but there was an idea to give the spectator a visual report of her emotional data recorded during the whole experience. In addition to that the end credit music will change according to the same data and thus giving an audible feedback of the experienced emotions. The interactive music is also spatialised so that each instrument group is positioned around the spectator in a 3D space.