Composing interactive music for a project like this would have been enough in itself for a three-month long involvement. However, I did not think about that too much when I first promised that I can make the music alongside the sound design. Later it turned out that music was actually the only interactive element in the experience and thus in the very core of the whole project. Knowing that earlier I could have spent more time on music, but then there should have been another person to work on the sounds.
The first bit of music I composed was the choice sequence music, an eight-bar long loop with theremin and piano. As the story is set in 1891 and 1938 Vienna with a lot of nervousness, anticipation, winds of change, mixed feelings, etc., I immediately got an idea to go for expressionists such as Schönberg and Webern for inspiration. I also wanted the music to be intimate, not orchestral or synthetic, but something that could have been played in a bourgeoisie Vienna home around the turn of the century. Piano, string quartet, etc. I also wanted to keep using theremin as I had already used it for the hypnosis sound effects. It would also add a level of eeriness and mystery that floats around the topics of hypnosis and Freud's psychotherapy.
I quickly found a piano version of Arnold Schönberg's Fünf Orchesterstücke, part III "Farben", which I thought would match the atmosphere of the story. The piece is composed in 1909 and, in my opinion it has a nice blend of calmness, peace, anticipation and nervousness. Conveniently enough, just like Sigmund Freud, Schönberg was also Austrian and from a Jewish family living in Vienna. Some of Schönberg's works from the same period has also been interpreted in the context of Freud's psychoanalytic theories, although that is a topic I don't know anything about.
The first version for the choice sequence music I composed used very much the same chords as the beginning of Schönberg's piece. The director liked that very much, and I moved on to other tasks. However, I was not entirely satisfied as the piece I had done felt more like a copy of the original than a composition of my own, so after a while I made a totally new piece using similar approach to chords and rhythm.
In the choice sequence the spectator can choose whether she wants to follow the story from Karl's or Freud's perspective. The choice is executed by gazing at the character for a few seconds. While staring either of the characters one can hear him producing some sounds, Karl playing with his hands and Freud coughing and making little sounds with his mouth. However, the original idea was that when Freud is gazed at the theremin melody fades up, and when Karl is looked at something else in the music changes. At some point we abandoned that idea and went for the foleys, so I just mixed the theremin so that it is the solo instrument all the time.
In the project there are three places with interactive music. In both films there is an interactive sequence where music starts to play and the EEG data is controlling it. Also the end credits are accompanied by music reacting to the emotion data collected during the whole experience.
In the films both Karl and Freud are seeing and hearing hallucinations. The overall mood is a bit daunting and depressing. As I already mentioned there were supposed to be two emotion parameters to be measured from the spectator, valence (positive–negative) and excitement (low–high). The director asked me to create interactive music that would reflect these emotions. The idea was to give feedback to the spectator of her/his feelings during the films.
Technically that would relatively easy, but there were two big questions:
1. How much can we allow the brain data to change the mood of the music so that the music still supports the story and the characters's feelings? For example there is a natural rhythm in the scene created by the actors and the director, but the spectator's excitement level might dictate a different energy level for the music. Will that lead to a fruitful contrast or will it break the flow and illusion?
2. What does positive or negative valence mean in terms of spectator's experience and how should that be reflected in music? For example if someone feels positive about the murky scene should the music change to happier or murkier?
With these questions hovering in my mind and after watching the raw version of the Freud hallucination scene I started to create some musical material for that. Using a MIDI piano connected to Pro Tools' MiniGrand software synth I started to improvise. I came up with a theme clearly related to that in the choice sequence. For the interactivity I simply decided to make multiple stems with different layers of textures and instruments and mix them according to the valence and excitement level.
For raised excitement I made a piano bass line using the 12-note system played with 16th notes. I think that sounded great and really fitted well with the crazy hallucination scene and Schönberg influences. Later it turned out that we could not measure the excitement level, so finally I had to redo the mapping and the 12-note bass line was dropped out.
For the changes in valence I added strings and higher octave material when going to the positive side and re-harmonising effects and bass content when going towards negative. The strings and higher content surely add new timbres to the music, but they don't contribute very much to the overall change of mood, so I don't know how much the spectator notices the changes. The re-harmonising effect on the other hand really makes the music crazy and murky, so that works fine. It was also very easy to create.
Planning multiple musical layers that all work together, performing them with Pro Tools, editing, transferring them to Wwise and creating the interactive stems there was a surprisingly slow process. Had it been faster and having more experience I could have concentrated more on the music itself.
For the Karl music the director suggested I used strings instead of piano. With a background in cello and chamber music I instinctively approached the task through a string quartet ensemble. I came up with a simple progression idea using parallel intervals in violins and a counter melody line in viola and cello.
The only problem was that although the music worked with the scene as it was, I could not come up with any ideas how it should change according to the emotion data. I postponed working on the Karl music until the very end of the project, which was actually a lucky decision, because only then we got the confirmation that only one parameter, the valence, was available. That obviously made the musical interaction much easier to create. For positive valence I added some faster textures with harp and strings, and for negative I used the same tools as with the Freud version: re-harmonising effect and lower notes.
I recorded and created all the music in Pro Tools. For the piano I used Avid's own MiniGrand software synth. For the strings I first used String Ensemble library from Native Instruments' Symphony Essentials collection. It has, however, quite artificial and "plastic" timbre without much details, so after returning home I invested in Spitfire's Chamber Strings library and remade all the strings with that.
Unlike with Freud I spatialised the Karl music in 3D. In Wwise I routed the music to an Ambisonic bus and assigned a different 3D position for each instrument. When the spectator turns her head the instruments should appear as if floating in the air around. As the last weeks were quite hectic and we didn't always have the HTC Vive at our disposal, I never had an opportunity to actually test the music with a VR headset as a part of the scene! I just hope it works.
For the opening credits I simply combined the Freud and Karl themes. The music starts with the Freud theme played with theremin and piano. The melody is framed by a very high and a very low long string notes creating a layer of suspensions. Soon a string quartet dwells in, and we are in the middle of the string progression of Karl's theme. The transition point between the two themes was surprisingly easy to find. Also when there became a need to make the music a bit longer it was easy to just extend the Karl strings progression as it is a flexible structure rather than a real melody.
The idea for the end credit music came only during the last weeks while visiting the Nantes team. The director wanted to give the spectator final feedback of the emotions felt during the experience. Instead of using just graphical charts or bars, she suggested making different version of the music, one for positive valence, one for negative, etc. The corresponding music would be selected according to the average emotions calculated during the watched films.
The idea was good, but as professor Vigier at the University of Nantes pointed out using the average value of the emotional data would not describe the experience correctly. I then suggested that I could create a piece of interactive music for the end credits. We could then record the changes in valence levels during the two films, and when the end credits start we could control the music with the recorded data. The data set from 10 or 20 minutes should of course be condensed into three minutes or so. Thus if someone's valence level stays constant during the film(s) the end credit music will not change much. On the other hand, if someone experiences rollercoaster emotions the music will also jump around accordingly. The concept was approved, but I did not have time to start with the music until I returned home.
As with the opening credit music I recycled the existing themes for the end credits. It starts with the Freud theme played three times. Instead of just creating multiple layers for different valence levels, I also rearranged the melody, which originally goes in a minor key, to play in a major key when the parameter rises above a certain level. The transition could not be realtime anymore, that would have sounded weird. Therefore I used Wwise's interactive functions to let the changeover happen always at the following bar line. The tempo changes I had used in the music caused a little bit of headache for me as Wwise is clearly made for fixed tempo material. However after some workarounds I managed to keep the changeovers at the bar lines even after a new tempo.
The major key version of the Freud theme sounds quite cheesy to me, but on the other hand, it servers the purpose of giving feedback of the recorded valence level. Of course we come back to the question what"positive valence" means in the first place. A melody in a major key might not represent what the spectator has felt when the EEG sensor has detected signs of "positive valence". But with EEG or even with fMRI we will not be able to see anyone's true feelings, so maybe we should not stress about that in this project.
After the Freud theme the music proceeds to the Karl's string progression. As that is neither in major or minor key, I could not make a similar rearrangement than in Freud. Instead, for the positive feedback, I just added the neurone theremins from the introductions. They are played back in 3D as in the introductions. For the negative levels the re-harmonising effect and some bass notes appear.
The music ends with the choice sequence music. This time the positive feedback is marked with added arpeggios.
Next:
Previous: