3. – Jargon Watch: Spatial Audio
There is a growing consensus among technologists and content makers that truly immersive virtual environments and experiences need to be multi-sensory ones, which means shifting our focus slightly from the visuals and paying attention to the important role that sound plays in tricking our brain that it is somewhere else. As Oculus Chief Scientist Michael Abrash puts it, 3D sound is not an addition to VR, but a multiplier that infinitely enhances the experience.
The method for recording and reproducing basic 3D audio, also known as binaural audio, goes back to the late 19th century, and in its simplest form requires only two microphones - placed roughly in the same distance as is the average position of human ears. In a slightly more complex method, both of the microphones are placed inside ear-shaped bodies at an optimum distance from one another as well as the source of the audio, which allows the recording to more closely mimic real-life sounds.
Spatial Audio refers to the science behind "placing" sounds directionally around the user, creating a dynamic 3D soundscape where the relative position of people and objects actively influences the delivery of audio to your ears - and your brain. Spatial audio is designed to mimic the pitch, volume, reverberation level and other audio cues the brain would expect during a real-world experience. It allows developers to create content whose sounds can come from any direction. It achieves that effect in VR using software algorithms that manipulate a program’s sound wave frequencies, creating audio levels that become louder or softer depending on the user’s distance from a virtual object. The sound also shifts from one headphone speaker to the other as the person moves their head from side to side or as the virtual objects move on their own.
Humans are hardwired to pay attention to sound and instinctively use it to map their surroundings, and we have in-built natural filters hardwired into us at an early age, which are called Head Related Transfer Functions (HRTF). So in order to design realistic spatial audio landscapes, companies like Microsoft created complex algorithms to account for the fact that, for example, different head shapes affect the way that we perceive audio.
In recent years the immersive technology industry has taken these principles to an entirely new level, most notably with devices such as the Microsoft HoloLens, which advanced on research the company had done for the Kinect gaming system and applied it to Mixed Reality.