More detailed information about the project can be found on the project website at http://resounding.place/
The vRSP network explores new forms of immersive performance experience in sites of historical and cultural import. It will start by examining new works by composer Michael Price who has been exploring the use of 360 video and audio to document performances in National Trust properties. The research network will examine how these performances have been recorded and converted to 360 videos, and offer new solutions using ongoing research from the University of Surrey’s Institute of Sound Recording, Center for Vision, Speech and Signal Processing, 5G Innovation Centre, and Digital Media Arts programmes.
Many spatial experiences are not well represented through current 360 video and immersive technologies. Despite Ambisonics being adopted as the de-facto soundfield capture and manipulation standard by Facebook and other 3D-audio content generation platforms, current 360 video experiences lack spatial audio detail and the ability to explore the soundfield convincingly. This leads to a considerably reduced immersive experience. While equipment manufacturers currently focus on increased image quality through higher pixel counts, there is currently a gap in both documentation and content of high-quality spatial audio which has been captured outside of game-engine based experiences.
The research network will explore how new forms of immersive experience can be created to allow future audiences to move through the space during the performance and explore the relationship between the performance and location. The network will also examine how extra layers or “maps” of the site can be added to allow the audience to explore both the history and significance of the site as well as the process of creating the work. Finally, the vRSP network will explore how an immersive experience can go beyond what is possible in a live performance to allow the experience to respond to the presence of the audience.
vRSP is developing outputs which are exemplars of a wholly new genre of art work, designed for virtual space but based on notable and rare historic places for which music will be written to integrate musical and social memory, and exploit unique architectural and acoustic environments. Live performances of newly composed works will be captured and disseminated to viewers and listeners as VR content through the application of a series of novel production techniques. These will enhance and address specific difficulties presented by current technologies, as described below, by applying interdisciplinary knowledge combining research from human perception, neuroscience, musical composition, animation, visual surface-mapping techniques and visual signal processing, stereoscopic 360 degree video and three- dimensional audio recording and reproduction methods. The network draws together experienced practitioners from all fields to inform the creation of a new work that will portray the intrinsic qualities of the historic performance space through site specific live performance and a combination of contemporary video- and animation-based techniques for VR generation.
The core project team includes Michael Price, one of the UK’s most sought-after composers, known for Emmy-winning compositions for BBC’s Sherlock and Unforgotten as well as work on films including Peter Jackson’s The Lord of the Rings trilogy, Richard Curtis’ Love Actually, Bridget Jones: The Edge of Reason and Alfonso Cuaron’s Children of Men. Alongside his Film/TV work, and signed with the label Erased Tapes Records, Price is releasing a new album recorded on six locations across the UK, ranging from a Tudor mansion to tunnels within the White Cliffs of Dover. Each piece was recorded onsite and composed in response to the site and its history. After enquiring with industry colleagues about methods of recording and reproducing immersive audio, Price began consulting directly with equipment manufacturers in search of solutions, becoming an associate member of the BAFTA-VR Advisory Group. vRSP will allow Price to work together with University of Surrey academics Kirk Woolford and Tony Myatt. Woolford is an expert on interactive media content creation with more than 150 interactive works created, 30 years’ experience developing immersive performances, including 20 years of VR performances, and 8 years’ work with AR. Myatt is the head of Music and Media at the University of Surrey, founding director of the University of York’s Music Research Centre, one of the foremost authorities on spatial audio. All three core partners are active creative practitioners and perform their works world-wide.
vRSP was funded as two AHRC projects at the University of Surrey, with Kirk Woolford as principal investigator, Tony Myatt as co-investigator and the composer Michael Price as industry partner. The first, “vRSP: virtual (Re)Sounding of Place” (AH/R010161/1), was funded under the AHRC/EPSRC Next Generation of Immersive Experiences call and ran from February to July 2018. A follow-on award for impact and engagement (AH/S010610/1) ran from January to May 2019.
The project looked for new ways to experience performance that are responsive, immersive and interactive, while also letting audiences explore the historic sites where the performances were made. New music by Price was written for historic venues, performed and recorded there, and distributed as virtual reality content. The work brought together site-specific performance, 3D audio, performance capture, live imagery, game-engine tools and VR/AR, along with research on perception, animation, visual processing, 360-degree video and spatial sound.
Partners included BAFTA, the University of Surrey’s Centre for Vision, Speech and Signal Processing (CVSSP), the 5G Innovation Centre, and games and VR studios in Guildford. The impact plan named film, broadcast, games and music companies, including Framestore, Double Negative, Supermassive, the Imaginarium, the BBC and The Foundry, with the Guildford School of Acting helping to shape actor training. The aim was methods, best practice and tools for creating convincing digital performers in 360-degree VR and AR. The 3D audio techniques were also relevant to games, film, education and simulation.
The first phase produced a prototype, built with CVSSP, in which a solo singer moves through a captured space. Viewers could circle the singer, but moving too close broke the illusion, because the low-resolution volumetric capture did not hold up at short range. The follow-on brought in Dimension Studios to re-capture the performance at professional resolution, so that audiences could physically walk through the virtual space and be inside a live music performance. The capture combined 360-degree video, mapped environment capture, sound and volumetric video.
The follow-on planned to co-produce three prototypes with partners and to share them through academic and industry networks. Planned venues included AHRC presentations, the G3 Futures games conference, film and VR festivals, and a showing to invited industry guests at Abbey Road Studios. Dimension Studios also connected the network to the Digital Catapult.
Source: UKRI Gateway to Research, grant AH/R010161/1 and AH/S010610/1. Kirk Woolford on Gateway to Research
