ARRI synchronizes video and light for the LIPS research project

LIPS stands for Live Interactive PMSE (Program Making and Special Events) Services. It is a research project funded by the German Federal Ministry for Economic Affairs and Energy (BMWi), in which several industrial and institutional partners work together on a professional production network. It is about the synchronization of audio, video, and light with extremely low latency to create the most immersive user experience possible. The LIPS project aims to interactively and immersively link live events performed at different locations by using new platforms and technical possibilities. The overall goal is to identify—based on newly developed IP-based audio/video production equipment—solutions to improve the access and participation of rural areas on creative, cultural, and political operation of metropolitan areas and vice versa. LIPS also makes significant contributions to inclusion, since it provides solutions which allow people with disabilities who are unable to travel to participate holistically and immersively in events.

LIPS consortium

To improve the immersive audio/visual experience for the connected users and to develop the technological background for connecting and merging various devices and locations into one production network, several parties had to bring in their individual know-how. Therefore the LIPS consortium was founded in 2018. It consists of the companies Sennheiser, TVN, Smart Mobile Labs, and ARRI, as well as the institutions Leibnitz University Hannover, Fraunhofer Heinrich Hertz Institute, and Friedrich Alexander University Erlangen-Nuremberg. Each partner has a different focus for the project. “ARRI was responsible for cameras and lighting with equipment and services,” explains Hermann Popp, Systems Architect at ARRI and technical project manager for LIPS. Within ARRI, numerous colleagues were involved with fundamental contributions.    

Use case networked music performance 

As a use case, the idea of realistic joint performances of musicians at remote geographic locations was born. One part of the student band—piano (Victor Ardelean Montelongo) and bass (Hans Koennecke)—was based at the University for Music and Theater Munich (HMTM), the other—drums (Max Dost), guitar (Julian Scarcella), and keyboards (Lukas Blecks)—570 kilometers (354 miles) north in Wedemark, near Hannover, at the Sennheiser headquarters. The playing of the “LIPS Brothers” took place in joint sessions, so-called field tests, which were transmitted via internet. A “virtual window” between the two linked rooms was created, which allows a visually and audibly plausible “look” in almost real-time into the other locations. In addition to the broadcast of video and audio, the lighting at both locations was also continuously adjusted. “The goal was to achieve the most immersive and authentic linking of these two exercise rooms. All participants should always have the same impression,” says Hermann Popp.

Both locations were equipped with the same setup and were connected via Wide-Area-Network with a data transfer rate of 1 Gbps. In both directions, six audio channels and one video stream as well as the lighting control information were transmitted.
 

Camera and display 

An immersive bidirectional image path with low latency was required. This related to the camera as well as the display at each side. The software and firmware of the cameras were specifically designed for LIPS by ARRI. These R&D cameras contain the same sensor and image processing like ARRI standard products with color filters matching characteristics of human eye receptors and high contrast capabilities. But they provide native all-IP functionality for image transfer, synchronization, and control. Each sends its image stream out of a 10-Gb/s-capable-Ethernet-copper-cable and, at the same time, receives the remote image stream over IP. A ProRes codec inside the cameras is used for image compression and decompression. Decompressed images were passed to a large format display via a synchronous serial digital interface (SDI) socket. Both cameras in Munich and Wedemark were synchronized via the Precision Time Protocol (PTP) and used the typical protocol stack for media production: Ethernet, IP, UDP, RTP, a media wrapper, and the compressed video payload. With the media wrapper MXF Live, data consistency and a seamless transition across stream and file was achieved.

LIPS aims a calibrated, linear image path from scene to image reproduction. With respect to displays, this required a defined and constant relation between a digital input value and the corresponding output luminance. The standard SMPTE ST 2084 defines a suited relation over a wide dynamic range and with an efficient perception-based quantization. After testing various kinds of displays large-format OLED displays were selected. They are able to provide this characteristic in a constant and adequate way.

Lighting 

Immersive integration of separate locations is essentially supported by matched lighting. At LIPS, natural available light was maintained as much as possible to preserve a natural vivid live environment and to avoid a studio feeling. To achieve this, available light, the spectral power distribution of light at each location, was measured and processed by a central lighting control. A common target value for both locations with respect to lighting color coordinates and lighting intensity was calculated. The target was achieved by adding some minimal artificial lighting. This was calculated specifically for each site.

This process was executed continuously and maintained identical lighting conditions at both locations even when the available natural baselight was changing. LIPS used LED spectral multichannel lighting devices from ARRI, which allow a smooth approximation to the calculated target value in a wide range. Color coordinates and brightness can be matched precisely. Two types of devices were in use at each location: two modified ARRI SkyPanel S60-C for a solid baselight in the scene, unfocused with large diffusors, and placed close to the windows for good superposition with natural light; and two ARRI L-Series L5-C Fresnel lights for scene forming and local enhancement, focused and directional with adjustable lenses.

LIPS final workshop in Munich  

After a long preparation period with system integration and networking, field tests were carried out over several months in Munich and Wedemark. During these sessions, students from both music universities continuously evaluated the development stages of the new technology, provided by LIPS. The final workshop including a joint concert of the musicians from the two locations took place on October 16, 2020 at the University for Music and Theater Munich (HMTM) and was streamed live on YouTube.

After such a successful event, how did the musicians experience the interactive concert? “99 percent realistic. The setup reaches very low latency,” judges Victor Ardelean Montelongo (piano) in Munich. “It could have been next door,” emphasizes Lukas Blecks (keyboards) in Wedemark Julian Scarcella (guitar) adds: “It is a great recording or rehearsal environment. But the interaction with the audience is still missing.”