Will Actual-Time Digital Units Exchange Inexperienced Screens?

Disney, Netflix, Amazon, Paramount and others rely heavily on digital sets, volumes and digital set extensions. Let's take a look.

Practical meets virtual. Digital devices are here to stay, and thanks to COVID-19, they are being adopted much faster. These systems are also known as XR or Cross-Reality and combine physical set pieces, cast and crew with digital sets or set extensions.

What differs from traditional rear projection techniques is that these sets are built into real-time game engines like Unreal Engine. The sets themselves can be changed during a shoot, which also means that the production world plays a new role.

At ILM (Industrial Light & Magic) they call this the Brain Bar – a group of 3D artists and technicians who work on set with the camera team and the lighting team to make these systems work.

A new role in production – the Brain Bar. Image via Disney + courtesy of ILMVFX.

They work hand in hand on the set with the top lights to light the set and adjust the colors. At the same time, they work with the camera team to track the camera position and lens settings so that the digital displays are at the correct angles, depths of field, exposures and more.

The 3D artists work with the camera team to track camera position, angles, etc. Image via Disney + courtesy of ILMVFX.

The biggest challenge? There aren't enough of these people in the industry. This type of job requires someone skilled in both filmmaking and video game production.

The ILM volume is a circular set 75 feet in diameter.

The circular set is 75 feet in diameter. Image via Disney + courtesy of ILMVFX.

It was originally designed for interactive light with a green screen background that an actor can follow. That way, there would be no need to re-light the screens between settings.

It was originally designed for interactive light with a green screen background. Image via Disney + courtesy of ILMVFX.

Producer and director Jon Favreau went to ILM to combine the filmmaking concepts and ideas he had from Elf, Iron Man, The Lion King and The Jungle Book.

Testing real-time environments during the production of The Lion King. Image via Disney + courtesy of Movie Trailer / PTP.

During the production of The Lion King, the film's VFX team began testing video game engines for digital sets.

The content of the screen has been tracked to the camera so that the position of the camera reflects a realistic view of the device through the camera lens. This enables perspective and parallax changes while filming.

The position of the camera reflects the real-time appearance of the device. Image via Disney + courtesy of ILMVFX.

Favreau credits Kathleen Kennedy of LucasFilm for providing the volume building resources for The Mandalorian.

The set is basically a huge lighting box. Image via Disney + courtesy of ILMVFX.

In the simplest way it is like in a big lighting box. When you place a subject in the room, your subject is lit realistically. And as Mandalorian's suit became more reflective with each episode, it gave the production team incredible freedom to light the Beskar helmet and suit.

The real cornerstone is that most of the technology is non-proprietary. LED screens, game engines, and 3D environments are all tools that humans have access to.

ILM is currently creating three more volumes around the world. Studio managers at Netflix, Amazon Originals and Paramount have announced that they are already using the technology and are building their own studios.

Now it's about getting the artists and technicians to push the concepts even further. So what are you waiting for?

Cover photo via Disney + courtesy of ILMVFX.

For more filmmaking tips, tricks, and advice, see the articles below: