Harry Shearer is a well-known and respected comedian with an incredible career that is most notable in the main cast in The Simpsons (who plays as many famous characters as Mr. Burns, Waylon Smithers, Principal Skinner and Ned Flanders) and in films such as This is Spinal Tap (1984), The Truman Show (1998) and A Mighty Wind (2003). Since 1983, Shearer has hosted the public radio comedy / music show that is now a podcast. In his weekly program, Shearer alternates between DJing, commenting on the day's news and performing original (mostly political) comedy sketches and songs.
This year he released a new album, The Many Moods of Donald Trump. The music and words are from Shearer and were recorded primarily at Esplanade Studios in New Orleans earlier this year. The first single from Shearer's new album was Son-In-Law.
Shortly before the COVID lockdown, Shearer was in Sydney and decided to use motion capture to produce the music video of the first single to create a political satire version of the US President. When COVID went broke, Shearer got on one of the last flights from Sydney back to the US, leaving his production team in Sydney to figure out how to do virtual indie motion capture over the Pacific. This wasn't Shearer's first contact with MoCap – he'd been working on a similar project, which was a comic short film in which President Obama showed the newly elected Donald Trump in 2016 in the Oval Office. This new project is both much more advanced and remote controlled collaboration from both sides of the Pacific Ocean.
Motion capture was recorded by DOP Matthew Mindlin in LA and with the MoCap team in Sydney, Australia. The entire project was designed and produced by Harry Shearer himself. The Australian team focused on Matt Hermans of The Electric Lens Company who was the technical director of the project. His team was supported by the MOD Studios team in Sydney led by Michela Ledwidge, the director and editor of virtual production.
The project is a strong political satire visually based on two technical phases, a CG President Trump that went offline based on the MoCap shoot with virtual production techniques, and an overlay of this footage of a "deep fake" of the President Face trained on publicly available footage.
We spoke to Harry Shearer in LA about this latest project and his remarkable career that began with his first film appearance in the film Abbott and Costello Go to Mars (1953) and almost all creative media, music, books, films, TV shows , Radio and podcasting.
Shearer has won a Primetime Emmy Award and has received several other Emmy and Grammy Award nominations. He was a two-time regular performer on Saturday Night Live. In his career he worked alongside a pantheon of comic stars, including Jack Benny, Mel Blanc, Christopher Gast, Michael McKean, Al Franken, Rob Riener, Matt Gröning and many more. He also studied political science at UCLA, attended Harvard University graduate school for a year and even worked for the Sacramento state parliament.
For Matt Hermans of Electric Lens Company (ELC), this was his first project with Shearer, and when he first signed up he had no idea how world events would affect the project so completely.
ELC invited Mod Productions on board to run Son in Law's virtual production side. “It was our first LA shoot from Sydney, commented Ledwidge. "Our job was broken down into R&D, filming, cameras and editing." The challenge for the team was to develop a remote virtual production model that would be feasible during the COVID19 lockdown when only one person was allowed to be in the room with Shearer. The LA shoot only had Shearer's longtime DOP Matt Mindlin in attendance. Mod was responsible for setting up systems that could capture Harry's performance as Trump and his army assistants from across the world in a real-time interactive MoCap session. "Thank god I had experience with it," comments Shearer. Because when you first do this it can get a little overwhelming and confusing. But I had worn the suit before and made MoCap so that it was almost comfortable. "
The "shoot" consisted of Ledwidge and Hermans at Mod's Haymarket studio in downtown Sydney. Harry Shearer and Matt Mindlin worked at Harry's LA home.
Ledwidge previously worked closely with Hermans, working together on how to test the feasibility of Matt's idea of making a deep fake out of the digital Trump character instead of a live action clip. Ledwidge used an early beta version of Epic Games' Live Link Face iPhone app to record face data. Shearer wore an Xsens suit for body tracking and a pair of Manus Prime 1 gloves for his hands. Mod's ShowControl app, written in TouchDesigner, was used to record and monitor multiple systems from Sydney, even when the buttons on the iPhone app were remotely controlled.
UE4.24 was used to provide a real-time preview of the Trump character in a real-time "magic mirror". However, all performance measurement data (face, body and hand) was also recorded externally in UE4 to keep the traditional CG pipeline options open and flexible.
For someone known for being hilariously funny, Shearer takes his comedy pretty seriously. “I take the execution pretty seriously. I don't like anything better than being with coworkers I want to play with and just having a good time – making each other laugh, but there's a lot more to it than that. Especially when you're making music… there's nothing funny or entertaining about bad music, ”he comments. Hopefully, at the beginning of developing the sketch, song, or movie, you're full of fun ideas and make yourself laugh from time to time. And yet, if you come to put the pieces together to make something of it, then is it's serious work. "
CG hand asset
After filming, Ledwidge synchronized the rushes and set up an editing pipeline for the sequences in the Oval Office. The scenes in the press room and in the rose garden were composed in the Nuke Studio at ELC. For the face capture data, Ledwidge wrote a Python tool for Houdini's Procedural Data Graph to generate FBX files from ARKit data saved as CSV by the Epic Live Link Face app. She then edited the piece with the UE4 Sequencer using character animation files that had been cleaned up in Maya at ELC, and recorded virtual camera movements with a hand-held Red Rock rig with HTC Vive tracker. Once editing was locked, Mod returned a base UE4 project to ELC to do the final visual enhancements and rendering using the real-time lighting settings from UE4 RTX.
There were three levels of quality for the Trump character. The proxy used for real-time preview captures real-time mocap shooting and editing, real-time RTX rendering, and the final appearance with a deep, fake face pieced together over the real-time character.
Given the complexity of this independent trans-Pacific virtual production, it took the video a while to reach a state that Shearer was happy with. “We've all tried to get three or four different technologies to play well together. And it's almost as difficult as getting three or four people to play nice together, ”he notes. “I think it was unprecedented in terms of all of these combinations and individual things… So everyone learned. We just shot the second video for another song in this collection, another Trump song that will be out in mid-September, and we've all learned from the first experiences, so it went faster. "
UE4 Oval Office
The project showed how remote working can support new ways of creative filming. ELC and Mod Productions continue to work with Shearer to refine how character design and storytelling can be combined in the virtual production for the second movie clip in production. Hermans admits that the new clip has some major advancements and improvements, particularly in the rigging of the body and fabric. Given the tiny crew, long distances, and great complexity of the first Son In Law project, the entire team was incredibly pleased with the results.
Shearer Would Like To Do More "It has always been my goal to take what I do on the radio in terms of the sketches I do, where I play all the characters and put them on a visual medium," he explains . “In this project I still only play one character and in some other experiments with this technique that are not that developed I played two or three characters in one scene, which really got me excited. For me there is more space to play in this special room. "
For more Harry Shearer, visit Harryshearer.com, listen to Le Show or follow him on Twitter