Behind the Detroit Opera Live Green Screen Adventure

DETROIT – A green screen, live actors, live musicians, digital cameras and a real-time game engine: is this the latest sci-fi fantasy coming to theaters? No, it’s the latest Detroit Opera venture opening this weekend at the Detroit Opera House.

While it might be a little harder to give an elevating pitch to this production, it’s this type of live experimental theater that puts Detroit on the opera front with Yuval Sharon at the helm. .

We spoke with Sharon, as well as Kaitlyn Pietras and Jason Thompson of PXT Studio who created the virtual world in which live performers live, about how this new technology brought Richard Wagner’s 1870 opera: Die Walkurie.

First of all, how did you get Sigourney Weaver involved in this?

Sharon: Since we’re only showing Act III, I wanted to make sure the audience didn’t feel like it was okay for them to come to this production without having seen the first two acts of Die Walkürieor the opera that precedes it, Das Rheingoldwhich is the first part of the Wagner cycle.

So to help make this piece stand out, it seemed important to have a voice to introduce us to the world of opera. Sigourney Weaver is the queen of science fiction, so she’s the perfect fit. I also thought it was a great opportunity to introduce us to the sci-fi aesthetic we’re going for, that kind of Tron meets the Matrix-like world we’re creating.

How did this idea of ​​working with a green screen come about?

Sharon: Well Jason, Kaitlyn and I have worked with green screen technology before. We actually did two other projects together in which we used green screens to help bring singers into a dimension and visual realm that is hard to reach within the three walls of a theater.

About a year ago, the LA Philharmonic asked me to conduct Act III of Die Walkürie for the Hollywood Bowl. And the Hollywood Bowl has these fantastic LED screens for image magnification, and it seemed like the perfect place to think about a production like this that would focus on that image magnification. So this focus on the jumbotron creates a dissonance between the live performance of the piece and the imaginary digital universe they would find themselves in.

Can you talk about real-time technology with the artists on stage?

Sharon: There are layers to this. We’re basically giving the audience a real-time video game and the singers are in front of that green screen and you’ll always be connected to the singers in real time. But the space in which they occur is very shallow and empty. The five digital cameras transport them to a digital environment that has been created using video game technology and a video game engine that has a three-dimensional component. So, although we have the singers live in a very shallow and empty space, they are also simultaneously participating in a kind of metaverse, this digital landscape that is created for them in real time by following the music. It is quite extraordinary.

Thompson: It’s the perfect project where the technology has gotten to the point where we’re now thinking about how video can react in real time. So we’re using Unreal Engine and it’s so powerful that the world is 3D. We send triggers to move virtual cameras inside and it all happens in real time every time we do. Shot for shot we create this cinematic experience and you get the duality of watching the performers on stage against a green screen and then also the composite happening in real time. It’s quite exciting and there are a lot of layers.

How has technology evolved to the point where we can host events like this?

Thompson: Technology is changing at an exponential rate, so the computing power we had when we started working together compared to now is completely different. It feels like light years away. Before, we waited for things to be rendered in 3D scenes. Now it happens in seconds. It’s happening so fast. And Yuval has always had such a cinematic spirit when he approaches operas like this. As we listen to the music, he sees these cinematic shots and rollovers and all these things and we are able to manipulate virtual cameras based on what Yuval sees.

Pietras: The first one we did was more of a 2D built environment that had a lot of Photoshop and AfterEffects in it. The second, all content was built on a program called Cinema 4D, which is more for motion graphics. It was more like waiting around rendering. This is the first time we’ve used game engine technology to create everything in 3D and also generate live stuff. So even things like snowfall, it’s completely different every time we see the show.

Sharon: At this point, the technology is incredibly exciting, but what’s really fascinating is how it really unlocks the music and unlocks that original story that Wagner demands with its massive, epic music. He tries to represent these eight women flying through the air on horses, and he tries to represent a magic fire that surrounds his daughter. These images can rarely be realized convincingly on stage. They’re often quite honestly quite embarrassing when the music is so amazing while the visuals of what’s actually possible on stage fall far short of the ideas you can infer from the music. So this technology feels like it’s able to realize that kernel of an idea that Wagner had and never could.

With green screen technology, you are pretty much limitless when it comes to design. How did you choose this retro look?

Sharon: It was a bit of trial and error. Our initial idea was a bit more naturalistic: mountain tops and rolling through the air with blue skies. But it just didn’t resonate at all with the music which sounds very futuristic, almost like science fiction.

So everything happened very quickly where we just said to ourselves: tron, we all love this movie. We all love The matrix. And we all responded to this aesthetic called vaporware of the 80s and 90s with this whole notion of the digital network and living on a digital plane was first starting to emerge and now feels very close to home which now feels very close to home because we’re doing this interview on Zoom. But tron et al seem really appropriate since it’s a fantasy, but in a sci-fi world. It is looking forward but looking backward at the same time.

What have been some of the challenges of using all this technology with live theater?

Pietras: Well, none of these technologies were designed to be used the way we use them. Video game software is designed for video games that are exported in a certain way and used by a user in a certain way. I lead a team of technical artists when we also didn’t know the software settings and I also try to explain to them how the theater works. So it was really cool to use this technology in a way that’s not meant to be used. It’s kind of our thing.

How do you think technology adds to the beauty of opera?

Sharon: The technology, as exciting as it is, the live performance is truly at its heart. In fact, the technology amplifies what the singers naturally bring to this piece by having the chance to have close-ups of these incredible singers. Christine Goerke owns this role of Brüunhilde – she has sung it all over the world. Having it here, but also being able to have an intimacy through this close-up that you don’t get anywhere else is such a gift. Alan Held, who plays Wotan, is an international star and seeing every nuance of emotion on his face brings this character’s life to the fore and it’s a truly remarkable experience.

So, with the performers live on stage and the giant screens displaying the rendered visuals in real time, how does the audience know where to watch?

Youval: There’s no right or wrong way to experience this: whether you only watch the screen, whether you only watch the singers, whether you close your eyes and just listen to the music. I like that there is a tension and that you have this choice of your own path through this performance. Ideally, you’re probably going back and forth a lot between the live image and the digital image. This means that each member of the public has their own experience of it. I was excited about this because opera opens up so many possibilities for experiencing and notion of these dual realities that you can choose how you want to compose them yourself. I think that’s part of the fun.

What was it like seeing this production at the Hollywood Bowl for the first time?

Pietras: The first time I watched it from outside the house, I really cried. It was really powerful. Were [Thompson] actually married and we have a son and he watched it with me.

Thompson: It’s one of the most impressive venues in LA and doing the show there is incredibly difficult but incredibly rewarding to be able to get there. It’s just amazing.

Are there any changes being made to the show as it heads to Detroit?

Pietras: We made a lot of improvements. The Hollywood Bowl was like proof that we could actually do this crazy idea, like a great proof of concept. I feel like we really took it to a new level. I think the whole concept we have is more suited to that environment. It’s indoors and the screens are a bit more accessible. I think it’s really more designed for this venue here and we’ve had a lot more time to tweak things and make little tweaks that actually had a huge impact on the whole thing.

One last word ?

Sharon: Jason, Kaitlyn and I have worked on so many projects that are incredibly hard to explain. But when you come to see it, you’re kind of like, “Oh, that’s what they were talking about.” It’s happened before even with the people who work here at Detroit Opera. But then they walk into the theater and their jaws are on the floor because they can’t believe what we’re actually doing. I hope it’s the same for the audience where it can still be very difficult to understand exactly what we’re doing, but if you trust us and come I think you’ll walk into this theater really experiencing something that you have never seen before.

The Valkyries is 87 minutes long and plays at the Detroit Opera House on September 17, 18 and 20. For timetables and tickets, visit DetroitOpera.org.