Oscars: Sound Mixing 101

Leading up to Sunday’s Oscars, EW.com will take a closer look at four categories that moviegoers may mistakenly think of as “technical.” After tackling Film Editing and Sound Editing, we now turn to Sound Mixing, with insights from nominees from Les Misérables, Skyfall, and Argo. Lincoln and Life of Pi complete the category. (Update: Cinematography completes this series.)

“The biggest misconception about what we do is that most people think it’s technical in nature because there’s intimidating mixing consoles involved. In reality, we are extremely creative in the use of sound to help tell the story,” says Argo sound re-recording mixer Gregg Rudloff, a five-time Oscar nominee with wins for his work on Glory and The Matrix. “But it goes back to what do you want people to notice and not notice? We want them to be watching the film and experiencing it and enjoying it. We don’t want them to be sitting there going, ‘Oh wow, that was a really cool sound thing that guy just did there.’ We don’t want to put them back in their seats. We want them to be totally immersed in the film. So in a way, we’re our own worst enemies. If we do a bad job, people notice what we do, and that’s not what we want. If we do it really well, people don’t notice it so much, but then they don’t understand the depths we’re working at.”

To fix that problem, we asked some of this year’s sound mixing nominees to walk us through key scenes to explain their roles and show us the kind of decisions their making. But first, the difference between sound mixing and sound editing, a separate Oscar category.

Essentially, the sound editorial team is responsible for assembling every sound you hear in the movie except the music and the dialogue that was recorded on set.

It’s the job of the production sound mixer to capture that dialogue as cleanly as possible to preserve the actor’s original performances, so sound editorial doesn’t have to have them do ADR (Automated Dialogue Replacement). Post-production, the sound mixing team takes all the material that has been prepared by sound editorial — dialogue, atmospheric sounds like the wind and ocean; hard effects, which are synched sounds for things like cars, gunshots, explosions, and helicopters; and Foley, or nuance sounds such as footsteps and people touching things with their hands — plus the score, and affixes them to the screen and positions them in the theater while also setting their volume levels relative to each other. “We get tens of thousands of pieces of sound in a film. It’s like a big chemistry set: How much of this do you put in to that? And how much of this do you put in to that, and how will that affect this?” says Skyfall re-recording mixer Scott Millan, whose nine Oscar nominations include four wins for his work on Apollo 13, Gladiator, Ray, and The Bourne Ultimatum. “All of us who do this work have a sensitivity and a touch that’s pretty unique. If you gave the same content to 10 individuals and said go ahead and mix this movie, you’d get 10 different perspectives. They wouldn’t be the same. And that’s what’s so much fun about it. It’s almost like handing sheet music to a bunch of musicians: How they touch it, and how they touch it collectively together as an ensemble, is unique. If you change any of those players, it would be different yet again. “

Since it all starts with the dialogue captured on set, let’s begin there.

NEXT: The art of production sound mixing

“The production sound mixer on any show is responsible for preserving the actors’ original performances and making sure that the feelings and the emotions during a scene on set are carried all the way through from the production process into the post-production process,” says Les Misérables’ production mixer Simon Hayes, a first-time Oscar nominee. “If the production sound mixer has done a good job, and recorded so perfectly that background noise won’t become an issue, that is the performance you see on-screen in the cinema.”

With director Tom Hooper deciding to have his actors sing live on set — piano players played in their earpieces, allowing the actors to dictate tempo — Les Misérables is obviously a showcase for the crucial role the production sound mixer plays. Hayes points to the scene in which Eponine (Samantha Barks) is walking down Rue Plumet singing “On My Own” in the rain.

Hayes: It’s a very, very challenging job to be able to record her singing clearly enough so that you don’t hear the rain overpowering her voice, so that you don’t hear the footsteps of the Steadicam crew underneath her voice, so that you don’t even hear her own footsteps, and all you’re trying to do is hone in on just her voice and record it as cleanly and upfront as possible so that the background noises on the set don’t in any way stand between the audience and the emotional performance of the actress. Every single piece of set that you don’t see on camera, we cover with soundproof material so that you don’t hear the rain hitting the set. We carpet the whole walk so that you can’t hear their footsteps. And it’s also a case of getting the microphone absolutely as close to the edge of the frame as possible. So literally, the boom microphone that recorded that scene is touching the edge of frame for the whole of her whole walk, and knowing exactly from the size of the lens the Steadicam is using how close he can get. And what we’re trying to do is to get the microphone absolutely as close possible, because the closer we get the microphone to the performance, the more of an emotional connection there is for the theater audience.

Though Anne Hathaway was stationary for “I Dreamed a Dream,” it presented its own challenge:

Hayes: We all knew when we were recording “I Dreamed a Dream” that we were witnessing something extremely precious and extremely special. Annie was in an extremely emotional state while she was singing that song, and what we tried to do was give her absolute freedom to go with whatever performance she wanted to do. And we started shooting without a rehearsal, because Tom and Annie decided that rehearsing could possibly waste a take. So we went into that not knowing whether Annie was gonna go loud or soft, and in fact, what we found out was Annie was gonna go loud and soft, so we basically mic’d that in a way that would be as unobtrusive as possible for her but really be able to grab the emotional aspect of her performance. And we did that by using a radio mic on her chest and also a boom mic, again right on the edge of frame, and we found that the boom microphone was really was fantastic on the louder pieces and on the emotional moments in between the words — for the breaths — the radio mic was a really important part of it. So I was really pleased to be able to use both of those mics so that afterward, we could do a blend of the pair of them and give the choice to Tom Hooper, and Andy Nelson, the re-recording mixer [who shares Les Misérables’ nomination with Hayes and re-recording mixer Mark Paterson], as to which microphone they felt was most appropriate for each piece of the performance. The re-recording mixer was taking my vocal recordings and blending them with the music and deciding what the overall balance between vocal and music would be in the finished product.

Hathaway’s factory number “At the End of the Day” was challenging because it involved blending her voice with 25 other women’s, Hayes says. “We were trying to be sure that Annie’s came through and every word of hers could be heard, but also that the other girls were beautifully recorded as well. And so we were really like artists playing with a palette of colors.”

Argo’s production sound mixer, first-time nominee Jose Antonio Garcia also had an added challenge because he and director Ben Affleck decided to let actors overlap on dialogue. “When you’re shooting a single of an actor, off-camera people generally have to wait for them to deliver the line, so the dialogue on-screen is cuttable. Then there’s no actor on top of him that would force you to use that overlap. Ben and I discussed it, because once you allow that you are married to that overlap, and we basically mic’d everybody at all times,” Garcia says. “That gave the performance a higher level of realism.”

NEXT: The art of the final mix

Skyfall Daniel Craig Javier Bardem — Francois Duhamel

Usually, the re-recording mixers (one typically handles the dialogue and music, and the other sound effects) will do a broadstroke first pass on the director’s first cut of the film independently of the director. “It’s what we think things will play like and what content we need in the film. Yet, we keep it all very separate, so that the choices can be changed when the filmmaker arrives,” says Skyfall’s Millan, who shares his nomination with re-recording mixer Greg P. Russell, a 16-time Oscar nominee, and twice nominated production sound mixer Stuart Wilson.

Once it’s time for the final mix, it’s a serious group effort. “When we get to the final, we have the director with us on stage, the sound editorial supervisor — the person who is responsible for assembling, finding, and creating all the sounds that we would ultimately be dealing with in the mixing process — almost always the picture editor, and often the sound editors as well. We’ll have the dialogue editor and the sound effects editor there,” Millan says. “Because of the parallel process of finishing films now — meaning the schedule’s so compressed and the picture is changing as we are in the process of making our final mix — things have to be more virtual than ever before. Even though we’ve created material that we thought was appropriate, the filmmaker may have another choice, or a visual effect may have changed or have been added that we didn’t know about. So there’s this constant evolution during the final mix of creating and integrating new content into the pre-dubbed material. “

Millan has worked on every one of director Sam Mendes’ films, so he understands and appreciates the director’s sensibilities when it comes to using sound to help tell the story. “The slightest nuance of the tone of a voice or the subtlety of a performance and how it might be dynamically supported is important to him,” Millan says. He points to the memorable scene in which Bond (Daniel Craig) and Silva (Javier Bardem) first come face-to-face.

Millan: One of the simplest scenes we did in the movie that took us one of the longest periods of time to get the right emotional tone was Javier Bardem’s entrance to the movie. It’s a very simple scene, but in a Bond movie, very gutsy. You have a five or six-minute scene between two principles, toe to toe, no music, no sound effects, no gunshots, nothing driving the sequence. The ambience of that room had to suck you in to what was going on between them. The vast space put on the voice is giving you the cavernous, cold, removed sensibility, yet everything had to be really intimate. Traditionally, a filmmaker might say, “Look at the room, there’s 1,000 computers in that room, and I don’t hear any. I need to hear tones and beeps and boops.” But Sam kept going in the opposite direction and pairing it back and pairing it back, looking for something that just sucked you in and engaged you in a way that wasn’t telegraphing anything. It was really allowing these two men to carry it. Again, that takes a lot of guts in a Bond movie, where usually the audience is looking for action and they’re looking for maybe quicker cutting. But to stay on certain shots and to allow this just to be performed, Sam spent a lot of time on that.

And to contrast with that, take the opening of the film, which is almost pop to pop action. That was one of the last reels we mixed with Sam. Our first take and mix of the reel before he was there was not far off what ultimately ended up being in the movie, but when he first looked at it, he wanted to start paring things back and getting it simpler and simpler and allowing music to drive it even more so. You go from all-out action to the quietest scene and he gives it equal amount of attention. But it’s really about emotionally telling the story and supporting the story. He has said at different times, “You’ve given me everything but nothing is important to this sequence.” It wasn’t a criticism. But he’s that focused on what’s on the soundtrack, and I love it. It makes me have a different sensibility that allows me to look and think about what’s on the screen multi-layers down, not the obvious. Just because there’s a car in the sequence or a gun, or someone in the background walking by, something might think, “Well, it’s there in the image, there should be sound for it,” but that’s not what’s important. What’s really important in storytelling and film is that you envelop the audience and they don’t think about that. They don’t think about the fact that it’s not there, but they don’t think about the fact that it is there.

Another example is one of Millan’s favorite moments from Mendes’ second movie, Road to Perdition.

Millan: One of the scenes that people have told me is their favorite, and one that I also enjoy tremendously, is when Paul Newman and Tom Hanks have the showdown near the end of the film, where Tom actually kills Paul Newman’s character, who is pretty much his adoptive father. Throughout the dubbing process, Sam was trying all sorts of different concepts about how to deal with that moment: Should it be stylized? Should it be literal? You see things in the long perspective, you see gun flashes, but you don’t see the image of who’s shooting. And then you see people in the foreground dying, and they shoot. We had all these different sounds, and rain was going on at the same time, and I kept trying to get less sound in that sequence. And it was really the 11th hour, and Sam hadn’t felt confident that he’d got the exact right emotional impact of the whole sequence, and we were talking about it, and I finally said, “Sam, why don’t we take everything out and just use the score? I think up to this point, everybody knows what’s going to happen. There’s a showdown coming — Tom Hanks in the scene before says, “There’s one last thing I have to do, and then it’ll be over,” and you know what that is. And he shot it like it was an opera — people dying in the foreground, people shooting in the distance. This kind of ominous sensibility. So he agreed, let’s try it. Tom Newman had created this beautiful piece of music, and we did it. It’s not what you see, it’s not what you would anticipate. That scene became so much more powerful by the absence of sound, so that when Tom Hanks walks down that street and ends up face to face with Paul Newman, and they do have a short dialogue scene and Paul Newman says, “I’m glad it’s you,” and then Tom slowly raises that Tommy gun and pulls the trigger at close range — the sound and impact of what happened was far more enhanced than if it was just loud and the things before that were there. Emotionally it brought it full circle.

NEXT: The art of Argo‘s tension

How many times have you heard someone say, “I was on the edge of my seat, and I knew what was going to happen” after seeing Argo. When something works that well, it starts with great writing, continues with great acting, directing, and picture editing, and ends with great sound. The Oscar-nominated sound editors took us through key scenes here. Now re-recording mixer Rudloff, who shares his nomination with production sound mixer Garcia and fellow re-recording mixer John Reitz, a five-time Oscar nominee with a win for The Matrix, tells us how they created that tension.

First, the scene after the prologue:

Rudloff: You’re instantly immersed in the demonstrations outside the Embassy, and then you’re taken through the takeover of the Embassy where the hostages are taken. Ben, as the director, wanted the audience to be just dropped into the middle of this utter chaos. You’ve got the screaming demonstrators, the anger, the hatred that they have both for America and the Shah. Then you cut inside the Embassy. When we first cut inside the Embassy, everything is kind of calm because this isn’t an unusual event for them — they’ve been experiencing these demonstrations — but as the scenes progress, they start to realize, wow, this is going beyond what we’ve been seen before, and eventually it becomes extremely chaotic for them. Now you’ve got their fear and their anxiety. So there’s all these raw emotions happening either outside with the demonstrations, and eventually inside with both the Embassy people and the Iranians who are inside the Embassy. So as the tension is starting to ratchet up, we started to mirror that with the mix. So early on in the scene, we’re more laid back. Once we’re inside, you still hear the offstage crowd, but it’s subdued to an extent. But as things start to progress and become more tense, we start increasing the rawness and edginess of the tracks that we’re playing. We start making abrupt cuts location to location. We start having sounds that jump out at you. It’s not smooth transitions from one camera angle to the other, or one location to another. So our mixing style actually starts to mirror what’s happening in the story. This is one of the things that we don’t necessarily want the audience to recognize that we’re doing, but what we’re doing with the sound is actually affecting the way that they’re perceiving the story. Ben wanted them to be on edge. He wanted them not to just be sitting back comfortably in their seats and watching this transpire. He wanted them to be intimately involved in this experience, so we used a style that was much more aggressive than in other areas of the film or that we may use in other films.

In that particular scene, in order to add to the realness, Ben chose not to use the score. Score is a wonderful vehicle for carrying the emotion of a scene, but audiences do kind of expect it at times. So the fact that Ben chose not to use it in that circumstance added to that rawness and realness that he was looking for. So again, we used the sound to help set the tone and emotional arc of the scene and we ratcheted it up just as the visuals did in the editing of the film. Billy Goldenberg did a wonderful job with that scene, because he would ramp the tension up in a location, and then he would break away. And then on the next location, he would start it and ramp it up again.

Throughout the film, Affleck had been trying to reinforce that the American houseguests were surrounded in a foreign land. It crescendos with the climactic airport finale.

Rudloff: When they first get inside the airport, there’s that parrot, and a real loud squawk of a parrot. It’s a story point that doesn’t really have anything to do with the story of these houseguests trying to get out, but it added another jarring moment for the audience. So we had all this hustle and bustle that had been going on throughout the film to keep not only the characters but the audience a little on edge, a little uncomfortable. Once they got in the airport, they had these checkpoints that they had to get past, each one was like its own hurdle. So we kind of ramp up to the first hurdle, and then they would get past that. And then they would ramp up to the next one. And here, again, it’s the choice of using sound or not using sound. When it got down to the stamp, the whole focus was that stamp of approval. They weren’t gonna get past that hurdle if they didn’t get that stamp of approval. We’d created all this hustle and bustle for the build-up to this, and then we started pulling back in and shedding some of those layers of sound. Slowly everything started to drift away, and everybody — the characters in the film, the audience — is just sitting there waiting to see if they’re gonna get that stamp. Ben is choosing to use the lack of sound in this case to focus the audience on what was important, and that happened to be that stamp. And when the stamp of approval is finally given, all the background sounds pop back in, and we’re back in reality.

How do sound re-recording mixers know when they’re ready to sign off on a scene like that? “When Ben says it’s done,” Reitz says. “When he’s happy with a scene, then we concentrate on the next one. That’s just the way it is.” Says Rudloff: “I asked the same question when I first started mixing. I used to say, ‘How do you know when it’s right?’ Over time, you learn you just have a feeling when it’s working. Unfortunately, there’s not a list of criteria you can point to and say, ‘Well, as long as I’ve done A, B, and C, we’re done.’ There’s always something else you can do. It’s subjective. It’s creative. You can always try something else. Like John says, ‘When the director says we’re done, we’re done.'”

Oscar-nominated cinematographers explain how they envelop you in the story

Oscar-nominated editors clear up the biggest misconception about their category (and explain the decisions you may not know they’re making)