Search This Blog

Emotions in "Inside Out" Movie, Dr. Paul Ekman and Science of Emotion

I just saw the new Pixar Movie "Inside Out" and the body language  in it was wonderful and I absolutely loved it. It is touching and heartwarming film that educates children on the role of emotions and memory in our lives.  There is one particular effective scene in the movie in which the character Sadness listens to the character Fear and helps him deal with his pain by mirroring his body language and reflecting his emotions.  Yes, a cartoon charterer used "Mirroring" and Reflective Listening."  A powerful message in the movie is that the emotion of sadness can be a important part of life and can actually move you to reflect on your life and make changes. I didn't begin to learn that until I was in college and my father passed away. How incredible for a small child to know that sadness does not make them bad, unlikable and less than. As a body language expert I knew I was going to be intrigued with the way each emotion was animated. I thought about all the research I have studied over the years about emotion and facial expressions done by Dr. Paul Ekman.
I was pleased to see he was consulted by Pixar for the move.
Here is an article on the science of emotions in the movie Inside Out.


Near the end of the closing credits of Pixar's Inside Out reads the line:
Special thanks to Dr. Paul Ekman and  Dacher Keltner for guiding us through this emotional journey.
Paul Ekman is well known to most readers as the father of FACS, the facial animation coding system that is now the cornerstone of facial animation rigs/blend shapes, and Dacher Keltner teaches at University of California, Berkley. Director Pete Docter (Monsters, Inc., Up), first learned about FACS and the work of Paul Ekman while serving as a supervising animator on the original Toy Story. Pixar used FACS even then as part of their animation process. Keltner's work came later, as he actually studied under Ekman (three years of post-doctoral work) and his research has led him to be a professor of psychology at Berkeley, where he directs the Berkeley Social Interaction Lab. His research focuses on the biological and evolutionary origins of compassion, awe, love, and beauty, and power, social class, and inequality. Keltner is also the co-director of the Greater Good Science Center. “I’ve spent 25 years of my career studying human emotion,” he says. “I’m interested in how we express emotions in our faces, voices and in touch.” Keltner influenced the idea of emotions playing off one another and almost co-existing, the fine line one has between emotions and how they are all key to a balanced individual functioning with others in a social or family group.
Much of Inside Out takes place inside the "Headquarters" or control room of 11-year-old Riley. There we follow her five Emotions all hard at work trying to make sense of the world. There is no universally agreed finite set of emotions. In fact, most people struggle to even define what emotion is or rather the boundaries of it. Like a mountain - most people can tell when they are on top of a big one - but where it actually starts or stops is rather harder to say. Given the lack of even a solid definition of what is or is not an emotion, Doctor turned to Ekman's research which states there are six fundamental emotions - joy, fear, disgust, anger, sadness and shock. Docter decided to fold shock into fear and the film had its five lead character emotions. Docter, himself a father, watched his own daughter enter her teen years and so felt this set the stage for some great character animation.
Before ending on the final five, the team did experiment with other characters such as Pride, "He would come in and say - 'Oh Joy I noticed no one applauded when I walked in the room' - very snooty with an upturned nose," explained Docter talking in Sydney recently. "Dr. Keltner’s work suggests that there are 21, with emotions like boredom, contempt, and embarrassment. There were so many possibilities in terms of character. It was fun to explore. We ultimately landed on five Emotions that pretty much make all of the researchers’ lists.”

Some of the rejected characters that the team explored along the way included "Schadenfreude - you know with a German Accent - "Your cries of pain amuses me!" Docter jokingly explained, but in the end the six key Ekman condensed into five seemed the right cinematic number. "Five seemed like a good group, as Surprise seemed like he might be a bit of a one-note character - just surprised all the time!".
In the film, there are these 5 emotional characters and also 'real world' characters such as Riley's parents. To present them differently the 5 emotions are not solid in their final render but rather boiling or constantly fizzing based on the idea that these emotions are in the mind and made therefore of energy, not solid matter. To add to the rendering complexity, the lighthearted and optimistic Joy (voice of Amy Poehler), emits lights. All of the other emotions - Fear (voice of Bill Hader), Anger (voice of Lewis Black), Sadness (voice of Phyllis Smith)   and Disgust (voice of Mindy Kaling) have the boiling skin in varying degrees, and to complete the challenge all the female character has strongly anisotropic iridescent hair.
When Riley's family relocates to a scary new city, the Emotions are on the job, eager to help guide her through the difficult transition. But when Joy and Sadness are inadvertently swept into the far reaches of Riley’s mind—taking some of her core memories with them—Fear, Anger and Disgust are left reluctantly in charge. Joy and Sadness must venture through unfamiliar places—Long Term Memory, Imagination Land, Abstract Thought and Dream Productions—in a desperate effort to get back to Headquarters, and Riley.
Director Pete Docter explaining the fundamental shapes of the 5 emotions in Sydney this week
Docter and the team designed each of the characters as representative shapes - Anger is, for example, a block - Fear is a raw nerve, long and angular, Sadness is an upside down teardrop, Disgust is based on a spear of broccoli and Joy is an explosion or star.
When it came to Joy (and the rest of the Emotions), the production team was committed to getting it right, committing resources, technology, imagination and research. “It’s all about the Emotions—they’re running the show,” says Docter. “We can control how we act, but we don’t get to choose how we feel.”


Joy


“The look and design of the Emotions had to remind people that they are personifications of feelings,” says Docter. “They’re not little people. They’re Emotions. They’re made of energy—they’re made up of thousands of particles, which kind of looks like energy. We wanted to capture what emotions feel like—the shapes, the colors—as well as their personalities.”
Albert Lozano, character art director, was inspired by production designer Ralph Eggleston’s early efforts. “The way that the chalk spattered on Ralph’s pastels, it reminded me of bubbles. Joy is effervescent. ... I do a lot of collage work, so I took the image of a sparkler, added a face, legs and arms, and that felt like Joy to me, too. I knew she had to emit joy.”
Filmmakers called on effects supervisor Gary Bruins and his team to figure out how to showcase that energy. “Pete wanted Joy to have particles that radiate and shoot off her skin throughout the entire film,” says Bruins. “That meant creating an effect that would appear in hundreds and hundreds of shots. It had never been done before.”
Joy, whose eyes have at least twice as many controls as any Pixar character before her, also serves as a light source, casting a yellow-blue glow around her. According to global technology pro Bill Reeves, a whole system needed to be built to achieve the look filmmakers wanted. “We tried dozens of ways of creating Joy’s glow and ended up with a volumetric solution. But since she’s in so many scenes, we needed to configure the software to be able to compute it.”
Perhaps the biggest challenge, however, when it came to lighting for Inside Out was Joy herself. The filmmakers felt that an Emotion that represents happiness should light up a room—literally. So Joy—who appears in nearly every sequence set in the Mind World—is actually a light source. “The problem is that if you take a picture of a lightbulb,” says White, “it’s just a flat bright thing. There’s no definition. We wanted Joy’s face to be round and appealing.” Angelique Reisch, who served as one of the lighting team’s lead technical artists, was brought on early in the production to tackle the challenge. Reisch took her lead from production designer Ralph Eggleston and the art department. “There was one pastel Ralph did early on that’s absolutely stunning,” she says. “He created an inner glow that’s really bright—brighter than her outer glow—and colored one side pink and one side white. It was beautiful.” The pastel inspired the team’s use of hue versus value to achieve the desired shaping of Joy’s features.
The use of color—lightest yellows to richer oranges and even red—does for Joy what adjustments in value traditionally do for a character. Joy as a light source presented some challenges that called for new technology. Inside Out became the first film to employ the use of a geometry light. "The geometry light was pretty interesting - we did not actually have access to that when we started developing joy - we heard that RenderMan was working on Geometry lights and it did not sound like it was going to be ready for us to use in production and if ever there was a film to use it on...this was the film!!" laughingly recounts Reisch.
After some internal discussion and some key discussions with the RenderMan team, they got the geo light working early and they used it in the production, but even so some scenes had been completed. "We actually went back into a sequence we had already lit using more of a mock set up of her light done with a series of sphere lights to mimic her head, her body and her limbs, but those were cumbersome and did not have nearly the sort of beautiful detail the geo light has - and it was more expensive. It was just beautiful - I can't say enough good things about it". With the new geo light, once the light was set up there was a small amount of tweaking but on the whole it just worked, and even as the scenes and environments changed, the same setup worked with Joy throughout the film.
With glowJoy only emits from her body, and it is emitting only from her body skin, not her hair, or clothes etc. In addition to the emitting light, there is a secondary volume around her, which while appearing as perhaps a post-production glow, is actually done in the same primary render pass as the principal character animation. Joy actually has two glows, her outer glow is a blueish volume, and then a strong but tight glow on her body which is interestingly "white on her key light side and saturated pink on her off-key side - that is also a volume and we called that her inner glow," explains Reisch.
If a character approaches Joy they are lit only by the yellow emit light, her blue hues don't affect any props or other characters. That decision was made to avoid complex and confusing color mixing that might be 'accurate' but work against the overall lighting design of the characters. Joy also does not cast a shadow. “We came up with a different approach for her," says Reisch. "The other Emotions receive light like any normal character would—master lighting from the set, plus some special lights for their glow and their volumes. But Joy has her own special rig, so she’s emanating light onto them. And she doesn’t receive light—like from the screen in Headquarters. Other sources of light don’t affect her because she is the brightest source."
With glowWithout glowWith glow
Like a good glass of champagne, Joy is also effervescent. Beneath the volume—the particles that make up the Emotions—is a body surface. “We blended her surface shading to give her that effervescent look,” says Reisch. “We also came up with tools and lighting so the lighters could work with the hard-surface version of Joy versus the volume version. That made her faster to light.” The effect is procedural, "those are geometry that are generated at render time, they were simulated to move around on her skin, and that work was done by the character department, but of course we worked very closely with them."
Each character had a different version of the boiling effect or what Pixar called 'solidity', Anger is the most solid, vs. Joy and Fear have their particles emanating out more "because they are more 'volume' (than solid) - for example, on Fear's nose the particles come out really far, and off his nose, and on joy we went so far as to basically have her wrist transparent, and that was very intentional - it was something that Pete (Docter) wanted to see in the film, to give the idea they were made of energy particles, to look how our emotions feel - our emotions change even as we are standing still - and so the characters change even as they are standing still."
Without the edge simulationWithout the edge simulationLighting finalLighting final
The other interesting aspect of the female characters, is their iridescent hair. Pixar first used iridescence on the hair of Kevin in UP. But since then the Pixar pipeline has moved to global illumination and the hair of these characters required the anisotropic iridescence shader to be rewritten from scratch with several revisions to balance the shininess and other properties, and even the hair and their hair to body shadows break up with the solidity effect.
Pixar managed to render all the characters together, without having to break out separate renders for the hair or the particle/solidity effect or even its glow. The team worked hard to reduce the render times and allow this more one pass approach. It has been a joke in the industry for some time that the easiest job in the world is being a compositor at Pixar as the TDs and lighting teams tend to nail the image in the render - in the one primary pass. This is changing on some of the newer films but for Inside Out it was still true. The rendering was in a version of PRMan with the geo lights and still using the Reyes renderer at the core.
Reisch feels that the biggest advantage of this approach is the consistency it provided to the characters, their glows were always the same, characters such as Joy held up in different lighting environments say between the memory dump and Headquarters, and also at different scales of the character in frame. "Of course her glow is going to look like it falls off quicker in a bright environment but the values were the same."
One challenge visually was 'abstract thought', the point in the film where several of the characters needed to be rendered in first a more cubist way and then finally as just 2D elements. Reisch and others were initially worried that this might pose complexity issues, but in reality it turned out to be more simple. "Once they pop into their solid version, which happens from their first pop - their first change, we just used a dome light - like an ambient light, with a very simple shadow. Joy's fancy light rig and all these tricks we have done to get things working in the rest of the film - actually becomes incredibly simple it is a very simplified setup, no more glow. We were all kind of worried about that sequence because of all the transitions but it ended up not being as tricky as we thought."
The World
Not only did the characters have to be lit and rendered but the two worlds of the film, the interior world of the mind and the more traditional human 'real' world that Riley inhabits.
inside-out-progression_image_4of7_Sets
While only on screen for a very brief period there is a sequence of 'classic' Pixar environments on the trip to San Francisco. Shots like these are hard as they are one off shots as part of a montage of the family driving west and yet each one was beautifully lit and clearly lovingly crafted, these shots were around mountains, across corn fields and eventually arriving at the Golden Gate bridge (which disappointingly to Riley was not made of gold).
Pixar's master lighting artist on these sequences was Josée Lajoie. "She lit most or all those one off shots, and they are beautiful, she did a lot of hard work, and for each of these she had to build them all up from scratch," explains Reisch.
The scenes were lit by a dome light for a fill light, the team may then add some spot lights to just add come color, and then for the key lights Lajoie used area lights, often a disc area light. Pixar has well-understood set ups for the sun which encompasses angle, time of day, color etc that mimics sunlight. After that the shot gets much more built up with bounce lights and character lights for the family in the car, plus "kicks and rims and all the atmosphere we need," says Reisch, "but primarily it is a combined approach of dome lights and disc area lights."
inside-out-d150_28acs.sel16.314
As a general rule the lighting team tries to light the characters from the light of the scene, certainly the fill light is normally from the environment. This is done to look the most natural, but in the case of characters in the car say, Reisch explains that "they will share the fill light from the set but they are each going to have their own key light, adjusted to get special shaping on each of them, because each of their face shapes is a little bit different, and probably we add some kick lights and rim lights to just pull them out of the background a little bit."
When the family is in San Francisco the lighting was more similar to Monsters University than other Pixar films such as UP - just because since Monsters University the approach has been global illumination, with physically based lighting and shading. But unlike MU the team had to light human skin with more realistic Sub Surface Shading (SSS). In this film the team used ray tracing SSS as opposed to the previous approach of having a pre-pass model of scattering. "It worked out great," says Reisch, "and we did not pay the price of that pre-pass and to just have it ray traced and add its nice softening around the character's noses and ears - and the fine detail on their faces."
Pixar's lighting team is presenting several papers on the characters and the lighting at SIGGRAPH 2015 in LA.

http://www.fxguide.com/wp-content/themes/fxguide3.1/images/branding/fxinsider-badge.png
Thanks so much for reading our article.

We've been a free service since 1999 and now rely on the generous contributions of readers like you. If you'd like to help support our work, please join the hundreds of others and become an fxinsider member.


Patti Wood, MA, Certified Speaking Professional - The Body Language Expert. For more body language insights go to her website at www.PattiWood.net. Check out Patti's website for her new book "SNAP, Making the Most of First Impressions, Body Language and Charisma" at www.snapfirstimpressions.com. Also check out Patti's YouTube channel at http://youtube.com/user/bodylanguageexpert.