Henson Digital Puppetry Studio.

Redefining animation and interaction through real-time performances

Intro.

Integrated with motion capture and Unreal Engine, Henson puppeteers use the award-winning Henson Digital Puppetry Studio to create complete CG character performances in real time. This includes the puppeteering of full facial animation for characters and creatures. With the Henson Digital Puppetry Studio, we can “shoot” CG scenes as if they were live action and bring the magic of Henson puppetry into the CG world.

Components

Motion Capture.

The Henson Performance Control System integrates into any motion capture stage. To capture a full body performance, the puppeteer works in tandem with a body performer (or mocap puppet). An optical tracking system or an inertial sensor suit captures the body animation. The data from motion capture streams together with the facial animation from The Henson Performance Control System. They combine in real time to create a complete character performance. 

Props Station.

The Henson props station is a slider box that can trigger prop actions and constraints. This allows us to trigger prop action or prop handoffs on the motion capture stage and reduces the need for keyframe animation for prop action and manipulation. 

Henson Performance Control System.

The core of the Henson Digital Puppetry Studio is the Henson Performance Control System. An evolution of the same system that runs animatronic puppets, the Henson Performance Control System allows puppeteers to perform CG characters with the same creativity, spontaneity, and fluidity that they bring to the performance of live-action puppets.

The system empowers the puppeteer to design the animation style of a character. Custom hand controls and foot pedals give the puppeteer 32 programmable dual-direction inputs. The puppeteer gangs together individual movements into expressions then assigns those expressions to the controls. The puppeteer can activate those expressions individually or combine them to create additional expressions and nuance.

Virtual Cameras.

Henson’s virtual camera controllers are designed with the functionality and feel of traditional TV pedestal cameras. Each camera can zoom, pan, tilt, crane, and dolly. Operators use the customizable buttons to preset shots. With three cameras operating during a mocap session, we can “shoot” our CG animation as if it were three-camera live action sitcom. 

Henson Take Recorder.

The Henson Take Recorder offers unparalleled flexibility in capturing and refining takes. It captures every input from the virtual cameras and puppetry performance control system. This multi-track capture allows the director to isolate and refine specific aspects of a take. By isolating individual tracks (while preserving others), the DP can refine a camera position, the puppeteer can fix a moment of lip sync or an eyeline, the director can adjust a bit of movement. 

All Henson Take Recorder Data can then be imported back into Unreal and/or Maya for a post-production workflow.

Note: use of the Henson Take Recorder is not required. The Henson Performance Control system can stream directly into Unreal Engine and is compatible with Unreal Engine’s Take Recorder

Components

Henson Performance Control System.

The core of the Henson Digital Puppetry Studio is the Henson Performance Control System. An evolution of the same system that runs animatronic puppets, the Henson Performance Control System allows puppeteers to perform CG characters with the same creativity, spontaneity, and fluidity that they bring to the performance of live-action puppets.

The system empowers the puppeteer to design the animation style of a character. Custom hand controls and foot pedals give the puppeteer 32 programmable dual-direction inputs. The puppeteer gangs together individual movements into expressions then assigns those expressions to the controls. The puppeteer can activate those expressions individually or combine them to create additional expressions and nuance.

Motion Capture

The Henson Performance Control System integrates into any motion capture stage. To capture a full body performance, the puppeteer works in tandem with a body performer (or mocap puppet). An optical tracking system or an inertial sensor suit captures the body animation. The data from motion capture streams together with the facial animation from The Henson Performance Control System. They combine in real time to create a complete character performance. 

Virtual Cameras

Henson’s virtual camera controllers are designed with the functionality and feel of traditional TV pedestal cameras. Each camera can zoom, pan, tilt, crane, and dolly. Operators use the customizable buttons to preset shots. With three cameras operating during a mocap session, we can “shoot” our CG animation as if it were three-camera live action sitcom. 

Props Station.

The Henson props station is a slider box that can trigger prop actions and constraints. This allows us to trigger prop action or prop handoffs on the motion capture stage and reduces the need for keyframe animation for prop action and manipulation. 

Henson Take Recorder.

The Henson Take Recorder offers unparalleled flexibility in capturing and refining takes. It captures every input from the virtual cameras and puppetry performance control system. This multi-track capture allows the director to isolate and refine specific aspects of a take. By isolating individual tracks (while preserving others), the DP can refine a camera position, the puppeteer can fix a moment of lip sync or an eyeline, the director can adjust a bit of movement. 

All Henson Take Recorder Data can then be imported back into Unreal and/or Maya for a post-production workflow.

All about the Henson Digital Puppetry Studio

Brian Henson with an early version of the system

The Henson Digital Puppetry Studio (HDPS) uses patented technology to allow performers to puppeteer and voice three-dimensional CG characters in real time.

In 2019, we began our relationship with Epic Games, giving us the tool of the Unreal Engine to integrate with HDPS. This partnership has helped us better interact in virtual worlds that are of stunning quality, in a high frame rate, without any latency.

We utilize this system in our Virtual Productions, Short Form Content Creation, and Interactive Location Based Experiences.

Benefits

Spontaneity

There is magic that happens when creative people can improvise and experiment together. A shoot with the Henson Digital Puppetry Studio runs like a live action set. With the director, cinematographer, and performers all in one place, and with everything streamed in real time, the creatives can be nimble. They can explore different approaches to the script. They can iterate quickly. They can discover things in the moment to make the story better.

Context

Realtime rendering of the environment, lighting, performance, and cameras on the stage means that the director, cinematographer, and production team have full context for creative choices on the set. They can see, moment-to-moment, the full performance, camera view, and environment and make changes as necessary. When those takes go to editorial, the editor, director, and producers have full context for the cut. They can review, make creative choices about, and approve an edit that is very close to the final version of the story. This avoids expensive changes in the post-production process.


Perfect for

Realtime Animation

Visual Effects

Previz & Animatics

Live Interactive Experiences

Video Game Cinematics

Efficiency

With the Henson Digital Puppetry Studio, we can “shoot” 10-12 minutes of script in a day. The takes (with their underlying animation and camera data) provide a big leap forward for animators, sometimes eliminating the need for keyframe animation entirely.

Character Animation Dev.

Social Media Campaigns

FAQ

Please reach out if the following answers don’t fully address your questions.

Productions & Projects

2020-Today

  • Monster Mix

  • Dani Can Dance

  • The Char Experience

  • Tall Poppy

  • Math Party

2015-2019

  • Earth to Ned

  • Word Party

  • Which Witch

  • Georgia Aquarium

  • Splash & Bubbles

Before 2014

  • Hershey’s Great Chocolate Factory Mystery In 4D attraction at Hershey’s Chocolate World

  • Sid the Science Kid

  • Frances

Awards & Milestones

SIGGRAPH Realtime Live 2024

Digital Puppetry Character Char (performed by Drew Massey) is the emcee of SIGGRAPH RealTime Live

Epic Mega Grant 2019

$500,000 grant to integrate the Henson Digital Puppetry Studio with Unreal Engine.

Primetime Emmy 2009

Outstanding Achievement in Engineering Development presented to Jim Henson’s Creature Shop for The Henson Digital Puppetry Studio

Academy Award 1992

Academy of Motion Picture Arts and Sciences Scientific and Engineering Award for the development of the Henson Performance Control System

Debut of Waldo C. Graphic 1989

Waldo C. Graphic, a CGI character performed in real time with a mitt controller by a puppeteer, is developed with Pacific Data Images for The Jim Henson Hour and Muppet*Vision 3D