Henson Digital Puppetry Studio.
Redefining animation and interaction through real-time performances
All about the Henson Digital Puppetry Studio
Brian Henson with an early version of the system
The Henson Digital Puppetry Studio (HDPS) uses patented technology to allow performers to puppeteer and voice three-dimensional CG characters in real time.
In 2019, we began our relationship with Epic Games, giving us the tool of the Unreal Engine to integrate with HDPS. This partnership has helped us better interact in virtual worlds that are of stunning quality, in a high frame rate, without any latency.
We utilize this system in our Virtual Productions, Short Form Content Creation, and Interactive Location Based Experiences.
Intro.
Integrated with motion capture and Unreal Engine, Henson puppeteers use the award-winning Henson Digital Puppetry Studio to create complete CG character performances in real time. This includes the puppeteering of full facial animation for characters and creatures. With the Henson Digital Puppetry Studio, we can “shoot” CG scenes as if they were live action and bring the magic of Henson puppetry into the CG world.
Components
Motion Capture.
The Henson Performance Control System integrates into any motion capture stage. To capture a full body performance, the puppeteer works in tandem with a body performer (or mocap puppet). An optical tracking system or an inertial sensor suit captures the body animation. The data from motion capture streams together with the facial animation from The Henson Performance Control System. They combine in real time to create a complete character performance.
.
Props Station.
The Henson props station is a slider box that can trigger prop actions and constraints. This allows us to trigger prop action or prop handoffs on the motion capture stage and reduces the need for keyframe animation for prop action and manipulation.
.
Henson Performance Control System.
The core of the Henson Digital Puppetry Studio is the Henson Performance Control System. An evolution of the same system that runs animatronic puppets, the Henson Performance Control System allows puppeteers to perform CG characters with the same creativity, spontaneity, and fluidity that they bring to the performance of live-action puppets.
The system empowers the puppeteer to design the animation style of a character. Custom hand controls and foot pedals give the puppeteer 32 programmable dual-direction inputs. The puppeteer gangs together individual movements into expressions then assigns those expressions to the controls. The puppeteer can activate those expressions individually or combine them to create additional expressions and nuance.
The Henson Performance Control System becomes the instrument of the puppeteer’s creativity. The flexibility of the technology combines with the creativity and skill of the artist to create virtuoso performances.
Virtual Cameras.
Henson’s virtual camera controllers are designed with the functionality and feel of traditional TV pedestal cameras. Each camera can zoom, pan, tilt, crane, and dolly. Operators use the customizable buttons to preset shots. With three cameras operating during a mocap session, we can “shoot” our CG animation as if it were three-camera live action sitcom.
.
Henson Take Recorder.
The Henson Take Recorder offers unparalleled flexibility in capturing and refining takes. It captures every input from the virtual cameras and puppetry performance control system. This multi-track capture allows the director to isolate and refine specific aspects of a take. By isolating individual tracks (while preserving others), the DP can refine a camera position, the puppeteer can fix a moment of lip sync or an eyeline, the director can adjust a bit of movement.
All Henson Take Recorder Data can then be imported back into Unreal and/or Maya for a post-production workflow.
Note: use of the Henson Take Recorder is not required. The Henson Performance Control system can stream directly into Unreal Engine and is compatible with Unreal Engine’s Take Recorder
Benefits
Spontaneity
There is magic that happens when creative people can improvise and experiment together. A shoot with the Henson Digital Puppetry Studio runs like a live action set. With the director, cinematographer, and performers all in one place, and with everything streamed in real time, the creatives can be nimble. They can explore different approaches to the script. They can iterate quickly. They can discover things in the moment to make the story better.
Context
Realtime rendering of the environment, lighting, performance, and cameras on the stage means that the director, cinematographer, and production team have full context for creative choices on the set. They can see, moment-to-moment, the full performance, camera view, and environment and make changes as necessary. When those takes go to editorial, the editor, director, and producers have full context for the cut. They can review, make creative choices about, and approve an edit that is very close to the final version of the story. This avoids expensive changes in the post-production process.
Perfect for
Realtime Animation
Visual Effects
Previz & Animatics
Live Interactive Experiences
Video Game Cinematics
Efficiency
With the Henson Digital Puppetry Studio, we can “shoot” 10-12 minutes of script in a day. The takes (with their underlying animation and camera data) provide a big leap forward for animators, sometimes eliminating the need for keyframe animation entirely.
Character Development / Reference for Animation
Social Media Campaigns
FAQ
Please reach out if the following answers don’t fully address your questions.
-
Our asset build schedule looks like a traditional CG asset build schedule. We model, texture, groom, and rig the character. Once the character is rigged, it is ready for our digital puppetry system. At this point, a puppeteer takes over. The puppeteer works with the system to design character expressions and customize the system for performance. Depending on the character and the production this takes anywhere from 1-5 days.
-
Yes! We can adapt any rigged character to be performed on our puppetry system. Adapting the rigging takes anywhere from a few days to a few weeks (depending on the existing rig). Once the asset is up and running on our system, we can help you cast a puppeteer to custom program the digital puppetry system in collaboration with your creative lead.
If your model does not have a rig, we can create one from scratch.
-
We can run 6 puppeteering rigs (combined with 6 additional mocap actors) to capture 6 characters at a time. If the scene requires additional characters, we can layer them in through additional takes.
-
Depending on the complexity of the scenes, we can shoot 10-12 minutes of animation per day.
-
Yes. Our hardware and software interface with both Maya and Unreal Engine. We can integrate into any CG workflow that uses those DCC packages.
-
No.
Productions & Projects
2020-Today
Monster Mix
Dani Can Dance
The Char Experience
Tall Poppy
Math Party
2015-2019
Earth to Ned
Word Party
Which Witch
Georgia Aquarium
Splash & Bubbles
Before 2014
Hershey’s Great Chocolate Factory Mystery In 4D attraction at Hershey’s Chocolate World
Sid the Science Kid
Frances
Awards & Milestones
SIGGRAPH Realtime Live 2024
Digital Puppetry Character Char (performed by Drew Massey) is the emcee of SIGGRAPH RealTime Live
Epic Mega Grant 2019
$500,000 grant to integrate the Henson Digital Puppetry Studio with Unreal Engine.
Primetime Emmy 2009
Outstanding Achievement in Engineering Development presented to Jim Henson’s Creature Shop for The Henson Digital Puppetry Studio
Academy Award 1992
Academy of Motion Picture Arts and Sciences Scientific and Engineering Award for the development of the Henson Performance Control System
Debut of Waldo C. Graphic 1989
Waldo C. Graphic, a CGI character performed in real time with a mitt controller by a puppeteer, is developed with Pacific Data Images for The Jim Henson Hour and Muppet*Vision 3D