Capturing 3-D Motion

BY MATT ROBERTS

Salim Zayat, Art Director of the Human Modeling and Simulation Lab (HMS)and a 2001 graduate of Penn Engineering’s Digital Media Design program , is put-ting some final touches on a short 3Danimation. Typically, the realistic human motion Franklin exhibits would take ages to model convincingly. Remarkably, Zayat and partner,2001 DMD graduate Neil Chatterjee, have managed to animate dead-on posture, gait and gesture —all in a matter of weeks.

The secret? While both are talented animators, it was not solely the duo’s artistic prowess that enabled such rapid articulation of Franklin’s sweeping wave and smug swagger. It was a technology known as 3D motion capture —MoCap for short —which allowed them to literally record an actor’s three-dimensional movement in space.

To use the system, an actor wears a special suit covered in infrared transmitters which can broadcast signal information nine hundred times per second. He then enters a room-sized, box-like array of infrared sensor bars —a piece of hardware known as the ReActor, produced by the specialty hardware manufacturer Ascension. The ReActor receives the infrared data and outputs usable computer data to PC’s running Kaydara’s Film Box software. Film Box can implant the real-time positional information from the movement of the actor’s joints into a three-dimensional model, in this case, Zayat’s irreverent Ben Franklin, created using Oscar award-winning Alias Wavefront’s Maya modeling and animation software.

Next, the model and the motion can be mapped and matched using software maker EonReality’s 3D integration package. Finally, powered by a dual-output nVidia graphics card, the animation can be projected stereoscopically in 3D onto screens in the back of the motion-tracking ReActor cage environment. Users or observers can don special stereo glasses that give the illusion of pulling the projection out of the screen and into the room. The end result is an apparent three-dimensional world in which natural looking character motions and gestures are rendered on the ?y. Two years in the making, the system as a whole has been dubbed LiveActor. But, while this motion capture system is an undeniably useful and powerful addition to the Human Modeling and Simulation Laboratory, graphics researchers primarily plan to use it as a base tool in a larger and potentially more important application being developed in the eastern wing of the Moore School. Dr. Norman Badler, Director of the HMS, and his cadre of computer graphics PhD’s have bigger plans for tracking and processing human movement than simply recording it —they are going to create systems which actually respond to a user’s movement, just like a real human might.

“The principle goal is to enable mutual interaction between real and virtual people,” explains Dr. Badler. Badler’s experience in placing humans in virtual environments is long standing: he lead the development of the pioneering “Jack ”virtual human modeling software in the 1980’s.

“The unique thing we did was marry the ReActor motion capture system with the stereo display,” continues Badler. The setup lays the ground work for simulations where a user’s gestures and movement can be treated as input to a system which examines the data and makes virtual characters respond appropriately. “We believe there are significant applications where having close, interpersonal relationships with individuals in a virtual space will provide training and educational opportunities that you can’t have when you are in a physical space of the same sort,” says Badler.

Such training opportunities abound in military, aerospace and security fields. Largely, though, these applications have been closely tied to learning tools or instrumentation, such as in flight simulation. While these simulators are extraordinarily successful teaching tools, there has always been a barrier between the user and the simulation, both physically and cognitively. A user’s ability to interact using natural language or gesture —our primary communication media as human beings —has been limited.

The research driving the LiveActor system hopes to change this. By facilitating the use of body movement and gesture in three dimensions as input to an intelligent, responsive simulation of virtual characters, LiveActor opens the door for more convincing and realistic means of experiencing scenarios which would be prohibitive to recreate in the real world due to cost, time or safety concerns.

One of the first projects in development extends a previously developed “checkpoint ”simulation into LiveActor’s 3D capture system. The simulation is a mockup of a security checkpoint where users from military or law enforcement can practice the correct procedure for identifying threats —a delicate and sometimes dangerous task. “Its probably good to experience the consequences of doing it wrong in the virtual space first,” says Badler.

Jan Allbeck, a PhD student and Systems Programmer at the HMS lab, is leading research on mapping a library of captured human gestures into a computer database. Similar to how a child learns to recognize verbal and non-verbal communication by associating occurrence with context, a neural-network based software system, developed at HMS and known as EMOTE, can “learn ”to identify3D movement and create a library of gestural information.

“We feed [the 3D motion input ] into our virtual characters, so now they know something about the state of mind or physiology of the real person,” explains Allbeck. She and her associates hope to create a system which would not only recognize didactic gestural commands(like ‘stop,’ ‘come’ or ‘go’)but also identify emotions: a bored user acts differently than an attentive or frightened one, for example. Humans perceive and respond to these non-verbal cues naturally, and deconstructing their specific attributes might allow a computer to do so as well.

Primary funding for the research comes from the NSF, NASA, Lockheed Martin and SEAS. NASA hopes to simulate in-space repair and maintenance in conjunction with a natural language interface, and PhD candidate Aaron Bloom field is working on a “tactor ”based system for them which would provide physical, vibrating feedback when users’ limbs touch virtual masses. This spring, as part of the undergraduate “Building Virtual Worlds ”course, taught by Badler and Dr. Stephen Lane, one group of students even used the motion capture system for their final class project. The LiveActor system has become a nexus of activity in the graphics department, and a lecture on and demonstration of the technology this spring drew a substantial crowd.

“Man, I wish this was here when I was an undergrad,” pines Zayat. Thanks to continuing grants and interest, LiveActor may be a tool that future Penn Engineering students get to experience first hand, and first person.