Ana səhifə

Partially Automated System for Synthesizing Human Facial Expressions in Interactive Media 2012

Yüklə 413.62 Kb.
ölçüsü413.62 Kb.
  1   2   3   4   5   6   7   8   9   ...   16

Partially Automated System for Synthesizing Human Facial Expressions in Interactive Media



Project Title: Partially Automated System for Synthesising

Human Facial Expressions in Interactive Media

Project Period: 2nd of february 2012 – 6th of June 2012

Department: Department of Media Technology

Project Supervisor: Stefania Serafin
Circulations: 3

Pages: 72 pages

Appendix Pages: 13 of pages

Attachments: DVD-Rom with Appendices & the game

Finished on: 2th of June 2010
Morten Havmøller Laursen

Kasper Søndergaard Pedersen



In order to enhance the quality of facial expressions used in interactive media, typically video games, a system for generating facial animation and influencing their characteristics and temporal change in real-time is proposed, implemented and evaluated. The proposed system takes into account the underlying muscles of the human face and how they interact in order to approximate a simulation-like quality, though appropriated for real-time rendered animations. All this is done using the Facial Action Coding System (FACS) developed by Paul Ekman [1][2] as well as studies related to gaze and head movement by Argyle & Cook [3] as well as Hader et. al. [4]
The system is partially controlled by the developer, but adds layers of detail in order to make the face feel less artificial. These extra details are added by the system. Further development would focus on developing an Acting Emulator capable of handling different emotions and perhaps eventually extract cues from speech audio files used as markers for conversational actions.

The content of this report is freely available, although publication can only be allowed after agreement with the authors.


This report has been prepared as a Master Thesis at Medialogy, Aalborg University. The project was commenced in order to research and develop a facial animation system, aimed at making it easier for animators and game developers to create believable facial animations.

Reader’s Guide

This project is split into two main parts, and should be read for what it is: Mainly a documentation of research and development of a proposed system. Research is mainly embodied in the part of the report called “Analysis”, and development is embodied in the part called “Implementation”. It is not to be confused with a scientific paper or thesis in the traditional sense; there is no central, refutable statement.
The proposed system is sometimes referred to as simply “the system” or in many cases “AECS” for “Animated Expression Compositing System”, which is the specific name given to the proposed system and its implemented counterpart.
The report details the research and development of the AECS and culminates in the evaluation in the capabilities of the system, in the form that it achieved towards the end of the project.
Evaluation data and work files are included in the digital appendix, in full. The digital appendix also contains the demonstration build, called “Sintel Expression”, which is the proof of concept demonstration used for the qualitative evaluation at the end of the report.
Sources cited from websites are also included in the digital appendix.
The CD also contains an audiovisual production (video) named “AECS demo video.mp4”, which provides a quick overview of the AECS, though referred to by its earlier name “Real-Time Facial Action Coding System”.
References that appear throughout the report will be summarized in the bibliography which can be found at the end of the report. References are numeric, meaning that references will be presented as following: [].

In the bibliography items will be referenced as following:

Books/Papers: (writer(s), year, title, publisher, edition, pages)

Games: (developer, year, title)

Webpages: (writer, year, title, url)

Figures, tables and charts are numbered according to chapter numbers. All figures and tables will come with a written explanation.


We would like to extend our most sincere thanks to the following people for making this project what it is today, and to us:

Stefania Serafin - our supervisor, and sparring partner in the first phases of the project
Mia Lerdam - Voice Actor

-who lended her great voice a few days before the birth of her second child (congratulations!)

Theodor B. Lancor - Qualitative Evaluation

Christian Førgaard Nielsen - Qualitative Evaluation

Mads Jacobsgaard - Qualitative Evaluation

Table of Contents


Synopsis 3

Abstract 3

Foreword 5

Reader’s Guide 5

Table of Contents 7

Indhold 7

Introduction 10

Motivation 11

Problem Statement 12

Hypothesis 12

Delimitation 13

Key Terms and Concepts 14

Facial Areas - Names and locations 14

Facial Animation 14

Visual Prosodic Movements 14

Facial Social Signals 14

Manipulators 14

Animation Combination 14

Action Units 15


A Brief History of Facial Animation 16

Facial Animation in Films and Games 17

State of the Art 18

Facial Animation 18

Visual Prosodic Movement 18

Existing Commercial Solutions 19

Analysis 20

Facial Movements and Expressions 20

Facial Social Signals 20

Head and Eye movement 24

Eye Behaviour in Social Encounters 24

Head Movement 26

Facial Action Coding System (FACS) 27

Workings of FACS 27

Motion & The Uncanny Valley 32

A Few Disney Principles of Animation 32

Follow Through & Overlapping 33

Secondary Action 33

Rigging & Animation Techniques 33

Implementation 35

Goals of the Implementation 35

Peripheral Work 36

Tools & Software 36

Unity3D 36

Autodesk Maya 37

Animation & Data Import 38

Rigging & Skinning Based on FACS 38

Animating the Action Units 39

Importing Action Units into Unity 41

Code Implementation 42

Code Ethic 42

Basic Action Unit Integration in Code 43

Blending the Action Units 44

Parts Not Described in Action Units 49

Editor Scripts 51

Integration with Sound and Lip Synchronization 54

Scene Setup 54

Defining the Evaluation 57

Quantitative Evaluation 57

Pilot test 58

Qualitative Evaluation 58

Choosing the Demonstration 58

Sound Acquisition & Processing 59

State of the System 60

Action Units 60

Sequencing Expressions 60

The Actor Emulator 60

Spectrogram Analysis 60

Results 62

Quantitative Evaluation Results 62

Presentation 62

Analysis 63

More general observations 64

Qualitative Evaluation Results 65

Presentation 65

Analysis 65

Conclusion 68

Discussion & Future Work 69

Bibliography 70

Appendices 73

Appendix 1: List of Action Units in the Facial Action Coding System 73

Appendix 2: Description of the 4 Bone Influence Limit in Unity3D 77

Appendix 3: The Uncanny Valley 78

Appendix 4: Questionnaire from Quantitative Study 79

Appendix 4, part 1: General questions 79

Appendix 4, part 2: Emotion recognition question (example: surprised) 79

Appendix 4, part 3: Overall impression + general comments 81

Appendix 5: Emotions used in the Questionnaire 82

Appendix 6: Questionnaire Responses 83

Appendix 7: Names and locations of facial areas and parts. 84

Appendix 8: Rig Hierarchy 85

  1   2   3   4   5   6   7   8   9   ...   16

Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur © 2016
rəhbərliyinə müraciət