The LIU Project Proposal

The LIU project will be broken into 3 separate but related areas concerned with MPEG-4 Facial Animation.

The research will expand upon the work done by John Stallo in his honours on adding emotion to speech, and by Quoc Huynh in his honours work on Facial Expressions and will expand upon and validate the Virtual Human Markup Language (VHML). The work will result in a large scale demonstration (see later).

The research will involve

literature review: the 3 students to each investigate one of the first 3 areas and that area's relevance to the last area - Talking Heads.
- The effect of Emotion on Speech, and Speech Utterances
- The effect of Emotion on Facial Expression, and Facial gestures
- The use of XML as a markup language for controlling Virtual Actors.
- The use and applications of Talking Heads. See later Case Study Demonstration and Evaluation.
inspection verification: the 3 students to each identify, verify and correct the implemented VHML tags in one of the first 3 areas and all to formulate a long term strategy for the development of the last area. This will result in a small XML based C/C++/Java application to manage "stimulus-reponse" Dialogue Manager databases. Students to seek expert advice from partners in the InterFace project via email interviews.
- Text to Speech markup - SML
- Facial Animation markup - FAML
- Emotion markup - EML
- Dialogue Manager markup - DMML
Case Study Demonstration and Evaluation of the HCI issues arising from a Talking Head being used to deliver information in response to user enquiries.

The above tasks allow for separate investigation by students with flexibility of choice (the first 3 areas in lists above) but drive the investigation to a single goal with a common thread (the last area).

Case Study Demonstration and Evaluation.

The Facial Animation Group at Curtin in 2nd Semester will consist of
Big Persons:

Andrew Marriott, Don Reid, Andrew Turpin
Middle Size Persons:

Simon Beard (PhD student), Hanadi Haddad (PhD Student)

Camilla Gustavsson (Masters), Linda Strindlund (Masters), Emma Wiknertz (Masters) (Swedish contingent)
Slightly Smaller persons:

Darren Wan (Honours), E-K Gulland (Honours).
Low level turnips:

Kate McInnes (Undergrad), Cara-leigh Patterson (Undergrad), Serge De Souza (Undergrad),

Brendan von Forslun (Undergrad), John McCabe-Danstead (Undergrad).

It has been decided that the entire group here at Curtin will set the following goal for a late November Talking Head Demonstration:

Existing: The demonstration will be Web based for PC's running Windows/Linux with Netscape or IE. This should not be a problem given the existing software base but integration of modules and accuracy of APIs will be important. The demo should equally be able to run stand alone.
Responsibility: Andrew Marriott, Simon.
Existing: The demonstration will be able to use heads (plural) - that is, we want to be able to show the Talking Head as various models, in various levels of realism. This will be 2 1/2D -> 3D photorealistic. This again should not be a problem given existing software. The heads will have an integrated personality.
Responsibility:
- Darren, E-K, Simon - integration of TTS-FAML-Personality sub-system with libraries.
  Also affects Kate, Serge, Cara-Leigh, Brendan
- Darren Wan - Specification of TTS Black Box API and implementation of system onto Elan Speech Cube.
  Also affects Kate, Serge, Cara-Leigh
- E-K - specification and implementation of "idle" personality
  Also affects Brendan
- John McCabe-Danstead - 2 1/2D face builder.
Partial: The demonstration should be INTERACTIVE. That is, it will connect to one of the Schools Dialogue Managers - Mentor or MetaFace. Users will interact with the Talking Head in some manner through the DialogueManager. This is working currently but will need to be brought in line with a tight API.
Responsibility: Andrew Marriott, Simon, Hanadi
New: The demonstration will use VHML. All reponses delivered will be marked up in VHML - EML, SML, FAML, XHTML.
Responsibility:
- Linda, Emma, Camilla - as above with verification/inspection of existing tags.
- Linda, Emma, Camilla - as above re XML-based "stimulus-response" DM database.
- Kate - new VHML tags for fear.
- Cara-Leigh - new VHML tags for emotional effects.
- Serge - accented voices plus vhml tags.
- Brendan - vhml tags for eye gestures.
- Andrew Marriott, Simon - integration and supervision.
New: The demonstration design. A significant design and implementation case study that will incorporate all the above research areas to highlight the functionality and versatility of the system.
The demonstration will consist of two experiments :
1. A story teller(probably of a childrens story) - this will demonstrate the emotional capability of the Talking Head and demonstrate the power of VHML. It is envisaged that multiple heads could be used with different voices. The story will be interactive in a trivial way in that the listener can direct where the story may go at certain branch points through a question-answer mechanism.
2. An information provider for enquiries about Talking Head Technology. The system will demonstrate the Dialogue Manager functionality of the Talking Head and its believability as a humane interface to information. The demonstration will also test the suitability of different head types to convey information and how well different personalities convey information.
Responsibility:
- Linda, Emma, Camilla - design/markup of story, supervising design of knowledge base domain, implementation of story branching mechanism, etc.
- Hanadi - design of heads, content, as well. To work with Linda, Emma, Camilla.
  Demonstrations must test research questions/hypothesis so we can evaluate and get useful information/results.
- Andrew Marriott, Simon - integration and supervision.
New: The case study will be concluded by an Evaluation of the users of the Talking Head via online questionnaire or similar. The questionnaire will be used to test the research questions/hypotheses posed by the group. The results of this questionnaire will form the base of one part of the data in the thesis "results" section.
Responsibility:
- Linda, Emma, Camilla
- Darren, E-K
- Hanadi, Simon, Andrew

Story telling 2-D Talking Head with characterisation

Information Presenter.