Elsa Sings Let It Go-Practice of Facial Animation

3DXchange5 Pipeline test series 17: Elsa sings Let It Go
2014. 11.24 Hirospot (Personal Site)

The Introductive Practice of Facial Animation by CrazyTalk 7 PRO

(YouTube)

Go to other Report →

Test Reports with 3DXChange5 & iClone5 Pipeline

印は日本語でのレポートです

iClone4 used to have a Facial Animation Pipeline with CrazyTalk 6 in which fine tuning of every Head Elements was possible in the Facial Timeline Editor.

Now, CrazyTalk7 (Pro) features Audio-driven Animation which uses voice energy to drive intensity of animations and make characters perform very natural.
CT7(Pro) provides many ways to animate the Face, such as Idle, Auto, Motion Clips and manual blending.

This report is the record of my practice to make Elsa sing 'Let It Go' within CT7Pro environment.

Tools	Manufacturer	Major Functions in this report
CrazyTalk7 PRO Ver:7.32.3114.1	Reallusion	Facial Images was auto-animated with audio sound and voice.
iClone5	Reallusion	MMD Elsa Face was rendered in iClone5.
Fireworks CS6	Adobe	Imaging and web tool for the report
Excel	Microsoft	Making chart of vocal text and frame number

Data Type	Name	Creator	Notes
Facial Model	MMD Elsa DL	0-0-Alice-0-0	Converted and inported to 3dxChange & iClone
Video (Vocal and Sound)	FROZEN - Let It Go Sing-along \| Official Disney HD	Disney UK	English Lyrics are displayed with the star to show the singing position.
Image	Background	isaver1	Disney-frozen-ice-castle-wallpaper

Referenced Video and Tutorial	Creator Name	Notes
CrazyTalk7 - New Features Guide	Reallusion	Dockablle UI, Enhanced Natural Movements: Audio-driven Animation and Smart Auto Motion Profiles: Talk, Sing, Listen, Dance.
CrazyTalk7 Tutorial - Getting to Know the Environment	Reallusion	Introductive Guidance of functions in CT7.
CrazyTalk7 Tutorial - Actor Creation-Face Fitting	Reallusion	Get to know how to import Elsa Face and fitting process.
CrazyTalk7 Tutorial - Actor Creation - Eye & Teeth Settings	Reallusion	How to adjust eyes and teeth settings for the custom characters.
CrazyTalk7 Tutorial - Import Audio & Basic Lip-syncing	Reallusion	Use of Lipsync Libraly, enhancing Lipsync, and replacing Phonemes.
CrazyTalk7 Tutorial - Advanced Singing Animation	Reallusion	Get to know how to import Let It Go Sound with Vocal and apply to Elsa Face.
CrazyTalk 7.3 Tutorial - Custom Motion Clip Strength	Reallusion	Use the ability to adjust the strength of motion clips.
CrazyTalk7 PRO Tutorial - Multi Audio & Auto Motions	Reallusion	Importing multiple audio tracks, and add different Auto Motions inside the PRO version of CrazyTalk7.
CrazyTalk7 PRO Tutorial - Advanced Key Editing & Puppet Settings	Reallusion	Introducing Face Puppet and Face Key Editor
CrazyTalk7 PRO Tutorial - Advanced Auto Motion 1 - Head & Body Movement	Reallusion	How to create natural and organic Body and Head movement, preview it and apply to the Model.
CrazyTalk7 PRO Tutorial - Advanced Auto Motion 2 - Muscle Control	Reallusion	How to create and enhance muscle movement on the facial parts: eye blow, eye lid, mouth and cheeks.
CrazyTalk7 PRO Tutorial - Advanced Auto Motion 3 - Global Settings	Reallusion	Adjustment of Reaction, Mode, Motion Settings and Blending Motion Strength in Auto Motion Settings.
CrazyTalk7 PRO Tutorial - Transform Editing & Camera Movements	Reallusion	How to transform model and simulate the camera movements on the timeline.
CrazyTalk 7.3 Tutorial - Custom Idle Motions for Web and iDevices	Reallusion	How to customize the idle motion.
User Manual for CrazyTalk 7	Reallusion	complete description of every feature in CrazyTalk7
CrazyTalk6 Tutorial - Create Your Script	CrazyTalk Tutorial
iClone4 Tutorial - Facial Animation Pipeline With CrazyTalk6	iClone Tutorial
Develop your facial animation in CrazyTalk 5.1 Pro for iClone	James Martin

1

Preparing Elsa Face Front View

Front View of Elsa Face is better for the test.
One of the way is to use MMD Elsa in iClone with long lens camera.

1-1

Open iClone5, load MMD Elsa, use Face Cam (use HotKey:J) to render her face.

1-2

Select Export/Image to export her Face for CrazyTalk7 Pro.

ElsaFace

1-3

Open CrazyTalk7, click Create New Actor

in the Top menu and select 'Import Image' to open Elsaface.

Actor Creator

1-4

Image Processing: Adjust her head region if needed and push the 'Next' button at the bottom.

ElsaCT7_v2

1-5

Auto-Fit Anchor Points: Drag the 4 control points to the indicated positons.

4Points_v2

1-6

Face Fitting Editor: Click 'Detail' icon, align the control points on each wireframe.

1-7

Preview the Animation:All, Head Rotate, Facial Expression, Eye Ball, Eye Blink and Mouth.

smile	angry	sad	happy	surprized	scared

1-8

Face Orientation: Select face style and rotate Blue mask to fit her head orientation.

Facce Style=Human, Strength=0.1

1-9

At Actor/Template: Click '03_Teeth' Settting icon, apply '03-White' or '06-Polished'.

1-10

Use Teeth Settings and adjust the size and position at Transform.

Teeth Settings: H-Size:=85, V-Size=85, Lip Corners=10 TeethSteeings_v1

1-11

Eye Settings: Click 'Eye Setting' icon

to open 'Eye Settings' panel.

1-12

Go to Actor/02_Eye and select Eyes from templates, or, prepare Elsa Eye Texture data and import it.

Eye Light can be added at the Eye Settings.

Eye1

Eye2

ElsaEye

1-13

Select Eye Ball-Iris, go to Diffuse, click 'Import' button to import Elsa eye.

Use Check Eyes

to compare the Eye Ball size and position to the original image.

[Case:Eye1 - MMD Elsa Eye (with Eye Light] EyeSettings_v1a

EyeSettings_v1a

[Case:Eye2 - Modified Eye] Eye Shadow: Opacity=33 / No Eyelash / Make Up: Opacity=0
EyeSettings_v2

EyeSettings_v2

1-14

Mask Editing: Click 'Mask Editing' icon

and Push 'Preview' button at the bottom. Push 'OK' button.

MaskEditor

1-15

Click the

button and save the imge in the Actor/01_Model/Custom as Elsa_CT7.

2

Importing Audio and Lip Sync Enhancement

2-1

Click the 'Import Audio' button, select 'Audio File' on the 'Import Audio' pop up.

ImportAudio

Top tool bar/Adding Voice and Motion/
'Import Audio' button

AudioImportOptions

Import Audio/ 'Audio File' button:
(.wav or .mp3)

Find the location and file: Let It Go.mp3

2-2

Select Talk Mode, Click 'Time Settings' select Frame, set Animation Length=6700 ～6800 Frames.

AutoMotionOption

2-3

Open "Let It Go" (Vocal and Sound).

2-4

Lipsync is generated by responding to all the Voice and Sound with Default Auto Motion.

Lipsync

2-5

Observe Lip Sync mouth shapes and enhance them by using Phonemes Library.

2-6

Replay the Audio and decide which Lip Syncs must be changed for better mouth replication.

2-7

Type the frame number into the 'Current Frame' space and push Enter to locate the Timeslider.

2-8

Use the

button on the Timeline Editor to undock and it can be expanded.

LPhoneme

By creating following tables (in Excel), editing work becomes easier in which
frame numbers are recorded under the corresponding music text.

(8 Tables in Total)

2-9

Push the target Lip Sync in the Lips track and double Click the new Phoneme icon in the Phoneme Library.

2-10

Organize the Timeline by deleting unnecessary Phonemes, then, replace or put new Phonemes.

LipSync

3

Adding Facial Animation in CrazyTalk7

3-1

Facial Animation for the character model can be established by using templates in the 'Auto Motion' Group and 'Motion Clip' Group in CrazyTalk7.

?

Type

Feature

Operation

Adjustment

AutoMotion

Animation
Generated from Audio

At Top Menu, Select Animation/Auto Motion Settings or click button.

Use Global/Reaction Settings, and adjust Strength.

0-150%

Motion Clip

Animation
Independent
from Audio

Use 'Clip Strength' by Mouse Right Click on each Motion Clip Bar in the Timeline.

ClipStrength

0-100%

No.	My Comments
1	'Auto Motion' is independent and generated by Audio Sound (Energy).
2	Timeline has both 'Auto Motion' track and 'Motion Clip' track separately.
3	Several Motion Clips can be applied on one selected Auto Motion in the Timeline.
4	Each 'Auto Motion' and 'Motion Clip' can be replaced, deleted and breaked in the Timeline.
5	Auto Motion can be copied to Motion Track and 'Face Key Editor' becomes available for further Key Frame Tweaking. This is useful to add the facial movement on the lipsync frame.
	Select the Keyframe on the Head bar and use 'Face Key Editor'.
6	'Face Key Editor' is shown by double-clicking the Face key in the Timeline.
7	By reducing 'Expressiveness' value on the Face key Editor, Lipsync setting become effective.
8		Face Puppet: ・Good to create short idle motions (in realtime manner). ・Can layer animations and blend data in a motion clip ・Can refine animations by selecting Personality Style and Mouth movement to animate the expressions. Profile: Male, Female, Youthful, Attractive, Wicked, Grumpy, Goofy
9		Auto Motion Settings: ・Can make Responding to Voice Energy (Voice Script) for more natural manner. ・Can add Body and Head motions to the Model. ・Can control motions on the Facial Parts: Brow, Nose, Eyebrows, Cheeks, Lips, jaw and Shoulder. ・Can make Global Settings such as Reaction Settings, Mode Settings and Motion Settings.

3-2

To confirm the Temlate structure and installed contents.

?

Find the suitable motions and clips in the Template Structure. (used in red color)

Type	Frame Length	Template List_1	Template List_2	Template List_3
Auto Motion	Full Length but can break	01_Functional	Auto Animation Library	Actor Swing, Head Swing, etc
			Muscle Enhancement	(omit)
			Rolling Eyes	(omit)
			Time Offset	(omit)
			（Others)	(omit)
		02_Scenario	01_Talk	(omit)
			02_Sing	AutoV1, Ballad, Happy Tune, Head Banging, Hip Hop, Jpyful, Lullaby, Opera, Opera_1, Opera_2, Pop Music, Sleepy Lullaby, Soul, Sweet Love Song
			03_Human sound	(omit)
			04_Reaction	(omit)
			05_Dance	(omit)
		03_Idle	(omit)	(omit)
Motion Clip	Partial Length	Auto Animation Library Vol 1	(omit)	(omit)
		Emotion	(omit)	(omit)
		Facial Expression Clips Vol 1	Anxious Mood	(omit)
			Funny Mood	(omit)
			Good Mood	Cheer, Curious, Excited, Laugh, Silly, Wink
			Mental Reaction	(omit)
			Physical Reaction	(omit)
			Sad Mood	Innocent, Sad, Sob, Desparate
			Thought	(omit)
		Movement	(omit)	(omit)
		Natural Facial Performance Vol 1	Angry	(omit)
			Charming	Affectionate
				Being Cute
				Blink
				Laugh
				Smile
				Tender
			Common	(omit)
			Proud	Act Innocent, Confident, Despise, Laugh, Sneer
			Sorrow	(omit)
			Treacherous	(omit)
			Weak	(omit)
		Natural Facial Performance Vol 2	(omit)	(omit)
		_Blank	(omit)	(omit)
		Kissing	-	-
		No	-	-
		Remembering	-	-
		Rolling Eyes_A	-	-
		Rolling Eyes_B	-	-
		Shivering	-	-
		Sneezing	-	-
		Yawning	-	-
		Yes	-	-

3-4

Right click on the folders and use the [Find File] for all file name and location.

3-5

To understand how to adjust Length of Motion Clip

Type	How to Adjust
Loop (Alt-L)	Streach the right edge of the Clip to Repeat it.
Speed (Alto-S)	Streach the right edge of the Clip to Extend it.

3-6

Apply some Motion and Clip for testing and observe the result.

3-7

Chose the suitable Motion in the Library to fit for the word of Song.

?

Editing Process Notes (Working stage and not a Final data)

	Words of Song (English)	(Japanese)	Frame	Applied Motion	Strength
	intro	前奏	1- 496	Default(Idle) Affectionate (1-477)	71
1	The snow glows white on the mountain tonight, Not a footprint to be seen. A kingdom of isolation, and it looks like I'm the queen. The wind is howling like this swirling storm inside. Couldn't keep it in, Heaven knows I tried.	降り始めた雪は足跡消して真っ白な世界に一人の私風が心にささやくのこのままじゃだめなんだと	496- 1340	Motion Clip/ FacialExpressionVol1/ Sad Mood/ Desperate_01 (488-1845)	60
2	Don't let them in, don't let them see. Be the good girl you always have to be. Conceal, don't feel, don't let them know. Well, now they know!	戸惑い傷つき誰にも打ち明けずに悩んでたそれももうやめよう	1340- 1845	Motion Clip/ FacialExpressionVol1/ Sad Mood/ Desperate_01 (488-1845)	60
3	Let it go, let it go. Can't hold it back anymore. Let it go, let it go. Turn away and slam the door. I don't care What they're going to say. Let the storm rage on, The cold never bothered me anyway.	ありのままの姿見せるのよありのままの自分になるの何も恐くない風よ吹け少しも寒くないわ	1845-2690	Motion Clip/ Natural Facial Performance Vol1/Proud/ Confident⇒change to Auto Motion	40
?	interlude	間奏	2690-2820	Auto Motion Motion Clip/Yes (2691- 3707)	70
4	It's funny how some distance, makes everything seem small. And the fears that once controlled me, Can't get to me at all It's time to see what I can do, to test the limits and break through. No right, no wrong, no rules for me. I am free!	悩んでたことが嘘みたいねだってもう自由よ何でも出来るどこまでやれるか自分を試したいのそうよ変わるのよ私	2820-3685	Motion Clip/Yes (3610- 3707)	70
5	Let it go, let it go. I am one with the wind and sky. Let it go, let it go. You'll never see me cry. Here I stand, and here I'll stay. Let the storm rage on.	ありのままで空へ風に乗ってありのままで飛び出してみるよ二度と涙は流さないわ	3685-4450	Auto Motion/ 02_Scenario/02_Sing/ Sing/Opera (3707-4325) Motion Clip/Yes (4320-4460)	22 50
	interlude	間奏	4450-4680	Auto Motion/02_Scenario/ 05_Dance/Head Shake (4460-4670)
6	My power flurries through the air into the ground. My soul is spiraling in frozen fractals all around And one thought crystallizes like an icy blast I'm never going back; the past is in the past!	冷たく大地を包み込み高く舞い上がる思い出描いて花咲く氷の結晶のように輝いていたいもう決めたの	4680-5532	Interested Auto Motion/ 02_Scenario/02_Sing/ Sing/Pop Music (4690-5974)	100
7	Let it go, let it go. When I'll rise like the break of dawn. Let it go, let it go. That perfect girl is gone Here I stand, in the light of day.	これでいいの自分を好きになってこれでいいの自分を信じて	5532-6214	Auto Motion/ 02_Scenario/02_Sing/ Sing/Pop Music Confident_01 (5974-6208）	100
8	Let the storm rage on! The cold never bothered me anyway (Door Closed Sound)	光浴びながら歩き出そう少しも寒くないわ	6214-6499 (End: 6621)	Auto Motion/ 02_Scenario/02_Sing/ Sing/Pop Music Blink (6393-6480) Surprize (6511- 6545) Blink_A (6613-6662)	100

4

Adding Movement and Emotion by Face Key Editor

4-1

Click 'Face Key Edit' icon on the Timeline panel, then, Face Key Editor opens.

4-2

Move the Timeslider to the target frame for the editing and open 'Muscle'.

4-3

Imagin the Elsa motion and select Facial Features (Parts) in the Feature Selection Pane.

4-4

Puppet Elsa Face by selecting buttons and move Mouse to express the expected movement.

Buttons (3D Movement

, Rotate ,Move

, and Head Front and Back

Mouse toward to puppet Elsa
FaceKeyEditor

FaceKeyEditor

4-5

Use 'Dafault Key' in the Face Key Editor to reset the edited data on that key.

4-6

Move the Timeslider to the next target frame and use the Mouth toward the next puppeting.

4-7

Observe the animation flow and move the Timeslider to the frame needed to correct.

4-8

Timeline View of the Practice

Timeline

By Using the 'Puppeteering' Panel, each and every muscle movement for her facial animation can be edited.

(CT6) Click the 'Emotive Tab to add an emotion to the overall Facial Animation data on the Timeline

5

Background Settings

5-1

Click 'Mask Editing' icon and push 'Preview' and 'OK' button to remove background color.

MaskEditor

5-2

Click 'Background Settings' icon, select 'Import Image' and open the smple image.

Background_v1

5-3

' Move', 'Rotate' and 'Scale' button is available in the Left-side Tool Bar for Elsa positioning.

6

Making of Report Video

6-1

Click the 'Export'

button at the top menu and 'Export Settings' panel is shown.

6-2

Open 'Video' tab and select Format, frame size, frame rate and Export Range.

Format	Quality Compressor	Frame Size	Frame Rate	Export Fange
WMV	Video=8, Audio=8	1024x600	30	All (6700)

6-3

Click the 'Export' button, type the name and start saving.

6-4

Open Project tab/Custom, and save the Project by clicking the 'Add|Save As' button.

7

Exporting the Script to iClone (Pending)

.CTS Script of Facial expression produced in CrazyTalk6 can be imported to iClone5.
In iClone, click the Animation Tab from the top and the Facial Animation button, then
locate the new CTS file in the Custom Folder, double click to apply.

However, Crazy talk7 uses .CT7Script format which is not imported to iClone5 at this moment.

3DXchange5 Pipeline test series 17: Elsa sings Let It Go
2014.11.24 Hirospot (Personal Site)