Creating an Instance for Interaction

Table of Content

• Creating a New Instance for interaction
• Designing a new multi-modal interaction episode
  • GRETA feeling GREAT
  • Making Greta count
    • Creating the 'Great' gesture
    • Creating the XML file
    • Defining the lexicon pair
  • Misc

Creating a New Instance for interaction

Now we can use the newly created expression and gesture to create a new sample interaction. To use the newly added gestures and expressions, we have to define the lexicon pair (Communicative act, multimodal behavior) in <GRETA_DIR>/bin/BehaviorPlanner/IntentionLexicon/lexicon_Camille.xml.

Open <GRETA_DIR>/bin/BehaviorPlanner/IntentionLexicon/lexicon{_<character>}.xml, manually add the gesture as follows:

<behaviorset name="{type}-{name}">
	<signals>
		<signal id="1" name="{type}={name}" modality="gesture">
		</signal>
	</signals>
	<constraints>
		<core>
			<item id="1"/>
		</core>
	</constraints>
</behaviorset>

Here is an example:

<behaviorset name="deictic-new_gesture1">
	<signals>
		<signal id="1" name="deictic=new_gesture1" modality="gesture"></signal>
	</signals>
	<constraints>
		<core>
			<item id="1"/>
		</core>
	</constraints>
</behaviorset>

• Whenever you do a change in a Library, you need to reload the Greta-Platform to be able to manipulate it. Libraries are loaded into the system only when it is first run.
• Reload the Greta platform to access the newly added gesture and expression.
• Now, to create a new interaction, create a copy of the 1-Welcome.XML file and try changing the existing gesture and emotion to the newly created ones.

Here is an example,

<fml>
	<performative id="p1" type="new_gesture1" start="s1:tm1" end="s1:tm2" importance="1.0"/>
	<deictic id="d1" type="selftouch" start="s1:tm2" end="s1:tm3" importance="1.0"/>
	<emotion id="e1" type="new_expression1" start="s1:tm2" end="s1:tm3" importance="1.0"/>
	<performative id="p2" type="propose" start="s1:tm3" end="s1:tm5" importance="1.0"/>
</fml>

Designing a new multi-modal interaction episode

Let's try to build a more complex instance for multimodal interaction that involves hand gestures, hand shapes, speech and facial expressions together. For a reference, go and watch the video on 'Common french gestures' and try to make Greta character do one of the gestures while speaking in French.

Here's the link to the video : https://youtu.be/tn2kew1FiWw

GRETA feeling GREAT !!

Let's go with the 'feeling great' gesture and try to build a better interaction involving this gesture.

We will create an instance where Greta asks the user to count from one to three together and ends the interaction with the 'great' gesture.

Here's the dialogue for the speech:

Hello,Let's count together from one to three.

Okay. Let's start.

One. Two. Three

Great

To design the interaction, it is better to break down the script into blocks that convey specific intents, gestures and emotions.

1. [Performative=Greet][Emotion=Joy] Hello, Let's count together from one to three.**
2. [Performative=Ask] Okay. Let's start.
3. [Performative=Count] One.
4. [Performative=Count] Two.
5. [Performative=Count] Three.
6. [Performative=Compliment][Emotion=Blend of Surprise and Happy] Great

Making Greta count

For the instances from 1 - 2 you can find predefined gestures in the Greta platform. So let's try to create few new gestures in Greta that involves both arm movements and handshape manipulations.

To make Greta count using fingers, we have to define the handshapes with one, two and three fingers respectively using the handshape editor.

• To edit the handshape graphically, add the HandShape Editor module from the 'Tools' category and connect it to the MPEG4 Animatable block .
• The hand shape, in this case, refers only to the finger positions. To create a new shape, do the following:
• Open <GRETA_DIR>/bin/BehaviorRealizer/AnimationLexicon/HandShape.xml
• Copy one of the hand shape, paste it after the last hand shape, give it a new name, and save the XML file and restart the Greta.
• Back to the Hand Shape Editor, choose the hand shape you have created, and start playing around with the joints there.

After creating the handshapes, make sure you reload Greta to make the changes effective.

Here's the XML equivalent for the handshapes for counting :

<HandShape id="count_one">
	<r_index0 x="0.000" y="0.000" z="0.000"/>
	<r_index2 x="0.000" y="0.000" z="0.000"/>
	<r_index1 x="0.000" y="0.000" z="0.000"/>
	<r_middle1 x="0.000" y="0.000" z="0.000"/>
	<r_index3 x="0.000" y="0.000" z="0.000"/>
	<r_middle0 x="0.000" y="0.000" z="0.000"/>
	<r_thumb3 x="-16.000" y="0.000" z="0.000"/>
	<r_pinky3 x="0.000" y="0.000" z="0.000"/>
	<r_thumb2 x="-7.000" y="0.000" z="0.000"/>
	<r_pinky2 x="0.000" y="0.000" z="0.000"/>
	<r_thumb1 x="-14.000" y="2.000" z="-2.000"/>
	<r_pinky1 x="0.000" y="0.000" z="0.000"/>
	<r_pinky0 x="0.000" y="0.000" z="0.000"/>
	<r_middle2 x="0.000" y="0.000" z="0.000"/>
	<r_ring3 x="0.000" y="0.000" z="0.000"/>
	<r_middle3 x="0.000" y="0.000" z="0.000"/>
	<r_ring2 x="0.000" y="0.000" z="0.000"/>
	<r_ring1 x="0.000" y="0.000" z="0.000"/>
	<r_ring0 x="0.000" y="0.000" z="0.000"/>
</HandShape>
<HandShape id="count_two">
	<r_index0 x="0.000" y="0.000" z="0.000"/>
	<r_index2 x="0.000" y="0.000" z="0.000"/>
	<r_index1 x="0.000" y="0.000" z="0.000"/>
	<r_middle1 x="0.000" y="0.000" z="0.000"/>
	<r_index3 x="0.000" y="0.000" z="0.000"/>
	<r_middle0 x="0.000" y="0.000" z="0.000"/>
	<r_thumb3 x="-16.000" y="0.000" z="0.000"/>
	<r_pinky3 x="0.000" y="0.000" z="0.000"/>
	<r_thumb2 x="-7.000" y="0.000" z="0.000"/>
	<r_pinky2 x="0.000" y="0.000" z="0.000"/>
	<r_thumb1 x="-14.000" y="2.000" z="-2.000"/>
	<r_pinky1 x="0.000" y="0.000" z="0.000"/>
	<r_pinky0 x="0.000" y="0.000" z="0.000"/>
	<r_middle2 x="0.000" y="0.000" z="0.000"/>
	<r_ring3 x="0.000" y="0.000" z="0.000"/>
	<r_middle3 x="0.000" y="0.000" z="0.000"/>
	<r_ring2 x="0.000" y="0.000" z="0.000"/>
	<r_ring1 x="0.000" y="0.000" z="0.000"/>
	<r_ring0 x="0.000" y="0.000" z="0.000"/>
</HandShape>
<HandShape id="count_three">
	<r_index0 x="0.000" y="0.000" z="0.000"/>
	<r_index2 x="0.000" y="0.000" z="0.000"/>
	<r_index1 x="0.000" y="0.000" z="0.000"/>
	<r_middle1 x="0.000" y="0.000" z="0.000"/>
	<r_index3 x="0.000" y="0.000" z="0.000"/>
	<r_middle0 x="0.000" y="0.000" z="0.000"/>
	<r_thumb3 x="-16.000" y="0.000" z="0.000"/>
	<r_pinky3 x="0.000" y="0.000" z="0.000"/>
	<r_thumb2 x="-7.000" y="0.000" z="0.000"/>
	<r_pinky2 x="0.000" y="0.000" z="0.000"/>
	<r_thumb1 x="-14.000" y="2.000" z="-2.000"/>
	<r_pinky1 x="0.000" y="0.000" z="0.000"/>
	<r_pinky0 x="0.000" y="0.000" z="0.000"/>
	<r_middle2 x="0.000" y="0.000" z="0.000"/>
	<r_ring3 x="0.000" y="0.000" z="0.000"/>
	<r_middle3 x="0.000" y="0.000" z="0.000"/>
	<r_ring2 x="0.000" y="0.000" z="0.000"/>
	<r_ring1 x="0.000" y="0.000" z="0.000"/>
	<r_ring0 x="0.000" y="0.000" z="0.000"/>
</HandShape>

You can also try to create your own handshapes in the Handshape Editor window.

Once you have created the handshapes, open the GestureEditor window to create the gestures for the same.

• The newly created gesture will be saved in <GRETA_DIR>/bin/BehaviorRealizer/gestuary.xml file.

For example, the XML description for the gesture for the counting positions are as follows:

<gesture category="performative" id="CountOne_POS_R">
	<phase>
		<hand side="RIGHT">
			<uniformPosition>
				<horizontalLocation fixed="false"
					overridable="false" value="0.33"/>
				<verticalLocation fixed="false" overridable="false" value="0.36"/>
				<frontalLocation fixed="false" overridable="false" value="0.5"/>
			</uniformPosition>
			<handShape overridable="false" value="COUNT_ONE"/>
			<wristOrientation global="false" overridable="false"
				w="0.8109630942344666" x="0.0"
				y="0.5850972533226013" z="0.0"/>
		</hand>
	</phase>
</gesture>
<gesture category="performative" id="CountThree_POS_R">
	<phase>
		<hand side="RIGHT">
			<uniformPosition>
				<horizontalLocation fixed="false"
					overridable="false" value="0.33"/>
				<verticalLocation fixed="false" overridable="false" value="0.36"/>
				<frontalLocation fixed="false" overridable="false" value="0.52"/>
			</uniformPosition>
			<handShape overridable="false" value="COUNT_THREE"/>
			<wristOrientation global="false" overridable="false"
				w="0.8109631195052179" x="0.0"
				y="0.5850972729404622" z="0.0"/>
		</hand>
	</phase>
</gesture>
<gesture category="performative" id="CountTwo_POS_R">
	<phase>
		<hand side="RIGHT">
			<uniformPosition>
				<horizontalLocation fixed="false"
					overridable="false" value="0.33"/>
				<verticalLocation fixed="false" overridable="false" value="0.36"/>
				<frontalLocation fixed="false" overridable="false" value="0.52"/>
			</uniformPosition>
			<handShape overridable="false" value="COUNT_TWO"/>
			<wristOrientation global="false" overridable="false"
				w="0.8109630942344666" x="0.0"
				y="0.5850972533226013" z="0.0"/>
		</hand>
	</phase>
</gesture>

Creating the 'Great' gesture

You can refer to this video of the gesture before starting with the editing.

• Link to the 'Great' gesture reference: https://youtu.be/CEMGcuzQzv4

Let's start with the Handshape editor first and try to create the 'thumbs up' shape needed for the gesture.

• To edit the handshape graphically, add the HandShape Editor module from the 'Tools' category and connect it to the MPEG4 Animatable block.
• The hand shape, in this case, refers only to the finger positions. To create a new shape, do the following:
• Open <GRETA_DIR>/bin/BehaviorRealizer/AnimationLexicon/HandShape.xml
• Copy one of the hand shape, paste it after the last hand shape, give it a new name, and save the XML file and restart the Greta.
• Back to the Hand Shape Editor, choose the hand shape you have created, and start playing around with the joints there.

Here's the XML equivalent of the handshape;

<HandShape id="thumbs_up">
	<r_index0 x="0.000" y="0.000" z="1.000"/>
	<r_index2 x="0.000" y="0.000" z="100.000"/>
	<r_index1 x="12.000" y="10.000" z="76.000"/>
	<r_middle1 x="6.000" y="0.000" z="86.000"/>
	<r_index3 x="0.000" y="0.000" z="90.000"/>
	<r_middle0 x="0.000" y="0.000" z="2.000"/>
	<r_thumb3 x="17.000" y="0.000" z="0.000"/>
	<r_pinky3 x="0.000" y="0.000" z="90.000"/>
	<r_thumb2 x="8.000" y="0.000" z="0.000"/>
	<r_pinky2 x="0.000" y="0.000" z="110.000"/>
	<r_thumb1 x="8.000" y="0.000" z="0.000"/>
	<r_pinky1 x="-8.999" y="-15.999" z="87.000"/>
	<r_pinky0 x="0.000" y="0.000" z="5.000"/>
	<r_middle2 x="0.000" y="0.000" z="110.000"/>
	<r_ring3 x="0.000" y="0.000" z="90.000"/>
	<r_middle3 x="0.000" y="0.000" z="90.000"/>
	<r_ring2 x="0.000" y="0.000" z="110.000"/>
	<r_ring1 x="-2.000" y="0.000" z="89.000"/>
	<r_ring0 x="0.000" y="0.000" z="4.000"/>
</HandShape>

Once you have created the handshape, now start creating the gesture using the arms.

Now, open the GestureEditor window and start editing the gesture. To make the character execute a natural movement during the gesture, we will make use of multiple strokes in the GestureEditor. For the 'Great'gesture we will break the gesture into three strokes with changes in the arm positions.

To add a new stroke, go to Gesture> Add Keyframe. Select the appropriate handshapes for each stroke and complete the gesture editing.

• The newly created gesture will be saved in <GRETA_DIR>/bin/BehaviorRealizer/gestuary.xml file. New gestures can also be defined in this file using an XML description of the gestures.

Example:

<gesture category="performative" id="Great_Ges_B">
	<phase>
		<hand openness="1.9399999999999995" side="LEFT">
			<uniformPosition>
				<horizontalLocation fixed="true" overridable="false" value="0.415"/>
				<verticalLocation fixed="true" overridable="false" value="0.2"/>
				<frontalLocation fixed="true" overridable="false" value="0.8"/>
			</uniformPosition>
			<handShape overridable="false" value="THUMBS_UP"/>
			<wristOrientation global="false" overridable="false"
				w="0.7757092118263245" x="0.22937114536762238"
				y="0.5638008117675781" z="-0.1667114943265915"/>
		</hand>
		<hand openness="2.0" side="RIGHT">
			<uniformPosition>
				<horizontalLocation fixed="false"
					overridable="false" value="0.4"/>
				<verticalLocation fixed="false" overridable="false" value="0.2"/>
				<frontalLocation fixed="false" overridable="false" value="0.8"/>
			</uniformPosition>
			<handShape overridable="false" value="THUMBS_UP"/>
			<wristOrientation global="false" overridable="false"
				w="0.7693592309951782" x="0.2498445063829422"
				y="-0.5591855645179749" z="0.1815919429063797"/>
		</hand>
	</phase>
	<phase>
		<hand openness="1.9399999999999995" side="LEFT">
			<uniformPosition>
				<horizontalLocation fixed="true" overridable="false" value="0.415"/>
				<verticalLocation fixed="true" overridable="false" value="0.8"/>
				<frontalLocation fixed="true" overridable="false" value="0.8"/>
			</uniformPosition>
			<handShape overridable="false" value="THUMBS_UP"/>
			<wristOrientation global="false" overridable="false"
				w="0.7757092118263245" x="0.22937114536762238"
				y="0.5638008117675781" z="-0.1667114943265915"/>
		</hand>
		<hand openness="2.0" side="RIGHT">
			<uniformPosition>
				<horizontalLocation fixed="false"
					overridable="false" value="0.4"/>
				<verticalLocation fixed="false" overridable="false" value="0.8"/>
				<frontalLocation fixed="false" overridable="false" value="0.8"/>
			</uniformPosition>
			<handShape overridable="false" value="THUMBS_UP"/>
			<wristOrientation global="false" overridable="false"
				w="0.7693592309951782" x="0.2498445063829422"
				y="-0.5591855645179749" z="0.1815919429063797"/>
		</hand>
	</phase>
	<phase>
		<hand openness="2.0" side="LEFT">
			<uniformPosition>
				<horizontalLocation fixed="false"
					overridable="false" value="0.4"/>
				<verticalLocation fixed="false" overridable="false" value="0.0"/>
				<frontalLocation fixed="false" overridable="false" value="0.8"/>
			</uniformPosition>
			<handShape overridable="false" value="THUMBS_UP"/>
			<wristOrientation global="false" overridable="false"
				w="0.7693592309951782" x="0.2498445063829422"
				y="0.5591855645179749" z="-0.1815919429063797"/>
		</hand>
		<hand openness="1.9399999999999995" side="RIGHT">
			<uniformPosition>
				<horizontalLocation fixed="true" overridable="false" value="0.415"/>
				<verticalLocation fixed="true" overridable="false" value="0.0"/>
				<frontalLocation fixed="true" overridable="false" value="0.8"/>
			</uniformPosition>
			<handShape overridable="false" value="THUMBS_UP"/>
			<wristOrientation global="false" overridable="false"
				w="0.7757092118263245" x="0.22937114536762238"
				y="-0.5638008117675781" z="0.1667114943265915"/>
		</hand>
	</phase>
</gesture>

Creating the XML file

Go on and create a new blank XML file for the new interaction. The XML file contains of both the BML and FML descriptions. Let's start with the BML part that deals with the speech content and parameters.

<bml>
	<speech id="s1" start="0.0" language="english" voice="marytts" type="SAPI4" text="">
		<description level="1" type="gretabml">
			<reference>tmp/from-fml-apml.pho</reference>
		</description>

		<tm id="tm1"/>
			Hello. Lets count together from one to three.
		<tm id="tm2"/>
			Okay. Lets start.
		<tm id="tm3"/>
			One.
				<tm id="tm4"/>
			Two.
				<tm id="tm5"/>
			Three.
		<tm id="tm6"/>
			Great.
		<tm id="tm7"/>


		<pitchaccent id="pa1" type="HStar" level="medium" start="s1:tm2" end="s1:tm3" importance="1"/>
		<boundary type="LL" id="b1" start="s1:tm7" end="s1:tm7+0.5"/>
	</speech>
</bml>

Now let's move on to the FML part that specifies the gestures and expressions for each time instances.

We will use the following gestures and facial expressions for creating the interaction.

1. [Performative=Greet][Emotion=Joy] Hello, Let's count together from one to three.**
2. [Performative=Ask] Okay. Let's start.
3. [Performative=Count] One .
4. [Performative=Count] Two.
5. [Performative=Count] Three.
6. [Performative=Compliment][Emotion=Blend of Surprise and Happy] Great

Now, let's write the FML for the same.

<fml>
	<performative id="p1" type="Greeting" start="s1:tm1" end="s1:tm2" importance="1.0"/>
		<emotion id="e1" type="Joyful" start="s1:tm1" end="s1:tm2" importance="1.0"/>
	<performative id="p2" type="Asking" start="s1:tm2" end="s1:tm3" importance="1.0"/>
	<performative id="p3" type="CountOne" start="s1:tm3" end="s1:tm4" importance="1.0"/>
		<performative id="p4" type="CountTwo" start="s1:tm4" end="s1:tm5" importance="1.0"/>
		<performative id="p5" type="CountThree" start="s1:tm5" end="s1:tm6" importance="1.0"/>
	<performative id="p6" type="Compliment" start="s1:tm6" end="s1:tm7" importance="1.0"/>
		<emotion id="e2" type="SurpriseandHappy" start="s1:tm6" end="s1:tm7" importance="1.0"/>
</fml>

Defining the lexicon pair

Now let us define the lexicon pair (Communicative act, multimodal behavior) in <GRETA_DIR>/bin/BehaviorPlanner/IntentionLexicon/lexicon_Camille.xml.

<behaviorset name="performative-Greeting">
	<signals>
		<signal id="1" name="performative=Greet_Ges_R" modality="gesture"/>
	</signals>
</behaviorset>

<behaviorset name="emotion-Joyful">
	<signals>
		<signal id="1" name="faceexp=joyStrong" modality="face" repetitivity="0" min="1" max="2" probability_start="1.0" probability_end="0.0"/>
	</signals>
</behaviorset>

<behaviorset name="performative-Asking">
	<signals>
		<signal id="1" name="performative=Ask_Ges_L" modality="gesture"/>
	</signals>
</behaviorset>

<behaviorset name="performative-CountOne">
	<signals>
		<signal id="1" name="performative=CountOne_Pos_R" modality="gesture"/>
	</signals>
</behaviorset>

<behaviorset name="performative-CountTwo">
	<signals>
		<signal id="1" name="performative=CountTwo_Pos_R" modality="gesture"/>
	</signals>
</behaviorset>

<behaviorset name="performative-CountThree">
	<signals>
		<signal id="1" name="performative=CountThree_Pos_R" modality="gesture"/>
	</signals>
</behaviorset>

<behaviorset name="performative-Compliment">
	<signals>
		<signal id="1" name="performative=Great_Ges_B" modality="gesture"/>
	</signals>
</behaviorset>

<behaviorset name="emotion-SurpriseandHappy">
	<signals>
		<signal id="1" name="faceexp=surprise-happy" modality="face" repetitivity="0" min="1" max="2" probability_start="1.0" probability_end="0.0"/>
	</signals>
</behaviorset>

Now that everything is defined, you can go send the newly created XML to Greta and see the output.

Here's the video of the final result: https://youtu.be/VcaYTNdeSfE

Misc

• To record a video: Use the CaptureController and VideoCaptureOutput modules to record the video of the player output. Connect the CaptureController to PlayerOgre and XuggleVideoCapture modules to start recording. Go to capture controller: press 'video'. It starts recording right away. You then need to run the FML file from FML File reader. Once the FML file is over, press Stop on the Capture Controller. An AVI file is saved in the main directory of the software.
• Do version the code. There is no undo function and most of the settings are saved in the project as text files. The easiest way to restore your project after you do something bad is by having it versioned in the first place, and then wiping out your destructive changes.
• After you edit an XML file manually (except the FML file), you have to restart Greta. Greta loads those XMLs while Greta is being started. Therefore, your changes on the XML file(s) are not going to be detected by Greta unless you restart Greta. The FML files are the exception of this rule.
• Sample videos of Greta

Side camera view : https://youtu.be/AU1zmMaYgIc