Kinect screen with Brainiac

From Engineered Arts Wiki
Jump to navigation Jump to search

Please note this screen is no longer available, and has been replaced in newer RoboThespians and in SociBot with the simplified Sensors screen. These pages are retained for reference only.

Overview

Please note this screen is no longer available, and has been replaced in newer RoboThespians and in SociBot with the simplified Sensors screen. These pages are retained for reference only.


The depth sensor is used by RoboThespian to detect people, track and focus on an individual he is interacting with, and detect and mimic the body positions of people in his field of view. An early video of RoboThespian mimicking body movements can be seen here - http://www.youtube.com/watch?v=VGt_W6hwnoo

The 'Auto' or 'Kinect Screen' (name varies depending on software version) allows control of the features accessible through the Kinect sensor. Please note that these are beta features and development and improvements are ongoing.

At the top of the screen there are two views as seen by the Kinect sensor:

  • The left view shows the image from the Kinect's colour camera
  • The right view shows the image from the Kinect's infra red camera, used to help calculate depth or distance from the Kinect of objects in its field of view


The 3 buttons at the left and right edge of the screen will apply labels to anyone detected by the Kinect

  • Show Label = label anyone in view of Kinect with a number in the order that they appear
  • Show Status = show what Kinect is doing with a person, for example, 'Looking for pose' or 'Calibrating'
  • Show Skeleton = shows what position Kinect thinks a person's skeleton is in. This is a simplified skeleton.

1280x720 kinect screen initial 3.jpg

(Click on the picture for a larger image)

Brainiac speech interaction

Brainiac is a feature (not currently in development) that offers interactivity through talking with the robot. Please see Brainiac

Follow Person Feature

RoboThespian is able to follow people in view of the Kinect. He will move his eyes and head to look at them.

This feature along with more complex behaviours is now achieved using Tritium Control Functions on 2015 and onwards model RoboThespians or SociBots running on Tritium

On older robots running on IOServe the following functions were achieved in the Touchscreen Interface software. This information is kept for reference:

Click on 'Follow Person' (also called 'Track Player' or 'Follow Player' in some versions) - the button should change colour to green.

In 'Auto Detect' mode RoboThespian will follow the first person to come into his field of view automatically. To stop RoboThespian following people automatically click on the 'Auto-Detect' button - it will change to blue. Click it again to turn this feature back on - it will turn green again.

You can manually select the person RoboThespian should follow by clicking on the number next to the person in either of the camera views at the top of the screen.


1280x720 kinect follow person.jpg


Mimic Person Feature

RoboThespian is able to mimic the movements of people in his view.

This feature along with more complex behaviours is now achieved using Tritium Control Functions on 2015 and onwards model RoboThespians or SociBots running on Tritium

On older robots running on IOServe the following functions were achieved in the Touchscreen Interface software. This information is kept for reference:

Movements mimicked currently comprise arm movement and torso rotation.

Hands and head are not mimicked. Robothespian will look at the person he is mimicking and follow them with his eyes and head.

Press 'Mimic Player' (also called 'Copy Player' in some versions) so the button turns green to enable this mode. If 'Auto Detect' is enabled (i.e. the 'Auto Detect' button is green), RoboThespian will be ready to mimic the first person he's seen. Alternatively chose a person to mimic by clicking on the number next to the person in either of the camera views at the top of the screen.

Once a person is selected show RoboThespian that you are ready to be mimicked by holding your arms in the air as shown below for several seconds. A simplified skeleton should appear over your body in both camera views. RoboThespian will begin to mimic your movements.

To stop RoboThespian mimicking you press 'Mimic Player' again so that the button turns blue.

You can see a short video of RoboThespian mimicking a person here - http://youtu.be/VGt_W6hwnoo

1280x720 kinect mimic person.jpg

(Click on the image for a larger version)

Verbal instructions

RoboThespian can give you verbal instructions and show you how to pose - press on the 'Speak to Player' button. To hear the instructions in other languages (currently German, but others are available on request), press the 'Deutsch' button.

640x480 - kinect verbal instructions.jpg


Simplified 'Auto' screen

The latest version of RoboThespian has a simplified Auto screen with on-screen instructions:

Minimal Auto Screen.jpg

Troubleshooting

RoboThespian is looking to the side / above / below the person he is following.

Check that the Kinect is correctly positioned. See Positioning_Kinect

If there are no views at the top of the screen...

If the two Kinect views are missing. First check that the Kinect is plugged in to the touchscreen PC and connected to the mains. Then press the 'Restart Kinect' button to restart the Kinect drivers.

If the camera views have frozen

Occasionally the views from the Kinect can freeze. Should this happen press the 'Reset Camera' button.

If people do not appear in the right hand (infra-red / depth) view

Increase the depth of view by using the 'Kinect Maximum User Range - 1000 to 6000' slider at the bottom of the screen

640x480 - kinect troubleshooting 2.jpg