Teacher’s Guide to Kinect and Gesture-Based Learning

Posted by johnnykissko on May 19, 2011

Kinect is more than a gaming device; realistically, it’s an input device that can facilitate gaming (think of it like the mouse and keyboard). With a few easy tweaks that anyone can do, we can use this device to transform learning in classrooms. You don’t have to be a programmer to do this, but you do need to have a passion for innovative learning.

If we can get a community of educators and students to understand what this means and how to do it, we can develop an entire bank of gesture-based “drivers” to support all software currently in existence.

In other words, with this tool, we can we easily take existing applications (Google Earth, flash-based activities, calculators, etc.) and integrate them with Kinect. Let’s start with the end in mind; watch my videos below to see what I’m talking about. Keep in mind that I did this in a matter of three minutes. My intent wasn’t to program it to perfection, but rather to show that the process is easy to grasp.

Playing piano with Kinect

All you’re doing is mapping your gestures to keyboard characters and mouse commands. Once you’ve grasped the concept, it’s very, very simple to do.

What can we do with this? Take all the existing software and make it Kinect compatible. We don’t have to wait; we, as a community, can easily develop and deliver gesture-based learning opportunities with the bank of software already in existence. After you watch the video and read the guide below, think of software that you want to see “Kinectified” and put it in the forums. From there, I’ll establish a directory of files that can be used for any software.

This video I created will give you a better visual of how to get started; I highly recommend that you watch it. Even with the video, you’ll still need to reference the guide below to find the list of commands.

Disclaimer:

This guide looks lengthy, but it’s only because it has a list of all the commands…..please don’t let that intimidate you. The full guide is available from Institute of Creative Technologies; I’ve just edited and restructured their guide to be more appealing to the K-12 education sector. If you don’t yet have a Kinect, please consider purchasing one from a source that will support the costs associated with hosting this site.

——————————————————————————————————–

Step 1: Install Software

To use FAAST, you will need to download and install the following software:

1. OpenNI Unstable Build for Windows v1.0.0.25
2. PrimeSense NITE Unstable Build for Windows v1.3.0.18
   During NITE installation, enter the free license key from OpenNI: 0KOIk2JeIBYClPWVnMoRKn5cdY4=
3. Hardware drivers for your sensor (only one of the following)
   • Microsoft Kinect: SensorKinect Module – download Win32 installer from Bin folder
   • PrimeSensor: PrimeSensor Module – this does NOT work with the Kinect
4. Download,unzip, and run FAAST

FAAST should then run out-of-the-box; no additional installation or setup is necessary. If you encounter an error on startup, you may also need to install the Microsoft Visual C++ 2008 SP1 Redistributable Package.

——————————————————————————————————–

Step 2: Mapping Keys with Gestures

Once you have the FAAST software running, click on the “Actions” tab. Here, you’ll see a list of default commands set to run with Kinect. They appear in this format:

action_name      action_threshold      virtual_event_type      virtual_event_name

action_name – The name of the pose or gesture action that the user performs.

action_threshold – The minimum threshold for activation (this varies by event).

virtual_event_type – The type of virtual event to be generated.

virtual_event_name – The specific event to be generated.

For example, the following command will press the “w” key when the right hand extends more than 18 inches in front of the shoulder, and then release it when it returns back behind that distance:

right_arm_forwards 18 key_hold w

Here is the list of all available actions (which fall under “action_name”):

Action List

Action Name	What the Action Means
lean_left	angular body lean left (degrees)
lean_right	angular body lean right(degrees)
lean_forwards	angualr body lean forwards (degrees)
lean_backwards	angular body lean back (degrees)
turn_left	angular amount of left body turn (degrees)
turn_right	angular amount of right body turn(degrees)
left_arm_forwards	forward distance from left hand to shoulder (inches)
left_arm_down	downward distance from left hand to shoulder (inches)
left_arm_up	upward distance from left hand to shoulder (inches)
left_arm_out	sideways distance from left hand to shoulder (inches)
left_arm_across	sideways distance from left hand across body to shoulder (inches)
right_arm_forwards	forward distance from right hand to shoulder (inches)
right_arm_down	downward distance from right hand to shoulder (inches)
right_arm_up	upward distance from right hand to shoulder (inches)
right_arm_out	sideways distance from right hand to shoulder (inches)
right_arm_across	sideways distance from right hand across body to shoulder (inches)
left_foot_forwards	forward distance from left hip to foot (inches)
left_foot_sideways	sideways distance from left hip to foot (inches)
left_foot_backwards	backwards distance from left hip to foot (inches)
left_foot_up	height of left foot above other foot on ground (inches)
right_foot_forwards	forward distance from right hip to foot (inches)
right_foot_sideways	sideways distance from right hip to foot (inches)
right_foot_backwards	backwards distance from right hip to foot (inches)
right_foot_up	height of right foot above other foot on ground (inches)
jump	height of both feet above ground (inches)
crouch	crouch distance, calculated as current height subtracted from standing height (inches)
walk	height of each step above ground when walking in place (inches)
push	velocity (inches/sec.)
swipe_up	velocity (inches/sec.)
swipe_down	velocity (inches/sec.)
swipe_left	velocity (inches/sec.)
swipe_right	velocity (inches/sec.)
circle	radius (inches)

  

Event List (What You Hope Happens After Performing a Specific Gesture)

Virtual Event Type	Virtual Event Name
key_press	key to press when the action is first detected (either a single character or a special key from the table below)
key_hold	key to hold down while the action is performed (either a single character or a special key from the table below)
key_type	a string of characters to type into the active window (no spaces allowed)
mouse_click	mouse button to click when the action is first detected (left_button, right_button, or middle_button)
mouse_double_click	mouse button to double click when the action is first detected (left_button, right_button, or middle_button)
mouse_hold	mouse button to hold down while the action is performed (left_button, right_button, or middle_button)
faast	input emulator control event (pause, resume, or stop)

Special Keys

backspace caps_lock up_arrow delete numpad_2 multiply f3 f11 f19 left_shift browser_refresh media_next_track

tab escape right_arrow help numpad_3 add f4 f12 f20 right_shift browser_stop media_previous_track

clear space down_arrow left_windows numpad_4 separator f5 f13 f21 left_control browser_search media_stop

enter page_up select right_windows numpad_5 subtract f6 f14 f22 right_control browser_favorites media_play_pause

shift page_down print applications numpad_6 decimal f7 f15 f23 left_alt browser_home start_mail

control end execute sleep numpad_7 divide f8 f16 f24 right_alt volume_mute media_select

alt home print_screen numpad_0 numpad_8 f1 f9 f17 num_lock browser_back volume_down start_application_1

pause left_arrow insert numpad_1 numpad_9 f2 f10 f18 scroll_lock browser_forward volume_up start_application_2

As another example, the following command will click the left mouse button when the left foot kicks forwards more than 12 inches in front of the hip:

left_foot_forwards 12 mouse_click left_button

Step 3: Calibrate and run the software!

To calibrate, the user must stand in this pose for several seconds until a stick figure appears, as shown in the image below.

Skeleton Calibration Pose

Once you’ve been detected and calibrated the software, click “Start emulator” , make your application you’re wanting to use gestures with become the active window, and use gestures to operate your software!

What software do you want to see support gestures? Check out the directory of applications that the community has (unofficially) made Kinect-compatible; additionally, use the forums for making a request or posting your own work.