Teacher’s Guide to Kinect and Gesture-Based Learning

Teacher’s Guide to Kinect and Gesture-Based Learning

Kinect is more than a gaming device; realistically, it’s an input device that can facilitate gaming (think of it like the mouse and keyboard). With a few easy tweaks that anyone can do, we can use this device to transform learning in classrooms. You don’t have to be a programmer to do this, but you do need to have a passion for innovative learning.

If we can get a community of educators and students to understand what this means and how to do it, we can develop an entire bank of gesture-based “drivers” to support all software currently in existence.

In other words, with this tool, we can we easily take existing applications (Google Earth, flash-based activities, calculators, etc.) and integrate them with Kinect. Let’s start with the end in mind; watch my videos below to see what I’m talking about. Keep in mind that I did this in a matter of three minutes. My intent wasn’t to program it to perfection, but rather to show that the process is easy to grasp.

Playing piano with Kinect


All you’re doing is mapping your gestures to keyboard characters and mouse commands. Once you’ve grasped the concept, it’s very, very simple to do.

What can we do with this? Take all the existing software and make it Kinect compatible. We don’t have to wait; we, as a community, can easily develop and deliver gesture-based learning opportunities with the bank of software already in existence. After you watch the video and read the guide below, think of software that you want to see “Kinectified” and put it in the forums. From there, I’ll establish a directory of files that can be used for any software.

This video I created will give you a better visual of how to get started; I highly recommend that you watch it. Even with the video, you’ll still need to reference the guide below to find the list of commands.

Disclaimer:

This guide looks lengthy, but it’s only because it has a list of all the commands…..please don’t let that intimidate you. The full guide is available from Institute of Creative Technologies; I’ve just edited and restructured their guide to be more appealing to the K-12 education sector. If you don’t yet have a Kinect, please consider purchasing one from a source that will support the costs associated with hosting this site.

——————————————————————————————————–

Step 1: Install Software

To use FAAST, you will need to download and install the following software:

  1. OpenNI Unstable Build for Windows v1.0.0.25
  2. During NITE installation, enter the free license key from OpenNI: 0KOIk2JeIBYClPWVnMoRKn5cdY4=
  3. Hardware drivers for your sensor (only one of the following)
  4. Download,unzip, and run FAAST

FAAST should then run out-of-the-box; no additional installation or setup is necessary. If you encounter an error on startup, you may also need to install the Microsoft Visual C++ 2008 SP1 Redistributable Package.

——————————————————————————————————–

Step 2: Mapping Keys with Gestures

Once you have the FAAST software running, click on the “Actions” tab. Here, you’ll see a list of default commands set to run with Kinect. They appear in this format:

action_name      action_threshold      virtual_event_type      virtual_event_name

action_name – The name of the pose or gesture action that the user performs.

action_threshold – The minimum threshold for activation (this varies by event).

virtual_event_type – The type of virtual event to be generated.

virtual_event_name – The specific event to be generated.

For example, the following command will press the “w” key when the right hand extends more than 18 inches in front of the shoulder, and then release it when it returns back behind that distance:

right_arm_forwards 18 key_hold w

Here is the list of all available actions (which fall under “action_name”):

Action List

Action Name

What the Action Means

lean_left

angular body lean left (degrees)

lean_right

angular body lean right(degrees)

lean_forwards

angualr body lean forwards (degrees)

lean_backwards

angular body lean back (degrees)

turn_left

angular amount of left body turn (degrees)

turn_right

angular amount of right body turn(degrees)

left_arm_forwards

forward distance from left hand to shoulder (inches)

left_arm_down

downward distance from left hand to shoulder (inches)

left_arm_up

upward distance from left hand to shoulder (inches)

left_arm_out

sideways distance from left hand to shoulder (inches)

left_arm_across

sideways distance from left hand across body to shoulder (inches)

right_arm_forwards

forward distance from right hand to shoulder (inches)

right_arm_down

downward distance from right hand to shoulder (inches)

right_arm_up

upward distance from right hand to shoulder (inches)

right_arm_out

sideways distance from right hand to shoulder (inches)

right_arm_across

sideways distance from right hand across body to shoulder (inches)

left_foot_forwards

forward distance from left hip to foot (inches)

left_foot_sideways

sideways distance from left hip to foot (inches)

left_foot_backwards

backwards distance from left hip to foot (inches)

left_foot_up

height of left foot above other foot on ground (inches)

right_foot_forwards

forward distance from right hip to foot (inches)

right_foot_sideways

sideways distance from right hip to foot (inches)

right_foot_backwards

backwards distance from right hip to foot (inches)

right_foot_up

height of right foot above other foot on ground (inches)

jump

height of both feet above ground (inches)

crouch

crouch distance, calculated as current height subtracted from standing height (inches)

walk

height of each step above ground when walking in place (inches)

push

velocity (inches/sec.)

swipe_up

velocity (inches/sec.)

swipe_down

velocity (inches/sec.)

swipe_left

velocity (inches/sec.)

swipe_right

velocity (inches/sec.)

circle

radius (inches)

  

Event List (What You Hope Happens After Performing a Specific Gesture)

Virtual Event Type

Virtual Event Name

key_press

key to press when the action is first detected (either a single character or a special key from the table below)

key_hold

key to hold down while the action is performed (either a single character or a special key from the table below)

key_type

a string of characters to type into the active window (no spaces allowed)

mouse_click

mouse button to click when the action is first detected (left_button, right_button, or middle_button)

mouse_double_click

mouse button to double click when the action is first detected (left_button, right_button, or middle_button)

mouse_hold

mouse button to hold down while the action is performed (left_button, right_button, or middle_button)

faast

input emulator control event (pause, resume, or stop)

Special Keys

backspace

caps_lock

up_arrow

delete

numpad_2

multiply

f3

f11

f19

left_shift

browser_refresh

media_next_track

tab

escape

right_arrow

help

numpad_3

add

f4

f12

f20

right_shift

browser_stop

media_previous_track

clear

space

down_arrow

left_windows

numpad_4

separator

f5

f13

f21

left_control

browser_search

media_stop

enter

page_up

select

right_windows

numpad_5

subtract

f6

f14

f22

right_control

browser_favorites

media_play_pause

shift

page_down

print

applications

numpad_6

decimal

f7

f15

f23

left_alt

browser_home

start_mail

control

end

execute

sleep

numpad_7

divide

f8

f16

f24

right_alt

volume_mute

media_select

alt

home

print_screen

numpad_0

numpad_8

f1

f9

f17

num_lock

browser_back

volume_down

start_application_1

pause

left_arrow

insert

numpad_1

numpad_9

f2

f10

f18

scroll_lock

browser_forward

volume_up

start_application_2

As another example, the following command will click the left mouse button when the left foot kicks forwards more than 12 inches in front of the hip:

left_foot_forwards 12 mouse_click left_button

Step 3: Calibrate and run the software!

To calibrate, the user must stand in this pose for several seconds until a stick figure appears, as shown in the image below.

Skeleton Calibraton Pose

Skeleton Calibration Pose

Once you’ve been detected and calibrated the software, click “Start emulator” , make your application you’re wanting to use gestures with become the active window, and use gestures to operate your software!

What software do you want to see support gestures? Check out the directory of applications that the community has (unofficially) made Kinect-compatible; additionally, use the forums for making a request or posting your own work.

Share
468 ad

1 Comment

  1. This is one heck of a resource, Johnny. I’m impressed that you have you’re frying so many fish in one pan.

    Do you think an average chrome-using iphone fella like myself can tackle this w/o much programming background ? Cheers, b

Leave a Reply

Switch to our mobile site