Kinect Hacked – New Teaching Paradigm?

Jump to the Telenoid section

Jump to the Telepresence section

Jump to the Kinect as an Innovation Tool section

Abstract: In this post, I will introduce some of the apps that have been developed for the Kinect to PC. Also I will show how the Kinect can be used as an innovation tool for educational purposes and lastly, how gesture-based interaction can change the way we interact in virtual environments.

In november 2010 i Gizmodo wrote this:

Adafruit’s $2,000 bounty for an open source Kinect driver hack was only offered up late last week and already someone has allegedly delivered, said Adafruit’s Phillip Torrone in an email to us just now. This was inevitable.


Immediately after reading that, I bought my first Kinect!

Microsoft lost the control of the device and hackers all around the world could now make applications for the kinect! In my head, I made all sorts of imaginations of what this could bring! And since this was made possible thanks to the open-source community, no one could control or limit the direction and focus.

I imagined:

Augmented Reality-boadgames being more interactive than ever.
– People controlling anything with just their body.
Object recognition, the Kinect knows where everything is and exactly what is in the room.
3D-modelling tools being more easy to use than software as we know it, based on mouse/keyboard.
3D-scans of anything.
– People entering virtual spaces with themselves.
Emotions analysed and digitally transferred in online communication.

Today all of these have more or less already been developed for all Kinect owners to use!

To me, it was especially this blog that kept me up to date and enabled me to download the apps (mostly for free)

Especially the creations of this “hacker” Oliver Kreylos made an impact on me, when he succeded in travelling to space in a Star Wars alike setting, play tic, tac, toe and fighting a friend using light sabers – all this via kinect…and some geeky nights i suppose:

Also this japanese guy turned himself into the japanese hero Ultra Seven:

He even proved the advantages of open source communities, when I wrote to him complaining about the bad measured calibration for activating the green laser gun. He replied that he made it using the standard measures of a japanese guy, and then made a custom version for me and my taller body! – thanks hogehoge335

But critics would categorize this as useless. What’s the point of being a superhero, playing “real” airguitar or fighting with light sabers??

Well, ive written some assignments about this. To me there is two important things the Kinect can contribute to:

1 Kinect as Telepresence
2 Kinect as Innovation-tool



According to Wikipedia, Telepresence is:

a set of technologies which allow a person to feel as if they were present, to give the appearance of being present, or to have an effect, via telerobotics, at a place other than their true location.

When I read Understanding Media by Marshall McLuhan, I find his idea of new media’s as being extensions of our presence, just as the hammer is an extension of the hand.
Basically he says: More “living human” and tacit knowledge transferred digital = Better communication!

Here is some examples of how the Kinect can provide that:

We have all participated in webinars with the keyspeaker presenting his PP-slides followed by his endlessly soporific voice-over. But to make these presentations more alive, we need the presenter to be standing in front of the slides. has made StagePresence, a very neat tool for that:

It is well suited for synchronous e-learning, online meetings and webinars. You can live stream a session or record it for offline use. And its free.

Another attempt to let Kinect bring humans together, is Kinected Conference:

This was developed by a group of MIT-students. Kinected Conference allows the participants to:

Focus the camera on the person who talks (the others become blurry)
Freeze the camera at a frame, if you don’t want it to record your moves (well suited for nose pickers)
– Use Augmented Reality-elements in online meetings

If you want to know more about this, please visit this page

Avatar Kinect

In his Avatar-thesis, Edward Castronova explains how we want to become our avatar, and look like our avatar. Microsoft has developed Avatar Kinect, allowing the user to control an avatar and enter web meetings with them. This allows the user to be present with our body online, but to keep some anonymity, which again lets the user be more relaxed. Unfortunately it is only for Kinect to Xbox.


One more exiting move in this direction is Telenoid. The japanese professor Ishiguro Hiroshi, whom I have been lucky to work with, made this robot as a low-budget/low-tech version of the Geminoid. The robot is opposite to Avatar Kinect. Telenoid lets the user become a real physical robot instead of a virtual avatar. When the user talks, moves, sees, the robot does the same, just like a mirror. The user can then get the feeling of being at the place where the robot is, from another place on earth.
Ditton & Lombard calls this Presence as Transportation, and splits it into three:
1. You are there – the user gets “moved” to another place
2. It is here – the place “moves” to the user
3. We are together – more users “moves” together

In Svendborg (Denmark) 2011 we examined how elderly people could interact with relatives via the Telenoid.

In a school context, I have this idea, that a long-term ill student who is forced to stay in hospital, shall be able to not only follow the subjects in the class, but also be a social player among the classmates.
The graphic to the left shows this scenario: The hospitalized student lying i bed. He’s got a laptop on his lap running the Telenoid software and facial API and a video stream of the class from the Telenoids point of view.

Below, we see how the Telenoid is placed on the absent students seat in the class. Will the student lying in the hospital bed get the feeling that “he is there” , “It is here” and/or “we are together”? Will his classmates?

Even though the technology behind the Telenoid isn’t exactly Kinect, the principle is the same. Telenoid uses a facial API that tracks the face movements through an ordinary webcam.

Download the paper here: (Telenoidopgaven.pdf)


I made a lesson plan for a class of students aged 14-16. The task was to see, if they could produce a body-based application from the following:

Scratch : A visual programming language developed by MIT. I’ve written about it before


– FAAST : A sort of API developed at University of Southern California, connecting the body movements captured by the Kinect to the computer. This program is based on simple, yet case-sensitive commands. FAAST provides the keybindings that determines how the computer shall act when a movement is detected, eg. #left_arm_up 20 key 3 (meaning when left arm moves 20 inches up the computer presses “3”) FAAST can be downloaded here.

The process looks like this:

The lesson plan is based on Hiim and Hippe’s didactic relation model and my entire Kinect-course can be downloaded  (pdf-file) lesson by lesson here.

The result of my Kinect-lesson was five innovative and funny applications developed by the students. The best was:

The body Alphabet

1- The participant places himself in front of the Kinect and makes body postures similar to letters in the alphabet.

2- In FAAST, the movements was translated to something like: legs 30 inches from center and arms 20 inces from the body = the letter “A”.

3- In Scratch, the group has programmed the letter that was activated to be typed on the canvas and the marker to move 20 pixels to the right, making space for the next letter.

This was more fun than practical. You had to jump and crouch to make an “o”. But the execution of the application was brilliant. A true bricolage-innovation.

WEB 2.0

We owe a lot to the open-source community. Thanks to the hackers and programmers, we can play, develop and interact with the Kinect on other devices than the Xbox. They have redesigned and revolutionized the possibilities and inspired others to do the same. Microsoft has finally realized that this development can’t be stopped, but is a bonus to everyone. The trend has definitely boosted the Kinect sales.

Participating in web 2.0 activities or open-source communities, makes you more engaged and open to the benefits of sharing. It raises the motivation and authenticity, so why should kids not be a part of that??


I recommend that all teachers considering using Kinect in education, read this blog to get inspired and this walk-through on how to install Kinect on your PC/MAC/Linux.

Finally there is an online forum for Kinect in educational use. It has grown big, and teachers can share lesson plans and learn from others! Feel free to read or write about experiences, using the Kinect in education here.


Oliver Kreylos and hogehoge335 and all Kinect application developers for inspiration!