Use your desk as mouse-free multitouch interface with MS Kinect

TouchTable is is Kinect powered application I am developing as part of my PhD research. It enables one to use bare hand laying on a regular office desk to control computer instead of mouse.

User can also move the cursor without any clicking. This can be used i.e. to hover over icons to display hint texts, which is possible for mouse but not for regular multi-touch screens.

TouchTable application features 

Following operations are possible:

  • Mouse cursor movement and hover events – by moving index finger on the surface of the office desk
  • Left mouse click – by touching the table with index finger (or by lifting the finger from a desk in reverse mode), while thumb is hidden in palm
  • Right mouse click – by touching the desk with index finger, while thumb is sticking out
  • Drag and drop – by placing index finger on the desk and dragging it across the surface

Demo applications

To get a better idea, please watch following videos:

  • Painting with finger in Microsoft Paint
  • Rotate & zoom using multitouch in Google Sketchup
  • Web browsing  with fingertip in Firefox with FireGestures

Painting with finger in Microsoft Paint

Finger is used as a natural drawing device. Even small children who cannot yet handle a pencil are able to draw pictures with finger into sand. Painting with TouchTable is more natural for them then with a mouse.

Rotate & zoom using multitouch in Google Sketchup

User rotates view by using left and right fingertips together (in the same way as on other multi-touch devices). Zooming is controlled by distance of fingers being dragged over the desk surface. 
Note: To install the sketchup part, copy SketchUpPlugin\Plugins from distribution ZIP to Sketchup plugins dir, choose View>Toolbars>nudgeOrbit and than enable it by clicking on the button with “nudgeOrbit:dialog” hint.

Web browsing  with fingertip in Firefox with FireGestures

This plugin integrates TouchTable with FireGestures plugin in FireFox. Index finger is used to perform gestures to go back, forward, open or close new tab. It thumb is extended, finger can open links like by using left mouse click. Double click mouse operation is possible if “thumb up flag” is on (thumb is extended and lifted from table).

If user is reading the page, whole hand can rest on the table which minimizes the strain in the finger muscles.

Why is Kinect placed above the table

There are other solutions utilizing Kinect for user interface, but most of them require to hold one’s arm in the air. This is quite uncomfortable after a while. Also precision of such movements is not great. In my solution, I have placed Kinect facing down approximately 1m above the table. I have mounted it on the arm of an adjustable table lamp. Thanks to it user does not need to be far away from screen but can sit as usually in the easy reach of his keyboard. If Kinect is placed facing the user, minimum distance of 50cm from device must be guaranteed for proper device operation.

How does it work?

Application processed data from Kinect in these simple steps:

  1. When first frame is processed, the depth map is stored as “environment depth map”. Basically, I remember the distance of each pixel in the view of Kinect as table pixel distance
  2. When a frame with a hand on it is processed, hand’s pixels are recognized based on comparing the actual distance of a pixel with the table pixel distance from previous step. If the distance is bigger than a threshold (f.e. 1cm), pixel is regarded as a hand pixel
  3. From the pixels of hand I extract hand contour (using OpenCV library)
  4. From hand contour I calculate the fingertips, based on the angle between neighboring pixels. If the angle is sharp, it is a hint that the pixel is at the fingertip sharp arc.
  5. I apply some heuristic to match detected fingertips to finger indexes. In this process, I try to first match fingers detected in previous frame based on their distance from the newly detected fingertips. I try to minimize the distance.
  6. I apply also some smoothing by mean averaging the fingertip coordinates from last few frames.

Can I use TouchTable in my application?

Non-commercial use

Please feel free to download and use this prototype for non-commercial purposes. If you create your own plugins, please share the code with the others. As this is an early release of the TouchTable library, it is very likely that the API will change in the future.

The download package contains:

  • Application binaries for Windows 64-bit and 32-bit
  • Presented plugins source codes

Writing TouchTable plugins is pretty simple, but if you can’t do that yourself, e-mail me and I can possibly write one for you. If you are NGO or non-profit organization, I’ll do it free of charge if I like the idea.

Commercial use

If you have some ideas how TouchTable can be used in real commercial usecase, please feel free to contact me at mail[AT]jetensky.net.

How to run it?

You need to install PrimeSense Kinect drivers and libraries first, before you attempt to run TouchTable. PleaKinect placement for TouchTable to work properlyse follow instructions in this post. Try running SimpleViewer application from Primesense OpenNI distribution to check your Kinect related settings. If you have any problems running it, you can post it in the discussion for this blog post and I will try to address it.

For proper operation, place your Kinect at least 70cm above table – see picture.  When started, listeners will be automatically activated based on the substring in the current exe path (f.e. Firefox plugin will be active if firefox.exe is present in the path). You need only to uncheck Disable listener checkbox on the application screen to start sending events to the listeners.

Support development of future versions

If you enjoy using this or find this work useful, it will be great if you can support development of future versions by donating.
1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5,00 out of 5)
Loading ... Loading ...

29 Comments so far

  1. amidream on Leden 3rd, 2012

    Wy not in Windows Kinect SDK drivers? :(

  2. Jety on Leden 6th, 2012

    Hi, Amidream, there are several reasons:
    - I have started the project with PrimeSense libraries before Microsoft SDK was released
    - MS SDK is considered to be a bit slower and I need to achieve high framerate(however has better skeleton recognition, but this is not important for this project)
    - PrimeSense libraries are free to use anywhere, MS SDK cannot be used in commercial projects yet

    Pavel

  3. amidream on Leden 8th, 2012

    Can you write me how can i install all i need 2 used your prog. Step by step. Thx.

  4. Jety on Leden 9th, 2012

    Hi, Amidream, thank you for your interest in using TouchTable.

    I presume you made an attempt to follow instructions in “How to run it” section of the post.

    Please let me know if you have managed to make OpenNI SimpleViewer running? If not, try googling for a solution (this post: http://jetensky.net/blog/2011/01/20/how-to-compile-and-run-kinect-related-samples/ may help you to).

    If you can run SimpleViewer, just try to place your Kinect above the desk and run exe from the distribution (32-bit or 64-bit based on your settings). You can e-mail me more details to pavel.jetensky[AT]seznam.cz.

  5. vvs551 on Leden 16th, 2012

    Good idea, but in order to use it for the (multi?) touch table you will need:
    1. calibration of the screen for four or more points
    2. the ability to control the mouse pointer of the operating system
    3. may be, support of TUIO
    Are you planning to add something of the following in your project?
    R

  6. Jety on Únor 5th, 2012

    Hi ws551, here are the answers to your questions:
    1. I am not much sure what calibration do you mean, please specify. What I am going to add is calibration of RGB camera to the Depth camera – same object that is seen by both have different coordinates – pixels are shifted a bit as both cameras are not on the same axis.
    2. Part of the distribution is MouseMappedFingerListener, that is sending commands to the operating system’s mouse pointer using Windows API
    3. TUIO is a good point, thanks, it would definitely be handy to implement this as a feature of TouchTable I will consider it

  7. vvs551 on Únor 6th, 2012

    Hi Jety!
    1. I mean point calibration (f.e. http://vimeo.com/24709976)
    2. I beg your pardon, from the description, I realized that the mouse only works in the three programs which are plug-ins.
    R

  8. Jety on Únor 9th, 2012

    Hi, ws551, regarding 1., there is no such calibration as mentioned in the video yet.

    Because display device is standard monitor, user is actually not touching the surface of the display unit and calibration is not necessary. However, we plan to combine touchtable with projected screen (i.e. micro LCD projectors) and in that case we will need similar calibration.

  9. Martin on Únor 9th, 2012

    Hi,
    very very nice and cool Program.
    Does it job well.
    2 Suggestions:
    1) What about scrolling – Moving with two fingers?
    2) Start/Stop would be nice. f.e. 5 Fingers on Table Starts/Stops the listener – every time listener starts again – the table will be recogniced again – if i lay something on it for instance….

    Twi finger gestures like panning/zooming (firefox) would be nice.

    Like your Program!

  10. Martin on Únor 9th, 2012

    Sometimes the mouse is a bit shaky. What do you recommend?

    its like 70cm above the table. settings are:
    speed 3.2 (otherwise i cant get to all edges – 1050p) – smooth 10, dilate 2, minimal 9, polygons 8,4

    thanks very much

  11. vvs551 on Únor 10th, 2012

    Hi Jeti!
    Really good idea to use pico projectors!
    You can see a very simple implementation of the algorithm calibration here:
    http://sharpboard.codeplex.com/SourceControl/changeset/view/44608#1251299

    R

  12. Martin on Únor 16th, 2012

    aaand a cool feature would be to set the dimensions of the touchable space.

  13. Jety on Únor 22nd, 2012

    Re Martin: Hi, happy that you made it working :) Regarding your questions:
    1) Regarding the shaky mouse – try increasing the FingerAnalyzerConfig.SmoothMovement (the higher the more frames are used to get a mean cursor position, but if too high, there is unwanted “following me” effect) – however 10 should be alright, try to check if there is some disturbance to the view (f.e. sleeves may be problem sometimes – check the depth preview)
    2) yes, limiting the scanned area on the table would be handy, no time to release a new version though :(
    3) Scrolling would be great, I even tried to implement but lack knowledge here – have not found easy way on Windows API – possibly if or anyone else find how, you can post and I will put it to new version, say with this signature: public void scrollActiveWindow(double scrollToPercent/*from 0 to 1)

  14. Jety on Únor 22nd, 2012

    Hi vvs551, thanks for the link, I have not found what is the project about, is there any link associated?

  15. vvs551 on Únor 22nd, 2012

    This is a similar project, but on the basis of a simple sensor – Wii remote. If you do not take into account the work with the device, much can be helpful.
    R

  16. Jety on Únor 24th, 2012

    Great, thanks for that vvs551 :)

  17. hadi on Duben 19th, 2012

    hi
    tanks for this
    but
    i can t run
    i install open ni and miicrosoft kinect sdk
    but i get this

    Could not load file or assembly ‘OpenNI.Net, Version=1.3.3.6, Culture=neutral, PublicKeyToken=6b43d0c6cf74ee7f’ or one of its dependencies. An attempt was made to load a program with an incorrect format.

    i added all dll to project

    please help

  18. Jety on Duben 23rd, 2012

    Hi Hadi, you do not need Microsoft SDK. I think your problem is related to using 32-bit version of TOuchTable on 64-bit version of Windows or the other way round. If you have correct TouchTable version, possibly you may have installed wrong drivers version during OpenNI install process. Let me know if it was the case please.

  19. hadi on Duben 23rd, 2012

    tanks for response
    i install nui 1.3.3.6
    first problem solve
    but now i have opencv error
    you think can use your project to make smartboard with kinect
    i make wiimote smart board (it is in net)
    but it work with ir led
    i want work without any pen on white board
    tanks for your attention

  20. core2 on Duben 24th, 2012

    Hello Jetensky

    I made an interactive whiteboard from your TouchTableLirary.
    it works very well on the wall with a projector,with the finger touching the wall :by inserting the calibration of the mouse.
    but its not working On an LCD TV, because of blobs (Lots of blobs).
    1 – how can I reduce the effect of blobs?
    2 – Can i use it in commercial?

    Sincerely yours

  21. hadi on Duben 24th, 2012

    hi core2
    i am looking for that you and jety made
    i am a teacher
    can you help me
    my yahoo id is
    hadiopera@yahoo.com

  22. Jety on Květen 23rd, 2012

    Hi, core2
    Question 1
    The trouble with LCD screen may be related to its reflective properties. Usually, shiny surfaces like mirrors example reflect not only visible light, but also infrared light, which is the way how Kinect calculates the distance. For example against mirror it will not work at all.
    Another issue is if Kinect is placed to look at sources of heat, what is probably not your case. But may be if there is light bulb behind LCD screen which is kind of warm this could cause the blobs as well.

    You can try to place a special glass that is transparent to visible light but opaque to infrared – they are used in buildings to reduce heat leakages, try to google for it.

    Please let me know how it went.

    Question 2
    Please note that using touchtable commercialy is not free. You can send me some details on planned usage and I will send you some license details.
    If you want to use TouchTable in your opensource or free non-commercial products, you do not need any license.

  23. ella on Listopad 22nd, 2012

    Hello. Is there a possible way to implement this same project using Kinect SDK? thanks

  24. Elite on Listopad 26th, 2012

    Hello!,
    is possible, use your Application for touch whiteboard?…or wall?…
    Thanks,
    Elite

  25. Jety on Prosinec 18th, 2012

    To Ella: Hi, it should be possible, if Kinect SDK API gives you access to the depth map coming from the Kinect Hardware (I am not sure about this)

    To Ellite: It should by ok, if surface does not reflect infrared light too much – f.e. it will not work against mirror. I have tried it on semitransparent polycarbonate desk and it worked.

    Please note that it may be tricky to position Kinect in this settings, so that user that is using TouchTable is not in the way of Kinect sensor (which is not an issue if Kinect is placed above table).

  26. Alexander G on Prosinec 1st, 2013

    Hello, my name is alex. My English is very bad :) .
    I have a problem :/ i want to run TouchTable.exe but it shows an Error:

    Error occured, see TouchTable.log for details. Exception:Can’t create any node of the requested type!

    I dont know how to fix it, please can you help me.

    Please dont hate me for my bad English. :D

  27. Alexander G on Prosinec 1st, 2013

    And the LOG…
    Can’t create any node of the requested type! bei OpenNI.WrapperUtils.CheckEnumeration(Int32 status, EnumerationErrors errors)
    bei OpenNI.Context..ctor(String xmlFile)
    bei TouchTableLibrary.application.Engine..ctor()
    bei SimpleViewer.net.Program.Main() in C:\prj\beanstalkapp\3d-keyboard\TouchTable\trunk\TouchTable\TouchTable\Program.cs:Zeile 38.

  28. inexpensive wall decals on Červen 1st, 2014

    inexpensive wall decals…

    Jetyho blog » Use your desk as mouse-free multitouch interface with MS Kinect…