Use your desk as mouse-free multitouch interface with MS Kinect
TouchTable is is Kinect powered application I am developing as part of my PhD research. It enables one to use bare hand laying on a regular office desk to control computer instead of mouse.
User can also move the cursor without any clicking. This can be used i.e. to hover over icons to display hint texts, which is possible for mouse but not for regular multi-touch screens.
TouchTable application features
Following operations are possible:
- Mouse cursor movement and hover events – by moving index finger on the surface of the office desk
- Left mouse click – by touching the table with index finger (or by lifting the finger from a desk in reverse mode), while thumb is hidden in palm
- Right mouse click – by touching the desk with index finger, while thumb is sticking out
- Drag and drop – by placing index finger on the desk and dragging it across the surface
Demo applications
To get a better idea, please watch following videos:
- Painting with finger in Microsoft Paint
- Rotate & zoom using multitouch in Google Sketchup
- Web browsing with fingertip in Firefox with FireGestures
Painting with finger in Microsoft Paint
Finger is used as a natural drawing device. Even small children who cannot yet handle a pencil are able to draw pictures with finger into sand. Painting with TouchTable is more natural for them then with a mouse.
Rotate & zoom using multitouch in Google Sketchup
User rotates view by using left and right fingertips together (in the same way as on other multi-touch devices). Zooming is controlled by distance of fingers being dragged over the desk surface.
Note: To install the sketchup part, copy SketchUpPlugin\Plugins from distribution ZIP to Sketchup plugins dir, choose View>Toolbars>nudgeOrbit and than enable it by clicking on the button with “nudgeOrbit:dialog” hint.
Web browsing with fingertip in Firefox with FireGestures
This plugin integrates TouchTable with FireGestures plugin in FireFox. Index finger is used to perform gestures to go back, forward, open or close new tab. It thumb is extended, finger can open links like by using left mouse click. Double click mouse operation is possible if “thumb up flag” is on (thumb is extended and lifted from table).
If user is reading the page, whole hand can rest on the table which minimizes the strain in the finger muscles.
Why is Kinect placed above the table
There are other solutions utilizing Kinect for user interface, but most of them require to hold one’s arm in the air. This is quite uncomfortable after a while. Also precision of such movements is not great. In my solution, I have placed Kinect facing down approximately 1m above the table. I have mounted it on the arm of an adjustable table lamp. Thanks to it user does not need to be far away from screen but can sit as usually in the easy reach of his keyboard. If Kinect is placed facing the user, minimum distance of 50cm from device must be guaranteed for proper device operation.
How does it work?
Application processed data from Kinect in these simple steps:
- When first frame is processed, the depth map is stored as “environment depth map”. Basically, I remember the distance of each pixel in the view of Kinect as table pixel distance
- When a frame with a hand on it is processed, hand’s pixels are recognized based on comparing the actual distance of a pixel with the table pixel distance from previous step. If the distance is bigger than a threshold (f.e. 1cm), pixel is regarded as a hand pixel
- From the pixels of hand I extract hand contour (using OpenCV library)
- From hand contour I calculate the fingertips, based on the angle between neighboring pixels. If the angle is sharp, it is a hint that the pixel is at the fingertip sharp arc.
- I apply some heuristic to match detected fingertips to finger indexes. In this process, I try to first match fingers detected in previous frame based on their distance from the newly detected fingertips. I try to minimize the distance.
- I apply also some smoothing by mean averaging the fingertip coordinates from last few frames.
Can I use TouchTable in my application?
Non-commercial use
Please feel free to download and use this prototype for non-commercial purposes. If you create your own plugins, please share the code with the others. As this is an early release of the TouchTable library, it is very likely that the API will change in the future.
The download package contains:
- Application binaries for Windows 64-bit and 32-bit
- Presented plugins source codes
Writing TouchTable plugins is pretty simple, but if you can’t do that yourself, e-mail me and I can possibly write one for you. If you are NGO or non-profit organization, I’ll do it free of charge if I like the idea.
Commercial use
If you have some ideas how TouchTable can be used in real commercial usecase, please feel free to contact me at mail[AT]jetensky.net.
How to run it?
You need to install PrimeSense Kinect drivers and libraries first, before you attempt to run TouchTable. Plea
se follow instructions in this post. Try running SimpleViewer application from Primesense OpenNI distribution to check your Kinect related settings. If you have any problems running it, you can post it in the discussion for this blog post and I will try to address it.
For proper operation, place your Kinect at least 70cm above table – see picture. When started, listeners will be automatically activated based on the substring in the current exe path (f.e. Firefox plugin will be active if firefox.exe is present in the path). You need only to uncheck Disable listener checkbox on the application screen to start sending events to the listeners.
Support development of future versions
Související články v kategorii 3D User interfaces
- Touchtable Multitouch Device With Rear Projection Ala Microsoft Surface - 3.3. 2012
- How to get OpenCV (EmguCV) Image from Primesense Kinect RGB camera - 24.2. 2012
- How to get OpenCV (EmguCV) Image from Primesense Kinect depth generator - 24.2. 2012
- Finding points in 3D space using point quad tree - java sources - 7.3. 2011
- 3DKeyboard project - 3.2. 2011
Související články v kategorii SW development
- Legend of Grimrock - Sword of Nex solution as Autohotkey script - 8.6. 2012
- How to get OpenCV (EmguCV) Image from Primesense Kinect RGB camera - 24.2. 2012
- How to get OpenCV (EmguCV) Image from Primesense Kinect depth generator - 24.2. 2012
- Finding points in 3D space using point quad tree - java sources - 7.3. 2011
- How to simulate vertical text (writing-mode:tb-rl) in Firefox - 29.7. 2009
Loading ...
Wy not in Windows Kinect SDK drivers?
Hi, Amidream, there are several reasons:
- I have started the project with PrimeSense libraries before Microsoft SDK was released
- MS SDK is considered to be a bit slower and I need to achieve high framerate(however has better skeleton recognition, but this is not important for this project)
- PrimeSense libraries are free to use anywhere, MS SDK cannot be used in commercial projects yet
Pavel
Can you write me how can i install all i need 2 used your prog. Step by step. Thx.
Hi, Amidream, thank you for your interest in using TouchTable.
I presume you made an attempt to follow instructions in “How to run it” section of the post.
Please let me know if you have managed to make OpenNI SimpleViewer running? If not, try googling for a solution (this post: http://jetensky.net/blog/2011/01/20/how-to-compile-and-run-kinect-related-samples/ may help you to).
If you can run SimpleViewer, just try to place your Kinect above the desk and run exe from the distribution (32-bit or 64-bit based on your settings). You can e-mail me more details to pavel.jetensky[AT]seznam.cz.
Good idea, but in order to use it for the (multi?) touch table you will need:
1. calibration of the screen for four or more points
2. the ability to control the mouse pointer of the operating system
3. may be, support of TUIO
Are you planning to add something of the following in your project?
R
Hi ws551, here are the answers to your questions:
1. I am not much sure what calibration do you mean, please specify. What I am going to add is calibration of RGB camera to the Depth camera – same object that is seen by both have different coordinates – pixels are shifted a bit as both cameras are not on the same axis.
2. Part of the distribution is MouseMappedFingerListener, that is sending commands to the operating system’s mouse pointer using Windows API
3. TUIO is a good point, thanks, it would definitely be handy to implement this as a feature of TouchTable I will consider it
Hi Jety!
1. I mean point calibration (f.e. http://vimeo.com/24709976)
2. I beg your pardon, from the description, I realized that the mouse only works in the three programs which are plug-ins.
R
Hi, ws551, regarding 1., there is no such calibration as mentioned in the video yet.
Because display device is standard monitor, user is actually not touching the surface of the display unit and calibration is not necessary. However, we plan to combine touchtable with projected screen (i.e. micro LCD projectors) and in that case we will need similar calibration.
Hi,
very very nice and cool Program.
Does it job well.
2 Suggestions:
1) What about scrolling – Moving with two fingers?
2) Start/Stop would be nice. f.e. 5 Fingers on Table Starts/Stops the listener – every time listener starts again – the table will be recogniced again – if i lay something on it for instance….
Twi finger gestures like panning/zooming (firefox) would be nice.
Like your Program!
Sometimes the mouse is a bit shaky. What do you recommend?
its like 70cm above the table. settings are:
speed 3.2 (otherwise i cant get to all edges – 1050p) – smooth 10, dilate 2, minimal 9, polygons 8,4
thanks very much
Hi Jeti!
Really good idea to use pico projectors!
You can see a very simple implementation of the algorithm calibration here:
http://sharpboard.codeplex.com/SourceControl/changeset/view/44608#1251299
R
aaand a cool feature would be to set the dimensions of the touchable space.
Re Martin: Hi, happy that you made it working
Regarding your questions:
1) Regarding the shaky mouse – try increasing the FingerAnalyzerConfig.SmoothMovement (the higher the more frames are used to get a mean cursor position, but if too high, there is unwanted “following me” effect) – however 10 should be alright, try to check if there is some disturbance to the view (f.e. sleeves may be problem sometimes – check the depth preview)
2) yes, limiting the scanned area on the table would be handy, no time to release a new version though
3) Scrolling would be great, I even tried to implement but lack knowledge here – have not found easy way on Windows API – possibly if or anyone else find how, you can post and I will put it to new version, say with this signature: public void scrollActiveWindow(double scrollToPercent/*from 0 to 1)
Hi vvs551, thanks for the link, I have not found what is the project about, is there any link associated?
This is a similar project, but on the basis of a simple sensor – Wii remote. If you do not take into account the work with the device, much can be helpful.
R
Great, thanks for that vvs551
hi
tanks for this
but
i can t run
i install open ni and miicrosoft kinect sdk
but i get this
Could not load file or assembly ‘OpenNI.Net, Version=1.3.3.6, Culture=neutral, PublicKeyToken=6b43d0c6cf74ee7f’ or one of its dependencies. An attempt was made to load a program with an incorrect format.
i added all dll to project
please help
Hi Hadi, you do not need Microsoft SDK. I think your problem is related to using 32-bit version of TOuchTable on 64-bit version of Windows or the other way round. If you have correct TouchTable version, possibly you may have installed wrong drivers version during OpenNI install process. Let me know if it was the case please.
tanks for response
i install nui 1.3.3.6
first problem solve
but now i have opencv error
you think can use your project to make smartboard with kinect
i make wiimote smart board (it is in net)
but it work with ir led
i want work without any pen on white board
tanks for your attention
Hello Jetensky
I made an interactive whiteboard from your TouchTableLirary.
it works very well on the wall with a projector,with the finger touching the wall :by inserting the calibration of the mouse.
but its not working On an LCD TV, because of blobs (Lots of blobs).
1 – how can I reduce the effect of blobs?
2 – Can i use it in commercial?
Sincerely yours
hi core2
i am looking for that you and jety made
i am a teacher
can you help me
my yahoo id is
hadiopera@yahoo.com
Hi, core2
Question 1
The trouble with LCD screen may be related to its reflective properties. Usually, shiny surfaces like mirrors example reflect not only visible light, but also infrared light, which is the way how Kinect calculates the distance. For example against mirror it will not work at all.
Another issue is if Kinect is placed to look at sources of heat, what is probably not your case. But may be if there is light bulb behind LCD screen which is kind of warm this could cause the blobs as well.
You can try to place a special glass that is transparent to visible light but opaque to infrared – they are used in buildings to reduce heat leakages, try to google for it.
Please let me know how it went.
Question 2
Please note that using touchtable commercialy is not free. You can send me some details on planned usage and I will send you some license details.
If you want to use TouchTable in your opensource or free non-commercial products, you do not need any license.
Hello. Is there a possible way to implement this same project using Kinect SDK? thanks
Hello!,
is possible, use your Application for touch whiteboard?…or wall?…
Thanks,
Elite
To Ella: Hi, it should be possible, if Kinect SDK API gives you access to the depth map coming from the Kinect Hardware (I am not sure about this)
To Ellite: It should by ok, if surface does not reflect infrared light too much – f.e. it will not work against mirror. I have tried it on semitransparent polycarbonate desk and it worked.
Please note that it may be tricky to position Kinect in this settings, so that user that is using TouchTable is not in the way of Kinect sensor (which is not an issue if Kinect is placed above table).