This posts introduces a result of our joined effort - Touchtable with rear project ala Microsoft Surface multitouch device. By combining these technlogies we have created a working prototype of multitouch interface that has similar features as Microsoft Surface but just fraction of costs.
The projector used is BenQ MP776ST with buying price approximately 1200€, depth camera tracking my fingers is Microsoft Kinect.

Video

Video shows whole hardware assembly and how the BenQ projector and Microsoft Kinect are placed. First I use my Sketchup plugin to show multitouch properties of the interface and at the and I control Crayone Deluxe game with my fingertip, drawing objects that immediately appear on the table.

More information about TouchTable project and how to download and install it can be found here: http://jetensky.net/blog/2011/12/18/touchtable/

If you are interested in using this table for your project, please contact Ivan Habrun (ifko5<..put..roller..here..>hotmail.com) for more information.

Solution

The code  takes RGB map from _rgbGenerator (OpenNI.ImageGenerator) and calculates RGB image (new Image<Bgr, Byte>(width, height)). Code presented uses unsafe sequences and array pointers to maximize the speed (you need to set up compiler with unsafe option).

Working C# code sample

This code is part of TouchTable project.

private unsafe void processDepthFrame()
{
var depthMd = new DepthMetaData();
// process pDepth
_depthGenerator.GetMetaData(depthMd);
var pDepth = (ushort*)_depthGenerator.DepthMapPtr.ToPointer();
ushort* pCurrentDepth = pDepth;
var width = depthMd.XRes;
var height = depthMd.YRes;
depthImage = new Image<Gray, Byte>(width, height);
var data = depthImage.Data;
var minimalFingerHeight = Config.MinimalFingerHeight;
fixed (int* pEnvDistances = _envDistances)
{
fixed (int* pTableDistances = TableDistancesYX)
{
fixed (byte* pImageData = data)
{
var pCurrEnvDistance = pEnvDistances;
var pCurrTableDistance = pTableDistances;
var pCurrImageData = pImageData;
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++, pCurrentDepth += 1, pCurrEnvDistance++, pCurrTableDistance++, pCurrImageData++)
{
var distFromKinect = *pCurrentDepth;
// Remember values
if (saveEnvDistances)
{
*pCurrEnvDistance = distFromKinect;
}
int distFromTable = (*pCurrEnvDistance – distFromKinect);
*pCurrTableDistance = distFromTable;
if (distFromTable > minimalFingerHeight)
{
// We will make pixel gray if it is considered a hand pixel
*pCurrImageData = 128;
}
}
}
}
}
}
}

Problem

OpenCV and its C# wrapper are great tools in your SW toolkit that helps processing of collected data from Kinect. To start using its power, you first need to instantiate an OpenCV Image class from  Kinect Depth.

Solution

The code  takes depth map from _depthGenerator (OpenNI.DepthGenerator) and calculates gray image (new Image<Gray, Byte>(width, height)). It works in a situation where Kinect is placed above some solid desk.

• When first depth frame is processed, distances from Kinect are remembered in array TableDistancesYX. This should happen when no object is on the desk.
• When succeeding depth frames are processed (imagine a hand is placed on the table), whenever depth differs more than minimalFingerHeight, it is marked as gray in result depthImage.
• Code presented uses unsafe sequences and array pointers to maximize the speed (you need to set up compiler with unsafe option).

The result EmguCV image has only black or gray pixels:

Working C# code sample

This code is part of TouchTable project.

Note:  Sorry guys for the poor quality of code formatting, please copy paste it to Visual Studio and reformat it. Hopefully I will upgrade soon to new version of WordPress to get nicer code samples but I am waiting for the migration of my MySQL database to version 5.

private unsafe void processDepthFrame()
{
var depthMd = new DepthMetaData();
// process pDepth
_depthGenerator.GetMetaData(depthMd);
var pDepth = (ushort*)_depthGenerator.DepthMapPtr.ToPointer();
ushort* pCurrentDepth = pDepth;
var width = depthMd.XRes;
var height = depthMd.YRes;
depthImage = new Image<Gray, Byte>(width, height);
var data = depthImage.Data;
var minimalFingerHeight = Config.MinimalFingerHeight;
fixed (int* pEnvDistances = _envDistances) {
fixed (int* pTableDistances = TableDistancesYX) {
fixed (byte* pImageData = data) {
var pCurrEnvDistance = pEnvDistances;
var pCurrTableDistance = pTableDistances;
var pCurrImageData = pImageData;
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++, pCurrentDepth += 1, pCurrEnvDistance++, pCurrTableDistance++, pCurrImageData++){
var distFromKinect = *pCurrentDepth;
// Remember values
if (saveEnvDistances){
*pCurrEnvDistance = distFromKinect;
}
int distFromTable = (*pCurrEnvDistance – distFromKinect);
*pCurrTableDistance = distFromTable;
if (distFromTable > minimalFingerHeight) {
// We will make pixel gray if it is considered a hand pixel
*pCurrImageData = 128;
}
}}
}
}
}
}

How to improve the solution

Possible there is a way how to copy whole block of memory from the Kinect depth map to EmguCV gray Image. This can be even faster than iterating through all rows and columns, however I was not able to find a way. If you have some better code sample please share in the comments and I will be happy to publish in this article.

User can also move the cursor without any clicking. This can be used i.e. to hover over icons to display hint texts, which is possible for mouse but not for regular multi-touch screens.

TouchTable application features 

Following operations are possible:

• Mouse cursor movement and hover events – by moving index finger on the surface of the office desk
• Left mouse click – by touching the table with index finger (or by lifting the finger from a desk in reverse mode), while thumb is hidden in palm
• Right mouse click – by touching the desk with index finger, while thumb is sticking out
• Drag and drop – by placing index finger on the desk and dragging it across the surface

Demo applications

To get a better idea, please watch following videos:

• Painting with finger in Microsoft Paint
• Rotate & zoom using multitouch in Google Sketchup
• Web browsing  with fingertip in Firefox with FireGestures

Painting with finger in Microsoft Paint

Finger is used as a natural drawing device. Even small children who cannot yet handle a pencil are able to draw pictures with finger into sand. Painting with TouchTable is more natural for them then with a mouse.

Rotate & zoom using multitouch in Google Sketchup

User rotates view by using left and right fingertips together (in the same way as on other multi-touch devices). Zooming is controlled by distance of fingers being dragged over the desk surface. 
Note: To install the sketchup part, copy SketchUpPlugin\Plugins from distribution ZIP to Sketchup plugins dir, choose View>Toolbars>nudgeOrbit and than enable it by clicking on the button with „nudgeOrbit:dialog“ hint.

Web browsing  with fingertip in Firefox with FireGestures

This plugin integrates TouchTable with FireGestures plugin in FireFox. Index finger is used to perform gestures to go back, forward, open or close new tab. It thumb is extended, finger can open links like by using left mouse click. Double click mouse operation is possible if „thumb up flag“ is on (thumb is extended and lifted from table).

If user is reading the page, whole hand can rest on the table which minimizes the strain in the finger muscles.

Why is Kinect placed above the table

There are other solutions utilizing Kinect for user interface, but most of them require to hold one’s arm in the air. This is quite uncomfortable after a while. Also precision of such movements is not great. In my solution, I have placed Kinect facing down approximately 1m above the table. I have mounted it on the arm of an adjustable table lamp. Thanks to it user does not need to be far away from screen but can sit as usually in the easy reach of his keyboard. If Kinect is placed facing the user, minimum distance of 50cm from device must be guaranteed for proper device operation.

How does it work?

Application processed data from Kinect in these simple steps:

1. When first frame is processed, the depth map is stored as „environment depth map“. Basically, I remember the distance of each pixel in the view of Kinect as table pixel distance
2. When a frame with a hand on it is processed, hand’s pixels are recognized based on comparing the actual distance of a pixel with the table pixel distance from previous step. If the distance is bigger than a threshold (f.e. 1cm), pixel is regarded as a hand pixel
3. From the pixels of hand I extract hand contour (using OpenCV library)
4. From hand contour I calculate the fingertips, based on the angle between neighboring pixels. If the angle is sharp, it is a hint that the pixel is at the fingertip sharp arc.
5. I apply some heuristic to match detected fingertips to finger indexes. In this process, I try to first match fingers detected in previous frame based on their distance from the newly detected fingertips. I try to minimize the distance.
6. I apply also some smoothing by mean averaging the fingertip coordinates from last few frames.

Can I use TouchTable in my application?

Non-commercial use

Please feel free to download and use this prototype for non-commercial purposes. If you create your own plugins, please share the code with the others. As this is an early release of the TouchTable library, it is very likely that the API will change in the future.

The download package contains:

• Application binaries for Windows 64-bit and 32-bit
• Presented plugins source codes

Writing TouchTable plugins is pretty simple, but if you can’t do that yourself, e-mail me and I can possibly write one for you. If you are NGO or non-profit organization, I’ll do it free of charge if I like the idea.

Commercial use

If you have some ideas how TouchTable can be used in real commercial usecase, please feel free to contact me at mail[AT]jetensky.net.

How to run it?

You need to install PrimeSense Kinect drivers and libraries first, before you attempt to run TouchTable. Please follow instructions in this post. Try running SimpleViewer application from Primesense OpenNI distribution to check your Kinect related settings. If you have any problems running it, you can post it in the discussion for this blog post and I will try to address it.

For proper operation, place your Kinect at least 70cm above table – see picture.  When started, listeners will be automatically activated based on the substring in the current exe path (f.e. Firefox plugin will be active if firefox.exe is present in the path). You need only to uncheck Disable listener checkbox on the application screen to start sending events to the listeners.

Support development of future versions

If you enjoy using this or find this work useful, it will be great if you can support development of future versions by donating.

Než jsem našel fungující verzi, vyzkoušel jsem řadu jiných prográmků (Wo Webcam Lite, IP Webcam,  SmartCam, USB Webcam), ale tohle byla první varianta, u níž jsem uspěl. Navíc bez nutnosti mít zapnutou Wifi nebo Blootooth, což je mi blízké, k provozu stačí připojení přes USB kabel.



Postup jsem vyzkoušel a funguje s následujícím HW a SW, ale není důvod, proč by nefungoval i s jinou podobnou konfigurací:

• Mobilní telefon HTC Desire HD
• Notebook ThinkPad R61
• Windows 7 64-bit
• Skype verze 5.1.0.112

Postup instalace

1. Do mobilu si stáhněte a nainstalujte aplikaci DroidCam
2. Do počítače si stáhněte a nainstalujte AndroidSDK
   Pokud se instalace nedaří kvůli tomu, že instalační program nemůže najít vaši nainstalovanou Javu, v dialogu klepněte Back a pak zase Next a už ji najde. Divné, co ? Ale mě to zafungovalo.
3. Spusťte android-sdk\SDK Manager.exe a doinstalujte si Android SDK platform tools, čehož součástí je i ADB (Android Debug Bridge).
4. Připojte mobilní telefon a v menu Nastavení -> Aplikace -> Vývoj zaškrtněte Ladění USB. V tomto okamžiku se vám doinstalují ovladače pro USB připojení k mobilu.
5. Spusťte DroidCam klienta na počítači a DroidCam aplikaci na mobilu
6. Ověřte, že ADB rozpoznal váš telefon a připojte se
   1. Spusťte příkazovou řádku cmd.exe a vní android-sdk\platform-tools\adb.exe
   2. Spusťte příkaz adb devices – váš mobil se tam musí objevit
      android-sdk\platform-tools>adb devices List of devices attached SH0CWRX00742    device
   3. Spusťte příkaz android-sdk\platform-tools>adb forward tcp:4747 tcp:4747
   4. V spuštěné aplikaci na počítači klepněte na Connect
      Pokud se objeví hlášení, že spojení je již využíváno nebo že se nelze připojit, tak restartujte adb server spuštěním nějakého příkazu (třeba adb devices) a znovu spusťte forward příkaz z předchozího kroku, případně zkuste restartovat aplikaci jak na mobilu, tak na počítači
7. Použití kamery ve Skypu – spusťte Skype a v menu zvolte Call > Video > Video settings. V rozbalovací nabídce Select Webcam vyberte DroidCam. Video z mobilní kamery by se mělo zobrazit v náhledu – viz. obrázek.

Odkazy

• Postup instalace, podle nějž jsem postupoval, na stránkách DroidCam - Windows verze (existuje i Unix verze)
• How To Set Up ADB/USB Drivers for Android Devices | TheUnlockr

 Co jsou tedy tito démoni, jaké mají vlastnosti?

Démony vnímám jako archetypy, programy lidského chování, které původně sloužily k nějakému praktickému účelu, ale došlo k jejich nezdravému přebujení. V mysli člověka jako živočišného druhu se tyto programy mohly vyvíjet po tisíciletí a v nepřebujelé verzi mu usnadňovaly přežití, v té přebujelé mu však už škodí. Mají tyto vlastnosti:

• Jedná se o součásti osobnosti, které se mohou za určitých okolností aktivovat a vedou k negativnímu jednání osoby (k něčemu, o čem by se řeklo „Ten se nezachoval dobře, to bylo od něj hnusné (hloupé)“. Toto chování může vést k ubližování sobě nebo druhým.
• Když člověk jedná podle démona, nerozhoduje se podle své svobodné vůle, ale jen podle toho naprogramování. V té chvíli není svobodný, ale neuvědomuje si to. Např. lakotný člověk si svoji lakotu nepřipouští, nepřizná, lakota je uhnízděná v temném koutě jeho duše. Přitom právě přiznat si vlastního démona je prvním krokem k tomu, aby se negativní vliv démona zmenšil nebo přímo eliminoval. O tom bych rád napsal více v pokračování tohoto seriálu, kde bude psát o průběhu situace, kdy nás posedne démon a uvedu nápady, jak to rozpoznat a jak s tím pracovat.

• Pokud člověk jedná podle démona, posiluje ho tím. To je v souladu s vlastností neuronových spojů (synapsí) v mozku. Pokud se v mozku nějaká dráha aktivuje často, zesiluje se a mozek má v budoucnu tendenci (jakýsi magnetismus) tuto silnou dráhu využívat daleko častěji než nějaká méně často používaná spojení. To je také důvod, proč se člověk špatně zbavuje zlozvyku a proč ritualizace pomáhá udržet nějakou aktivitu (např. pravidelné cvičení jógy nebo běhání).

Alegorie démonů jako černých puntíků na naší duši se mi moc líbí, protože je z toho patrné, že nejsou naší součástí, jen jsou na nás třeba právě nalepení. Zastiňují pak naši osobnost, berou nám energii a negativně ovlivňují naše chování. Snižují naši kapacitu štěstí.

Seznam rozpoznaných démonů

Abych vnesl do tématu více světla, uvedu několik rozpoznaných typů  démonů.

Démoni v lidské kultuře

Židovská Kabala

O démonech se vyučuje v židovské kabale. Tato část není nijak magická, spíš je to psychologie osobnosti.

„From the year 1540 and onward, the basic levels of Kabbalah must be taught publicly to everyone, young and old. Only through Kabbalah will we forever eliminate war, destruction, and man’s inhumanity to his fellow man.“

„Od roku 1540 a dál, základy Kabaly musí být veřejně vyučovány a dostupné všem, mladým i starým. Pouze skrz Kabalu dokážeme navždy eliminovat válku, ničení a vzájemnou lidskou nehumánnost.„

(zdroj: The Kabbalah Centre)

 Červená kniha K. G. Junga

Démony se zabýval i Gustav Jung, kamarád Míra mi vyprávěl o jeho z Jungova pohledu velmi důležitém díle – Červené knize (viz. rozhovor o knize v Televizní akademii s vydavatelem jejího českého vydání Zdeňkem Jančaříkem). Tuto knihu jsem však ještě nečetl, takže víc k tomu teď říct nemůžu. Velkou měrou se démony zabývá i kniha Jozefa Kariky Zóny stínů (viz. detail knihy na kosmas.cz).

Uvítám vaše připomínky i osobní zkušenosti s démony v diskuzi pod článkem.

Všechny informace zde jsou v podstatě opisem z knihy Martina Hirteho Očkování – pro a proti.

Spalničky

• Léčba
  • Nejsou známé žádné léky, které by je vyléčily
  • Nedostatek vitamínu A může přispět k závažnému průběhu. Podání vysoké dávky vit. A lze redukovat počet komplikací a úmrtí až o 80%.
  • Typicky doporučované snižování horečky během onemocnění pomocí léků je diskutabilní, protože zvyšuje úmrtnost 7x
• Onemocnění může být nebezpečné výskytem encefalitidy (zánět mozku)
  • šance je 1:15000 ve věku do 4 let, 1:5000 ve věku 5 – 9 let a 1:2500 nad 10 let
  • 15 % na následky zemře, 35% má trvalé následky, 60% bez následků
• Prodělání nemoci v dětském věku má prokazatelně pozitivní vliv na zralost imunitního systému
  • Snižuje riziko vzniku rakoviny prsu a uzlin, sklerózy multiplex, alergie na prach, neurodermatitidy
  • Může vyléčit některé chronické nemoci, např. lupénku, epilepsii nebo neurodermatititu způsobenou potravinovou alergií
• Očkování
  • posouvá výskyt onemocnění do pozdějšího věku (jedinec je neprodělá v „ideálním“ věku, tj. mezi 1. a 9. rokem)
  • očkování matky může způsobit, že dítě onemocní v 1. roku života (neochrání ho matčiny protilátky), což je riziková skupina
  • vede k častějšímu výskytu senné rýmy, může vést k nervovým onemocněním, jako je poškození zrakového nervu, ochrnutí lícního nervu nebo svalů oka a hluchotě (ta může být způsobena očkovací složkou proti příušnicím).
  • zvyšuje riziko atopického exému do věku 15 let 2x a chronického zánětu střeva (MMR vakcína)

Příušnice

• Příušnice probíhají u dětí typicky bez problémů, spontánně odezní a ve většině případů proběhnou dokonce skrytě. Velmi zřídka může onemocnění vyvolat poškození sluchu.
• Jsou na pokraji vymizení. Děti dnes téměř nemajé možnost prodělat tuto nemoc v dětském věku a tím získat doživotní imunitu.
• Očkovací látka je v současnosti pouze v rámci MMR-vakcíny, není dostatečně prozkoumaná z hlediska bezpečnosti a možných dlouhodobých následků.

Zarděnky

• Zarděnky jsou bezproblémovým onemocněním dětského věku, které i bez léčby probíhá bez následků.
• Pokud se zarděnky objeví během těhotenství, hrozí riziko potratu případně poškození plodu. Proto nejpozději na začátku puberty by měla být děvčata, která zarděnky neprodělala, proti nim očkovaná. Pro optimální hladinu protilátek jsou potřebná dvě očkování.
• Očkováním se zarděnky posouvají do dospělého věku spolu s rizikem přibývání případů embryopatie. Zabránit tomuto vývoji by měla co nejvyšší míra proočkovanosti, případně úplné vymýcení zarděnek. Neočkovaní nemají za těchto okolností téměř šanci prodělat zarděnky a tím získat doživotní imunitu.
• Očkování proti zarděnkám může vyvolat chronické onemocnění svalů nebo kloubů.

O MMR vakcíně

• MMR vakcína je sdružená vakcína proti spalničkám, příušnicím a zarděnkám.
• V USA bylo v letech 1990 – 2000 zaznamenáno přes 40 000 hlášení nežádoucích účinků. Mezi ně patří encefalitida, chronická artritida. Riziko závažných nežádoucích účinků bylo vypočítáno na 1:13000, studie v Rusku (vakcína MMR Triplovax) v roce 1996 zjistila 1,3% výskutu „závažných nežádoucích účinků po očkování“. Větší šance na nežádoucí účinky je u ženského pohlaví.
• U 5 až 15 % očkovaných se objeví několikadenní horečka, tpypicky mezi 7. – 15. dnem po očkování. U 3 – 5% se tak0 může objevitmalátnost, zánět spojivek, kašel, fibrilní křeče (ty však nemívají pozdější následky na zdraví) nebo spalničková vyrážka.
• V raném dětském věku při očkování MMR-vakcínou hrozí riziko poškození neurologického vývoje

Odkazy

• Detail knihy Martina Hirteho Očkování – pro a proti v e-shopu outdooring.cz – zdroj informací o MMR vakcíně

This post introduces an working algoritm that can be used for searching through a 3D space filled with 3D points. Each point is represented by its x, y, z coordinates. Search is performed using search rectangular box, specified by its left bottom back and right top front corners.

As a result, all points from 3D space that belong to the search box are returned. See this demo live:

See video of the demo:

Work on it is done as part of my Phd studies at University of Pardubice under supervision of my teacher, Kavička Antonín, prof. Ing. Ph.D. .

Demo demonstrating the algoritm implementation

• Demo first generates fixed number of 3D points in the working area between -100,-100,-100 and 100, 100, 100. These points are added to the QuadTree3D instance using add method. This is the part when quad tree is built in memory:

for (int i=0;i< numberOfPoints;i++) {

    int x = random.nextInt(quadrantWidth+1) - quadrantWidth/2;

    int y = random.nextInt(quadrantWidth+1) - quadrantWidth/2;

    int z = random.nextInt(quadrantWidth+1) - quadrantWidth/2;

    quad.addPoint(new Point(x,y,z));

}

• Next step is performing a search in this 3D space, using search method. It accepts two arguments defining a search box, leftBottomBack and rightTopFront points:

Point searchFrom = new Point(fromX, fromY, fromZ);

Point searchTo = new Point(toX, toY, toZ);

List<Point> found = quad.search(searchFrom, searchTo);

• Finally, 3D space is rendered on the OpenGL canvas. Found points are displayed as white, other points are green. Quadrant’s cubes are rendered as cyan and search box is transparent purple.

How to execute demo

If you are running Windows, just download and extract the zip file, execute bin/go.bat from command line. On other OS you can run is as standard java application:

java -Djava.library.path=. -DnumberOfPoints=%POINTS% -DfromX=%fromX% -DfromY=%fromY% -DfromZ=%fromZ% -DtoX=%toX% -DtoY=%toY% -DtoZ=%toZ% -DshowQuadCubes=%showQuadCubes% -classpath quadtree3ddemo.jar;timer.jar;jogl.jar net.jetensky.quadtree3d.QuadTree3DDemo

Just replace the %% variables with desired values. You can use the default settings:

SET POINTS=30
# Whole searchable region is from -100,-100,-100 to 100,100,100, FROM and TO search window should be subset of this
SET fromX=-100
SET fromY=-50
SET fromZ=-0
SET toX=-50
SET toY=110
SET toZ=70
SET showQuadCubes=false

Technical background – Region Point Quad Tree

At the core of the demo is implementation of PR Quadtree (Point Region Quad Tree), as defined by Hanan Samet in Foundations of Multidimensional and Metric Data Structures in chapter 1.4.2.2 PR Quadtrees. There is a slight difference in the implementation of logic behind choosing, wheter actual quadrant should be searched:

• Samet (in figure 1.27) suggests to compare the root of the quadrant to the position of the search box and enumerates all the options.
• My implementation simplifies the solution to answering, whether the search box and actual quadrant intersects. If there is no intersection, whole quadrant can be skipped as it is completely outside the search box

      public boolean intersects(Point leftBottomBack, Point rightTopFront) {

          return !(

                  center.getX() - halfWidth > rightTopFront.getX() ||

                  center.getX() + halfWidth < leftBottomBack.getX() ||

                  center.getY() - halfWidth > rightTopFront.getY() ||

                  center.getY() + halfWidth < leftBottomBack.getY() ||

                  center.getZ() - halfWidth > rightTopFront.getZ() ||

                  center.getZ() + halfWidth < leftBottomBack.getZ()

          );

      }

Used technologies

Algoritm and demo is implemented in Java. Graphical representation of points is rendered using Java OpenGL library (jogl). Whole demo, including sources and working executable, can be downloaded:

• 3DQuadTree.zip

Please feel free to use the source code for your needs, however I would be glad if you let me know how you use it by writing a comment to this post.

About 3DKeyboard

Recently, after some research and solving some problems with C++ code, I have finished initial version of 3DKeyboard. Work on it is done as part of my Phd studies at University of Pardubice.

3DKeyboard is an application utilizing Microsoft Kinect, that enables you to type
text with moving one of your hands in the air. It’s purpose is to minimize need
of a real keyboard in situations, when you want to control computer in a touchless
manner.

See video of 3DKeyboard in action:

Note: The application is in prototype stage and has not been properly tested. However, if you really want to try it, you can download the Windows version 3DKeyboard v0.0.1.zip (please read readme.txt in the package for instructions).

Keyboard layout

Keyboard consists of 7 cubes in a cross-like pattern. Crosses are parallel to
user screen, so normally you see only front cube (blue) and middle layer (green)
cubes.

Side view of 3D Keyboard	Front view of 3DKeyboard

How to type letters

• Start with waving your hand in front of kinect – white sphere appears
  at the position of your hand
• Place the sphere in the central green cube, so that it highlights
  - this is starting position for typing
• Move your hand to some other cube(s) and back to the center – this
  writes a letter and adds it to the text displayed at the top of the screen

This table shows the movements to be done so that particular letter is
typed:

Logic behind the letters and their typing movements

You may have noticed, that pattern how the letters are distributed in the
keyboard is similar to that of 8pen, which
can be used on smart phones. That is right. This picture shows 8pen keyboard, which
can be thought as a 2D version of 3DKeyboard.

To type letter y using 8pen, you need to:

• Put your finger to the central circle
• Move your finger up
• Move your finger left and down (crossing the blue line)
• Move your finger back to the center

To type letter y using 3DKeyboard:

• Put the hand to the central green cube
• Move your hand up to the top cube
• Move your hand left and down to left cube
• Move your hand back to center

If you study the movements table for 3DKeyboard, you will realise that:

• Letters closest to the 8pen circle (i, y, dot, comma, e, t, o, a) are typed
  the same way as using 8pen in a plane parallel to the screenImagine that you place your right palm in front of your face (thumb up)
  and draw on it 8pen diagram. Then type letters using this diagram, f.e. „a“
  you would type moving your hand right, up and little left and than down to the
  middle cube.
• Letters with distance of 2 from the 8pen circle (d, x, s, n, l, h, u, r)
  are typed the same way as using 8pen, but in a vertical plane orthogonal
  to the screenImagine that you place your right palm in front of your face (thumb up),
  draw 8pen diagram on it and rotate the hand in the elbow right. Then type letters
  using this diagram, f.e. „r“ you would type moving your hand close to your body
  (forth), then up and back closer to the screen and than down to the middle cube.
• Letters with distance of 3 from the 8pen circle (g, k, c, m, p, b, w, f)
  are typed the same way as using 8pen, but in a horizontal plane orthogonal
  to the screenImagine that you place your right palm in front of your face (thumb up),
  draw 8pen diagram on it and rotate the palm facing up. Then type letters using
  this diagram, f.e. „f“ you would type moving your hand left, than back closer
  to the screen and forth to the middle cube.
• Letters with distance of 4 from the 8pen circle (z, apos, v, j, q, @, !
  and ?)  have no relation to the 8pen layout, just see the movements table
  how to type them.

Used technologies

3DKeyboard was created using these technologies:

• Microsoft Kinect device for XBox 360
•  Openni Kinect drivers
• Primesense Nite middleware

What next?

To improve the application, I have several thinks in mind:

• Render letters in some sensible way above the sides of the cubes, so that
  it is clear without reference table, how to write different characters
• Instead of inputting text on the top of screen, which is good for prototype
  but actually useless for real needs, type them to different active window (f.e.
  search box in browser)
• Introduce calibration of the cube cross, so that user can choose, where
  in his real space his keyboard resides. This would help to work naturally perpendicular
  to screen even if kinect is not centered to the screen (f.e. placed to the right
  of the screen).

This post describes problems I needed to solve to make run Kinect examples. These examples are part of distribution of OpenNI and Nite libraries. As I am utter newbie in C++ and MS Visual Studio, please consider these as „work for me“ and „may not be best“ solutions. However I hope they may be helpful to someone. If you encounter these or different problems related to running the samples, please share with us your experience (and possibly solution) in the comments.

My environment is Windows Vista 32 bit SP1, Intel Core Duo 2GHz, JDK 1.6.0_03.

Content

Problems debugging OpenNI and Nite in MS Visual Studio

Note: The fix procedures described here are valid for Debug configuration profile, but they work as well for Release, just swap Debug in the instructions for Release.

Failed to save the updated manifest to the file

This is a build time error. It appeared, when I converted project to newer format and tried to built it. Oddly enough, this error disappears if you close MS Studio and reopen the project using *.sln file.

This is the full text of the error:

mt.exe : general error c101008a: Failed to save the updated manifest to the file "Debug\NiSimpleRead.exe.embed.manifest". The parameter is incorrect.

fatal error RC1015: cannot open include file ‘afxres.h’

This is a build time error. If this happens to you, it is related to difference between your development environment to those used in distribution, as suggested in this discussion. To fix it, replace afxres.h with windows.h in NITE.rc file.

The program PointViewer has exited with code 131076

This is happens if you try to run sample and have incorrectly set working directory. To fix it:

• Right click  your project in Solution Explorer and choose Properties
• Go to Configuration properties\Debugging\Working Directory and change $(ProjectDir) to ..\Bin\Debug
  Note: I have changed Intermediate Directory as well, but I am not sure, if it was necessary.

The system cannot find the file specified

This may happen if you run sample from Visual Studio, even if it runs all right from Windows Explorer. I believe this is caused by mess in output folders and project name settings, as they are distributed in Nite samples’ vcproj files. Project name is f.e. PointViewer, but output file is set to Sample-PointViewer. Also, the default build path is set to ..\Bin\Debug, but settings point to Debug folder. To fix this:

• Right click  your project in Solution Explorer and choose Properties
• Go to Configuration properties\Target Name and replace variable $(ProjectName) with Sample-PointViewer (for different samples it will be similar, f.e. Sample-SimpleRead)
• Go to Configuration properties\General\OutputDirectory and change Debug\ to ..\Bin\Debug\

The program ‘[5540] Sample-StickFigure.exe: Native’ has exited with code 131076 (0×20004)

Alternative symptoms of this error is that if you run the application exe file from command line, following error appears:

InitFromXml failed: Device Protocol: Bad Parameter sent!

This is caused by Kinect not accepting resolution 320×240.  To solve it:

• In all 3 XML files under directory Change Nite unstable 1.3.0.18\Data\ change <MapOutputMode xRes=“320″ yRes=“240″ FPS=“30″/> to <MapOutputMode xRes=“640″ yRes=“480″ FPS=“30″/>

Running OpenKinect.org samples

Download compiled library freenect.dll and freenect_sync.dll

I spent a lot of time getting FreenectTest.java to run. One of the steps required and described at OpenKinect getting started Windows page was to build freenect.dll. First I tried to download it somewhere, but only found that lot of people in the discussions are looking for it too. So I went through the compile process – here are steps I needed to do (to name a few):

1. Install proper kinect drivers
2. Install and use cmake.exe to generated MS Visual Studio project files
3. Compile these files and generate freenect.dll

If you want to skip these steps, download my compiled version of freenect.dll and freenect_sync.dll. Hopefully it will work for you, I was compiling using MS Visual Studio 10 on Windows Vista.

Note: If using java wrapper as me, place it in the lib folder alongside libusb0_x86.dll and run with -Djava.library.path=lib, when I did so I was able to run OpenKinect-libfreenect-2ea3ebb\wrappers\java\src\test\java\org\openkinect\freenect\FreenectTest.java.

New release of windows libfreenect port addresses compilation issues (updated on 26/01/2011)

Since this post has  been published, Marcos Slomp and libfreenect team released new version of windows port, that adresses compilation issues I was facing. Fortunatelly, building your own version of libfreenect.dll on Windows should be easier. See readme.txt for Windows port.

Problems running CLNUI and CLUNI4J sample (new)

Unable to load library ‘CLNUIDevice.dll’

There can be multiple causes for this problem, like missing dll file in java.library.path. Even if you have library there, the error can still pop up. To fix this, I needed to following:

• Fix wrong DLL identifier in CLUNI4J distribution
  On line 11 in CLNUI.java remove .dll from libraryFilename pointer and rebuild project.

      public static String libraryFilename = "CLNUIDevice";

• Always start the project in Debug mode
  To be honest, I do not understand this one. However, when I run PointCloudApp using Run mode in my IntelliJ Idea, it fails with the UnsatisfiedLinkError. If I run it using Debug mode, it does not fails (however fails with Couldn’t start camera problem described later in this post).

Full error message is:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'CLNUIDevice.dll': The specified module could not be found.

    at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:164)

 CLUNI4J – (KinectDevice)  Couldn’t start camera

This happens, when I try to run PointCloudApp. It happens at line 185 of cl.nui.KinectDevice class. Then follows this error:

Exception in thread "Animation Thread" java.lang.NullPointerException

    at cl.nui.KinectDevice.getDepthAt(KinectDevice.java:776)

Up till now, I was unfortunatelly unable to find reason for this error and this is the reason I am abondoning my experiments with CLUNI4J library. If you have any ideas I will be gratefull if you give me a hint in the comments section.