3DKeyboard project


About 3DKeyboard

Recently, after some research and solving some problems with C++ code, I have finished initial version of 3DKeyboard. Work on it is done as part of my Phd studies at University of Pardubice.

3DKeyboard is an application utilizing Microsoft Kinect, that enables you to type
text with moving one of your hands in the air. It’s purpose is to minimize need
of a real keyboard in situations, when you want to control computer in a touchless
manner.

See video of 3DKeyboard in action:

Note: The application is in prototype stage and has not been properly tested. However, if you really want to try it, you can download the Windows version 3DKeyboard v0.0.1.zip (please read readme.txt in the package for instructions).

Keyboard layout

Keyboard consists of 7 cubes in a cross-like pattern. Crosses are parallel to
user screen, so normally you see only front cube (blue) and middle layer (green)
cubes.

Side view of 3D Keyboard Front view of 3DKeyboard

How to type letters

  • Start with waving your hand in front of kinect – white sphere appears
    at the position of your hand
  • Place the sphere in the central green cube, so that it highlights
    - this is starting position for typing
  • Move your hand to some other cube(s) and back to the center – this
    writes a letter and adds it to the text displayed at the top of the screen

This table shows the movements to be done so that particular letter is
typed:

Key 1. move 2. move 3. move 4. move
[Space] BACK CENTER    
! RIGHT DOWN LEFT CENTER
UP LEFT DOWN CENTER
, LEFT DOWN CENTER  
. LEFT UP CENTER  
? RIGHT UP LEFT CENTER
@ DOWN RIGHT UP CENTER
a RIGHT UP CENTER  
b FRONT RIGHT CENTER  
[Backspace] LEFT CENTER    
c LEFT BACK CENTER  
d UP FRONT CENTER  
e DOWN LEFT CENTER  
f RIGHT BACK CENTER  
g BACK RIGHT CENTER  
h DOWN FRONT CENTER  
i UP RIGHT CENTER  
j LEFT DOWN RIGHT CENTER
k BACK LEFT CENTER  
l DOWN BACK CENTER  
m LEFT FRONT CENTER  
n BACK DOWN CENTER  
o RIGHT DOWN CENTER  
p FRONT LEFT CENTER  
q DOWN LEFT UP CENTER
r FRONT UP CENTER  
s BACK UP CENTER  
t DOWN RIGHT CENTER  
u FRONT DOWN CENTER  
v LEFT UP RIGHT CENTER
w RIGHT FRONT CENTER  
x UP BACK CENTER  
y UP LEFT CENTER  
z UP RIGHT DOWN CENTER

Logic behind the letters and their typing movements

You may have noticed, that pattern how the letters are distributed in the
keyboard is similar to that of 8pen, which
can be used on smart phones. That is right. This picture shows 8pen keyboard, which
can be thought as a 2D version of 3DKeyboard.

To type letter y using 8pen, you need to:

  • Put your finger to the central circle
  • Move your finger up
  • Move your finger left and down (crossing the blue line)
  • Move your finger back to the center

To type letter y using 3DKeyboard:

  • Put the hand to the central green cube
  • Move your hand up to the top cube
  • Move your hand left and down to left cube
  • Move your hand back to center

If you study the movements table for 3DKeyboard, you will realise that:

  • Letters closest to the 8pen circle (i, y, dot, comma, e, t, o, a) are typed
    the same way as using 8pen in a plane parallel to the screenImagine that you place your right palm in front of your face (thumb up)
    and draw on it 8pen diagram. Then type letters using this diagram, f.e. “a”
    you would type moving your hand right, up and little left and than down to the
    middle cube.
  • Letters with distance of 2 from the 8pen circle (d, x, s, n, l, h, u, r)
    are typed the same way as using 8pen, but in a vertical plane orthogonal
    to the screen
    Imagine that you place your right palm in front of your face (thumb up),
    draw 8pen diagram on it and rotate the hand in the elbow right. Then type letters
    using this diagram, f.e. “r” you would type moving your hand close to your body
    (forth), then up and back closer to the screen and than down to the middle cube.
  • Letters with distance of 3 from the 8pen circle (g, k, c, m, p, b, w, f)
    are typed the same way as using 8pen, but in a horizontal plane orthogonal
    to the screen
    Imagine that you place your right palm in front of your face (thumb up),
    draw 8pen diagram on it and rotate the palm facing up. Then type letters using
    this diagram, f.e. “f” you would type moving your hand left, than back closer
    to the screen and forth to the middle cube.
  • Letters with distance of 4 from the 8pen circle (z, apos, v, j, q, @, !
    and ?)  have no relation to the 8pen layout, just see the movements table
    how to type them.

Used technologies

3DKeyboard was created using these technologies:

What next?

To improve the application, I have several thinks in mind:

  • Render letters in some sensible way above the sides of the cubes, so that
    it is clear without reference table, how to write different characters
  • Instead of inputting text on the top of screen, which is good for prototype
    but actually useless for real needs, type them to different active window (f.e.
    search box in browser)
  • Introduce calibration of the cube cross, so that user can choose, where
    in his real space his keyboard resides. This would help to work naturally perpendicular
    to screen even if kinect is not centered to the screen (f.e. placed to the right
    of the screen).
1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5,00 out of 5)
Loading ... Loading ...

7 Comments so far

  1. Jan Novotný on Únor 4th, 2011

    I am really wondering how far will you get it. But for the first try it is really big advance. Keep us informed, please :)

  2. Jety on Únor 5th, 2011

    Thank you Jan for your encouragement. Today there is an article about 3D Keyboard at KinectHacks.net – http://kinecthacks.net/8pen-in-3d-3d-keyboard-with-kinect/

  3. Jety on Březen 10th, 2011

    This is something that rocks – http://www.youtube.com/watch?v=IosmAD10AKg&feature=mfu_in_order&list=UL. Controlling Windows with avatar through virtual environment, having visual feedback of the actions. In this direction I would like to go too.

  4. Libor on Červenec 20th, 2011

    Hello
    Myslim, ze se v oboru bezdotykoveho ovladani pocitace objevil jeden skok v opensourcovni eyewriter viz http://www.ted.com/talks/lang/eng/mick_ebeling_the_invention_that_unlocked_a_locked_in_artist.html
    see ya
    L

  5. Jety on Červenec 22nd, 2011

    Ahoj Libore, díky za tip.

  6. seo dallas Texas on Duben 17th, 2013

    seo dallas Texas…

    Jetyho blog » 3DKeyboard project…

  7. Bandar Judi Poker Terbesar on Říjen 17th, 2017

    Thanks for another fantastic article. Where lse could anyone
    get that kiund of info in such an ideal method oof writing?
    I have a presentation subsequent week, and I’m onn thee search for such
    info.