openleap / pyleapmouse

A Leap Motion based mouse in Python

OpenLeap

An initiative for an open source driver for the Leap Motion sensor.

I recently got my hands on Leap Motion devices [1].

The so-called "Linux support" is in fact only a x86/x64 Ubuntu 12.04 LTS proprietary binaries. So there is no luck for other platform/system aficionados.

Since the device may become mainstream, with the Leap motion HP deal for example [2], an open source alternative to access to the device would be welcome. Moreover, it would be nice to see the device work on new kind of devices, like Android/ARM platforms. I initially wanted to wait until the release of the Leap Motion to publish this code. The release date was postponed [3] and I waited too long, so here it is.

Plugging the device will present a standard USB Video Class device (like a USB webcam). A modern system even create the /dev/videoN device for you, but trying grab a video stream from this device will result to errors or black images.

By looking a bit to the Leap SDK, I saw lots of open source BSD-like packages: libusbk, Qt4, turbojpeg, OpenSSL...

I was a bit disappointed to see an almost standard device using so many open source BSD code not working on an open system. So I decided to dig a bit to make the device work. Initially I wanted to access the raw depth map, a bit like OpenNI for the Kinect. It turned out it's not directly possible since the leap motion hardware seems to be very simple (or simpler than the Kinect, at least). From what I understood, the leap motion device is "only" a stereo infra-red UVC webcam with a wide-angle lens. For an announced price of 80$ I was expecting something more complex. I may have missed something, but the real magic seems to be in the software part. I looked a bit in the device. The main components are: a Cypress EZ-USB FX3 controller [4], a serial EEPROM [5] [6], two camera sensors (not sure if it's CCD, CMOS or something else), few IR LEDs and a bunch of small components linking everything together. For understanding FX3 based devices, Cypress Application Notes are a good source of information [7], more especially the image sensor one [8].

I wrote two small programs to access to the device:

• "low-level-leap": for initializing the leap motion device and grabbing raw data on its standard output.

• "display-leap-data-opencv": which read the raw data from its standard input, unpack frames, and display them. There is also a sdl version.

These two programs are very simple proof-of-concept code. Since this was coded in few hours, the code quality is really poor. Some frame drop / image breaks may appear due to the synchronous I/O.

For the device initialization, I did not reverse engineered the device communication. I just captured a normal initialization of the Leap SDK with Wireshark, and made a Wireshark Lua script to generate libusb C code to replay the data. I expect there is some calibration in there and a replayed initialization like this may result to a poor functioning. But it's a start. I don't include my initialization file, because I don't know what kind of data there is in it (a device serial number for example).

I saw two operating modes of the device: one with 640x240 frames with IR LEDs always ON. And one other with 640x120 frames with IR LEDs going ON and OFF at each frame. I suppose this 2nd mode is the "bad lightning / robust mode" (for subtracting ambient IR sources). To be sure, one has to play with the control panel of the device, and see what happen on the USB bus. Moreover, the device advertise other resolutions in its descriptor, but I didn't saw them during my testing which was USB 2.0 only. Maybe they are used in USB 3.0.

Frames are not compressed. Each pixel has two components of 8 bits each. The two components are the Left/Right camera pixels (i.e. pixels from the two cameras are interleaved). Each component is the IR illumination. The last few pixels are data that does not seems to be the illumination.

Moreover, the Leap SDK may need to download the device firmwares and some "calibration" data. I currently have no idea what it is. Could be lens distortion info, learning data sets, etc...

How to see Leap Motion images:

• Plug the Leap,

• run the command: "lsusb -d f182:0003" and note the bus and device address. Check if your device has a different VID/PID. Replace it if needed.

• This step is needed only if there is no udev rule file installed. Change device permission with the command, as root: chmod 666 /dev/bus/usb/BUS/DEV_ADDRESS (replace BUS and DEV_ADDRESS with values from the previous command).

• in case the usbmon kernel module is not loaded, run: "modprobe usbmon" as root

• Run a capture of the USB bus traffic, with the command, as root: "tshark -i usbmonX -w leap-init.pcap" where X is the number of the USB bus. You may also use the graphical Wireshark as well.

• Run the original Leap software, make sure the device is properly detected and initialized. If unsure, test with the visualizer that everything is working.

• Stop the Leap software and the capture.

• Generate the Leap initialization C code with the command: "make_leap_usbinit.sh DEV_ADDRESS leap-init.pcap > leap_libusb_init.c.inc"

• Build the low-level-leap and display-leap-data-opencv programs with the "make" command.

• You should be able to see images from the Leap device with the command: "./low-level-leap | ./display-leap-data-opencv"

  If there is lots of frame skips/break, try the following things:

• close some CPU consuming applications;

• set the CPU frequency governor to "performance";

• add some poor buffering with: "./low-level-leap | cat | ./display-leap-data-opencv";

• work offline: "./low-level-leap > leap.dat" then "./display-leap-data-opencv < leap.dat";

• implement it the right way (async IO, ring buffer, threads)

I don't think the publication of this code will do any harm to the Leap Motion company, since it's only a low level frame display, and there is no finger tracking code. On the other hand, I would like to see if the community can come up with other ideas with the device. Ideally, the Leap Motion could also give few details for a low level acces to the device, or maybe a simpler driver than the current SDK. An access to the depth map would be nice too, if possible.

Since I will not have lots of time to put on this little project, I expect some people in the community to continue it. Here is some hints for future works:

• Make a proper driver based on the uvcvideo driver. Maybe it's possible to add some quirks into the kernel driver: it would be nice to see the leap motion as a V4L2 video input. See [9] [10].

• Play with OpenCV stereo image calibration to reconstruct a depth map from the video flow. See [11] [12].

Have fun!

Elina.

[1]. https://www.leapmotion.com/

[2]. http://www.bbc.co.uk/news/technology-22166424

[3]. http://blog.leapmotion.com/post/48872742284/release-date-update

[4]. Cypress EZ-USB FX3 datasheet: http://www.cypress.com/?mpn=CYUSB3014-BZXC

[5]. FLASH Memory: Newer leap (rev5+?) 32Mb version, MX25L32 datasheet: http://www.mxic.com.tw/QuickPlace/hq/PageLibrary4825740B00298A3B.nsf/h_Index/6F878CF760C559BD482576E00022E6CC/?OpenDocument&EPN=MX25L3206E

[6]. FLASH Memory: Early version 8Mb version, m25p80 datasheet: http://www.micron.com/parts/nor-flash/serial-nor-flash/m25p80-vmw6g?pc={6BAC3112-DFB2-4ADB-BAB0-158F28467649}

[7]. Cypress EZ-USB FX3 application notes: http://www.cypress.com/?id=3526&rtID=76

[8]. Cypress EZ-USB FX3 application note: AN75779 - How to Implement an Image Sensor Interface with EZ-USB FX3 in a USB Video Class (UVC) Framework http://www.cypress.com/?rID=62824

[9]. USB Video Class 1.1 specification: http://www.usb.org/developers/devclass_docs/USB_Video_Class_1_1_090711.zip

[10]. Linux kernel uvcvideo driver: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/media/usb/uvc

[11]. OpenCV StereoBM operator: http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#stereobm-operator

[12]. OpenCV stereo image match sample: https://github.com/Itseez/opencv/blob/master/samples/cpp/stereo_match.cpp

pyleapmouse's Issues

Which version of the Leap SDK?

I'm new to all this, and haven't been able to get PyLeapMouse to work so far. For my first step in debugging I figure I should make sure I'm using the right SDK version. So, which version of the Leap SDK does PyLeapMouse use?

Also, if this information could be included in the readme that would be very helpful.

Bind commands with gesture

Hello,

I did a little project to bind command with gestures. I think it can be useful to add it into yours. It's not working very well for the moment but I will improve it.

I don't like to use the mouse with leap because it's very imprecise. I prefere to start music with a circle, start the browser, etc.

What you think?

Will not run

crashes on load, any idea why?

Fatal Python error: PyThreadState_Get: no current thread

I'm not sure where to even begin debugging this.

$ python  PyLeapMouse.py --palm
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6

I'm on OS X use somebody with a fairly standard homebrew set up. (Python is provided by homebrew in this case.)

If I run with the system /usr/bin/python and it starts up okay. But the cursor doesn't respond to my hand over the leap of motion.

Tracking stops when window out of focus

On Windows 8, when the window goes out of focus the mouse stops moving.

I've narrowed it down that it has to do with the Leap not giving events when the window goes out of focus. At first I thought the mouse events were the problem however I tested out Sample.py in the Leap SDK and it stops printing messages as soon as the window loses focus - implying this is something to do the Leap Controller listener.

Any ideas why this might be? Or how to fix it?

The Leap Screen freature isn't available in the newest update.

The newest update in the Leap API disables the screen feature and makes the screen API unavailable. Therefore, users that would like to use the default feature of using their finger for the PyLeapMouse script are left with an error message:
Please calibrate your Leap screen feature.

There should be some fix to use the field of view from the Leap Motion device as some kind of makeshift screen for this functionality. With it being an inverted pyramid from the POV of the device, it would be kind of difficult to scale the screen to the field of vision from the device. Testing and ease of usage would be concerns for those that use the library as a intermediary point between their device and their screen, but I'm just bringing up this point for the latest update to the leap api.

not work well on Ubuntu 12.04 with GNOME classic

Hi

when I run it, it just move really slow and it even not move at all from time to time.
The console print all the movement but the cursors almost not moving.

Leappython module

when I try to start the PyLeapMouse.py it gives me an error in the leappython module saying
AttributeError: 'module' object has no attribute 'delete_BoolArray'
I have Windows 7 64bit and Python 2.7.5

Finger mode fails to move cursor

When program is started in default mode, once my hand or finger is within detection range I get: 'NoneType' object has no attribute 'intersect' instead of typical coordinates as in --palm mode and the mouse cursor does not move. Nevertheless clicking in still seems to work.

I am using the leap on Fedora 19/KDE 4.11.2 if that helps any.

PyLeapMouse Pointer Goes to Corner

Hello,

Using PyLeapMouse in Linux Mint (Ubuntu 13.04) when I insert a finger into the frame the mouse will go all the way to the top left corner and sit there, it will not do anything else.

Can anybody comment on the potential cause of this?

Remove Windows and OSX leap sdk files

I think would be better removing all the LeapMotion sdk files from the repository and ask the users to get them from their sdk folder ..

failed on Linux -> ImportError: No module named pymouse

just clone it and copied the following to the Linux dir :
LeapSDK/lib/Leap.py
LeapSDK/lib/x86/LeapPython.so
LeapSDK/lib/x86/libLeapCSharp.so
LeapSDK/lib/x86/libLeap.so
LeapSDK/lib/x86/libLeapJava.so

then I got this error:

> python PyLeapMouse.py

Traceback (most recent call last):
File "PyLeapMouse.py", line 4, in
from leap import Leap, Mouse
File "/data/homeData/git/PyLeapMouse/leap.py", line 7, in
import Linux.Mouse as Mouse
File "/data/homeData/git/PyLeapMouse/Linux/Mouse.py", line 1, in
from pymouse import PyMouse
ImportError: No module named pymouse

Thumb detection not working very well

Hi,

Good job on your implementation of the mouse with Leap Motion.

Just wanted to give you some feedback.

1. I don't seem to be able to use the thumb gesture to click. It barely works 1 out of 12 times I try it.
2. You might want to implement a little "softener" on the move of the cursor; the Leap Motion device is a very precise tool and it'll catch very small movement of the finger (shake) and make the cursor shake.

Just want to help you.

Thanks again!

Other methods to click

I find the thumb-click thing not relay working too well. Maybe because of the previously mentioned lack of accuracy on the thumb recognition in issue #4 ?
Some ideas:

1. Scissors Click (saw that in a video): Point at the screen with the indexfinger and middlefinger. Bring the two fingers together as if you cut something with a scissor.
2. Use of the Tap Gesture (will get the "mouse" jumpy?)
   3 ...

OSX Startup Fail : PyThreadState_Get: no current thread

xx:PyLeapMouse-master xc$ python PyLeapMouse.py
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6

Ubuntu 14.04 - install

Hello,
i have bought the famous leap motion.

i want control my ubuntu with it, officiali i found nothing, but i found u program, i want to try iot.

but sorry the install procedure not really explicit 👍

LINUX USERS: You must copy the Leap.py file and all required library files (.sos) from your Leap SDK folder to the "Linux" folder (same reason as for Windows); alternatively, add the directory (or directories) containing them to your PYTHONPATH. Additionally, you must have the PyUserInput and Xlib Python modules installed.

ok so, where is the Leap SDK folder ?

so please how i cna meke the script works

thanks

Could someone figure out what is needed in the windows library folder?

I feel like there's way too much crap in there right now. Would someone mind figuring out what is and isn't needed?

Error on running pyleapmouse on Arch Linux

I receive the following error on running pyleapmouse:
Traceback (most recent call last):
File "PyLeapMouse.py", line 4, in
from leap import Leap, Mouse
File "/opt/pyleapmouse/leap.py", line 7, in
import Linux.Leap as Leap
File "/opt/pyleapmouse/Linux/Leap.py", line 28, in
LeapPython = swig_import_helper()
File "/opt/pyleapmouse/Linux/Leap.py", line 24, in swig_import_helper
_mod = imp.load_module('LeapPython', fp, pathname, description)
ImportError: /opt/pyleapmouse/Linux/LeapPython.so: undefined symbol: _ZNK4Leap5Image10rayOffsetXEv

I had installed PyLeapMouse using AUR. I couldn't figure out the what is causing this error

Duplication

Hey @wyager!

This is an awesome project! I remember you recently asked me about it and OpenLeap.

Unfortunately, it looks like you cloned it (rather than transferred it) to OpenLeap. If this was because you don't know how (the process isn't all that intuitive) an explanation is available at https://developer.leapmotion.com/forums/forums/projects-and-collaborations/topics/leap-open-source-github midway down the page.

If this was because you wanted admin rights, fear not! You can transfer a repo to OpenLeap and still have complete admin rights (which include the option to transfer the repo back to yourself if you so choose.) Just email [email protected] to get them!

I've deleted the clone so now you can transfer this repo at will.

Thanks,
Owen

Palm mode works, Finger doesn't

Ubuntu 13.10 x64

I get the following when running the software in finger mode:

Using finger mode...
Adding Listener.
Please calibrate your Leap screen feature.

Palm mode works, but is quite twitchy. I'm using exactly the command.ini you have in the readme.

Xlib error

invalid

Works in --palm mode but are these the correct gestures?

This is great stuff. +1 for the two hand approach which is much more ergonomic. 💯
I'm not sure about the gestures though, tilting a palm left or right nicely drives the pointer on the axis, but for moving the pointer up or down I seem to use a gesture that's way too original (no matter the orientation of the device, I need to tilt my hand/palm down to move the pointer up, and tilt my hand/palm up to move the pointer down.

Is it supposed to work much differently?

ImportError: No module named Leap

Hi there,

Running this on LM 14 (Debian/Ubuntu). I have copied the files as instructed and when I launch PyLeapMouse with python I get this:

Traceback (most recent call last):
File "PyLeapMouse.py", line 4, in
from leap import Leap, Mouse
File "/home/zero/Downloads/PyLeapMouse-master/leap.py", line 6, in
import Linux.Leap as Leap
ImportError: No module named Leap

I have the SDK properly installed and can run Visualizer and other non python Leap apps.

When I run located Leap.so I cannot even find a Leap.so file on my system, there is libleap.so but no Leap.so

Always 5 fingers for my gesture hand

Hi everyone,
I do not know how to click or scroll in palm mode. After adding a few print in the code source, I found my problem : when the leaps detects the second hand (left one), it always recognize 5 fingers instead of 2 or 1.
Any ideas about how to fix this problem ?

Invalid Syntax?

$ >> python PyLeapMouse.py
File "PyLeapMouse.py", line 9
print "----------------------------------PyLeapMouse----------------------------------"
^
SyntaxError: invalid syntax

The syntax error it actually points to is the quote. See screenshot for the real view of the output

http://i.imgur.com/5fSajTh.png

Clarification of "relative scrolling"?

I am planning to migrate PyLeapMouse to using https://github.com/SavinaRoja/PyUserInput for all mouse functions, so it can handle the cross-platform support. Potentially this could make the code entirely OS-agnostic (as Leap.py is included in the SDK and could be linked to).

As part of this, I pointed the PyUserInput maintainer to the scrolling code in PyLeapMouse so they could add full cross-platform scrolling support (needed before a migration would be possible). They have a question that I am unable to answer, however:

I'm not terribly familiar with relative cursor scrolling, and what distinguishes it from absolute scrolling; could you provide me with a few brief statements of the motivation and uses of it? It should help me become educated on the subject faster.

Would you be able to explain how your relative scrolling code came into being? Either here (and I will copy across), or you could reply directly at SavinaRoja/PyUserInput#10

Error in running PyLeapMouse.py

The function initialising the listener never returns, and the code get stuck at that function call.
"listener = Palm_Control_Listener(Mouse) "