Integrating the Leap Motion with the HoloLens in Unity

From my experience using the HDK (HoloLens Dev Kit) and searching forums online, I believe the case to be that it is not possible to create custom gestures for the HoloLens. Because of this, I decided to integrate the Leap Motion with the HoloLens in order to create more granular interactions for AR. The Leap allows for skeletal tracking of each index, so further down the line we would be able to create our own gestures, and we can interact with diegetic interfaces in AR.

I won’t get into the project it was specifically used for, but I wanted to write something up so that people looking to integrate the two for their Unity applications would have more information than what’s currently out there. At the time, when I started hacking together this prototype, I couldn’t find any information on integrating the two save for one GitHub post.

Here’s how to do it with Unity, it’s pretty simple:

  1. Plug in the Leap Motion into your PC. The mini-usb 2.0 port on the HoloLens is only for charging and debugging, so it won’t accept any input devices.
  2. Using the HoloLens specific version of Unity, configure your project for HoloLens.
  3. Import the Leap SDK.
  4. Open one of the Leap demo scenes, and configure it for the HoloLens per the documentation.
  5. You’ll be able to interact with your project with the two pieces of hardware via HoloLens remoting.

This is the quickest way to get to a usable solution, but it’s not ideal. At this point you are tethered to the PC, based on how long your USB extension cable is for the Leap. Another solution could include the same setup, with a full application build to the HoloLens instead of remoting.  It could possibly be done in a way that the Leap streams the data to the application on the HoloLens from the PC, like what I referenced earlier. But in that case, you’re still tethered to the PC physically, which may not be a problem depending on the use of your application.

As for the hardware, I mounted the Leap on top of the computer hat so as to not interfere with the gesture recognition already in the system, just so I can use the menus in the OS. Here’s a before and after, where the before was my first attempt as sticking the Leap on the computer hat, don’t put it where I had put it:

 

Getting the angle of the Leap correct for aligning the real and virtual hands was tricky and took some time to get the putty placement right. The putty was initially used because the materials that are on the Leap and HoloLens don’t facilitate tape or adhesion. Eventually we updated it to be more stable with some bendable metal pieces – during demos, my coworkers would frequently knock it off the top when they go to take off the HoloLens.

That’s it. Have fun touching things in AR.

Leave a Reply

Your email address will not be published. Required fields are marked *