Integrating the Leap Motion with the HoloLens in Unity
I’ve used the HDK (HoloLens Dev Kit) for years and the gesture tracking is pretty limited. I haven’t found great evidence that it’s possible to create custom gestures for the HoloLens, through searching forums online. Because of this, I decided to integrate the Leap Motion with the HoloLens in order to create more granular interactions for AR. The Leap allows for skeletal tracking of each index, so further down the line.
When I started hacking together this prototype, I couldn’t find any information on integrating the two save for one GitHub post.
I’m here to tell you about two ways to you could achieve tracking with the Leap using the HoloLens:
- Using Unity Play mode with holographic remoting. (Easy)
- Sending the Leap tracking data over a network (probably using a networking plugin like Photon). (Hard)
In both scenarios, you’ll be tethered to the computer by the Leap, but using a super long USB cord you can get more movement. Unfortunately the mini-usb 2.0 port on the HoloLens is only for charging and debugging, so it won’t accept any input devices. Bummer.
The first method is easier because it doesn’t involve networking or building the application to the HoloLens, bypassing the build process in Unity, and deployment with Visual Studio. Here’s a high-level overview of that method, and the way I accomplished Leap/HoloLens integration for my project:
- On your HoloLens, go to the Microsoft Store and install the Holographic Remoting Player app
- Plug in the Leap Motion into your PC.
- Using the HoloLens specific version of Unity, configure your project for HoloLens.
- Import the Leap SDK.
- Open one of the Leap demo scenes, and configure it for the HoloLens per the documentation.
- You’ll be able to interact with your project with the two pieces of hardware via HoloLens remoting.
This is the quickest way to get to a usable solution, but it’s not ideal. The ideal would be skeletal tracking in the device that is fully configurable.
As for the hardware, I mounted the Leap on top of the HoloLens so as to not interfere with the gesture recognition already in the system, so I can use the menus in the OS with gestures like ‘bloom’. My first attempt at attaching the Leap resulted in obscuring the vision of the HoloLens, so I had to remove it and move it on top of the visor. Here’s a before and after, don’t put it where I had first attached it:
Getting the angle of the Leap correct for aligning the real and virtual hands was tricky and took some time to get the putty placement right. The putty was initially used because the materials that are on the Leap and HoloLens don’t facilitate tape or adhesion. Eventually we updated it to be more stable with some bendable metal pieces – during demos, my coworkers would frequently knock it off the top when they go to take off the HoloLens.
That’s it. Feel free to contact me if you have any questions regarding the implementation or project at firstname.lastname@example.org.
Have fun touching things in AR.