Activiz holographic remoting: Stream VTK to the Hololens 2

This is a huge step towards Augmented Reality (AR) support in VTK. It is now possible to stream holograms from a PC to the Hololens 2 using Activiz! Thanks to the ongoing work around OpenXR support in VTK, we are able to release a dedicated version of Activiz that supports streaming holograms to mixed reality (XR) devices in real time.

Some devices such as the Hololens 2 only support Universal Windows Platform (UWP) applications, and do not provide any support for OpenGL. As a result, it is a huge challenge to deploy VTK-based applications on such devices.
Holographic remoting consists in a player application running on the device, and a VTK-based remote application running on a standard Windows machine. The remote application receives camera information and rendering resources from the player. It renders the VTK scene before streaming back the resulting texture to the player application. Hence, this streaming approach allows us to keep VTK on a desktop machine. This way we avoid the need to add UWP support in VTK, and we can also keep using VTK’s OpenGL-based rendering pipeline.

Still, DirectX must be used to fill the texture to be streamed back to the Hololens. To achieve this we create a texture shared by both a DirectX and OpenGL context. This is possible thanks to the NV_DX_interop extension available on almost every recent GPU.

Activiz

Activiz is a .NET library for advanced 3D visualization. It provides the VTK public API in C# to build interactive 2D/3D applications in the .NET environment. In addition to the state-of-the-art algorithms and rendering techniques offered by VTK, Activiz also interfaces well with existing .NET applications such as Unity.

Read more about the latest improvements and visit the Activiz website for more information about this outstanding library.

How to

It is quite straightforward to get started. By following those 3 steps, you can get VTK running into the Hololens.

Player application

• Download Microsoft Holographic player application Version_2.5.0 and follow the instructions to build the “player” solution.
• Deploy the player application to the Hololens by passing the IP address of the Hololens as first parameter.
• When the player is running, you should see the following message: “Waiting for connection on …”

You can now run the remote app.

Remote application

• Fill the contact form to request a special version of Activiz compatible with the Hololens.
• Download and install the Activiz For Hololens installer and open the HolographicRemoting example.
• Run the application by passing the IP address of your Hololens as 1st parameter.

You should see a sphere floating approximately 1 meter in front of your head.

Warning: If nothing gets displayed in the device and you have both an Intel and NVidia GPU in your laptop, try disabling one of the Display Adaptors in the Device Manager.

Status and next steps

This version supports rendering but does not yet offer the possibility to interact with the objects in the scene. We are actively working around the following items.

• Implement interactions to transform and pick holograms
• Refactor VTK’s existing VR support into a generic module offering interactive menus
• Finalize XR support in VTK and extend it to Paraview, 3DSlicer, and other VTK-based application
• Improve holograms stability
• Support depth information for a better integration with the physical environment and possible registration of holograms.

Stay tuned for further updates !