Following the initial proof of concept that was implemented in ActiViz, Kitware is pleased to officially announce support for rendering inside the Hololens 2 using VTK.
The recent addition of the OpenXR runtime in VTK allowed us to build a dedicated module for streaming rendered content to the Hololens 2. The new module, available in VTK 9.2.2, leverages the OpenXR Holographic Remoting extension to allow VTK users to bring advanced algorithms and visualization techniques to the Hololens 2, and interact with the scene in real time.
So far we mainly focused on building a common infrastructure for both augmented reality (AR) and virtual reality (VR). This first integration in VTK provides basic interactions for users to grab and move objects around in real time.
It also ensures that the advanced features offered by VTK, such as Volume Rendering or 3D widgets, can be effectively used and rendered in virtual scenes.
This initial support for the Hololens 2 defines a generic architecture that allows for building more complex feature in the future, including:
- Integration into Paraview and 3D Slicer
- Enriched hand interaction thanks to finger tracking
- Depth occlusion support for a better integration with the physical environment
Reminder about the approach
Some devices such as the Hololens 2 only support Universal Windows Platform (UWP) applications, and do not provide any support for OpenGL. As a result, it is a huge challenge to deploy VTK-based applications on such devices.
Holographic remoting consists in a player application running on the device, and a VTK-based remote application running on a standard Windows machine. The remote application receives camera information and rendering resources from the player. It renders the VTK scene before streaming back the resulting texture to the player application. Hence, this streaming approach allows us to keep VTK on a desktop machine. This way we avoid the need to add UWP support in VTK, and we can also keep using VTK’s OpenGL-based rendering pipeline.
Still, DirectX must be used to fill the texture to be streamed back to the Hololens. To achieve this we created a texture shared by both a DirectX and OpenGL context. This is possible thanks to the NV_DX_interop extension available on almost every recent GPU.
Download the Microsoft MixedReality HolographicRemoting samples and follow the instruction to build the player application.
⚠️ The version number in the branch name must match the version of the
Microsoft.Holographic.Remoting.OpenXr package used below by the remote application.
When the player is deployed, you should see the following message: Waiting for connection on XX.XX.XX.XX where XX.XX.XX.XX describes the IP address the remote application should connect to.
VTK Remote application
Enable the CMake option
VTK_MODULE_ENABLE_VTK_RenderingOpenXRRemoting when building VTK, and set the
OpenXRRemoting_INCLUDE_DIR to provide the path to the OpenXR Remoting headers and binary directory.
Microsoft.Holographic.Remoting.OpenXr Nuget package that provides this dependency is available here on nuget.org.
⚠️ The version of the
Microsoft.Holographic.Remoting.OpenXr package must match the branch name of the player application above.
RenderingOpenXRRemoting module depends on the existing
RenderingOpenXR module which requires the OpenXR SDK. Download and build the latest version (1.0.25) of the OpenXR-SDK from here, in order to provide the VTK CMake configuration with the
When successfully built, use the OpenXR and OpenXRRemoting dedicated rendering stack in your application, as highlighted below.
The address of the player application to connect to must be set using
vtkOpenXRRemotingRenderWindow::SetRemotingIPAddress("XX.XX.XX.XX") before starting the interactor.
See the TestOpenXRRemotingInitialization test for a complete example.
⚠️ Make sure to provide the content of the OpenXR Remoting binary directory in the system PATH or next to the executable before running the program.
Check the Troubleshooting section in the README file of the OpenXR Remoting module if you encounter any issue while trying the new module.
This work is funded by the CALM-AA European project (cofunded by the European fund for regional development), and the United States National Institute of Standards and Technology (NIST) under Kitware Contract 1333ND21PNB770285 “Modifications and Enhancements to the ParaView /VTK Software for Immersive Scientific Visualization”