Using VTK with the Meta Quest for Immersive 3D Visualization

Since several minor releases, VTK has introduced support for virtual and augmented reality (XR) devices. This allows users to interact with data in more dynamic and intuitive ways through immersive visualizations and interactive 3D experiences. One key advantage of VTK’s support for XR is its compatibility with many devices, allowing seamless integration in various XR platforms.

The Meta Quest head mounted display (HMD) is widely adopted across other emerging XR devices due to its perfect balance of affordability and high-quality performance. This blog post outlines a method for integrating VTK with this device, enabling you to harness its full power.

Getting Started

To use VTK with XR devices, enable the CMake option VTK_MODULE_ENABLE_VTK_RenderingOpenXR when building VTK, and set the OpenXR_DIR variable to provide the path to the OpenXR SDK install directory.

Then use the dedicated OpenXR rendering stack in your VTK application:

  vtkNew<vtkOpenXRRenderer> renderer;
  vtkNew<vtkOpenXRRemotingRenderWindow> renderWindow;
  vtkNew<vtkOpenXRRenderWindowInteractor> interactor;
  renderWindow->AddRenderer(renderer);
  interactor->SetRenderWindow(renderWindow);

  // Enable interactions
  // Set the path to the directory containing vtk_openxr_actions.json
  interactor->SetActionManifestDirectory("XXX")

  // Add controller models
  // Set the path to the directory containing openxr_controllermodels.json
  interactor->SetModelsManifestDirectory("XXX")

Remote connection

Rendering in the Meta Quest requires using the OpenGLES or Vulkan rendering backend which is not yet supported by the VTK-OpenXR interface.

To address this limitation, a solution is to connect your Meta Quest to your PC using AirLink, which allows you to wirelessly access the VTK-OpenXR app running on the desktop. With this approach, not only VTK can use the OpenGL context provided by the workstation, but it also leverages its hardware capabilities to create performant advanced visualizations.

Make sure your device and your workstation are connected to the same network and follow the instructions to enable AirLink before enjoying content rendered with VTK in the Meta Quest.

Some users have also reported that the same approach was possible using SteamLink or VDXR.

As an alternative, it is possible to connect your Meta Quest to the PC using the Link cable.

Interactions

New JSON interaction profiles have been contributed to VTK to control the scene using the Meta Quest controllers.

While the controller models are not provided by the OpenXR runtime, it is possible to provide your own models by following the instructions in the Controller Model Rendering section of the OpenXR README file.

In the above demo, we used the Meta Quest 2 models provided by SteamVR and converted them to GLTF using Paraview. The content of the openxr_controllermodels.json file was modified as follows:

  {
    "interaction_profile": "/interaction_profiles/oculus/touch_controller",
    "asset_paths": {
      "left_controller": "quest2/left/scene.gltf",
      "right_controller": "quest2/right/scene.gltf"
    }
  }

Future work

Pass-through is a feature on Meta Quest headsets. It lets you see a real-time view of your surroundings. We experimented with the XR_FB_passthrough OpenXR extension to enable this feature. It uses a real-time view of the physical world as the background for the VTK scene. This turns the Meta Quest HMD into an augmented reality (AR) device. Reach out to us if you would like to help contributing this functionality to VTK.

Troubleshooting

If the VTK-OpenXR app hangs forever on start-up, make sure the Meta Quest Link is set as the active OpenXR runtime in the Oculus app settings :

Partnering with Kitware

Kitware is committed to driving innovation in scientific computing and visualization technologies. Our team has the expertise to support your exploration of AR and VR technologies, helping you seamlessly integrate these XR experiences into your projects. Whether you’re a research institution, university, or commercial enterprise, collaborating with Kitware will enable you to maximize the potential of this technology. Contact us for more information.