Rendered Realism at (Nearly) Real Time Rates

About Ray Tracing
Ray tracing is the standard approach to produce photo realistic images. The simplicity of the ray tracing algorithm makes it trivial to achieve realism. The algorithm casts a ray for each pixel and determines the color of the first hit object, then it identifies its adherence to geometric optics, and it takes into consideration the classical physics model of light transport (in which light rays bounce and bend at reflective and refractive objects). Shadows, reflections, translucency, depth of field, antialising and motion blur are all trivially computed in a ray tracer, by simply casting more rays.

The ray tracing algorithm is simple, but it is not particularly fast, especially when more and more rays are traced to obtain a desired level of realism. There are a large number of rays that need to be traced to begin with, and memory reference locality is worse when tracing rays than when doing traditional rasterization. Consider that a rasterized triangle is only touched once, while any ray might hit any or all triangles in the scene. Fortunately, space sorting acceleration structures and the algorithm's trivially parallel nature make the algorithm attractive. Especially when rendering large data sets on high performance computers.

For a long time is has been possible to achieve realism for scenes rendered in VTK, by resorting to non-interactive post processing. One did so by setting up a scene in a low quality interactive session and then attaching an exporter to the renderwindow (vtkRIBExporter, vtkPOVExporter) to convert the visible scene into a format that can be post processed by a high quality offline rendering engine. Thanks to recent work from LANL, it is now possible to have realism while you create and interact with the scene, by interfacing VTK directly with the Manta interactive ray tracing engine.

Figure 1: Details in a protein data bank dataset, demonstrating implicit sphere and cylinder glyphs, shadows and translucency.

About Manta
Over the past two decades, advances in both hardware performance and ray tracing implementations have made interactive ray tracing feasible. The Manta open source ray tracing engine [1] is one of the most advanced, and most flexible interactive ray tracers currently available.

Manta is a mulithreaded application and library, in which processors independently trace different sets of pixels simultaneously. Within each thread, packets of rays are traced together to improve memory locality, and within packets rays are traced simultaneously using SIMD instructions to make use of intraprocessor parallelism.

Manta is unique in that it is not only fast but also very flexible. The rendering engine can be asked to scale up in terms of image quality (at a cost of reduced interactivity), and it supports a large number of primitive types and rendering modalities (volume rendering, direct isosurface rendering, particle rendering).

About VtkManta
vtkManta is, as the name implies, a bridge between VTK and Manta. It acts as a substitute for the eight most important OpenGL based classes in VTK's rendering pipeline:

• vtkMantaCamera,
• vtkMantaLight,
• vtkMantaProperty,
• vtkMantaTexture,
• vtkMantaPolyDataMapper,
• vtkMantaActor,
• vtkMantaRenderer and,
• vtkMantaRenderWindow.

For example, vtkMantaPolyDataMapper is a new sibling to vtkOpenGLPolyDataMapper which makes calls like Manta::Mesh::addTriangle() instead of calls like gl_Begin(GL_TRIANGLE).

These new Manta-based rendering classes have been wrapped into a ParaView parallel "View" plug-in, which makes it easy to obtain ray traced image quality on very large visualization problems. The default settings in the plug-in are to render at Manta's lowest level of realism, which closely approximates standard OpenGL rendering and obtains the greatest frame rate. The advantage of the vtkManta bridge is that the settings can be changed at run time to increase realism, at a corresponding decrease in interactivity. Note that ray tracing scales linearly and that the number of rendering processors can be changed at runtime to use as many cores as you have on your platform for greatest interactivity.

How to Use in VTK
To use vtkManta in a VTK program, you must compile three projects, all of which are configured through CMake. These are VTK, Manta, and vtkManta. Compile VTK in the standard way, just be sure to choose shared instead of static libraries. Next obtain and compile Manta by following instructions on the build instructions page of the Manta webpage. It is particularly important to use the Release build type, as one can expect a roughly 10x slowdown when compiled under Debug build. It is also important to note that Manta currently does not build without modification on Microsoft Windows.

vtkManta is currently only available from within the ParaView source tree, but we expect to move it into VTK proper when it matures a bit more. Until then, obtain the ParaView source tree and then point CMake directly at the ParaView/Plugins/Manta/VTK directory when you make your initial build configuration. When you configure vtkManta, you must give it the locations of your VTK build, and of the Manta source and build directories. There are starter programs to learn from within the Examples and Testing subdirectories.

Note that the vtkManta classes are not currently interoperable with the standard OpenGL-based classes, thus they must not be used together. You can enforce the creation of only vtkManta classes by instantiating only concrete classes by name in your code (see example below), or you can try VTK's object factory override mechanism. Factory overriding tells VTK to instantiate a Manta-based concrete subclass whenever your program calls for the instantiation of an abstract parent class. Set this up by setting the VTK_AUTOLOAD_PATH environment variable to the directory where the vtkManta library is built before you run your program. The following is an excerpt from a Python example program that demonstrates direct instantiation of Manta classes by name for the rendering pipeline:

How to Use in Paraview
To use vtkManta in ParaView, simply build Manta as described above, and then turn on both shared libraries and the Manta plug-in when you build ParaView. You will then have to specify the location of the Manta source and build trees as described above.

Due to the interoperability restriction mentioned above, you must run ParaView in client-server mode to use Manta within ParaView. Multiple render views are not supported for the same reason. To do so, start the ParaView server and connect to it, then use the Tools -->Plugin manager to load the Manta plug-in on both the client and server processes. When this is done, close the default 3D view and create a Manta view in its place.

There are settings to control the Manta engine (such as the number of rendering threads, enabling/disabling shadows) on ParaView's options dialog. There are also settings to control each actor's material properties (metal, dielectric, refractive indices, reflectivity) on the bottom of the Display Tab of the object inspector. Note that the exotic material types are not compatible with color mapped rendering (because of the need for data fidelity) and thus, they have no effect unless the object is rendered in solid color mode.

Technical Details
There are several paradigm shifts between rasterization and ray tracing, and general peculiarities of interfacing VTK with a multithreaded API worth noting.

Consider for example VTK_VERTEX and VTK_LINE cell types. With rasterization these 1 and 2D primitives cover at least an entire pixel, and making them larger is a trivial screen space operation. In ray tracing, these infinitely thin objects are missed by nearly all of the infinitely thin rays that are sent through the scene. In vtkManta these cell types are converted to Manta::Sphere and Manta::Cylinder primitives and given a somewhat arbitrary radius/diameter in world space. Manta and OpenGL rendered scenes then may look subtly different. For example, Manta lines look smaller the farther they are from the view and look relatively smaller in relation to large objects. OpenGL rendered lines are a constant width regardless. An easy way to demonstrate is by zooming in onto ParaView's center of rotation widget.

Figure 3: Left - line segments in OpenGL are drawn in screen space units, usually 1 px wide. Right - line segments in Manta are drawn as cylinders with a radius defined in world space units.

Another difference is that with rasterization, scene setup is most often considered to be free while in ray tracing, scene setup is the slowest part of rendering because a ray tracer needs to sort objects in space in order to render them at a reasonable speed.

Scene setup is by far the slowest part of rendering (O(n) versus O(log N)). Because of this, the default expectation in the VTK rendering classes is to expect primitives not to be drawn, and in Manta the default is to expect them to be drawn (to better amortize setup cost). That is, in OpenGL one begins by clearing the screen and then redrawing every primitive, whereas in Manta one begins by clearing the screen and then primitives are assumed to keep the same status they had in the previous frame.

For example, in every frame one begins by re-renderering a light into the scene in OpenGL, but in Manta one assumes the light is still there. Another example is that the acceleration structure for a Manta mesh is only created once, which takes seconds for moderate-sized data, but then that acceleration structure is reused until something changes.

Fortunately, VTK maintains Modified times for all vtkObjects, which allows vtkManta to determine exactly when an object needs to be reprocessed. Once an acceleration structure is created, rendering an object in Manta that is an order of magnitude more complex is only linearly slower, while doing the same with rasterization makes the rendering an order of magnitude slower.

The difference in controlling the rendering engine is exacerbated by the fact that Manta is a multithreaded API, which has to coexist in a process with a VTK thread (which is notoriously non-thread safe except in special circumstances). To control Manta, special care had to be taken then to ensure that neither the VTK thread nor the Manta threads interfere with the other at an inopportune moment.

For example the VTK thread cannot ask its vtkMantaActor to modify or delete its own Manta::Mesh structure when it needs to be changed unless it is sure that the Manta threads are not simultaneously accessing it. To ensure sufficient synchronization between the two sets of threads, we have adopted a two part approach.

Figure 4: Intra thread control. The Manta Manager class ensures that the Manta Engine persists until it is no longer needed. Helper structures, such as "actor 1 helper" exist to pass control of allocated structures from the VTK thread to the manta threads so that they may be safely destructed.

First, we use a reference counted VTK class that acts as an interface to the Manta engine as a whole. This class is called vtkMantaManager. It is a singleton that is instantiated by the Renderer, when it first calls into the Manta API. All vtkManta classes then register to the singleton, and unregister themselves on their own deletion. Reference counting then is what ensures that the Manager, and thus the engine that it acts as a proxy for, outlives all of the vtkManta instances that might need to make calls into the Manta API. The vtkMantaManager shuts down and destroys the Manta engine threads in its own destructor, because at that point it is guaranteed that no other classes will need to call into it.

Second, calls into the API must take effect only at safe points in time. Manta provides a callback mechanism for this purpose, which is a means of asking the Manta threads to execute arbitrary functions at some safe point of time in the future. The default safe point of time is during Manta's pre-frame setup phase, when none of its threads are actively accessing the scene data structures.

It is also important that the Manta thread does not interfere with the VTK thread in those callbacks. This would happen for example when the Manta callback calls delete on some object held by a VTK class, or when the callback directly calls some method in the VTK class. To solve this problem we use temporary helper classes. They exist primarily to take over ownership of pointers, after which the VTK objects can reassign their internal pointers. The helper classes are created by the VTK thread, and handed over to the Manta threads and exist only until the callback function processes their contents, which generally means freeing their pointed to objects.

Figure 5: Sort first within sort last parallel rendering in parallel vtkManta

Lastly, the particular class of parallel rendering exhibited by vtkManta within ParaView is worth describing, because it negates some of the realism benefits of ray tracing. Ray tracing is a trivially parallel algorithm only when the entire scene is globally addressable. That is, if the scene is small, one can give all processors a copy of it (or access it in fast shared memory) and then divide the screen amongst themselves. This is known as sort first or image space parallelism. ParaView's rendering is at the opposite end of the parallel rendering spectrum. ParaView makes use of sort last, or data space parallelism because ParaView's primary design goal is scaling in terms of data size. Thus, ParaView divides up space instead of the screen, and uses depth compositing to produce a final image.

vtkManta then operates in a hybrid mode, best described as sort first within sort last parallel rendering. Here data is first broken up amongst the nodes of a distributed memory cluster and then on each node the local data is rendered by many Manta threads in an image parallel fashion over the entire image. The incomplete images are depth composited together. Because data is broken up and not globally addressible, only primary rays are valid. Bounce rays miss all objects that reside on other nodes (i.e. (N-1)/N of the objects are missed), thus shadows, reflection, and translucency will be incorrect in distributed memory parallel configurations.

We should note however that standard single bounce rendering as exemplified by OpenGL in ParaView is quite sufficient for many visualization tasks, especially on very large datasets. vtkManta is at least this realistic, even without secondary rays. vtkManta has the added benefit of being a pure software solution, which means that (potentially very large) data does not have to be moved off the CPU to be rendered, which can be the greatest bottleneck to large data visualization.

Results
We are in the process of benchmarking and optimizing vtkManta and results are not yet complete. However, as a point of reference, on an 8 core (2.8Ghz Xeon) machine with 10GB of RAM and an NVidia GeForce 8800 GT with 512MB RAM and running the following benchmark which compares Manta with all 8 cores against OpenGL (with 1 core and the GPU), both with and without the use of display lists on the same scene. The results given are the first frame setup time and the per frame render time both in seconds. In practice, during pipeline parameter changes frame rates are (1 / setup time), while during camera manipulations frame rates are(1 / render time).

From these preliminary results we see that Manta can be quite interactive, and that at high triangle counts, VTK's use of Manta begins to become competitive with VTK's use of the GPU in terms of both setup and render times.

Table 1: Comparison of first frame preprocess time and frame delay between Manta, and NVidia GPU (both with and without display lists). The command line to run these tests was “MantaBenchmark -screensize 500 -noChanges -fuzziness 0.0” appended by the arguments at the top and left of the table.

Future Work
vtkManta is very new and we are actively working on fixing a handful of known bugs, adding features that are missing in the standard OpenGL-based VTK classes, and improving general robustness. Even in its current state it is obvious that the inclusion of realistic rendering in VTK and ParaView shines a light (pun entirely intended) on some interesting new possibilities.

One example is that light placement in a scene with shadows is very important aesthetically, and the number of lights greatly affects the rendering performance. It is expected that the existing lighting dialog in ParaView will need to be revamped to let the user more finely control and inspect the light setup.

There are many capabilities of Manta that we are not yet exercising because we have limited the work so far to rendering meshes. Manta is perhaps most interesting when we use it to make transformational changes, such as comparing glyphing against direct rendering of sphere primitives, as shown in Figure 1. Manta fully supports instancing, which makes glyphing with arbitrary geometry particularly efficient. The benefit in this case is similar to what one can expect for point sprite mapping in GPU hardware. Another example is in direct rendering of isocontours, which may be faster because there is no need to run an isocontour algorithm or produce intermediate geometry. If we write new actors and mappers to supplement the existing vtkMantaPolyDataMapper, we can also put some of Manta's high quality and explicitly parallel volume rendering, direct isosurfacing, and implicit surface rendering capabilities to use.

There are also architectural and technical improvements that we plan to make. The most pressing of those is a restructuring of vtkManta so that the interface to Manta is entirely constrained within the Renderer class. Once that is done Manta and OpenGL can coexist more naturally and one will be able to have GL and Manta windows side by side and even intermixed.

Using a ray tracer in a scientific visualization application exercises the ray tracer in a manner that is little studied today; ray tracing has primarily been studied in static settings because of its long setup time. In scientific visualization, the default state is dynamic change, whether due to the need to inspect time varying data or the frequent pipeline parameter modification that occurs in the process of data discovery. We anticipate investigating and making improvements in this area as we continue to make vtkManta more and more useful to the scientific community.

Acknowledgements
This work has been funded by Los Alamos National Laboratories through subcontract number 76433-001-09.

References
[1] http://mantawiki.sci.utah.edu/manta/index.php/Main_Page

  Li-Ta Lo is a Technical Staff Member in Los Alamos National Laboratory. His first encounter with ray-tracing was Standford RayShade on HP-9000 workstations in the early 90'. It took hours to render a simple demo scene and he was blamed for having fun wasting precious computing resources. After almost 20 years, he is happy to be able to help bringing real-time ray-tracing to real world application doing real jobs.

  David DeMarle is a member of the R&D team at Kitware where he contributes to both ParaView and VTK. He frequently teaches Kitware’s professional development and training courses for these product applications. Dave's research interests are in systems level aspects of visualization, in particular memory optimizations for parallel visualization of large data sets.