Verification and Validation of iMSTK
iMSTK Overview
The Interactive Medical Simulation Toolkit (iMSTK) is an open-source toolkit that allows fast prototyping of surgical simulators and skill trainers. iMSTK features advanced high-performance libraries for physics simulation, haptics, advanced rendering/visualization, user hardware interfacing, geometric processing, collision detection, contact modeling, and numerical solvers. These features allow for medically relevant simulation tasks, such as tissue tearing and burning, needle-tissue interaction, bone cutting, and tool-to-tool interactions. We have recently added a V&V framework to iMSTK to measure the accuracy and repeatability of our physics engine. This addition also greatly improves integrated testing support for existing and upcoming interactions supported by iMSTK.
Simulations of medically relevant tasks completed in iMSTK. Clockwise from the largest image: tissue tearing, needle-tissue interactions, tissue burning, tool-to-tool interactions, and bone cutting.
iMSTK Verification and Validation Framework Overview
Verification and validation are critical to maintaining the quality of modeling and simulation software. iMSTK provides a set of unit tests to ensure the stability of its fundamental algorithms but has lacked the ability to ensure integrated functionality is accurate and maintained as new features are implemented. By creating an automated framework for assessing integrated functionality, we are now able to verify functionality is maintained during development and validate our solver accuracy against well known scenarios. This new V&V framework provides a way to create and script object movements within a simple scene. A programmable controller was created to translate scripted movements to the movements of one or more iMSTK scene objects. Our framework also writes object state data and solver metrics for every simulated time step to a Comma Separated Value (CSV) file as well as a ParaView file for scene visualization. As iMSTK is discrete, repeated execution of these scripts results in the same result each execution. A comparison algorithm can then compare two CSV files and report any differences between two given scene data sets. If the results vary by more than 2%, the test scene is reported to have failed.
Top: VTK visualizing a deformable test scene as it runs. Bottom: ParaView visualizing the resulting data file generated by our V&V Framework.
A set of CSV files generated by all our test scenes is reserved as a baseline result. As development proceeds, we are able to generate and compare CSV files generated by the latest commit to ensure these updates do not break any of the current features within iMSTK. All test scene failures or deviations are reviewed by the iMSTK team to determine if the changes are acceptable or not. If these deviations are acceptable, we replace the current baseline CSV with this new CSV, resulting in a new baseline file to be used for future development.
Our V&V framework also contains several analytic implementations of various standard test cases (ex. Bending beam, unit cube strain, spring mass system) used across various industries. These implementations also output CSV files containing the same data as in our verification tests. Using our V&V framework, we are able to create similar scenes in iMSTK and compare iMSTK results against these analytic results. We have updated iMSTK to generate data to ensure that our results are consistent and within an acceptable error range when comparing these results.
iMSTK Validation Result
Currently, iMSTK has three scenes to verify the accuracy of the solver. The first is a bending beam scene where the nodal positions returned from the solver are compared to the analytic solution using Bernoilli-Euler beam theory. We ran this test while varying the interaction count used for the xPBD solver and the resolution of the mesh, and we found that even for the coarse mesh and low iteration counts our computed displacements were within 4% of the analytic solution.
The bending beam scene simulated a long thin beam undergoing a prescribed displacement. The positions along the central axis of the beam are compared to the analytic solution. The RMS error in position shows the accuracy of our solver. We are reasonably accurate even with low interaction counts and coarse meshes.
The second scene compares the total strain energy of the system for a unit cube undergoing 20% tension, compression, and shear. We ran this scene at multiple mesh resolutions and iteration counts using the strain energy (FEM like) constraints.
Testing the behavior of iMSTK during the simulation of a sample undergoing tension, compression, and shear.
The last scene tests the distance constraints and compares them to the analytic solution of a spring mass system. The simulation was set up with two point masses with a one meter spring between them. The top point mass is fixed, and the bottom point is pulled to 20% strain, then released. The force and position match well with the analytic solution, except for a constant error in position induced due to the inherent numerical dissipation in the xPBD solve.
Distance constraints in an xPBD solver behave as a spring between two masses. The RMS error for force and position comparing the analytic solution of position and spring force evolution in a mass spring system. The force calculation comes directly from the constraint value from the xPBD solver. *Differences less than 1E-8 are ignored.
In the coming months, we are working to improve and extend iMSTK with:
- Improved suturing with haptic feedback
- Updates to the iMSTK Unity Asset
For more information on our efforts and our users, visit our website or read our other blog posts. If you would like to feature your iMSTK use case, please email us at kitware@kitware.com.
Acknowledgments
The research reported in this publication was supported, in part, by the following awards: NIH Grant numbers 1R01EB025247 and R01EB031808.