Medical imaging creates images of the human body, aiding in the diagnosis, monitoring, and treatment of medical conditions by visualizing internal structures non-invasively. Technologies such as X-rays, MRI, CT scans, nuclear bone scans, and DEXA (Dual-Energy X-ray Absorptiometry) are widely used to visualize both soft and hard tissues. Each modality comes with its own advantages and disadvantages. X-ray imaging, for example, is fast, cost-effective, and involves relatively low radiation exposure. But X-rays produce only 2D projection images with limited visualization of overlapping tissues. In contrast, CT scans provide highly detailed 3D visualizations of anatomical structures, enabling evaluation of complex fractures, anatomy, and lesions. However, this comes at the cost of higher radiation exposure and greater expense. Selecting the most appropriate modality, therefore, requires balancing diagnostic accuracy against safety and resource considerations.

To overcome these trade-offs, advances in shape analysis, mathematical modeling, and computational reconstruction have enabled the derivation of 3D information from inherently 2D imaging techniques. This innovation not only reduces reliance on CT by enabling 3D reconstructions from multiple X-rays but also extends 3D visualization to modalities that are fundamentally 2D, such as ultrasound, angiography, and endoscopic videos. By doing so, clinicians gain 3D anatomical insight from standard imaging workflows, thereby lowering the radiation burden, reducing costs, and expanding the diagnostic and interventional capabilities of existing modalities. This opens new opportunities for precision care in orthopedics, cardiology, neurosurgery, dentistry, gastroenterology, and emergency medicine.

• Orthopedics: Multiple X-rays can be reconstructed into 3D models of skeletal structures, aiding in the evaluation of fractures, deformities, and surgical planning with lower radiation and cost than CT.
• Cardiovascular Medicine: Echocardiography and angiography, traditionally performed in 2D, can be reconstructed into 3D models of the heart and vessels, enhancing visualization for valve repair, stent placement, and aneurysm treatment.
• Neurosurgery: X-ray-based reconstructions enable 3D visualization of brain tumors, spinal anatomy, and neural structures, supporting safer and more precise surgical planning.
• Dental and Maxillofacial Surgery: Panoramic and cephalometric X-rays can be transformed into 3D models of the jaw and teeth, aiding in orthodontic planning and dental implant placement while reducing the need for cone-beam CT scans.
• Gastrointestinal and Urologic Procedures: Endoscopic video frames can be combined into 3D models of the colon, bladder, or prostate, supporting lesion detection and minimally invasive interventions.
• Trauma and Emergency Medicine: In urgent or resource-limited settings, portable X-rays combined with reconstruction algorithms provide 3D anatomical insights when CT is unavailable, supporting critical decision-making.

Why 2D Imaging Alone Isn’t Always Enough

While 2D images are often sufficient for medical professionals to make some diagnostic decisions or plan certain surgical interventions, 2D-to-3D reconstruction software can enhance this process in two fundamental ways:

1. Automating parts of the measurement and planning process
2. Improve measurements or planning

2D-to-3D reconstruction offers benefits by converting standard medical imaging into accurate three-dimensional representations of patient anatomy. This technology enables surgeons to perform more reliable and repeatable measurements, which is critical for planning precise incisions, implant placements, or anatomical corrections. Reducing subjectivity in assessments supports more objective decision-making and minimizes variability between practitioners. The enhanced precision of 3D models provides a clearer understanding of complex structures, improving spatial awareness during surgery. The intuitive, user-friendly tools make preoperative planning and intraoperative navigation more efficient and accurate.

However, 2D-to-3D reconstruction is not a trivial task. Two leading approaches are: deep learning (DL) and statistical shape modeling (SSM).

Deep Learning: A Powerful but Data-Hungry Approach

Deep learning is one approach for 2D-to-3D reconstruction. Neural networks can model the complex, nonlinear relationships between 2D projections and 3D structures, making them powerful tools in this domain. However, their effectiveness depends heavily on access to large, well-annotated datasets that capture the variability of real-world anatomy and imaging conditions [1–9].

In domains like autonomous driving, obtaining such datasets is feasible—cars equipped with cameras can capture massive amounts of labeled training data. However, in the medical domain, annotated datasets are scarce. Generating labeled 3D ground truth from 2D images often requires expert radiologists and specialized software applications, making data collection both expensive and time-consuming. Sometimes, this is unavoidable. A common strategy is to use CT scans as the reference 3D ground truth and generate corresponding 2D projections. Tools such as Kitware’s X-ray Genius provide a practical solution by automating this process and enabling the creation of synthetic 2D/3D training pairs (Figures 1 and 2).

Figure 1: WebUI of XRay-Genius allows users to specify projection parameters with immediate preview.

Figure 2: Higher fidelity X-Ray(s) can be downloaded soon after starting the simulation.

Statistical Shape Modeling: A Robust Alternative

When large datasets are unavailable, statistical shape modeling (SSM) provides a compelling alternative. Unlike deep learning’s “black box” approach, SSM is a white-box method that allows for greater interpretability. SSMs are mathematical representations that capture the geometric properties of a class of shapes, allowing for the analysis of shape variation within that cohort as well as the creation of new cases within the typical variations of the model. SSM can leverage any number of publicly available 3D anatomical datasets to construct statistical models, which can then be used to infer 3D structures from 2D images. A larger number of training datasets will lead to models that are more generalizable, but SSMs can be built with any number of datasets. Public datasets can be combined with private customer dataset(s).

Figure 3: Key advantages of SSMs.

Key advantages of SSM include:

• Less data dependency: Instead of requiring vast labeled datasets, SSM relies on a large number of existing 3D anatomical models.
• Transparent optimization: Unlike deep learning’s randomized initialization and lengthy training process, SSM follows a well-defined optimization approach. While the optimization method and the exact metric being minimized will vary depending on the application, they are always well-known.
• Robustness in medical applications: SSMs are generally more predictable in the outputs they produce, as the underlying statistical distribution of anatomical shapes constrains their variation. In contrast, neural networks can occasionally generate highly unrealistic or unstable outputs, particularly when faced with data outside their training distribution. Moreover, errors in SSM-based reconstructions are typically easier to recognize and flag. In contrast, failures in neural networks can be subtle, difficult to detect, and in some cases nearly impossible to identify without ground truth.

Real-World Applications

The primary objective of 2D-to-3D reconstruction is to extract clinically relevant 3D information from 2D imaging data. While generating an explicit 3D model is often a helpful intermediate step, it is not always required. In practice, the technology can support a range of applications, including:

• Reconstruction of 3D anatomical structures: By combining two or more orthogonal 2D projections, it is possible to approximate the 3D geometry of anatomical features, providing richer spatial context than individual images alone.
• Accurate quantitative assessment of anatomical geometry: For anatomical structures that are not optimally aligned with the available 2D views, reconstruction methods enable a more precise assessment of lengths, angles, and volumes, thereby reducing errors that arise from projection-based distortion.

Imaging Software & Shape Analysis at Kitware 

Kitware’s team brings deep expertise in SSMs. These models are crucial for 3D reconstruction, implant design, biomechanics, and generating patient-specific anatomical digital twins. Kitware also develops and contributes to open-source toolkits, such as ShapeWorks and SlicerSALT, enabling clinicians and researchers to build, visualize, and analyze high-precision 3D models from imaging data. These contributions support both static and dynamic (e.g., temporal, cycle-based) anatomical modeling, enriching applications across orthopedics, cardiology, neurosurgery, and beyond.

In summary, 2D-to-3D reconstruction offers a way to bridge the gap between accessible, low-cost 2D imaging and the diagnostic richness of 3D visualization. Orthopedics is a leading application, but the technology is equally valuable in cardiology, neurosurgery, oncology, dentistry, ophthalmology, trauma care, and beyond. By reducing radiation exposure, lowering costs, and improving accessibility, this approach has the potential to enhance imaging, surgical planning, and patient care across multiple disciplines.

How Kitware Can Help

If your team is exploring how to integrate 2D-to-3D reconstruction into device design or a clinical software application, Kitware can help. Our expertise spans the spectrum from deep learning-driven pipelines to statistical shape modeling frameworks, ensuring that whether you have access to large datasets or only limited imaging, there is a clear and effective pathway forward. Contact our team to discover how we can assist you in achieving your goals.

References

1. Ha HG, Lee J, Jung GH, Hong J, Lee H. 2D-3D Reconstruction of a Femur by Single X-Ray Image Based on Deep Transfer Learning Network. IRBM [Internet]. 2024 Feb 1 [cited 2025 Sept 9];45(1):100822. Available from: https://www.sciencedirect.com/science/article/pii/S1959031824000034 
2. Minhas S, Wu TH, Kim DG, Chen S, Wu YC, Ko CC. Artificial Intelligence for 3D Reconstruction from 2D Panoramic X-rays to Assess Maxillary Impacted Canines. Diagnostics [Internet]. 2024 Jan [cited 2025 Sept 9];14(2):196. Available from: https://www.mdpi.com/2075-4418/14/2/196
3. Maas KWH, Pezzotti N, Vermeer AJE, Ruijters D, Vilanova A. NeRF for 3D Reconstruction from X-ray Angiography: 2023 Eurographics Workshop on Visual Computing for Biology and Medicine, VCBM 2023. Hansen C, Procter J, Raidou RG, Jönsson D, editors. EG VCBM 2023 – Eurographics Workshop on Visual Computing for Biology and Medicine, Full and Short Paper Proceedings [Internet]. 2023 Sept 19 [cited 2025 Sept 9];29–40. Available from: http://www.scopus.com/inward/record.url?scp=85178290623&partnerID=8YFLogxK
4. Maken P, Gupta A. 2D-to-3D: A Review for Computational 3D Image Reconstruction from X-ray Images. Arch Computat Methods Eng [Internet]. 2023 Jan 1 [cited 2025 Sept 9];30(1):85–114. Available from: https://doi.org/10.1007/s11831-022-09790-z
5. Yang CJ, Lin CL, Wang CK, Wang JY, Chen CC, Su FC, Lee YJ, Lui CC, Yeh LR, Fang YHD. Generative Adversarial Network (GAN) for Automatic Reconstruction of the 3D Spine Structure by Using Simulated Bi-Planar X-ray Images. Diagnostics [Internet]. 2022 May [cited 2025 Sept 9];12(5):1121. Available from: https://www.mdpi.com/2075-4418/12/5/1121
6. Yang CJ, Lin CL, Wang CK, Wang JY, Chen CC, Su FC, Lee YJ, Lui CC, Yeh LR, Fang YHD. Generative Adversarial Network (GAN) for Automatic Reconstruction of the 3D Spine Structure by Using Simulated Bi-Planar X-ray Images. Diagnostics [Internet]. 2022 May [cited 2025 Sept 9];12(5):1121. Available from: https://www.mdpi.com/2075-4418/12/5/1121
7. Ge R, He Y, Xia C, Xu C, Sun W, Yang G, Li J, Wang Z, Yu H, Zhang D, Chen Y, Luo L, Li S, Zhu Y. X-CTRSNet: 3D cervical vertebra CT reconstruction and segmentation directly from 2D X-ray images. Knowledge-Based Systems [Internet]. 2022 Jan 25 [cited 2025 Sept 9];236:107680. Available from: https://www.sciencedirect.com/science/article/pii/S0950705121009424
8. Shiode R, Kabashima M, Hiasa Y, Oka K, Murase T, Sato Y, Otake Y. 2D–3D reconstruction of distal forearm bone from actual X-ray images of the wrist using convolutional neural networks. Sci Rep [Internet]. 2021 July 27 [cited 2025 Sept 9];11(1):15249. Available from: https://www.nature.com/articles/s41598-021-94634-2
9. Kasten Y, Doktofsky D, Kovler I. End-To-End Convolutional Neural Network for 3D Reconstruction of Knee Bones from Bi-planar X-Ray Images. In: Deeba F, Johnson P, Würfl T, Ye JC, editors. Machine Learning for Medical Image Reconstruction. Cham: Springer International Publishing; 2020. p. 123–33.