Augmented reality (AR) and virtual reality (VR) technologies, broadly categorized as extended reality (XR), are actively being explored for a wide range of medical applications. The initial applications for XR in healthcare have focused on wellness, education, and pain management. However, advancements in display and sensor technology have expanded potential XR applications into medical imaging, surgical planning, and image-guided interventions. While XR technology used for medical imaging is still in the early stages, these technologies have the potential to significantly impact clinical decision making and clinical workflow. Medical devices leveraging XR hardware have been demonstrated for medical specialties including radiology, orthopedic, neurology, cardiology, ophthalmology, therapeutics, behavioral health, and pain management. Given the potential cross-cutting impact of these technologies, research focused on evaluation methods, clinical studies, and clinical implementation is critical. The collection of papers in this JMI special section provides insights into these different applications, arising challenges, and potential improvements to the standard of care through the implementation of XR technologies.

The first topic addressed by this collection of papers was training and surgical planning. In the review paper by Queisner and Eisentrager (doi 10.1117/1.JMI.11.6.062603), the authors conducted a systematic review of recent literature (spanning April 1, 2021, to May 10, 2023) on the use of head-mounted VR devices for surgical planning. A notable aspect of this review is its exclusive focus on studies utilizing patient-specific images. The studies were categorized based on their nature (retrospective vs. prospective, case studies vs. comparative studies) and analyzed from several technical perspectives, including VR devices, software, imaging modalities, and segmentation approaches. Despite significant variability in study designs, sample sizes, surgical procedures, technical setups, and reporting details, the review identified a promising emerging trend: surgical decision-making in VR often surpasses the quality achieved through standard methods of reviewing planar images.

The review paper by Lauinger et al. examines the benefits and challenges of XR in medical applications like training and surgery (doi 10.1117/1.JMI.11.6.062608). XR enables immersive visualization of 3D patient anatomy, aiding surgeons in understanding complex pathologies. The authors highlight the importance of segmentation, registration, and 3D modeling algorithms for XR integration. Barriers to adoption include high costs, the need for technological improvements in accuracy and realism, and regulatory challenges for device validation. However, the authors express optimism that with ongoing research and collaboration between academia and industry, XR has the potential to overcome these hurdles and improve patient outcomes.

In a third review paper related to pre-operative planning, Kantor et al. present a high-level overview of the use of immersive technologies in integrating medical images into surgeries and procedures (doi 10.1117/1.JMI.11.6.062607). The team employed AI tools to screen relevant literature, resulting in a final review that synthesizes findings from 59 articles on the topic. The paper discusses both pre-operative planning and intra-operative augmentation applications, highlighting the advantages of using XR, which range from improved anatomical understanding and outcomes to enabling tele-mentoring and remote collaboration. It provides an analysis of XR’s potential, while also addressing the challenges hindering its broader clinical adoption and proposing potential solutions.

Jacquemyn, Bamps, et al. present original research on a multi-user and multi-device application for collaborative planning of cardiovascular procedures (doi 10.1117/1.JMI.11.6.062606; on the issue cover). The study provides both quantitative and qualitative assessment of the application through measurement tasks in the application using different AR and VR head mounted displays and results were compared to the ground-truth CT images. The study also provides methods for validating planning applications across AR/VR hardware.

Another aspect of this collection of papers focuses on evaluation of image-guided procedures. The first work, by Zhang et al., presents original research focusing on a HoloLens 2-based AR visualization approach for ultrasound-guided procedures (doi 10.1117/1.JMI.11.6.062604). The solution enables operators to view live ultrasound images directly within their line of sight, eliminating the need to look away, as required in traditional methods. A purpose-built device is described that streams images from any vendor’s ultrasound system to the HoloLens. Additionally, the HoloLens app supports gesture and voice controls, allowing users to position, resize, and interact with the ultrasound images seamlessly. A pilot study involving vascular access procedures in 30 pediatric patients (15 control, 15 interventional) demonstrated the clinical feasibility of the approach. The results highlight the potential to reduce procedure completion times and minimize head adjustments through AR visualization.

Related original research presented by Gadodia et al. conducted an accuracy study of navigational system for AR needle-guidance in biopsies and ablations (doi 10.1117/1.JMI.11.6.062602). The study used a cadaveric model with implanted targets. The error was evaluated using the targeting registration error (TRE) defined as the Euclidean distance between the placed tip and the target measured by ultrasound. Image fusion registration error (IFRE) is the registration error between the ultrasound and CT images. The study gives the TRE and IFRE from a user study showing targeting and registration errors for multimodality AR image guidance and the cadaveric study shows promise for further studies.

The final topic addressed focused on perceptual evaluation of contrast for VR, which is important for diagnostic and surgery planning applications. Bhansali et al. developed a test platform to investigate monocular and binocular contrast perception through a user perceptual experiment (doi 10.1117/1.JMI.11.6.062605). Users with different interpupillary distances (IPDs) compared to the IPD setting on the HMD. The user tests were performed with targets in different locations in the field of view of the HMD. The test can be used to evaluate monocular and binocular image quality on virtual reality head mounted displays for medical applications, which is important for evaluating HMDs for surgery planning and diagnostic tasks.

We encourage the readers to peruse the review papers and original research in this JMI special section to understand the current landscape, future trends, and unresolved challenges for using XR technologies in medical imaging applications.