2.1.

Case Identification

When selecting a patient with the potential for success in VR-based pre-surgical planning, one must consider the complexity of the 3D anatomic/pathologic representation. In our experience, complex congenital heart disease and surgical solid tumor cases represent a straightforward initial focus for this technology. Converting these anatomies into 3D models achieves two goals. First, 3D modeling improves the mental representation of the surgical anatomy as it will be encountered in the surgical field.²⁵ Surgeons utilizing VR for surgical planning have independently stated: “I felt like I had been there before.” Long gone are the days of “I’ll figure it out when I get in there.”²⁶ This improved mental representation leads to the next benefit of expertise-dependent situational awareness. Rather than utilizing working memory to marry the surgical field to the surgery conference presentation of 2D multi-modal imaging, the operator more efficiently orients the surgical field to the memorable landmarks of the 3D mental representation provided by the VR pre-surgical interaction.²⁷ This long-term memory of the 3D anatomy effectively replaces what historically required years of operator experience to develop. Practically speaking, in hypertrophic obstructive cardiomyopathy, successful myectomy (surgically removing “just the right amount” of left ventricular outflow tract myocardial tissue) has been linked to years of expertise.²⁸ For example, an expert will study medical imaging similar to a novice, but when an expert opens the surgical field and encounters the anatomy in real-life 3D view, they can tap into years of long-term memories of similar cases to exercise their expertise in optimal execution of a procedure on this specific patient. The extraction and visualization of the virtual 3D models can add some of this expertise to an operator’s view, derived through years of exposure to various cases by creating operator experts on individual patients.²⁹ Due to the 3D model, the operator now becomes more of an “expert” on the specific patient in front of them. In addition to the improvement in 3D mental representation, VR affords the opportunity to selectively visualize merged 3D imaging data, toggling on and off layers such as tractography of the brain or functional data. It also enables the presentation of non-traditional anatomic information in new visual formats. For example, converting stereo electro-encephalographic data into 4D digital animated models allow for the operator to “see” a seizure propagate through the brain.^30,31 It is not possible to achieve time-sequential 3D representations of anatomy and pathology with traditional approaches of 3D printing, enabling extended applications to unlock dynamic physiological visualizations, in our opinion.

3.1.

3D Imaging Acquisition and Segmentation

With XR applications driving the need for extracting high-quality 3D virtual models of anatomy from imaging data, traditional radiology imaging protocols, which are focused on 2D, high-resolution in-plane views but thick slices, are challenged to adapt. The creation of patient-specific 3D virtual anatomy models involves segmenting 3D blocks of grayscale digital imaging and communications in medicine (DICOM) data into clinically relevant, tissue-specific, 3D digital segments of anatomy. In addition, once the focus of diagnostic imaging is transitioned from 2D representations to the extraction of 3D models, the extracted models themselves provide new opportunities in diagnostics and analytics.³² When one considers the diagnostic reasoning of the radiologist, they utilize their understanding of human anatomy as it is represented (in multi-modal formats) on a 2D screen and look for deviations from the norm. This can only occur when there is a variation in signal intensity between two tissues. This contrast allows for the segmentation and mental representation of anatomy/pathology in the mind of the reading radiologist. When imaging fails to represent, in a visual format, the variation between tissues; imaging is rendered unsuitable. This impractical nature of medical imaging can be understood in the analysis of seizure foci or cardiac arrhythmia foci. In both instances, cellular dysfunction is the source of pathology and is not captured in current imaging techniques. However, when a trained diagnostic imaging expert can detect subtle differences in signal intensity on a medical image, they are able to “see” and interpret pathology that is not perceived by the general practitioner.³³ This characteristic of expertise in medical imaging formulates the foundation of one of the benefits of 3D modeling segmentation. As the radiology expert identifies segments, slice by slice, of “like” tissue and builds 3D models of the anatomy and pathology of like tissues, they are converting their subjective knowledge into objective 3D models. This commitment to an objectively defined 3D model represents a new ground truth by which all members of the treating team can evaluate imaging pathology. This objective rendering of anatomy and pathology from 3D DICOM grayscale blocks of data into 3D segmental puzzle pieces of patient-specific anatomy represents a significant benefit of 3D; however, extracting accurate 3D structures from the data also represents the main barrier to scaled deployment due to the need for isotropic imaging protocols and trained raters or algorithms to perform accurate segmentations. If the clinically relevant components can be seen on medical images, then they can be segmented into 3D models provided the source data was acquired in a 3D near-isotropic format.

This tissue-specific segmentation is critical for 3D modeling in medicine. As radiologist translates 2D images to 3D mental anatomical models, they do so in a manner that reproduces their analysis of each 2D image in their mind’s eye. As the radiologist reviews an axial computed tomography (CT) slice through the chest, the radiologist can segment (in their mind’s eye) the various tissues of the lung parenchyma, airways, bone, myocardial tissue, contrast-filled vessels, bone, muscle, fat, and skin. The radiologist can utilize contrast and brightness of the image to help distinguish between various tissues. However in VR, we are not aiming for new methods of visualizing data in a 2D cut-plane; rather, we seek superman-like “X-ray vision” of these 3D segments in a meaningful, clinically relevant viewing experience. Now, imagine this same 3D grayscale DICOM dataset in VR. If contrast and brightness controls of a 2D image were exchanged for opacity and brightness, any global effort to transform the 3D DICOM dataset would fail to achieve the superman X-ray effect because as the skin, fat, and muscular signal intensity layers are washed away to reveal the internal anatomy, critical components of the internal anatomy are also rendered transparent due to the similar signal intensity characteristics shared by the external anatomy. Therefore, the process of image segmentation is required to perform tissue-specific segmentation on the source images to link the expert categorization of signal intensity of like tissues into separate 3D segmental models. This results in 3D segmented and labeled datasets that allow transparency control over segments that align with clinical decision making. As a result, in VR, the clinician can choose to vary the opacity of the skin, fat, muscle, and bone, to reveal the internal anatomy in a clinically relevant manner, achieving a true 3D understanding of the anatomy. This objective segmentation by the expert grows in impact as the 2D tissue differentiating qualities diminish.

Automated segmentation using machine learning/deep learning (DL) approaches has grown tremendously over the past 5 years, driven by open science data competitions to make the best glioma segmentation algorithm, for example, see Refs. 20 and 34 and Fig. 1. These competitions have brought together large datasets for training segmentation algorithms, including heterogeneous imaging data from a variety of clinics and accompanying ground truth segmentations. A potential disadvantage of the readily available training data is that many algorithms do not explore the use of common preprocessing pipelines for imaging data, such as bias field correction, which can have a significant impact on segmentation accuracy.³⁵

Fig. 1

Glioma patient segmentation examples and VR visualization. (a) Segmentation of the tumor into edematous, enhancing, and necrotic regions. (b) Visualization of ML-algorithm auto-segmentation of tumor core. (c) Gold-standard manual segmentation of tumor core of the same patient for comparison. (d) VR visualization of a patient with a large glioma, showing gray matter (in pink), ventricles (in blue), and tumors in yellow.

Segmentation needs to include both the pathology of interest and the underlying anatomy. All relevant features of the patient anatomy must be extracted from the images as objects, with separate segmentations required for tissue types, pathological regions, bones, blood vessels, and other separate anatomical structures, for example, see the 3D models in Figs. 1 and 2.

Fig. 2

VR experience using a Vive VR headset of a congenital heart surgeon reviewing a complex heart prior to surgery.

3.2.

Medical Imaging Optimized

The imaging datasets need to be acquired in a suitable near-isotropic resolution 3D format, and second, the images should be selected for clinical delineation of anatomic and pathologic features. In our experience, CT imaging is frequently adequate, but due to historical constraints of digital storage, many CT datasets were resampled and compressed, and critical information was not saved. With the transition of the imaging team’s mindset to enabling 3D modeling, these cases are now stored in high isotropic resolution. Magnetic resonance imaging (MRI) presents a different dynamic and frequently excels at delineating soft tissue variance in tissue characteristics. However, again, due to historical practice patterns, thick 2D imaging stacks in 3D planes (axial, sagittal, and coronal) are typically acquired rather than an equivalent near-isotropic 3D dataset. As there is a shift in the use of the data for VR and for automated segmentation and diagnostics, the imaging protocols must adapt. Fortunately, with recent advances in imaging speed enabled by parallel imaging and compressed sensing, transitioning to 3D imaging can occur without sacrificing existing diagnostic imaging efforts or extending the imaging session.³⁶

3.3.

Registration for Augmented Reality

For merging of information into 3D models from multiple medical imaging modalities such as CT and MRI, there are many image registration methods using both standard alignment algorithms, with optimization functions such as mutual information³⁷ for merging grayscale information from disparate modalities, and machine learning/DL approaches.³⁸ However, a more challenging registration is required for using the 3D models in AR interactively in the surgical suite. Image registration algorithms are needed to align the 3D volumetric data with the surface of the 3D object and a view angle of the operator or depth indicator from the surgical instrument or other navigation device. This process requires knowing the position of the patient’s relevant anatomy in the camera’s field of view in the operating room and merging in the patient’s grayscale imaging data or 3D volumetric models in that same coordinate system, all while providing the observer the correct projection of this complex and layered information relative to their viewing position. This can be done with fiducial markers (such as a small object affixed to the patient that is imaged across multiple modalities) carefully placed and localized to establish correspondence between the 3D model from imaging and the visual space of the operator. More recently, it can be done with markerless registration to use features of the patient’s anatomy to provide the basis for registration. We also note that the location of surgical instruments or other intervention devices may also need to be tracked, and location information merged with the 3D model anatomical information to provide real-time surgical navigation or incorporation of optical probe information. These approaches seek to enable sensitive surgical procedures to be performed through a laparoscopic intervention but require algorithms that can handle registrations with limited information and take into account the deformable structure of the tissues.³⁹ Although automated registration and deformation tracking algorithms are drastically improving in accuracy, manual adjustment of registration that takes into account local landmarks must often be done to ensure alignment.⁴⁰ Other technologies, including ultrasound and electromagnetic probes, are also being developed and improved upon that may aid in AR registration prior to surgery.^41,42 Although not used previously with AR, placement of the SaviSout radar system prior to lumpectomy has shown the ability to provide wireless localization of breast tumors.⁴³

AR-based image overlays during surgery merge high-resolution pre-surgical anatomical information with the interactive procedure. Surgical interventions on soft tissues result in deformations of those tissues and this displacement must be accounted for during the procedure to maintain accuracy of the AR overlay. In addition to the intervention itself, differences in gravity between the presurgical scan and the surgical positioning may also need to be addressed.⁴⁴ This can be done by performing rigid registrations followed by finite element method (FEM)-based deformation estimations.^45,46 Often, fiducial points on the surface of the skin are used to provide an initial correspondence with pre-surgery imaging data and to drive updates of soft tissue deformation on the surface to guide deformation estimates based on FEM.⁴⁷ These systems have become very accurate with current systems showing less than a few mm of positioning error, enabling more reliance on the registered imaging data during the intervention.⁴⁸ Surgical AR provides unprecedented views into the patient and a minimal footprint, often being incorporated into a holographic headset. This technology is set to change surgery practice as several companies, such as Medivis and Mediview, have recently achieved Food and Drug Administration (FDA) approval.

3.4.

Considerations for 3D Visualization

3.5.

Use of Artificial Intelligence and Deep Learning with Mixed Reality Imaging

Artificial intelligence (AI) and DL have a significant role to play in enabling the high dimensionality information management that has to happen with a VR/XR system to integrate with patient medical imaging data. First, creating the patient-specific anatomy into actionable objects requires precise segmentation of medical imaging data into 3D structures, capturing both healthy tissue and lesions. Specific segmentation algorithms have been trained using 2D and 3D convolution neural networks (CNNs) using the framework of a U-Net⁵³ to identify specific anatomy and pathology in the bones in the hand,⁵⁴ parcellations of cortical and subcortical structures of the brain, and myocardium and blood pool from the heart,^55,56 among many other specific algorithms in a rapidly growing research area. The need for specific segmentation tools for each task, trained with a particular task-specific set of training data such as cardiac magnetic resonance (MR) with a particular pulse sequence, is a significant limitation as the right tool must be chosen for application to a particular imaging task. To counter this and enable a broader use of automated segmentation algorithms, multimodal and multiorgan segmentation tools have been constructed from CNNs to handle a variety of tissue types⁵⁷ and pathology,^58–60 with some algorithms merging diverse contrasts from multiple imaging modalities.⁶¹ In contrast to task-specific segmentation, there are other general image segmentation algorithms that have been applied to medical images, such as the Segment Anything Model⁶² which will segment structures in generic images and generate labels of different structures without prior training data specific to that data type. These models have shown some promise in medical imaging data,⁶² with a significant advantage in not requiring large amounts of manually labeled training data. However, improvements are needed to surpass the performance of specially designed segmentations for a particular task. Recent algorithms, such as nnU-Net and TotalSegmenter,^63,64 represent an excellent tradeoff between robustness and serving multiple tasks with a high degree of accuracy by bundling adaptive algorithms with smart preprocessing workflows. With TotalSegmenter, CT data can be segmented into 104 anatomical structures automatically. An important aspect of this tool is that extensive training data are made available to help users test their own algorithms in comparison and to provide a full description of the diverse training data to enable users to understand the range of data that the algorithm was trained on.

Further roles for AI in VR/XR will be developed to enable the clinician to interact with a broad set of data, including normative models, predictive progression models, and data from recent literature, all while viewing the high dimensional data of a specific patient and discussing the case with the AI assistant. Similar to segmentation tasks, the integration of AI into these tasks may be driven by the rapidly developing medically focused language learning models.^65–67

6.1.

Human-Centered System Design

As is true for most technology design, and especially so for medical technology, usability and safety issues are critical for eventual clearance or approval by the FDA. Adoption of the technology by end-users also depends on whether the new technology disrupts workflow or creates additional cognitive and/or physical workload. To increase the likelihood of success, medical innovation must satisfy the “market-product fit” criterion—the technology addresses a real need/pain point, and customers are willing to pay for it. Human-centered approaches to design and development ensure that system constraints and complex relationships in the work environment are made explicit and that metrics for assessment and validation can be defined. Until valid assessment tools are created and applied to demonstrate the value of XR, there is a barrier to the wide adoption of this technology into the medical space.

6.2.

Technological Challenges

This paper describes the ongoing evolution of XR in medical imaging and related applications but also lays out large untapped potential limited by current technologies. A challenge with XR is that there is no one silver bullet that will address the technological barriers today. The accuracy and realism needed for real-time AR registration and 3D reconstruction in the described scenarios require significant algorithmic advances in computer vision and graphics, driven by both classical and recent, more transformative, AI techniques. Due to the accuracy required in medical procedures, these systems must be accurate within millimeters and have realistic representations of the procedure that are not achieved by today’s vision transformers. For user comfort, these compute-intensive algorithms must run on much smaller form factor headsets and wearables than we have today, without the need for tethering to large PCs. This requires either highly energy-efficient sensing and computing on the headset with long battery life or efficient offloading of the computation to larger servers through wireless networking. The former is made challenging due to barriers to reducing atomic-scale dimensions of current semiconductor devices (commonly referred to as the end of the era of Moore’s law and Dennard scaling). The latter is challenging due to the stringent latency requirements (especially for AR, e.g., $<5\mathrm{ms}$ motion to photon latency) and the high bandwidth needed for transmitting rich reconstructions in real time. Current network technologies are specifically a key limitation for telesurgery and remote medicine due to the poor reliability and latency of the systems. Commercial systems have barely scratched the surface of new modalities such as haptics that would provide a major leap in the realism of the user experience for medical application scenarios. Accompanying all the above is the lack of consideration for security and privacy in XR systems today.⁸⁶ Finally, there is little work on the assessment and benchmarking of XR end-to-end systems in general and for medical applications in particular. A large missing component is the science of the design and evaluation of XR systems. Although some of these challenges are common to all XR systems today, some are particularly so in the context of the medical domain (e.g., the need for provably high registration accuracy, reconstruction fidelity, and privacy). Recent projects such as ILLIXR (Illinois Extended Reality testbed) and ARENA (Augmented Reality Edge Networking Architecture) have begun to provide open-source community infrastructures to address some of the above challenges by enabling rapid prototyping of research technologies and end-to-end system benchmarking and assessment.^87–89

6.3.

Regulatory Issues

The Medical Extended Reality (MXR) domain faces major regulatory science and technology gaps and challenges, which also were identified by the Medical XR Program at the FDA’s Center for Devices and Radiological Health. This program researches to ensure the safety and effectiveness of innovative XR-based medical devices, tackling various issues, among them are as follows: (i) MXR platforms lack characterization and evaluation methods, mainly for critical medical applications such as interventional procedures, surgery, and rehabilitation; (ii) consumer-grade sensors employed in MXR platforms, including accelerometers, inertial measurement units, gyroscopes, magnetometers, and cameras lack validation for use in clinical contexts; and (iii) there is a dearth of usability assessment tools for devices such as cognitive load measurement. These tools are critical for enhancing device safety and efficacy for surgical and diagnostics applications.⁹⁰

The Medical XR Program bridges the knowledge gaps by undertaking research inquiries related to the design, development, and assessment of novel MXR devices and their applications,⁸⁸ thus forming a scientific foundation for developing a regulatory framework pertaining to innovative MXR devices.⁹¹

6.4.

Industry-University Collaboration

Engagement of academia with industry, national labs, and governmental agencies is critical to the successful translation of XR applications in the biomedical domain. Some of the examples of evolving innovative partnerships are the Center for Medical Innovations in Extended Reality (MIXR) and IMMERSE-Center for Immersive Computing at Illinois. Addressing these challenges will require interdisciplinary collaborations between academic and industry researchers in the various underlying technologies, medical domain experts, and human factors researchers to assess these systems from a human-centric viewpoint. Although such efforts are challenging in themselves, in Illinois, we have recently engaged in the MIXR Center, which is a National Science Foundation Industry-University Cooperative Research Center which was developed initially in 2022 between University of Michigan (Dr. Mark Cohen), the University of Maryland (Dr. Amitabh Varshney), and Maryland Shock Trauma Center (Dr. Sarah Murthi) as well as launching the campus-wide Center for Immersive Computing (IMMERSE).

MIXR’s mission is to advance the global democratization of XR in improving health, training, and recruiting the next generation for a more diverse XR workforce, and bringing together computer scientists, engineers, and healthcare providers with access to diverse patient populations to advance medical XR.⁹² As an industry-academic partnership, it has brought together companies such as Microsoft, Sony, Magic Leap, and the FDA to discuss the future opportunity of XR in healthcare as well as how industry-academic partnerships can solve barriers to adoption and scalability of these technologies in healthcare. Illinois’ expertise in this domain has led to ongoing collaborations with the MIXR center with a goal of expanding its capabilities and impact.

IMMERSE brings together campus strengths in XR technologies, the college of medicine, and human factors.⁹³ We are bringing together research in sensing, hardware acceleration, network protocols, computer vision, graphics, generative AI, and frameworks for human-centric assessment. We are building end-to-end testbeds to prototype and evaluate these new techniques and build new benchmarking methodologies with the goal to accelerate and democratize the development and adoption of XR in the medical domain.