SignificanceMinimally invasive surgery (MIS) has shown vast improvement over open surgery by reducing post-operative stays, intraoperative blood loss, and infection rates. However, in spite of these improvements, there are still prevalent issues surrounding MIS that may be addressed through hyperspectral imaging (HSI). We present a laparoscopic HSI system to further advance the field of MIS.AimWe present an imaging system that integrates high-speed HSI technology with a clinical laparoscopic setup and validate the system’s accuracy and functionality. Different configurations that cover the visible (VIS) to near-infrared (NIR) range of electromagnetism are assessed by gauging the spectral fidelity and spatial resolution of each hyperspectral camera.ApproachStandard Spectralon reflectance tiles were used to provide ground truth spectral footprints to compare with those acquired by our system using the root mean squared error (RMSE). Demosaicing techniques were investigated and used to measure and improve spatial resolution, which was assessed with a USAF resolution test target. A perception-based image quality evaluator was used to assess the demosaicing techniques we developed. Two configurations of the system were developed for evaluation. The functionality of the system was investigated in a phantom study and by imaging ex vivo tissues.ResultsMultiple configurations of our system were tested, each covering different spectral ranges, including VIS (460 to 600 nm), red/NIR (RNIR) (610 to 850 nm), and NIR (665 to 950 nm). Each configuration is capable of achieving real-time imaging speeds of up to 20 frames per second. RMSE values of 3.51±2.03%, 3.43±0.84%, and 3.47% were achieved for the VIS, RNIR, and NIR systems, respectively. We obtained sub-millimeter resolution using our demosaicing techniques.ConclusionsWe developed and validated a high-speed hyperspectral laparoscopic imaging system. The HSI system can be used as an intraoperative imaging tool for tissue classification during laparoscopic surgery.
Neuroblastoma is the most common type of extracranial solid tumor in children and can often result in death if not treated. High-intensity focused ultrasound (HIFU) is a non-invasive technique for treating tissue that is deep within the body. It avoids the use of ionizing radiation, avoiding long-term side-effects of these treatments. The goal of this project was to develop the rendering component of an augmented reality (AR) system with potential applications for image-guided HIFU treatment of neuroblastoma. Our project focuses on taking 3D models of neuroblastoma lesions obtained from PET/CT and displaying them in our AR system in near real-time for use by physicians. We used volume ray casting with raster graphics as our preferred rendering method, as it allows for the real-time editing of our 3D radiologic data. Some unique features of our AR system include intuitive hand gestures and virtual user interfaces that allow the user to interact with the rendered data and process PET/CT images for optimal visualization. We implemented the feature to set a custom transfer function, set custom intensity cutoff points, and region-of-interest extraction via cutting planes. In the future, we hope to incorporate this work as part of a complete system for focused ultrasound treatment by adding ultrasound simulation, visualization, and deformable registration.
Biopsies play a crucial role in diagnosis of various diseases including cancers. In this study, we developed an augmented reality (AR) system to improve biopsy procedures and increase targeting accuracy. Our AR-guided biopsy system uses a high-speed motion tracking technology and an AR headset to display a holographic representation of the organ, lesions, and other structures of interest superimposed on real physical objects. The first application of our AR system is prostate biopsy. By incorporating preoperative scans, such as computed tomography (CT) or magnetic resonance imaging (MRI), into real-time ultrasound-guided procedures, this innovative AR-guided system enables clinicians to see the lesion as well as the organs in real time. With the enhanced visualization of the prostate, lesion, and surrounding organs, surgeons can perform prostate biopsies with an increased accuracy. Our AR-guided biopsy system yielded an average targeting accuracy of 2.94 ± 1.04 mm and can be applied for real-time guidance of prostate biopsy as well as other biopsy procedures.
While minimally invasive laparoscopic surgery can help reduce blood loss, reduce hospital time, and shorten recovery time compared to open surgery, it has the disadvantages of limited field of view and difficulty in locating subsurface targets. Our proposed solution applies an augmented reality (AR) system to overlay pre-operative images, such as those from magnetic resonance imaging (MRI), onto the target organ in the user’s real-world environment. Our system can provide critical information regarding the location of subsurface lesions to guide surgical procedures in real time. An infrared motion tracking camera system was employed to obtain real-time position data of the patient and surgical instruments. To perform hologram registration, fiducial markers were used to track and map virtual coordinates to the real world. In this study, phantom models of each organ were constructed to test the reliability and accuracy of the AR-guided laparoscopic system. Root mean square error (RMSE) was used to evaluate the targeting accuracy of the laparoscopic interventional procedure. Our results demonstrated a registration error of 2.42 ± 0.79 mm and a procedural targeting error of 4.17 ± 1.63 mm using our AR-guided laparoscopic system that will be further refined for potential clinical procedures.
Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. However, there are still prevalent issues surrounding intracorporeal surgery, such as iatrogenic injury, anastomotic leakage, or the presence of positive tumor margins after resection. Current approaches to address these issues and advance laparoscopic imaging techniques often involve fluorescence imaging agents, such as indocyanine green (ICG), to improve visualization, but these have drawbacks. Hyperspectral imaging (HSI) is an emerging optical imaging modality that takes advantage of spectral characteristics of different tissues. Various applications include tissue classification and digital pathology. In this study, we developed a dual-camera system for high-speed hyperspectral imaging. This includes the development of a custom application interface and corresponding hardware setup. Characterization of the system was performed, including spectral accuracy and spatial resolution, showing little sacrifice in speed for the approximate doubling of the covered spectral range, with our system acquiring 29 spectral images from 460 to 850nm. Reference color tiles with various reflectance profiles were imaged and a RMSE of 3.56 ± 1.36% was achieved. Sub-millimeter resolution was shown at 7cm working distance for both hyperspectral cameras. Finally, we image ex vivo tissues, including porcine stomach, liver, intestine, and kidney with our system and use a high-resolution, radiometrically calibrated spectrometer for comparison and evaluation of spectral fidelity. The dual-camera hyperspectral laparoscopic imaging system can have immediate applications in various surgeries.
Hyperspectral imaging (HSI) is an emerging modality for digital pathology. The purpose of this study is to develop an extended depth of field (EDOF) method for mosaic hyperspectral images acquired with a snapshot camera. EDOF is a technique for ensuring that an image is in focus at all points. A stack of mosaicked hyperspectral images of hematoxylin and eosin (H&E)-stained histologic slides were acquired at different positions along the z-axis and used to output a hyperspectral histologic image that was in-focus at every point. Three different methods were compared to achieve a fully focused image. We compared conventional patch-based methods to our proposed growth-based and band-based methods. The Brenner function was used to quantitatively measure the focus quality of each image measured. The results show that both of our proposed methods performed better qualitatively and quantitatively than the patch-based method, with the band-based method performing the best, as it leveraged dividing pixels into their proper wavelengths in addition to spatially, giving the algorithm better contrast to measure. In terms of speed, the band-based method was the fastest, followed by the patch-based method, with the growth-based method being the slowest. Our proposed extended depth of field hyperspectral imaging methods can have immediate applications in digital pathology, especially whole slide imaging, and other microscopic imaging.
Significance: Hyperspectral imaging (HSI) provides rich spectral information for improved histopathological cancer detection. However, acquiring high-resolution HSI data for whole-slide imaging (WSI) can be time-consuming and requires a huge amount of storage space.
Aim: WSI using a color camera can be achieved with fast speed, high image resolution, and excellent image quality due to the established techniques. We aim to develop an RGB-guided unsupervised hyperspectral super-resolution reconstruction method that is hypothesized to improve image quality while maintaining the spectral characteristics.
Approach: High-resolution hyperspectral images of 32 histologic slides were obtained via automated WSI. High-resolution RGB histology images were registered to the hyperspectral images for RGB guidance. An unsupervised super-resolution network was trained to take the downsampled low-resolution hyperspectral patches (LR-HSI) and high-resolution RGB patches (HR-RGB) as inputs to reconstruct high-resolution hyperspectral patches (HR-HSI). Then, an Inception-based network was trained with the HR-RGB, original HR-HSI, and generated HR-HSI, respectively, for whole-slide histopathological cancer detection.
Results: Our super-resolution reconstruction network generated high-resolution hyperspectral images with well-maintained spectral characteristics and improved image quality. Image classification using the original hyperspectral data outperformed RGB because of the extra spectral information. The generated hyperspectral image patches further improved the results.
Conclusions: The proposed method potentially reduces image acquisition time, saves storage space without compromising image quality, and improves the image classification performance.
Hyperspectral imaging (HSI) has many advantages in microscopic applications, including high sensitivity and specificity for cancer detection on histological slides. However, acquiring hyperspectral images of a whole slide with a high image resolution and a high image quality can take a long scanning time and require a very large data storage. One potential solution is to acquire and save low-resolution hyperspectral images and reconstruct the high-resolution ones only when needed. The purpose of this study is to develop a simple yet effective unsupervised super resolution network for hyperspectral histologic imaging with the guidance of RGB digital histology images. High-resolution hyperspectral images of hemoxylin & eosin (H&E) stained slides were obtained at 10× magnification and down-sampled 2×, 4×, and 5× to generate low-resolution hyperspectral data. High-resolution digital histologic RGB images of the same field of view (FOV) were cropped and registered to the corresponding high-resolution hyperspectral images. A neural network based on a modified U-Net architecture, which takes the low-resolution hyperspectral images and high-resolution RGB images as inputs, was trained with unsupervised methods to output high-resolution hyperspectral data. The generated high-resolution hyperspectral images have similar spectral signatures and improved image contrast than the original high-resolution hyperspectral images, which indicates that the super resolution network with RGB guidance can improve the image quality. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will potentially promote the use of hyperspectral imaging technology in digital pathology and many other clinical applications.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.