2.1.

Imaging Setup

The imaging setup included a 10-mm, 0-deg laparoscope (WA4KL100 UHD, Olympus America, Center Valley, Pennsylvania, United States) and a 150-W halogen source (OSL2IR High-Intensity Fiber-Coupled Illuminator, Thorlabs, Newton, New Jersey, United States) with an operating wavelength range of 400 to 1750 nm. The SC configuration in Fig. 1(a) includes a 20-mm C-mount adapter (Stryker, Kalamazoo, Michigan, United States) and a 2× C-mount fixed focal length lens extender. In this work, three snapshot hyperspectral cameras with different wavelength profiles were used. The visible (VIS) hyperspectral camera utilizes a $4\times 4$ mosaic pattern and captures 16 bands covering a spectral range of 460 to 600 nm. Meanwhile, the red/near-infrared (RNIR) camera covers 15 wavelength bands ranging from 600 to 870 nm with a $4\times 4$ mosaic configuration as well. Finally, the near-infrared (NIR) hyperspectral camera utilizes a $5\times 5$ mosaic configuration and captures 24 bands from 660 to 960 nm. Each camera weighs 32 g, leading to a total additional weight to the SC setup of 162 g with the attached optics (not including the laparoscope itself). This setup was used for all spectral and spatial evaluations, as well as the phantom study. For the ex vivo studies, the DC configuration shown in Fig. 1(b) was utilized, which includes a 605-nm dichroic mirror (Thorlabs), two 35-mm VIS-NIR C-mount lenses (Edmund Optics, Barrington, New Jersey, United States), and two snapshot hyperspectral cameras capable of capturing up to $120\text{hypercubes}/\mathrm{s}$. Fabricated adapters couple the eyepiece of the laparoscope to the dichroic mirror, and another adapter was made to attach the fiber light guide from the halogen source to the light post of the scope.

Fig. 1

Laparoscopic HSI system configurations. (a) Single camera. (b) Dual camera.

2.2.

Image Acquisition

Once the imaging system was constructed, various considerations were taken for controlled, accurate, reproducible image acquisition. First, all imaging was done at a working distance of 7 cm. An adjustable lab jack (Thorlabs) was used to ensure a fixed working distance for each procedure. We then focused on the respective hyperspectral cameras and set the exposure time to allow maximal intensity when imaging a white reference (WR) reflectance tile without saturating the imaging sensor. Other optical parameters of the system were set as well, including f/#, focal length, and exit pupil distance. Last, before acquiring images of the target/tissue, we acquired white and dark reference images for HSI calibration. The dark reference/dark current image must be acquired by either shutting off the light source or blocking any incoming light at the distal end of the scope. Conversely, the WR image was taken with a 95% reflectance surface (Spectralon, Labsphere Inc., North Sutton, New Hampshire, United States) at the approximate working distance of the target or specimen. To mitigate the effects of noise, we employed temporal averaging by capturing multiple hyperspectral images of both the tile and the imaged target/tissue. For each validation procedure, $\sim 30$ images were acquired for each tile and target. We increased this for the ex vivo study, imaging each sample $\sim 65$ times. Room lights were turned off to avoid variation due to stray light during acquisition.

2.3.

Software Development

To efficiently acquire and process images with the system, we developed an in-house software application written in C++ using the QT graphical user interface framework. This enables the consolidation of image acquisition, processing, and implementation of tools and features, such as demosaicing and temporal averaging, in a single interface. The application, as a universal tool, allows up to five hyperspectral cameras to be connected simultaneously with the runtime parameters for acquisition and processing individually set for each camera. This allows the simultaneous acquisition of hyperspectral images with multiple cameras, a feature leveraged in our DC configuration. For efficient imaging with our stationary system, a burst capture feature was implemented, allowing users to specify a number of images or an amount of time for which to capture. Figure 2 shows an example of hypercube acquisition using a snapshot camera.

Fig. 2

Software application for concurrent hyperspectral imaging with the dual-camera system. The left pane shows the live video feed of the camera as raw, mosaic images. The right pane shows a single band from a captured hypercube.

2.4.

System Validation

To validate the spectral accuracy of the hyperspectral cameras in the system, standard reflectance tiles were used (Spectralon, Labsphere Inc.) to compare the spectral footprints from hyperspectral images with standard reflectance curves generated with a calibrated spectrophotometer. The used tiles are characteristically Lambertian in their reflectance profile, unlike the tissues used in our ex vivo study. Eight tiles with different colors were included in the spectral analysis. Each tile produces a unique spectral footprint throughout the wavelength range of 360 to 830 nm, allowing us to validate the accuracy throughout the intensity range of the system. Another reflectance tile with a calibrated spectral profile extending into the NIR range was used to assess the NIR hyperspectral camera (WCS-MC-020, Labsphere Inc.). The spectral accuracy of the system was quantitatively evaluated using the root mean squared error (RMSE) metric. To evaluate the spatial resolution capabilities of the setup, a 1951 USAF Resolution Test Target (Thorlabs) was imaged, and Michelson contrast was calculated. To assess the system’s capabilities when imaging non-Lambertian surfaces such as those seen intraoperatively, spectral validation with ex vivo tissues was performed with a calibrated spectrometer (HR2000+ES, Ocean Insight, Orlando, Florida, United States) by fixing the probe in the field of view (FOV) of the laparoscope so that ground truth spectral footprints of the tissue in the hyperspectral camera FOV could be measured for comparison. The spectrometer was calibrated using a radiometrically calibrated source (HL-3P-CAL, Ocean Insight) for absolute irradiance measurements from 190 to 1100 nm.

2.5.

Demosaicing

Unprocessed images acquired with SRDAs produce an image with higher resolution than the final processed hypercube, as the entire image sensor is used. However, due to the repeated pattern of micro-scale optical filters and different relative transmissions, a “mosaic” pattern overlays the grayscale image. To take advantage of the larger image raster size of the unprocessed images and their potential to improve the spatial resolution of the system, we performed a demosaicing process on the unprocessed, mosaic images. This process can be visualized in Fig. 3, which shows the removal of the mosaic pattern seen throughout the image. Three demosaicing methods were evaluated for their performance: white reference calibration (WRC), low-pass filtering (LPF), and filter convolution (FC).

Fig. 3

Conceptual visualization of demosaicing. (a) Raw mosaic image is processed into the (b) demosaiced image.

The WRC technique is straightforward—deriving its name from the common HSI preprocessing step. This method requires the mosaic image of a WR tile in conjunction with the image to be demosaiced (referred to as the original image henceforth). The final image is produced via element-wise division of the original image by the WR mosaic

The LPF method was developed by intuiting that the repeated mosaic pattern can be treated as a form of high-frequency noise that can be filtered out using standard image processing techniques. Therefore, the pattern may be removed by performing a 2D Fourier transform on the original image, isolating the magnitude component, and then applying a threshold to remove high-frequency signals from the magnitude plot. We made sure to retain those signals that occur in the central region of the plot, as it includes low-frequency information vital to the image. Finally, an inverse 2D Fourier transform was performed to recover the newly demosaiced image. The threshold value was determined experimentally for the test target data.

Last, the FC method was assessed. We form two kernels related to the filter responses of the hyperspectral camera. The first was formed by performing a filter-wise sum across the original image to produce a kernel equivalent in size to the mosaic configuration of the camera, either $4\times 4$ or $5\times 5$. Figure 4 shows the kernel formation process. The second kernel is the reciprocal of the first. Each kernel is convolved separately over the original image. The two resulting images were normalized and averaged to form the final demosaiced image. Finally, an unsharp filter is applied to increase image sharpness.⁴⁰ An overview of the FC method is shown in Fig. 5.

Fig. 4

Kernel formation. The final kernel is the resultant matrix of a sum performed across the original image at each filter sub-unit. The final filter characterizes the response of each filter in the array.

Fig. 5

Filter convolution demosaic method. (a) Original mosaic image. (b) Resulting image from convolution performed with filter $H$. (c) Resulting image from convolution performed with filter $H^{-1}$. (d) Final demosaiced image.

2.6.

Spatial Resolution Evaluation and Quality Evaluation

To measure spatial contrast, a Michelson contrast cutoff of 20% was used as shown in Eq. (2)

As shown in Fig. 6, the line elements were analyzed to produce a plot of intensities along the line pairs. A sine curve was then fit to the plot to get $S_{\max }$ and $S_{\min }$ values. $S_{\max }$ corresponds to the peak of the sine curve produced by the intensities of the line elements, and $S_{\min }$ corresponds to the minimum of the curve. Smaller elements and groups of the resolution target were analyzed until the contrast threshold was surpassed. The contrast was analyzed on full-resolution demosaiced images, including WRC, LPF, and FC methods outlined in Sec. 2.5.

Fig. 6

Spatial resolution assessment. (a) Demosaiced image of the test target. (b) Line elements being analyzed. (c) Resulting sinusoidal curve from the group of line elements.

In addition to gauging the resolution using contrast, we evaluated our demosaicing techniques using a perception-based image quality evaluator (PIQUE).⁴¹ PIQUE is an unsupervised and opinion-unaware no-reference algorithm that estimates distortion and local variance, with lower scores indicating higher-quality images. As image brightness affects PIQUE scores, the demosaiced images were normalized and brought to similar brightness levels by scaling each image to the same mean intensity over the assessed region to produce comparable results.

2.7.

Ex Vivo Tissue Imaging

To further validate our system for laparoscopic imaging, porcine visceral organs were used for ex vivo tissue imaging, including the stomach, liver, intestine, and kidney. To cover a larger spectral range and avoid the need to change cameras, our DC configuration was used with the VIS and RNIR hyperspectral cameras. Specimens were imaged at a real-time rate of 25 frames per second for 50 to 60 unprocessed frames. After hypercube reconstruction, temporal averaging was performed to reduce the effect of noise on the hyperspectral images and subsequently generated spectral curves. Ground truth spectral reflectance values were obtained via a radiometrically calibrated spectrometer alongside our system to validate the spectra of the tissues. Absolute reflectance values from the tissues were then linearly transformed for comparison with the relative reflectance produced by the system. Figure 7 shows the imaging setup used for imaging the visceral tissues and capturing simultaneous ground truth spectral curves from the spectrometer. Manual segmentation of the hyperspectral images from each camera was performed to produce masks for registration among the cameras. Preprocessing for registration included averaging the hypercube along the spectral dimension to produce a single grayscale image and binarizing each resulting image at an experimentally determined threshold. Further refinement was performed before performing affine, control point registration of the RNIR hypercube to the VIS hypercube. As the cameras were fixed within the system, the same transformation matrix could be used for each hyperspectral image. A proposed region of interest (ROI) corresponding with the captured spectrometer readings was assumed for each tissue for comparison of spectral curves.

Fig. 7

Imaging system and spectrometer placement over ex vivo tissue scene. (a) Full imaging system view. (b) Active imaging view.

To further simulate intracorporeal procedures, abdominal tissues were positioned within a phantom model, and a hyperspectral image sequence was captured, as depicted in Fig. 8. In this procedure, a porcine liver, intestine, and two kidneys were introduced into the phantom for imaging. This facilitated the examination of the system’s mobility and maneuverability in a dynamic environment, which is more akin to abdominal MIS than the static setting used for controlled imaging and experimentation. The phantom consisted of a modified hollow mannequin portion to accommodate ex vivo tissues. We bored a 0.5-in. (1.27-cm) aperture in the abdomen below the umbilicus for the insertion of the 10-mm laparoscope.

Fig. 8

Phantom assessment of laparoscopic HSI system. (a) External view of SC configuration. (b) Pseudo-RGB formed from the hyperspectral frame in a video sequence. (c) Internal view of the distal end of the laparoscope and visceral tissues.

2.8.

Tissue Spectra and Pseudo-RGB Construction

To analyze the tissue spectra derived from the hyperspectral cameras, a binary mask of the tissue was generated. Hyperspectral images were calibrated using white and dark reference images according to our previous work.³⁸ Pseudo-RGB images were created for each tissue image by selecting the nearest bands from the VIS hyperspectral camera corresponding to RGB color channels, 700.0, 546.1, and 435.8 nm, respectively. Calibrated reflectance values were then acquired over ROIs approximating the FOV of the spectrometer as gauged by pseudo-RGB images to produce the spectral footprints of each imaged tissue. A threshold was applied to the pixels in the ROI to mitigate the effects of glare. Glare pixels were not removed outright as this processing could not be mirrored in the ground truth spectroscopy data.

3.1.

Spectral Accuracy and Spatial Resolution

Spectral validation was performed on the reference tiles mentioned in Sec. 2.4 with the SC configuration of each camera. Spectral footprints of each of the eight color tiles were used to assess the VIS and RNIR hyperspectral cameras. The spectra covered a range of 360 to 830 nm, with a 1-nm step size. For spectral validation of the NIR camera, the NIR tile is used with a calibration spectral footprint that covers 235 to 2000 nm, with a 1-nm step size as well. The VIS and RNIR hyperspectral cameras covered 461 to 597 and 614 to 853 nm, respectively, with the VIS capturing 16 wavelengths and the RNIR capturing 15 wavelengths. As a result, the HSI curve of the RNIR camera extended marginally beyond the reference curves for the tiles. These values were excluded from the quantitative analysis. The resulting spectral curves from the VIS and RNIR cameras are plotted in Fig. 9 along with the ground truth calibration curves. The NIR hyperspectral camera captured a spectral range of 669 to 949 nm, and its resulting spectra are shown in Fig. 10. Overall, the recorded reflectance values from each of the systems conformed to the reference curves well with the largest RMSE of the VIS camera (blue = 6.25 and yellow = 6.31) resulting from a slight shift in the steep region of the tile reflection profile. The error also seemed to accumulate in the extreme regions of the captured spectral range, especially toward the longer wavelength region of each system, as seen in the blue spectral curve and RMSE of the RNIR camera and the NIR spectral plot. The standard reflectance values of the tiles used overall cover a broad range of intensities, showing the sensitivity of the system to accurately discern spectral features at different relative reflectance intensities. Overall, the average RMSEs achieved by the VIS and RNIR cameras were $3.51\pm 2.03\%$ and $3.43\pm 0.84\%$, respectively, whereas the NIR camera achieved an RMSE of 3.47%. Table 1 outlines the RMSE values for each tile.

Fig. 9

Result of the spectral accuracy of VIS and RNIR cameras. HSI spectral curves (solid) against the tile standard reflectance (dashed). (a) Red reflectance tile. (b) Green reflectance tile. (c) Blue reflectance tile. (d) Yellow reflectance tile. (e) Violet reflectance tile. (f) Cyan reflectance tile. (g) Orange reflectance tile. (h) Purple reflectance tile.

Fig. 10

Result of the spectral accuracy assessment for the NIR hyperspectral camera. HSI (yellow) plotted as mean ± SD (shaded area) versus reference curve (purple).

Table 1

RMSE for each reflectance tile.

Spatial validation showed sub-millimeter horizontal and vertical resolution for each camera and each demosaicing technique (besides vertical resolution with NIR camera with WRC) as shown in Table 2. The WRC demosaic narrowly outperformed FC, followed by LPF for the VIS camera. Vertical and horizontal resolutions of 0.250 and 0.223 mm were achieved, respectively. For the RNIR camera, FC was superior, equally distinguishing vertical and horizontal elements spaced 0.353 mm apart. The other two techniques performed equally with the RNIR camera in each direction, achieving 0.500 and 0.446 mm vertical and horizontal resolutions, respectively. Resolutions achieved with the NIR camera were larger than those of the other cameras likely due to the larger mosaic configuration of $5\times 5$. It’s also worth noting that the reflectance profile of the line elements of the test target is progressively more reflective moving into the RNIR–NIR wavelength range, also contributing to decreased contrast and overall performance of the RNIR and NIR cameras. That said, FC outperformed the other techniques by a significant margin for the NIR camera. Besides the performance of the VIS camera, the FC technique is the superior demosaicing method for optimal spatial resolution. Qualitative analysis suggests its superiority as well, as LPF results retain some high-frequency information and WRC outputs seemingly lack global contrast, each resulting in lower-quality images. The qualitative results of each method are shown in Fig. 11 as a small region of each image is highlighted.

Table 2

Spatial resolution (mm) and PIQUE scores achieved with each hyperspectral camera.

Fig. 11

Qualitative results of each demosaicing method. (a) Original mosaic image. (b) WR calibration demosaic output. (c) Filter convolution demosaic output. (d) Low-pass demosaic output.

The resulting PIQUE scores from each camera suggest superior quality for the WRC output for both the VIS and NIR cameras despite it being the poorest performing method with the RNIR camera. Instead, the RNIR camera saw superior results from FC demosaicing. There are few global patterns across each camera, suggesting the importance of method selection depending on the given hardware.

3.2.

Ex Vivo Tissue Spectra

Tissue imaging was carried out using the DC configuration. Blood absorption coefficients for oxygenated and deoxygenated hemoglobin vary distinctly in the RNIR region, providing clear tissue distinctions and estimation of blood content.⁴² Tissues were imaged individually and collectively, and masks were created for each specimen of spectrometer ROIs after thresholding glare pixels to a maximum value of 1. Figure 12 shows the pseudo-RGB images produced from the VIS hypercubes as well as the mean reflectance values for each tissue within the highlighted ROI throughout the spectral range of the VIS and RNIR cameras. The resulting spectral curves produced from the calibrated spectrometer were also plotted for reference. The values measured by the spectrometer underwent a similar calibration to the hypercubes, as each tissue spectral curve was normalized to the spectral curve of the white reference tile under similar lighting conditions and working distance. After calibration, a least squares regression was performed with one of the tissues to slightly adjust the scale further as the laparoscopic HSI system produces relative reflectance values and not absolute measurements. We note that the specimens are distinguishable by their reflectance intensities, especially in the RNIR range.

Fig. 12

Ex vivo experimental results. Pseudo-RGB endoscopic image of (a) intestinal, (b) stomach, (c) liver, and (d) kidney tissues, each with ROI highlighted. (e) Spectral footprint of liver, stomach, intestinal, and kidney tissues from the laparoscopic system (solid) plotted alongside the spectrometer output (dotted).

As seen in a few of the validation spectral curves of the reference tiles, the features in the short VIS range from the laparoscopic HSI system did not align as well with the spectrometer curves as in other wavelength ranges.

3.3.

Phantom Imaging

The SC configuration of the system was used to acquire a hyperspectral sequence of images using the VIS camera. The pseudo-RGB feed was visualized for each of the abdominal tissues in the phantom. The system was shown to be maneuverable and lightweight, so as not to disrupt the standard workflow. Significant variation in image brightness was observed with varying working distances without a feature equivalent to auto exposure time as seen in RGB systems.