2.1.

Multispectral Line-Scanning Confocal Microscope

The layout of the multispectral line-scanning confocal microscope is illustrated in Fig. 1. The fluorescence excitation light is provided by an iFLEX Viper laser system that has three laser sources operating at wavelengths of 488, 640, and 780 nm. The three lasers, with operator-controlled power levels, are coupled into a single-mode fiber that delivers the fluorescence excitation light to the MSCE imaging system. The multiple laser lines allow for a broad range of fluorescent dyes to be used, and although we have investigated several different multidye and laser combinations, in the tissue imaging work presented here we have used only proflavine and rose bengal, which are both excited well by the single 488-nm fluorescent excitation laser.

Fig. 1

System layout of the MSCE. The blue box shows the cross section of the MSCE in the $x-y$ plane. The red box shows a subframe collected at one position of the scan mirror.

The excitation laser light exiting the single-mode fiber passes through a collimator lens and a cylindrical lens, which together with the objective lens (Olympus $10\times$ Plan Fluor, EFL 18 mm, NA 0.3), focuses the light to a line illumination on the face of the fiber bundle probe. The dichroic mirror reflects the fluorescence excitation light toward the scan mirror, which scans the line illumination across the proximal face of the fiber-bundle probe.

The line illumination beam is relayed to the distal face of the fiber-bundle probe, which is in contact with the tissue. The fluorescence emission from the tissue is relayed back to the proximal end of the fiber-bundle probe, descanned by the scan mirror, transmitted through the dichroic mirror, and focused by a tube lens (Thorlabs TTL-100A EFL 100 mm) onto the confocal slit aperture. This creates a confocal system with optical sectioning at the distal tip of the fiber-bundle probe. The magnification from fiber bundle to slit is $100/18=5.56$.

To achieve multispectral imaging capability, the light passing through the slit is imaged by a second tube lens (same specifications as the first tube lens) and an imaging lens (Nikon $4\times$ Plan Fluor EFL 50 mm, $\mathrm{NA}=0.13$) onto the image sensor with the spectral dispersion of the light provided by a prism (N-BK7, wedged angle 11 deg) located between the second tube lens and the imaging lens. The magnification from slit to sensor is $50/100=0.5$. For a fixed position of the scan mirror, the CMOS image sensor (Basler acA640-750um) collects a 2D data set, called a subframe, which consists of one spatial dimension (along the slit direction: 640 pixels) and a spectral dimension perpendicular to the slit (20 pixels in the spectral dispersion direction). As the scan mirror moves the illumination across the fiber bundle, more and more subframes are collected. Collection of 430 subframes covers the full $750\text{-}\mu \mathrm{m}$ field of view of the fiber probe in contact with the tissue. A full frame of multispectral data consists of 20 spectral images with $480\times 430\text{pixels}$ per spectral image. The mode of data collection is shown in Fig. 2.

Fig. 2

The mode of data collection and image formation. The fluorescent signal emitted from a certain line position of the object is collected by the sensor as a subframe. After 430 subframes are collected, different rows of subframes are mapped to different spectral channels to get 20 ($480\times 430$) images across the spectral range.

The reduction from 640 pixels to 480 pixels in the along-slit direction is done to better match the field of view with some flexibility in positioning of the circular field of view in the multispectral data set. The exposure time of the sensor can be set to $202\mu \mathrm{s}$ for each subframe, which correspond to 4950 subframes per second. But due to the calculation time and display time between each multispectral frame, the system can only collect 4300 subframes per second, which yields a full multispectral frame rate of 10 frames per second. In this paper, the exposure time is set to $340\mu \mathrm{s}$ for each subframe to have a higher signal-noise ratio and yields a frame rate of six full multispectral frames per second.

The MSCE system shown in Fig. 1 was carefully designed to be essentially diffraction limited between the proximal face of the fiber bundle and the image sensor over the wavelength range of 500 to 750 nm without any compensation or adjustments required. Because the multispectral system images the full spectrum simultaneously, it has a high requirement for the correction of longitudinal chromatic aberration.

2.2.

Fiber-optic Probe

A Fujikura coherent fiber bundle (FIGH-30-800G) was chosen as a flexible optical relay to image lung tissue from inside the body to the external confocal microscope. The bend radius of this fiber bundle is 9 cm, which should allow sufficient flexibility to connect the fiber bundle to the MSCE instrument over a distance of $\sim 3\mathrm{m}$. This coherent fiber bundle has 30,000 fiber elements covering a 0.75-mm field of view and a diameter of 0.8 mm when the external silicone resin protective coating is removed (0.950 mm with protective coating present). The fiber bundle with the protective coating can fit through the lumen of a 19-gauge introducer needle that is typically used with a 20-gauge core biopsy needle for tissue extraction. The core size of the individual fibers in the bundle is $\sim 3\mu \mathrm{m}$ with an NA of 0.3. The core-to-core distance is $4.1\mu \mathrm{m}$.

The spatial resolution of this imaging system is fundamentally limited by the fiber bundle sampling, which depends on the core-to-core distance between fibers in the fiber bundle. The maximum spatial frequency that can be imaged without aliasing has a period of twice the core-to-core distance or $8.2\mu \mathrm{m}$, giving a maximum spatial frequency of $122\text{cycles}/\mathrm{mm}$ in object space. The overall magnification from fiber to sensor is $5.56\times 0.5=2.78$. Therefore, the core-to-core distance measured at the image sensor is $11.4\mu \mathrm{m}$, and the maximum spatial frequency without aliasing at the sensor is $43\text{cycles}/\mathrm{mm}$. The sensor itself has a pixel size of $4.8\mu \mathrm{m}$, so its maximum unaliased spatial frequency is $104\text{cycles}/\mathrm{mm}$, well above the maximum spatial frequency of the fiber bundle.

2.3.

Dyes, Tissues, and Dye Delivery System

Exogenous fluorescent dyes provide contrast for identifying critical cellular pathology, such as abnormally large nuclei and/or excessive proteinaceous connective tissue. The fluorescent dyes used in this work are FDA approved, so in principle, they can be used off-label by any physician. Although multiple dye combinations were evaluated as described in Ref. 10, the work presented here employed proflavine as a nuclear stain and rose bengal as a proteinaceous connective tissue stain. Multispectral imaging was done using these dyes on ex vivo human lung core-biopsy tissue samples obtained under UA IRB Protocol 1706540415, and on rat lung resected from euthanized rats under UA IACUC Protocol 16-209. In total, tissue samples from 16 of 19 human subjects, and 2 of 2 rat lung samples produced high-quality multispectral images such as those shown in Figs. 7 and 8, respectively.

To minimize risk and limit dye exposure to human subjects during future in vivo clinical validation studies, fluorescent dyes will be delivered topically to the lung tissue. Figure 3 shows the series of events that will be employed for in vivo dye delivery and imaging. First, an introducer needle with stylet in place is inserted into the patient under CT guidance, which is a normal part of the core biopsy procedure. The stylet is then removed, and the fiber bundle probe is inserted through the introducer needle and pushed to a position where the fiber tip is extended $\sim 1\mathrm{mm}$ past the end of the introducer needle. When the fiber bundle probe is in place with its distal end at the tip of the introducer needle, a small volume of the fluorescent dye combination (on the order of $25\mu \mathrm{l}$) will be delivered to the tissue. Subtle forward motion of the probe into the tissue will be accomplished to achieve tissue contact. In real time, the operating physician will evaluate the images on a monitor and assess whether the tissue being imaged is appropriate for biopsy (i.e., that the needle is in the nodule and at a location where it is giving relevant diagnostic information to proceed with the core biopsy extraction). At that point, the fiber probe would be removed, and the standard core-biopsy needle inserted to collect one or more core biopsy tissue samples from the selected site.

Fig. 3

Steps in the dye delivery, MSCE imaging, and core biopsy extraction procedure.

Steps 1 and 2 have been demonstrated while imaging the normal lung of a euthanized rat. The images obtained from these animal experiments are presented in Sec. 3. Step 3 is a standard part of the core biopsy procedure in humans but was not tested in the rodent model due to the small size of the organ.

The fiber bundle probe and dye delivery apparatus will have to be assembled and sterilized prior to use in any clinical core biopsy procedure. Use of vaporized hydrogen peroxide to sterilize endoscopy instruments is standard clinical practice and should be a straightforward procedure. A maximum time-averaged irradiance of $3.5\mathrm{mW}/{\mathrm{mm}}^2$ can be delivered to the tissue during the imaging process. However, a time-averaged irradiance of $1\mathrm{mW}/{\mathrm{mm}}^2$ appears to be sufficient for this lung imaging application.

2.4.

Software Interfaces

To synchronize the scan mirror and image sensor readout, both devices are connected to a function generator. The whole system is connected to a PC and controlled by a LabView program. The sensor sends a trigger signal to the function generator when it collects every subframe. Then the function generator is triggered and starts sending a ramp-shaped voltage signal to the scan mirror. This ramped signal continues to rise linearly for 167 ms, which is the time it takes to collect an entire data set of 430 subframes. The necessary image processing algorithm is written and applied in LabView, which has sufficient calculation speed to work under real-time imaging conditions.

A Gaussian blur filter was used to reduce the pixelization caused by the fiber bundle. This feature can be turned on or off by the operator. The interface program controls the laser intensity as well as sensor parameters, including exposure time and gain.

Figure 4 shows data from a piece of paper stained with two different fluorescence marker pens. The windows displayed are: (a) a multispectral image window with a suitable color mapping function to the RGB color display; (b) a spectrum of the fluorescence emission over the whole image field of view; (c) a gray-scale image summed over all 20 spectral channels; and (d) an image at spectral channel 16 in the red end of the spectrum. The units in Fig. 4(b) are spectral channels that run over a range from 500 to 750 nm, although there is significant nonlinearity in this spectral mapping. During operation, the windows can be displayed individually or simultaneously together. The spectral distribution plot can be data at a single spatial pixel or averaged over any selected spatial region of interest in the image sensor space.

Fig. 4

Display windows in LabView front panel. Images are collected from a piece of paper marked by two fluorescent marker pens: (a) this window shows the color image after the program has color mapped the signals of the 20 spectral channels to the RGB standard. (b) A spectral plot, the $x$ axis is the channel number; the $y$ axis corresponds to the average signal intensity of each channel in arbitrary units. (c) The gray-scale image (sum over all 20 spectral channels). (d) The gray-scale image of one channel, changing the number above this window changes the displayed channel.

2.5.

Spatial and Spectral Resolution of the MSCE

The MSCE consists of a fiber-coupled laser illumination system to excite fluorescence and a fluorescence emission detection system that itself consists of two back-to-back imaging systems. One is the imaging system from the object (tissue) to the slit, and the other is the imaging system from the slit to the image sensor. Complicating matters for the MSCE system is that there is a fiber bundle optical relay placed between the objective lens and the object, which affects both the fluorescence excitation intensity and emission detection sensitivity. Without the fiber bundle, the diffraction-limited resolution of the line-scan confocal microscope with a finite-width slit aperture is determined by the product of the three-dimensional point spread function of the illumination times and the three-dimensional detection sensitivity function, taking into account the finite size slit. Based on the mathematical model for a line-scan confocal microscope,¹² the lateral resolution would be $1.0\mu \mathrm{m}$ for an excitation wavelength of 488 nm and emission wavelength of 550 nm. The situation changes considerably in the MSCE system because of the fiber bundle probe that is employed. The fiber bundle can be thought of as a sampling function, each core producing one sample. With a sampling periodicity of one sample every $4.1\mu \mathrm{m}$, the maximum spatial frequency that can be measured without aliasing is one cycle every $8.2\mu \mathrm{m}$ or $122\text{cycles}/\mathrm{mm}$ (air force bar target = group 7, element 1). Thus the fiber bundle becomes the key element limiting the lateral spatial resolution in the MSCE system.

The axial resolution is calculated using the same product of the three-dimensional illumination point spread function times the three-dimensional detection sensitivity function taking into account the slit aperture. The difference is that the resulting three-dimensional product needs to be integrated over the $x$, $y$ plane at each $z$ position. The full-width half-max of this result as a function of $z$ (depth) position yields a value for the axial resolution. At the same excitation and emission wavelengths of 488 and 550 nm, respectively, and diffraction limited image of the slit image width at the focal plane is $15/5.56=2.7\mu \mathrm{m}$, and the axial resolution is $14.0\mu \mathrm{m}$. With the fiber bundle in place, one would expect a similar performance for the axial resolution because the individual core diameters of the fiber bundle are $3.0\mu \mathrm{m}$, which is well matched to the slit width in the object space. Thus the fiber bundle is not expected to have a major detrimental effect on the axial resolution of the MSCE.

To evaluate the axial resolution experimentally, a system was built using a thick uniform fluorescent slide in the object space. The fluorescence in the slide was excited with the 488-nm laser. The probe was placed in contact with the fluorescent slide and a series of images were collected as the slide was moved away from the fiber bundle probe by a motorized translation stage. The approach is like the slanted edge test used to measure lateral resolution. A derivative operation on the set of depth images produces a curve that delivers the axial resolution response. The signal is highest when the probe is in contact with the slide and drops off rapidly as the slide is moved away from contact. The axial resolution measured as the half-width at half-max of the derivative curve gave a value of $6.0\mu \mathrm{m}$. This measured axial resolution is significantly better than $14.0\mu \mathrm{m}$ stated above because the measurement with a bare fiber bundle gives signal only from the probe surface out, which is half of what would be expected from a situation where there is an imaging system whose focal plane is some distance away from the probe. The probe contact requirement is an advantage in terms of axial resolution for the MSCE instrument. It should be noted that it is difficult to know how well the probe is in contact with the slide. The first few images in the depth scan showed circular interference fringes, which indicates the probe surface is not flat and this could account for the better than expected axial resolution result.

The spectral resolution of the MSCE multispectral imaging system is determined by the diffraction-limited image of the slit on the sensor and the dispersion introduced by the prism. The $15\text{-}\mu \mathrm{m}$ wide slit maps to a $7.5\text{-}\mu \mathrm{m}$ wide slit image at the sensor. The diffraction-limited point spread function of the imaging system from slit to sensor is $2.35\mu \mathrm{m}$ wide at 500 nm and $3.52\mu \mathrm{m}$ wide at 750 nm, so the diffraction-limited image of the slit on the sensor is about $9.85\mu \mathrm{m}$ wide at 500 nm wavelength and $11.02\mu \mathrm{m}$ wide at 750 nm wavelength. Due to the nonlinear wavelength-dependent dispersion of the prism, this results in a spectral resolution of 14 nm at 500 nm wavelength and 60 nm at 750 nm wavelength.

Since this is a multispectral imaging system that simultaneously collects image data across a broad spectral range, longitudinal chromatic aberration must be kept low. Figure 5 shows that the geometrical RMS spot size simulated in Zemax is smaller than the diffraction limit in the system without fiber bundle, indicating the longitudinal chromatic aberration is acceptable and will not significantly affect the multispectral image quality.

Fig. 5

RMS spot size versus wavelength diagram for the MSCE system without the fiber bundle probe. Field 1: on-axis field (blue); field 2: 0.7 field (green); and field 3: full field (red).

2.6.

Dye Combination

To obtain distinguishable color differences between cell nuclei and connective tissue, the nuclear binding dye, proflavine, and protein binding dye, rose bengal, were chosen for this initial system validation work. In prior work in Ref. 10, a bench top confocal microscope with three spectral channels was used to investigate different dye combinations. Although many dye combinations produced useful image contrast, proflavine and rose bengal appeared to have the fastest binding to the desired targets.

2.7.

Linear Spectral Unmixing Algorithm

Many potential dye combinations that could be imaged with the MSCE have partially overlapping spectra. With general RGB color mapping,¹³ the color contrast, which is set by channel wavelength bands and RGB gains, may not be sufficient to distinguish the color differences between the two dyes. In general, the spectral signal from dye combinations is, to first order, a linear mixture model. To improve the color contrast between multiple dyes, a linear spectral unmixing algorithm can be applied to decompose the mixed fluorescence signal into separate dye images that can then be mapped to RGB display channels.

Assuming the normalized spectral signal for a specific contrast agent is $\hat{s}(\lambda )$ with a signal weight of $f$, the spectrum ${\widehat{s}}_{\mathrm{mix}}(\lambda )$ for a combination of $n$ different dyes is

At every pixel in the multispectral image, the signal weight for each dye can be obtained by

LabView software was written to implement both RGB color mapping and real-time spectral unmixing for the two dyes proflavine and rose bengal. The user has the ability to select either of the two options.

3.1.

Results of Linear Spectral Unmixing

Figure 6 shows two rows of images from different tissue samples. The first row of images is from a core biopsy sample of human lung tissue, and the second row is from within the lung tissue of a euthanized rat. The human core biopsy tissue sample has a diameter slightly smaller than the field of view of the MSCE fiber probe. The rat lung tissue is larger and covers the full field of view. In gray-scale mode, where all the spectral channels are added together and mapped to a gray-scale image intensity (first column of images in Fig. 6), the cell nuclei and connective tissue are not easily distinguishable. In RGB color mapping (second column), the color contrast allows better identification of cell nuclei and connective tissues. In the spectral unmixing images (third column), the color contrast between proflavine (green) and rose bengal (red) is significantly improved. The red box in images b2 and b3 indicates a region where the color contrast in the spectral unmixed image in b3 is much better than the RGB color mapped image in b2.

3.2.

Rat Lung Image from Biopsy Simulation

This section shows MSCE lung images collected from a normal Spague Dawley rat. In order to simulate the clinical biopsy situation, the normal lung of a euthanized rat was imaged using the introducer needle, fiber bundle probe, and dye delivery system discussed in Sec. 2.3. The process of obtaining images at different depths was achieved by pushing the fiber probe and introducer needle through the rat lung.

Figure 7 shows still frames from videos of normal healthy rat lung (Videos 1 and 2). Figure 7(a) and Video 1 are from the probe in contact with surface of the rat lung. When the probe moves over the surface of the lungs, one can clearly see the dense arrangement of cell nuclei, but a relatively small amount of connective tissue since that tissue lies deeper below the surface. When the probe is pushed into the lung [Fig. 7(b) and Video 2], the distribution of nuclei becomes sparser, and the connective tissue of the alveolar structure more apparent. The static images of Figs. 7(a) and 7(b), as well as Videos 1 and 2 clearly show that the MSCE can be used to identify critical lung tissue anatomy in real time. It is worth noting that no tissue rinsing procedure was employed in the rat imaging experiments as the selected dyes, proflavine and rose bengal rapidly bind to their intended targets.

Fig. 7

Healthy rat lung image from a core biopsy simulation: (a) still image from Video 1 collected at surface of the lung and (b) still image from Video 2 collected deeper inside the lung (Video 1, MP4, 5.04 MB; [URL: https://doi.org/10.1117/1.JOM.3.1.011002.s1] Video 2, MP4, 2.20 MB [URL: https://doi.org/10.1117/1.JOM.3.1.011002.s2]).

3.3.

Ex Vivo Human Lung Tissues

Figure 8 shows MSCE images obtained from fresh human core-biopsy samples. All samples were collected under IRB approved protocols with informed consent from patients undergoing clinical core-biopsy procedures. Three different types of tissue are shown along with corresponding H&E histology slide images, where hematoxylin is used for nuclear staining and eosin for cytoplasmic and protein staining. The H&E histology images were evaluated by a highly trained pathologist (Dr. Klein) whose diagnoses were that [Fig. 8(a)] is from necrotic tissue, Fig. 8(b) shows tissue with a significant inflammatory response, and Fig. 8(c) shows squamous cell carcinoma.

Fig. 8

Ex vivo human tissue from lung needle biopsy: (a)–(c) obtained from MSCE with proflavine (DNA-binding, green) and rose bengal (protein binding, red); (d)–(f) H&E images from same tissue regions as top row.

Although the magnification and orientations of the histology images are slightly different than the corresponding MSCE images, histology and MSCE images are from the same region of the human tissue sample.

The first MSCE image [Fig. 8(a)] from a necrotic sample shows the proflavine nuclear staining agent diffusely distributed. In necrotic tissue, cellular structures break down, leaving a soup-like mixture composed of both DNA and proteins. The image characteristics of the corresponding H&E image [Fig. 8(d)] support this necrosis diagnosis showing essentially no intact nuclei and little viable tissue structure. Similarly, in the MSCE image [Fig. 8(a)], nuclei are not observed and the staining pattern of the two dyes is patchy, rather than forming the highly ordered structure of living tissue. Instead of being confined within cellular nuclei, the proflavine (green) dye more evenly stains the distributed DNA material in the sample. The rose bengal (red) dye does not display the lacey pattern of staining observed in lung connective tissue. Proflavine is distributed deeper in the center part of the tissue, whereas rose bengal stays on the outside layer of the sample, which may be due to the different permeabilities of the two dyes in staining necrotic tissue.

The second MSCE image [Fig. 8(b)] is from human tissue showing an inflammatory response. In the H&E image, the arrows identify areas of inflammation and fibrosis. Different than the previous necrotic sample, the nuclei in the H&E image are well stained and show up dark purple. In the MSCE image [Fig. 8(b)], cell nuclei are identifiable and stained green by proflavine. At the same time, one can also see the enhanced connective fibrous tissue stained red by rose bengal. Compared with ordinary lung tissue, the connective tissue in this figure is thicker and more fibrotic, a consequence of chronic inflammation. The images in the third column [Figs. 8(c) and 8(f)] are taken from a sample that was diagnosed as squamous cell carcinoma. Both in the H&E image [Fig. 8(f)] and in the MSCE image [Fig. 8(c)], the nuclei of the cancer cells are larger than the nuclei of noncancerous cells. It would be appropriate and useful to show MSCE images of normal human lung tissue. However, this was not possible as the tissue samples obtained are from within lung nodules that are the target for the clinical evaluation of pathologic changes. Nevertheless, one can obtain an appreciation for what normal lung tissue looks like from the rat images shown in Fig. 7. Figure 7(b) shows what the MSCE images look like when the needle is inserted into the lung, which would be the case for the clinical application considered in this work.

Finally, Fig. 9 shows a still frame from a video of ex vivo human lung tissue (Video 3). This sample was extracted from a 20-gauge biopsy needle, so the core size is smaller than the 7.5-mm FOV of the MSCE. In this video, as the probe moves slowly along the sample, one can clearly see changes in the cellular and fibrotic distributions. This indicates that the MSCE can provide real-time images of tissue at the distal tip of the fiber probe, which is expected to help clinicians diagnose or at least select appropriate tissue sampling positions. It is noteworthy that there is little motion artifact observed in Video 3. In imaging ex vivo tissue samples, there is a delicate balance in applying appropriate pressure to maintain tissue contact while at the same time not applying too much pressure such that the delicate tissue sample gets torn or stuck to the end of the fiber probe. This should be less of a problem for in vivo imaging of human lung because the introducer needle and fiber probe move together through the tissue rather than sliding along the top of the tissue as is the case with the imaging of ex vivo core biopsy tissue samples.

Fig. 9

Image from a video of inflammatory ex vivo human lung tissue sample. Dye applied: proflavine (DNA-binding, green) and rose bengal (protein binding, red) (Video 3, MP4, 3.38 MB [URL: https://doi.org/10.1117/1.JOM.3.1.011002.s3]).