2.1.

Snapshot Multispectral Endoscope for In Vivo Clinical Imaging

In the future, SRDAs could be deployed as “chip-on-tip” cameras at the distal end of an endoscope, however, such integration requires significant miniaturization of optics and changing a clinically approved device, which substantially extends the timeline required for first-in-human testing. Furthermore, the spectral properties of esophageal tissue are not well characterized nor reliably measured using ex vivo tissue.⁴⁴ Thus, to facilitate clinical testing, we combined the SRDA with a CE-marked “babyscope” (PolyScope, PolyDiagnost, Germany), a small fiber-bundle-based endoscope that can be inserted through the accessory channel of a standard endoscope to relay light between external optics and the esophageal lumen. We have previously shown that SRDAs can be implemented in combination with such an imaging-fiber bundle without reducing resolution, since the resolution of fiberscope-based imaging is limited by the size of individual fiberlets rather than by the sensor resolution.⁴⁵ The babyscope system included: a single fiber to relay illumination from outside the patient and direct it onto the tissue; and a 10,000-fiberlet imaging bundle to relay diffusely reflected light from the esophageal tissue to custom external detection optics [Fig. 1(a)]. The 3 mm diameter of this babyscope allowed it to be introduced through the accessory channel of a therapeutic gastroscope [Fig. 1(b)]. Utilizing this unmodified CE-marked device to relay light between the esophagus and external optics facilitated clinical application of the spectral endoscope.

Fig. 1

A compact snapshot multispectral endoscope for esophageal imaging. (a) Light from a broadband supercontinuum light source enters the patient via a single illumination fiber. Reflected light is returned via a 10,000 fiberlet imaging bundle. (b) The “babyscope” enters the esophagus via the working channel of a standard of care endoscope. (c) Light is detected by an SRDA. (d) This includes a $3\times 3$ mosaic of color filters, eight narrow bands, and one broad band (B) deposited directly on the sensor pixel. (e) The spectral response of the detection channel of the multispectral endoscope is shown. (f) Light is delivered to the esophageal lumen via the babyscope illumination fiber and diffusely reflected light is collected by the imaging fiber bundle.

At the proximal end of the babyscope, the illumination fiber was coupled to a broadband supercontinuum light source (SuperK COMPACT, NKT Photonics, United Kingdom), which provided illumination from 450 to 875 nm (Fig. S1 in the Supplementary Material). The proximal end of the imaging fiber was imaged and magnified using an objective lens (UPLFLN20x, Olympus, Japan). The light was split using a 90:10 plate beam splitter (BSN10R, Thorlabs, Germany) and focused using an achromatic doublet lens ($f=100\mathrm{mm}$, ACA254-100-A, Thorlabs, Germany); 10% of the light was focused onto a red–green–blue (RGB) camera (Grasshopper 3.0, IDS, Germany) to record reference images and the remaining 90% was focused onto the SRDA (CMS-V, SILIOS, France) [Fig. 1(c)]. The SRDA consists of nine spectral filters deposited as a $3\times 3$ super-pixel [Fig. 1(d)] across a complementary metal oxide semiconductor sensor (NIR Ruby sensor, UI1242LE-NIR, IDS, square pixel size $5.3\mu \mathrm{m}$ matched to spectral filter size). The nine spectral filters comprise eight narrow bands of average full-width half-maximum 30 nm with center wavelengths 553, 587, 629, 665, 714, 749, 791, 829 nm, along with 1 broad band; 500 to 850 nm [Fig. 1(e)]. The optics were securely housed inside a light tight enclosure and mounted on an optical breadboard (MB4545/M, Thorlabs, Germany), which was fixed to a stainless-steel trolley (FW2901-3, Freeway Medical, United Kingdom) with a footprint of $512\mathrm{mm}\times 480\mathrm{mm}$, allowing the system to be easily and safely transported in a busy and often crowded clinical setting.

This distal end of the multispectral endoscope accessed the esophagus via the working channel of a therapeutic gastroscope, directed by the articulation of the gastroscope to collect diffuse reflectance images alongside standard-of-care imaging [Fig. 1(f)], thus causing minimal disruption to clinical workflow.

Endoscope settings were controlled using an interface developed in LabVIEW (National Instruments) running on a PC and a tablet (Surface Pro, Microsoft). Images captured by the SRDA were saved as 8-bit $1280\times 1024\text{-}\text{pixel}$ bitmap image files (to enable fast acquisition and immediate review of the images). These images contain two artifacts: a comb structure due to the imaging fiber bundle and a mosaic pattern due to the spectral filters of the SRDA [Fig. 2(a)]. For analysis, these were removed in a process of demosaicing and decombing as described previously,⁴⁵ resulting in a $1280\text{pixel}\times 1024\text{pixel}\times 9$ band image cube [Fig. 2(b)] with a spatial resolution of $240\pm 20\mu \mathrm{m}$ at a working distance of 1 cm. For real-time display, three of the nine bands were decombed and displayed as a false-RGB image at up to 15 fps. For analysis, columns of the image cube can be treated as nine-point “per-pixel spectra” [Fig. 2(c)]. As high-dimensional image data would be challenging for an operating clinician to interpret in real-time, machine-vision techniques can be used to classify per-pixel spectra to produce a tissue map for endoscopic image guidance [Fig. 2(d)].

Fig. 2

Data processing for snapshot multispectral endoscopy. (a) Raw collected data contain comb and mosaic artifacts from the imaging fiber and SRDA, respectively. (b) These are removed by a demosaicing and decombing algorithm to produce a spectral data cube. (c) For a given pixel in the image cube, a nine-band per-pixel spectrum can be determined. (d) Using these per-pixel spectra, machine-learning-based classification algorithms can interpret the image cube to generate a classification map, thus helping to guide the endoscopist.

2.2.

Color Chart Imaging

To test the accuracy of the snapshot multispectral endoscope in a controlled setting with known reference data, we performed handheld imaging of a Macbeth color chart (ColorChecker Classic Mini, x-rite) prior to four of the in vivo trials (Fig. 3). The color calibration target consists of 24 squares of painted samples, 18 of which contain colors whose spectral reflectance is intended to mimic those found in natural objects, and 6 of which are a uniform gray lightness scale.

Fig. 3

Color chart imaging. (a) A Macbeth color chart was used to validate the multispectral endoscope. (b) Handheld imaging of the Macbeth color chart took place prior to four of the in vivo trials. (c) The mean per-pixel spectra for each color on the Macbeth color chart are distinct, but overlap within their variation. Shaded region shown is standard deviation/5 for clarity. (d) PCA not only shows clustering of the per-pixel spectra by color but also shows the overlap of similar colors.

2.3.

First-in-Human Pilot Study of Snapshot Multispectral Endoscopy

The snapshot multispectral endoscope was deployed in a pilot clinical study to acquire in vivo images. This prospective pilot cohort study was carried out at the Cambridge Clinical Research Centre, Cambridge University Hospitals, United Kingdom. Eligible patients were adults (at least 18 years old) with a previous diagnosis of BE with an endoscopic length of at least 2 cm if circumferential (circumferential extent $C\ge 2$ according to the Prague classification) or at least 3 cm if not circumferential (maximal extent $M\ge 3$ according to the Prague classification). Exclusion criteria are given in the Supplementary Material. The trial was reviewed by Cambridgeshire Research Ethics Committee and was approved in March 2018 (18/NW/0134). The trial was registered at ClinicalTrials.gov (NCT03388047).

2.4.

Endoscopic Procedure and Histopathology

The clinical procedure undertaken is summarized in Fig. S2 in the Supplementary Material. Each procedure was performed by a single experienced endoscopist (MdP).

For analysis, the most advanced disease present in each biopsy determined the label for the region such that three classes of spectra were acquired: (1) normal squamous; (2) nondysplastic BE (NDBE); and (3) neoplasia (consisting of dysplasia and intramucosal carcinoma).

2.5.

Coregistration of Multispectral Images with Diagnosis from Histopathology

To enable matching of multispectral image data with the diagnosis from histopathology, the HR-WLE video stream and the video stream from the multispectral endoscope were synchronized and placed side-by-side for analysis. The HR-WLE video was carefully inspected to extract frames where the multispectral endoscope was sampling the marked regions of interest that were biopsied to give gold-standard diagnosis from histopathology. Since the esophageal lumen is quite featureless on MSI, the frames were identified using the cautery marks and other visible landmarks in the HR-WLE video stream to estimate the field of view of the multispectral endoscope. The tip of the multispectral endoscope was visible within the field of view of the HR-WLE endoscope, which facilitated identification of appropriate multispectral images that could be related to the gold-standard diagnosis made by histopathology.

2.6.

Spectral Data Processing

Briefly, raw images were checked for saturation ($\text{pixel values}>250$), dark subtracted then decombed, to account for the structure introduced to the image by the individual fiberlets, and demosaiced, to separate the nine spectral bands, creating an image cube as outlined previously.⁴⁵ In the image cube, low signal pixels were removed (max per-pixel $\text{spectrum value}<50$) as they are more likely to be affected by noise. Finally, each per-pixel spectrum was divided by a white light reference spectrum and normalized to the maximum of the per-pixel spectrum, such that the per-pixel spectra represent max-normalized reflectance. The white reference spectrum was an average of images captured from a white reflectance standard prior to each procedure to avoid variations introduced depending on experimental conditions.

For classification, the spatial dimensions were binned into $32\times 32\text{pixels}$ as a compromise between retaining spatial information and increasing classification accuracy by reducing noise (Fig. S3 in the Supplementary Material). This resulted in $40\text{-}\text{pixel}\times 32\text{-}\text{pixel}\times 9$-band multispectral image cubes. Each multispectral image cube therefore contains 1280 nine-band spectra. By averaging these across a region of interest drawn in the center of the image around the center of the fiber bundle, a mean spectrum was determined for each image (this occurs prior to reflectance normalization such that brighter pixels contribute more to the mean image spectrum). By averaging the spectra across images of the same tissue type within a given patient, mean “patient spectra” were determined. These different categories enabled us to test the performance of different classification methods on a per-pixel, per-image, or per-patient basis.

To quantify the variance in the data, standard deviations were calculated according to

2.7.

Classification of Multispectral Image Cubes to Discriminate Tissue Pathologies

For classification, the multispectral image cubes were randomly split into 80% for training and 20% for testing. Spectra calculated per-pixel and per-image were classified using seven methods commonly used in spectral data classification.⁵⁰ Linear discriminant analysis (LDA) classifies spectra by finding a linear combination of features that maximizes the separation between classes relative to within-class variance in the feature space. K-nearest neighbor (KNN) algorithms classify spectra by choosing the most frequent classes of KNN data points in the feature space ($k=3$, 5, and 7 were tested). Both LDA and KNN classifiers were applied using the nine-band spectrum directly as a nine-element feature vector. While LDA assumes linear decision boundaries, the KNN algorithm is nonparametric so makes no assumptions about the shape of the decision boundaries. LDA also assumes variables are Gaussian distributed. The data were also projected onto principal component axes using principal component analysis (PCA) and the weights used for KNN classification. Spectral angle mapping (SAM) calculates the $n$-dimensional angle between a target spectrum and a reference spectrum; in this case, $n=9$. The reference spectra are the mean spectra per-pathology within the training dataset; thus three spectral angles are calculated for each target spectrum — ${\theta }_{\text{target-squamous}}$, ${\theta }_{\text{target-}\mathrm{NDBE}}$, and ${\theta }_{\text{target-neoplasia}}$. For a simple-SAM classification, the minimum of these three angles was taken as the predicted class. Alternatively, the angles were treated as a three-element feature vector and classified using LDA or KNN.

Finally, a neural network (NN) was used for classification. NNs perform classification by passing an input vector, in this case a nine-band spectrum, through a series of artificial neurons, with each neuron outputting some nonlinear function of its inputs with some weight that is adjusted during training. The output values of the final layer determine the classification. In contrast to LDA, NN classification does not make assumptions about the distribution of input data nor the shape of decision boundaries. The NN was implemented using a two-layer feedforward network, with a sigmoid transfer function in the hidden layer (10 neurons) and a linear transfer function in the output layer, using the MATLAB (MathWorks) Neural Network Pattern Recognition app. The 20% testing image cubes were randomly split into 50% for validation and 50% for testing (10% of all image cubes each). In summary, the seven classification methods compared were LDA, KNN, PCA-KNN, simple-SAM, SAM-LDA, SAM-KNN, and NN.

Accuracy for per-pixel classification was calculated in three ways: per-pixel accuracy, the percentage of pixels classified correctly over all pixels; per-image accuracy, the average percentage of pixels classified correctly in an image; and majority-pixel-per-image, the percentage of correctly classified images based on a relative majority decision (plurality) of all pixels within an image (if there is a tie between two classes, the image was counted as incorrectly classified). Accuracy for per-image-spectrum was the percentage of correctly classified mean-image-spectra.

3.1.

Clinical Study Recruitment

Between May 2018 and December 2019, a total of 20 subjects were recruited to this pilot clinical trial (Table S1 in Supplementary Material). All patients provided written informed consent. Of these, one subject was considered unfit for endoscopic procedure due to concomitant acute comorbidity and one was excluded as the visible lesion was too small for spectral imaging. Of the 18 subjects that underwent standard-of-care endoscopic procedures, three were excluded from analysis because of insufficient illumination for spectral imaging using the first prototype ($n=2$) and failure of the standard-of-care recording unit ($n=1$).

The multispectral endoscope was deployed to acquire in vivo esophageal image cubes. The study design prioritized the collection of image cubes matched to gold-standard diagnosis from histopathology (Fig. S1 in the Supplementary Material). To do this, cautery marks were made in two distinct regions of the esophagus, one deemed suspicious and one a control (Fig. S2A in Supplementary Material). The marked regions were inspected with the multispectral endoscope (Fig. S2B in the Supplementary Material), then the regions were biopsied (Fig. S2C in the Supplementary Material) and diagnosed based on the histopathologic analysis of the biopsies (Fig. S2D in the Supplementary Material).

Data were collected in 15 subjects at 44 distinct regions: 9 regions of squamous tissue, 24 regions of NDBE, and 11 regions of neoplasia. Of the 35 BE tissue regions, 34 were confirmed with histopathological assessment from 45 collected samples (33 biopsies, 14 endoscopic mucosal resections). In patient number 12, the control region identified on endoscopy was diagnosed as dysplastic by histopathology, so spectra that were incidentally captured from another region of NDBE were taken as NDBE without confirmation from histopathology. Coregistration between imaging data and histopathological diagnosis was successful for 30 regions (20 NDBE and 10 neoplasia), yielding a total of 570 labeled image cubes for analysis, plus an additional 157 image cubes from nine squamous regions (Table 1, Fig. S4 in the Supplementary Material). The mean exposure time was $200\pm 50\mathrm{ms}$. MSI lengthened the procedure time by $<10\min$ per trial (data captured for mean of $5.2\pm 1.5\min$ per trial).

Table 1

Summary of the collected image cubes and biopsies for matched regions.

3.2.

Machine Learning Enables Classification of Reference Color Spectra

To validate the performance of the snapshot multispectral endoscope, handheld imaging of a Macbeth color chart was performed prior to imaging of four patients, resulting in 568 image cubes totaling 46,459 spectra for per-pixel classification [Figs. 3(a) and 3(b)]. The mean spectra per-pixel show distinct shapes but with significant overlap [Fig. 3(c)]. For classification, 454 image cubes (36,714 spectra) were used for training and 114 image cubes (9,745 spectra) were used for testing. Visualization of the training spectra by PCA not only shows clustering by color but also demonstrates the overlap between classes [Fig. 3(d)]. Despite this overlap, the classification was possible with 89.5% accuracy on a per-pixel basis ($96.1\pm 11.4\%$ mean ± standard deviation per-image) and up to 99.1% when taking a majority decision per-image using PCA-KNN with $k=3$ [Fig. 4(a)]. On per-image-spectrum classification basis, the accuracy was 96.5%. As expected, per-pixel classification results for each color show that misclassification often occurred between similar shades [Fig. 4(b)] and at the edges of the illuminated area, where noise is more apparent [Fig. 4(c)]. Examples of classified image cubes representing colors from the middle row of the Macbeth color chart are shown in Fig. 4(c).

Fig. 4

Machine learning enables classification of multispectral images of color chart. (a) Seven classification approaches were compared for classification of color chart image cubes. The best classification approach was PCA-KNN. (b) Per-pixel accuracies are shown as pie charts for each color in the Macbeth color chart. (c) Example classification maps are shown for $40\text{-}\text{pixel}\times 32\text{-}\text{pixel}\times 9\text{-}\text{band}$ image cubes of the middle color row of the Macbeth color chart. White arrows highlight misclassified pixels, which often lie close to the edge of the illuminated region. *The dark hole in the center of the “moderate red” image is a region of specular reflection. **Many of the “orange yellow” pixels are misclassified but are of a similar shade to the correct “orange yellow” color.

3.3.

Spectra Show Promising Differences Between Tissue Types Despite Considerable Intra- and Interpatient Variation

We first evaluated the variance within our data. There is considerable variance among the spectra, per-pixel [Fig. 5(a)], per-image [Fig. 5(b)], and per-patient [Fig. 5(c)], but the average spectra across patients show clear differences among different histopathological categories [Fig. 5(d)]. We therefore sought to better understand the nature of the variance in the dataset (Fig. 6). The variance over all per-patient spectra, and over all per-pixel spectra, increases with progression of disease (0.213, 0.246, 0.321 for per-patient spectra and 0.459, 0.474, 0.479 for per-pixel spectra in regions of squamous, NDBE, and neoplasia, respectively). This is likely due to the increasing heterogeneity of the diseased tissues compared with healthy squamous tissue. Interestingly, the per-image variance within-patients is largest in NDBE. This might be due to intrapatient heterogeneity as multiple distinct regions of NDBE were imaged in four of the patients. It might also reflect the mosaic of cell types found in NDBE. The overall variance of spectra calculated per-image is smaller than the overall variance in the per-pixel spectra (0.226 versus 0.459, 0.385 versus 0.474, 0.276 versus 0.479 for squamous, NDBE, and neoplasia, respectively), perhaps due to noise removal by averaging in the images [Fig. 6(a)]. The same effect can be observed in PCA plots, where the spectra per-pixel [Fig. 6(b)] cluster less well than the spectral per-image [Fig. 6(c)]. Spectra per-pixel and spectra per-image cluster by pathology [Figs. 6(b) and 6(c)] and by patient [Figs. 6(d) and 6(e)], which suggests significant interpatient differences in the dataset, further supported by the standard deviation calculations [Fig. 6(a)]. The within-patient variance in per-image spectra is smaller than the total variance (0.180 versus 0.226, 0.227 versus 0.385, 0.192 versus 0.276 for squamous, NDBE, and neoplasia, respectively), suggesting there are notable interpatient differences, which would make generalizable classification challenging.

Fig. 5

Spectra show promising differences between tissue types. (a) Per-pixel spectra for a single image of NDBE in patient 16. (b) All per-image spectra for patient 16. (c) Per-patient spectra for patient 16 and patient 11 are shown. The shaded region is the standard error from the standard across per-image spectra. (d) The trial-wide spectra. The shaded region is the standard error from the standard deviation across per-patient spectra.

Fig. 6

Spectra show considerable variance between patients. (a) The standard deviation is shown for per-pixel spectra, per-image spectra, and per-patient spectra for each pathology class. (b) PCA for per-pixel spectra shows clustering by pathology but with considerable overlap. (c) PCA of per-image spectra shows a more defined clustering per-pathology. Both (d) per-pixel spectra and (e) per-image spectra show visible clustering by patient, making clear the interpatient heterogeneity in the dataset.

3.4.

Snapshot Multispectral Endoscopy Shows Promising Accuracy for In Vivo Tissue Classification

For classification, the patient data were split into training data comprising of 581 multispectral image cubes (121,600 spectra) and test data comprising of 146 image cubes (31,405 spectra). Seven classification methods were compared [Fig. 7(a)]. Despite the substantial aforementioned variance in the dataset, the classification per-pixel was possible with 74.2% accuracy ($71.2\pm 20.2\%$ mean ± standard deviation per-image), rising to 87.0% accuracy when taking a majority pixel decision per-image using PCA-KNN with $k=3$ [Fig. 7(a)]. Per-image-spectrum classification increased accuracy to 96.5%. The classification results for each pathology show that squamous tissue was most reliably classified, whereas neoplasia was most difficult to classify [Fig. 7(b)]. This is in line with our expectations given the variance described in the dataset, as well as the familiar clinical challenge of distinguishing neoplasia. Nevertheless, on majority-pixel-per-image classification, accuracies of 95.8%, 90.7%, and 76.1% were achieved for squamous, NDBE, and neoplasia, respectively, and on per-image-spectrum classification, these rose to 95.8%, 98.6%, and 95.5%, respectively. Confusion matrices are shown alongside calculated values of sensitivity, specificity, positive predictive value, and negative predictive value for neoplasia classification in Fig. S5 and Table S2 in the Supplementary Material.

Fig. 7

Snapshot multispectral endoscopy shows promising accuracy for in vivo tissue classification. (a) Seven approaches were compared for tissue classification. (b) The best classification approach was PCA-KNN with $k=3$. Per-pixel, majority-pixel-per-image, and per-image-spectrum accuracies are shown for each pathology. Black sectors represent images where majority pixel decision could not be reached. (c) Example classification maps of $40\text{-}\text{pixel}\times 32\text{-}\text{pixel}\times 9\text{-}\text{band}$ image cubes are shown for each pathology alongside the matched false-RGB image generated by assigning bands 2, 3, and broadband to R, G, and B colors, respectively. *Specular reflections are excluded from the classification and marked as white in the classification maps.

We then created classification maps representing the three pathologies [Fig. 7(c)] to display alongside RGB images, composed by assigning bands 3, 2 and 1 to red, green and blue, respectively. For visualization, map pixels whose four nearest-neighbor pixels are all a different class (to the said pixel) are defined as noise and replaced by the mode class of the four nearest-neighbor pixels. The maps illustrate how classification data could be presented to an operating endoscopist in real-time.

To calculate sensitivity, specificity, positive predictive value, and negative predictive value for detection of neoplasia in BE, the classification models were retrained for two-way classification using only NDBE and neoplastic image cubes (Fig. S6 and Table S3 in the Supplementary Material). For per-pixel classification PCA-KNN with $k=3$ achieved 60.7% sensitivity, 88.1% specificity, 71.7% positive predictive value, and 81.8% negative predictive value. In per-image-spectrum classification, this increased to 97.7% sensitivity, 100% specificity, 100% positive predictive values, and 98.6% negative predictive value.