This PDF file contains the front matter associated with SPIE Proceedings Volume 12471, including the Title Page, Copyright information, Table of Contents, and Conference Committee information.

High-grade gliomas are aggressive forms of brain cancer associated with a poor prognosis of 12-15 months. The mutation of isocitrate dehydrogenase I (IDH) and O6-methylguanine-DNA-methyltransferase (MGMT) promoter methylation are biomarkers relevant to disease classification and prognosis. Current methods to identify patients’ molecular status are expensive and can be resource prohibitive. Morphological attributes as captured on histopathology contain rich phenotypic information indicative of the underlying molecular processes. Recently, deep learning methods have demonstrated diagnostic and prognostic value via computational analysis of histopathology to incorporate complex and subvisual morphometric features that may not be visually accessible to pathologists. We hypothesize that a computational deep learning approach applied to Hematoxylin and Eosin (H&E)-stained digitized tissue slides will be able to reliably predict the molecular (IDH, MGMT) status in high-grade glioma patients. Specifically, we present a deep learning approach that employs self-supervised and multiple instance learning, on a total of n=325 H&E stained high-grade glioma slides, for identifying (a) IDHmutant versus IDH-wild-type (WT) and (b) MGMT promoter methylated versus unmethylated tumors. The approach addressed challenges of patch selection and the unbalanced instances that arise from only subregions of whole-slide images providing diagnostic value. The deep learning approach achieved accuracy values of 91.17 (+/- 3.47) and 86.11 (+/- 4.45), for the prediction of IDH mutation and MGMT promoter methylation status respectively, demonstrating an improvement of over 5% compared to the reported accuracy values in previous studies.

Abnormally phosphorylated tau proteins are known to be a major indicator of Alzheimer’s Disease (AD) with strong association with memory loss and cognitive decline. Automated generation of pixel-wise accurate neurofibrillary tangles (NFTs) and neuropil threads (NTs) segmentation is a challenging task, due to lack of ground truth segmentation data of these abnormal tau pathology. This problem is most prominent in the case of segmenting NTs, where the small threadlike morphology makes pixel-wise labeling a laborious task and unrealistic for large-scale studies. Lack of ground truth data poses a significant limitation for many learning-based methods to generate accurate segmentations of NFTs and NTs. This work presents an automated pipeline for pixel level segmentation of NFTs and NTs that does not rely on ground truth segmentation data. The pipeline is composed of four main steps: (1) color deconvolution is used to separate histopathology images into staining channels (DAB, Hematoxylin, and Eosin), (2) Otsu’s thresholding is used on the DAB stain channel to generate pixel level segmentation of abnormal tau proteins staining, (3) a weakly-supervised learning paradigm (WildCat), using only global descriptors of images, is used to generate density maps of potential regions of NFTs and NTs, and (4) density maps and segmentations are then integrated using connected component analysis to localize NFTs and NTs in the detected tau segmentations. Our results show high global classification accuracy for NFTs (Acc:0.96) and NTs (Acc:0.91), and statistically significant distinctions when evaluating the percent area occupied of the detected NTs relative to expert ratings of NTs severity. Qualitative assessment of the NFTs and NTs results showed accurate pixel-level segmentations of the NFTs, while modest performance for NTs.

CMYC positivity is an important prognostic factor for diffuse large B-cell lymphoma. However, manual quantification of CMYC can be subjective and may show intra- and inter-observer variability. Therefore, we sought to develop an automated method to quantify CMYC. Our method applies attention-based multiple instance learning to regress the proportion of CMYC positive tumor cells from pathologist-scored tissue microarray cores. The results of our experiments indicate a high Pearson correlation of 0.8421+/-0.1268. Additionally, we show that regardless of cross-validation methodology, this correlation remains relatively stable. When utilizing a standard clinical threshold of 40% for positivity, our method results in a sensitivity and specificity of 0.7600 and 0.9595. Finally, using clinical outcomes, we found that regressions provided more significant and robust stratification when compared to pathologist scoring. We conclude that proportion of positive stain can be regressed using attention-based multiple instance learning.

In situ RNA capturing represents an excellent opportunity for bridging transcriptomic data to a spatial domain, making it possible to map the gene expression to the corresponding anatomical structure. As a result, scientists can better understand the transcriptional heterogeneity with spatially resolved, anatomical, and pathophysiological contexts. However, high throughput sequencing technologies paired with histological images suffer from lower resolution mapping between transcriptome and imaging data. Here, we present Spatial Transcriptome Auto-encoder and Deconvolution (ST-AnD), a scalable deep generative model for predicting gene expression at cellular or nuclei level based on H&E imaging and in situ RNA capturing, thus allowing a better understanding of the tissue microenvironment.

ER, PR (estrogen, progesterone receptor), and HER2 (human epidermal growth factor receptor 2) status are assessed using immunohistochemistry and reported in standard clinical workflows as they provide valuable information to help treatment planning. The protein Ki67 has also been suggested as a prognostic biomarker but is not routinely evaluated clinically due to insufficient quality assurance. The routine pathological practice usually relies on small biopsies, such that the reduction in consumption is necessary to save materials for special assays. For this purpose, we developed and validated an automatic system for segmenting and identifying the (ER, PR, HER2, Ki67) positive cells from hæmatoxylin and eosin (H&E) stained tissue sections using multiplexed immunofluorescence (MxIF) images at cellular level as a reference standard. In this study, we used 100 tissue-microarray cores sampled from 56 cases of invasive breast cancer. For ER, we extracted cell nucleus images (HoverNet) from the H&E images and assigned each cell nucleus as ER positive vs. negative based on the corresponding MxIF signals (whole cell segmentation with DeepCSeg) upon H&E to MxIF image registration. We trained a Res-Net 18 and validated the model on a separate test-set for classifying the cells as positive vs. negative for ER, and performed the same experiment for the other three markers. We obtained area-under-the- receiver-operating-characteristic-curves (AUCs) of 0.82 (ER), 0.85 (PR), 0.75 (HER2), 0.82 (Ki67) respectively. Our study demonstrates the feasibility of using machine learning to identify molecular status at cellular level directly from the H&E slides.

Hyperspectral imaging is a label-free and non-invasive imaging modality that seeks to capture images in different wavelengths. In this study, we used a vision transformer that was pre-trained from video data to detect thyroid cancer on hyperspectral images. We built a dataset of 49 whole slide hyperspectral images (WS-HSI) of thyroid cancer. To improve training, we introduced 5 new data augmentation methods that transform spectra. We achieved an F-1 score of 88.1% and an accuracy of 89.64% on our test dataset. The transformer network and the whole slide hyperspectral imaging technique can have many applications in digital pathology.

We established a translation dataset that contains pixel-wise registered H&E and multiplexed immunohistochemistry (mIHC) staining images. Deep learning models were trained to translate H&E inputs into their corresponding mIHC image versions. Comparison experiments have been carried out to validate the translation performance between TransUNet, U-Net, and pix2pix models. We also compared the impact of different Losses on model performances. The TransUNet model could achieve 0.862 SSIM score for L1 loss and 0.805 for L2 loss, surpassing U-Net and pix2pix model in both settings. This demonstrates the potential benefit of the Transformer module in stain translation tasks.

Hyperspectral imaging (HSI) is an emerging modality for digital pathology. The purpose of this study is to develop an extended depth of field (EDOF) method for mosaic hyperspectral images acquired with a snapshot camera. EDOF is a technique for ensuring that an image is in focus at all points. A stack of mosaicked hyperspectral images of hematoxylin and eosin (H&E)-stained histologic slides were acquired at different positions along the z-axis and used to output a hyperspectral histologic image that was in-focus at every point. Three different methods were compared to achieve a fully focused image. We compared conventional patch-based methods to our proposed growth-based and band-based methods. The Brenner function was used to quantitatively measure the focus quality of each image measured. The results show that both of our proposed methods performed better qualitatively and quantitatively than the patch-based method, with the band-based method performing the best, as it leveraged dividing pixels into their proper wavelengths in addition to spatially, giving the algorithm better contrast to measure. In terms of speed, the band-based method was the fastest, followed by the patch-based method, with the growth-based method being the slowest. Our proposed extended depth of field hyperspectral imaging methods can have immediate applications in digital pathology, especially whole slide imaging, and other microscopic imaging.

The Tangram algorithm is a benchmarking method of aligning single-cell data to various forms of spatial data collected from the same region. With this data alignment, the annotation of the single-cell data can be projected to spatial data. However, the cell composition of the single-cell data and spatial data might be different because of heterogeneous cell distribution. Whether the Tangram algorithm can be adapted when the two data have different cell-type ratios has not been discussed in previous works. In our practical application that maps the cell-type classification results of single-cell data to the Multiplex immunofluorescence spatial data, cell-type ratios were different. In this work, both simulation and empirical validation were conducted to quantitatively explore the impact of the mismatched cell-type ratio on the Tangram mapping in different situations. Results show that the cell-type difference has a negative influence on annotation mapping accuracy.

Multiplex immunofluorescence (MxIF) is an emerging imaging technology whose downstream molecular analytics highly rely upon the effectiveness of cell segmentation. In practice, multiple membrane markers (e.g., NaKATPase, PanCK and β-catenin) are employed to stain membranes for different cell types, so as to achieve a more comprehensive cell segmentation since no single marker fits all cell types. However, prevalent watershed-based image processing might yield inferior capability for modeling complicated relationships between markers. For example, some markers can be misleading due to questionable stain quality. In this paper, we propose a deep learning based membrane segmentation method to aggregate complementary information that is uniquely provided by large scale MxIF markers. We aim to segment tubular membrane structure in MxIF data using global (membrane markers z-stack projection image) and local (separate individual markers) information to maximize topology preservation with deep learning. Specifically, we investigate the feasibility of four SOTA 2D deep networks and four volumetric-based loss functions. We conducted a comprehensive ablation study to assess the sensitivity of the proposed method with various combinations of input channels. Beyond using adjusted rand index (ARI) as the evaluation metric, which was inspired by the clDice, we propose a novel volumetric metric that is specific for skeletal structure, denoted asclDice_SKEL. In total, 80 membrane MxIF images were manually traced for 5-fold cross-validation. Our model outperforms the baseline with a 20.2% and 41.3% increase in clDice_SKEL and ARI performance, which is significant (p<0.05) using the Wilcoxon signed rank test. Our work explores a promising direction for advancing MxIF imaging cell segmentation with deep learning membrane segmentation. Tools are available at https://github.com/MASILab/MxIF_Membrane_Segmentation.

Imaging Mass Cytometry (IMC) is an emerging multiplexed imaging pathology technology which can detect the spatial distribution of up to 40 markers at cellular resolution. However, the high content and complexity of this spatial information is underutilized in downstream analysis. To overcome this limitation, we develop an interpretable graph convolutional network (GCN) trained by IMC data with 30 cell markers from 238 patient breast cancer samples, with both marker expressions and cell locations. The network enables triple negative breast cancer (TNBC) classification from other clinical types, including HR+HER2+, HR+HER- and HR-HER+. More importantly, with an embedded self-attention pooling module, cell communities with high diagnostic values can be detected based on the attention scores. The proposed GCN framework is benchmarked with a fully connected artifical neural network (ANN) without spatial information. With a stratified 8-fold cross validation, GCN performs slightly better than ANN for tissue-level classification (class balanced accuracy: 0.8283±0.0964 to 0.8123±0.0989; area under the curve: 0.8548±0.1252 to 0.8298±0.1407). Nevertheless, GCN remarkably outperforms ANN on potentially interested cell community detection, especially in TNBC tissues, regarding the spearman correlation coefficient (SCC) between attention scores and marker expressions. The average SCC differences between GCN and ANN range from 0.0532 to 0.1876 for Cytokeratin 5, 7, 14, 8/18, 19, and pan Cytokeration. With comparisons of selected markers on tissues with different clinical types and attention scores, the cell marker expressions correlate with their clinical types and diagnostic values, which further validate the proposed framework. Overall, our GCN enables interpretable triple negative breast cancer detection and has the potential to be widely implemented in other diseases and highly multiplexed imaging techniques for enhanced microenvironment analysis.

Quantifying elastic fiber in the tissue specimen is an important aspect of diagnosing different diseases. In conventional pathology, special staining technique such as EVG (Verhoeff’s Van Gieson) is applied physically for this purpose which is expensive and time-consuming procedure. Though H&E (Hematoxylin and Eosin) staining is routinely used, less expensive and most common tissue staining technique, elastic and collagen fibers cannot be differentiated using it. This study proposes a modified CycleGAN based unsupervised method for the computerized generation of RGB EVG stained tissue from hyperspectral H&E stained one to save the time and cost of conventional EVG staining procedure. Our proposed method is designed to utilize the sufficient spectral information provided by the H&E hyperspectral image (HSI) without reducing the spectral dimension. For doing so, we have faced challenges to calculate one of the training losses (identity loss) of CycleGAN that requires reducing the channel dimension of H&E HSI to be the same as RGB EVG stained image. We have addressed the issue by adopting intentionally designed three basis functions that can reduce the channel dimension of HSI into three without losing the essential color of elastic fibers. The set of this function includes Linear Discriminant Function (LDF) and the transmittance spectrum of Eosin and Hematoxylin which has proved to best preserve the underlying important features of EVG stained image while reducing the dimensionality of hyperspectral H&E. The experimental result proves the feasibility of our proposed method to generate realistic EVG stained image from its corresponding H&E stained one.

Quantitative phase imaging (QPI) is an advanced label-free imaging technique that can quantify pathlength changes in biological samples at the nanometer scale without staining or tagging. White blood cells (WBCs) are the key components of the body’s immune system. However, automatic detection and segmentation of WBCs from QPI data can be challenging as QPI may lack the needed intrinsic specificity and contrast compared to a stained brightfield image. Morever, typical supervised learning methods require a large number of annotations for model training which involves time-consuming efforts and clinical expertise, and may not be available often. Motivated by the effectiveness of the recently developed contrastive learning-based semi-supervised segmentation methods and considering the intrinsic properties of QPI, the main contribution of this work is to implement an efficient semi-supervised segmentation method to segment WBCs from QPI data when only a small amount of ground-truth labels are available. A DeeplabV3+ and DeeplabV2 architecture were used as a backbone of the segmentation network with a modified convolution module, specifically to capture the subtle morphological details of WBCs in QPI. The preliminary results showed significant improvement in terms of quantitative metrics compared to the supervised training methods. An exploratory study was also performed to investigate the effect of the contrastive learning module on segmentation performance. To the best of our knowledge, this is the first application of the semi-supervised contrastive learning method in the WBC segmentation framework from QPI.

Tissue examination and quantification in a 3D context on serial section whole slide images (WSIs) were labor-intensive and time-consuming tasks. Our previous study proposed a novel registration-based method (Map3D) to automatically align WSIs to the same physical space, reducing the human efforts of screening serial sections from WSIs. However, the registration performance of our Map3D method was only evaluated on single-stain WSIs with large-scale kidney tissue samples. In this paper, we provide a Docker for an end-to-end 3D slide-wise registration pipeline on needle biopsy serial sections in a multi-stain paradigm. The contribution of this study is three-fold: (1) We release a containerized Docker for an end-to-end multi-stain WSI registration. (2) We prove that the Map3D pipeline is capable of sectional registration from multi-stain WSI. (3) We verify that the Map3D pipeline can also be applied to needle biopsy tissue samples. The source code and the Docker have been made publicly available at https://github.com/hrlblab/Map3D.

Accurate quantification of renal fibrosis has profound importance in the assessment of chronic kidney disease (CKD). Visual analysis of a biopsy stained with trichrome under the microscope by a pathologist is the gold standard for evaluation of fibrosis. Trichrome helps to highlight collagen and ultimately interstitial fibrosis. However, trichrome stains are not always reproducible, can underestimate collagen content and are not sensitive to subtle fibrotic patterns. Using the Dual-mode emission and transmission (DUET) microscopy approach, it is possible to capture both brightfield and fluorescence images from the same area of a tissue stained with hematoxylin and eosin (H&E) enabling reproducible extraction of collagen with high sensitivity and specificity. Manual extraction of spectrally overlapping collagen signals from tubular epithelial cells and red blood cells is still an intensive task. We employed a UNet++ architecture for pixel-level segmentation and quantification of collagen using 760 whole slide image (WSI) patches from six cases of varying stages of fibrosis. Our trained model (Deep-DUET) used the supervised extracted collagen mask as ground truth and was able to predict the extent of collagen signal with a MSE of 0.05 in a holdout testing set while achieving an average AUC of 0.94 for predicting regions of collagen deposits. Expanding this work to the level of the WSI can greatly improve the ability of pathologists and machine learning (ML) tools to quantify the extent of renal fibrosis reproducibly and reliably.

The detection of prostate cancer within H&E-stained tissue slides helps identify the presence, location, and extent of the disease. Machine learning approaches have been developed to accomplish this task for biopsies and radical prostatectomies. Deep learning approaches, primarily using convolutional neural networks (CNN), have demonstrated substantial ability to identify cancer in each of these sample types. Some models trained on biopsies have even been approved for clinical application. Biopsies are small tissue samples acquired through a syringe to establish cancer diagnosis while radical prostatectomies are large planar sections of prostate tissue acquired post-surgery. This leads to differing morphology and heterogeneity between sample types, making it unclear whether algorithms trained on biopsies are robust enough to perform well on radical prostatectomies and vice versa. Our goal was to investigate whether morphological differences between sample types affected the performance of radical prostatectomy trained CNN models when applied to biopsies and vice versa. Radical prostatectomies (N=100) and biopsies (N=50) were acquired from The University of Pennsylvania to train (80%) and validate (20%) a CNN using the Densenet architecture for biopsies (M^B), radical prostatectomies (M^R), and a combined dataset (M^B+R). Model performance was compared using sensitivity, specificity, and F1 score. To isolate the effect of morphological differences on performance, we acquired all data from the same institution, ensured no batch effects were present via UMAP, applied data augmentation during training, and applied stain normalization. Additionally, the radical prostatecomy training set was optimized for transfer to biopsies by reducing heterogeneity (M^R*). This reduction was based on distance from the centroid of a texture feature distribution. Models performed well when applied to their own sample type (F1 > 0.88) but performed poorly when applied across sample types (F1 < 0.65). Reducing heterogeneity to optimize the radical prostatectomy training set resulted in M^R* (F1=0.72) noticeably outperforming M^R (F1=0.53) on biopsies. This indicates that differences in morphology and heterogeneity drive performance differences between cancer detection models trained on different sample types. Our results suggest that these morphological differences can be overcome with sample type specific models or training set optimization methods.

Measuring hyperplasia in Atlantic salmon gills can give important insight into fish health and environmental conditions such as water quality. This paper proposes a novel histology image classification technique to identify hyperplastic regions using an emerging signal decomposition technique, Empirical Wavelet Transform (EWT) in combination with a fully connected neural network (FCNN). Due to its adaptive nature, we hypothesise and show that EWT effectively represents unique features of gill histopathology whole slide images that help in the classification task. Our hybrid approach is unique and significantly outperformed regular deep learning-based methods considering a joint speed-accuracy metric.

Digital and computational pathology tools often suffer from a lack of relevant data. Although more and more data centers publicize the datasets, high-quality ground truth annotations may not be available in a timely manner. Herein, we propose a knowledge distillation framework that can utilize a teacher network that is already trained on a relatively larger amount of data and achieve accurate and robust performance on histopathology images by a student network. For an effective and efficient knowledge distillation, we introduce a quintet margin loss that pushes the student network not only to mimic the knowledge representation of the teacher network but also to outperform the teacher network on a target domain. We systematically evaluated the proposed approach. The results show that the proposed approach outperforms other competing models with and without different types of knowledge distillation methods.

Diabetic nephropathy (DN) in the context of type 2 diabetes is the leading cause of end-stage renal disease (ESRD) in the United States. DN is graded based on glomerular morphology and has a spatially heterogeneous presentation in kidney biopsies that complicates pathologists’ predictions of disease progression. Artificial intelligence and deep learning methods for pathology have shown promise for quantitative pathological evaluation and clinical trajectory estimation; but, they often fail to capture large-scale spatial anatomy and relationships found in whole slide images (WSIs). In this study, we present a transformer-based, multi-stage ESRD prediction framework built upon nonlinear dimensionality reduction, relative Euclidean pixel distance embeddings between every pair of observable glomeruli, and a corresponding spatial self-attention mechanism for a robust contextual representation. We developed a deep transformer network for encoding WSI and predicting future ESRD using a dataset of 56 kidney biopsy WSIs from DN patients at Seoul National University Hospital. Using a leave-one-out cross-validation scheme, our modified transformer framework outperformed RNNs, XGBoost, and logistic regression baseline models, and resulted in an area under the receiver operating characteristic curve (AUC) of 0.97 (95% CI: 0.90-1.00) for predicting two-year ESRD, compared with an AUC of 0.86 (95% CI: 0.66-0.99) without our relative distance embedding, and an AUC of 0.76 (95% CI: 0.59-0.92) without a denoising autoencoder module. While the variability and generalizability induced by smaller sample sizes are challenging, our distance-based embedding approach and overfitting mitigation techniques yielded results that suggest opportunities for future spatially aware WSI research using limited pathology datasets.

Deep learning (DL) systems obtain high accuracy on digital pathology datasets that are within the same distribution as the training set. When applied to unseen datasets, performance degradation occurs due to differences in acquisition hardware/software and staining protocols/vendors. This issue poses a barrier to translation since developed models cannot be readily deployed at new labs. To overcome this challenge, we present silver standard (SS) annotations as a method to improve the performance of deep learning architectures on unseen Ki67 pathology images. An unsupervised technique referred to as IHCCH was used to generate SS masks for Ki67⁺ and Ki67⁻ nuclei from the target lab. A previously validated architecture for Ki67, UV-Net, is trained with a combination of the gold standard (GS) and SS masks to enhance performance consistency. It was found that adding SS masks from the unseen center to the training pool improved performance over clinically relevant PI ranges.

Spectral decoupling, weight normalization, and L1 loss have been applied to varying degrees in studies concerning computational pathology. These methods induce sparsity and tend to improve overall model performance. However, no work has combined these methods to improve model performance. The work presented here combines and compares these three methods in an attention-based multiple instance learning model to classify whole slide histopathology images from Camelyon16. We observed that spectral decoupling improves accuracy and area under the curve (AUC), but that weight normalization and L1 loss do not. However, when either of the latter is combined with spectral decoupling, accuracy, and AUC further improve over just spectral decoupling. Finally, we demonstrate that varying the magnitude with which these three methods affect model training considerably affects the resulting testing accuracy and AUC.

Whole slide image (WSI) classification methods typically use fixed-size patches that are processed separately and are aggregated for the final slide-level prediction. Image segmentation methods are designed to obtain a delineation of specific tissue types. These two tasks are usually studied independently. The aim of this work is to investigate the effect of region of interest (ROI) detection as a preliminary step for WSI classification. First, we process each WSI by using a pixel-level classifier that provides a binary segmentation mask for potentially important ROIs. We evaluate both single-resolution models that process each magnification independently and multi-resolution models that simultaneously incorporate contextual information and local details. Then, we compare the WSI classification performances of patch-based models when the patches used for both training and testing are extracted from the whole image and when they are sampled from only within the detected ROIs. The experiments using a binary classification setting for breast histopathology slides as benign vs. malignant show that the classifier that uses the patches sampled from the whole image achieves an F1 score of 0.68 whereas the classifiers that use patches sampled from the ROI detection results produced by the single- and multi-resolution models obtain scores between 0.75 and 0.83.

Detecting the background is often one of the first tasks when analyzing whole slide images. However, these methods greatly vary and are generally not reported in sufficient detail for subsequent studies to be able to reproduce. Therefore, we sought to determine the effect of varying background detection methods on downstream performance metrics. Specifically, we applied attention-based multiple instance learning, CLAM, and Attention2majority to classify whole slide images from Camelyon16 using four different background detection methods. Our results show that performance metrics (i.e., accuracy and area under the curve - AUC) differ when different background detection methods are utilized. Furthermore, all classification methods perform worse when using hand-drawn tissue annotations for background detection. Finally, we show that depending on the background detection method, the best performing method changes. We conclude that background detection methods must be fully reported so that results can be accurately reproduced and techniques can be fairly compared.

Fluorescence imaging is used to visualize biological processes at molecular and cellular level. However, it has several limitations e.g., potential toxicity of fluorescent dyes, photobleaching and sensitivity to the environment. On the contrary, differential interference contrast (DIC) microscopes provide pseudo-3D images by enhancing contrast in unstained specimens and are harmless, and non-invasive compared to fluorescence imaging. This study proposes a massive-training artificial neural network (MTANN) scheme to generate simulated fluorescence images from DIC images. Experimental results showed that the proposed method generated fluorescence images of the proteins (sum of troponin T and vimentin) with SSIM value 0.878 which is close to the corresponding ‘gold standard’ images. Thus, it can be said that the proposed method contributes to harmless cell imaging.

Deep-learning techniques have been used widely to alleviate the labour-intensive and time-consuming manual annotation required for pixel-level tissue characterization. Our previous study introduced an efficient single dynamic network - Omni-Seg - that achieved multi-class multi-scale pathological segmentation with less computational complexity. However, the patch-wise segmentation paradigm still applies to Omni-Seg, and the pipeline is time-consuming when providing segmentation for Whole Slide Images (WSIs). In this paper, we propose an enhanced version of the Omni-Seg pipeline in order to reduce the repetitive computing processes and utilize a GPU to accelerate the model’s prediction for both better model performance and faster speed. Our proposed method’s innovative contribution is two-fold: (1) a Docker is released for an end-to-end slide-wise multi-tissue segmentation for WSIs; and (2) the pipeline is deployed on a GPU to accelerate the prediction, achieving better segmentation quality in less time. The proposed accelerated implementation reduced the average processing time (at the testing stage) on a standard needle biopsy WSI from 2.3 hours to 22 minutes, using 35 WSIs from the Kidney Tissue Atlas (KPMP) Datasets. The source code and the Docker have been made publicly available at https://github.com/ddrrnn123/Omni-Seg.

Automatic cell quantification in microscopy images can accelerate biomedical research. There has been significant progress in the 3D segmentation of neurons in fluorescence microscopy. However, it remains a challenge in bright-field microscopy due to the low Signal-to-Noise Ratio and signals from out-of-focus neurons. Automatic neuron counting in bright-field z-stacks is often performed on Extended Depth of Field images or on only one thick focal plane image. However, resolving overlapping cells that are located at different z-depths is a challenge. The overlap can be resolved by counting every neuron in its best focus z-plane because of their separation on the z-axis. Unbiased stereology is the state-of-the-art for total cell number estimation. The segmentation boundary for cells is required in order to incorporate the unbiased counting rule for stereology application. Hence, we perform counting via segmentation. We propose to achieve neuron segmentation in the optimal focal plane by posing the binary segmentation task as a multi-class multi-label task. Also, we propose to efficiently use a 2D U-Net for inter-image feature learning in a Multiple Input Multiple Output system that poses a binary segmentation task as a multi-class multi-label segmentation task. We demonstrate the accuracy and efficiency of the MIMO approach using a bright-field microscopy z-stack dataset locally prepared by an expert. The proposed MIMO approach is also validated on a dataset from the Cell Tracking Challenge achieving comparable results to a compared method equipped with memory units. Our z-stack dataset is available at https://tinyurl.com/wncfxn9m.

Computer-aided/AI-driven tools are increasing being developed for use with digital pathology images. Whether a given scheme makes it into clinical use depends on a multitude of factors, perhaps most importantly whether it has an impact on a clinician’s decision-making process thus on patient care and outcomes. To have a positive impact, clinical decision support tools must be well-integrated into routine clinical workflows and thus require assessment from a human factors perspective that includes attention to ways these tools impact users’ perceptual and cognitive information processing mechanisms. Methods from implementation sciences (the scientific study of methods to promote systematic uptake of research findings and other evidence-based practices into routine practice to improve the quality and effectiveness of health services) can also be used to prepare users for and then assess the impact of introducing these tools into the clinical workflow. This paper will provide an overview of these perspectives, drawing on the history of medical image perception research in radiology and the growing application of these principles and methods in pathology.

Endometrial cancer (EC) is the most common gynecologic malignancy in the US and complex atypical hyperplasia (CAH) is considered a high-risk precursor to EC. Treatment options for CAH and early-stage EC include hormone therapies and hysterectomy with the former preferred by certain patients, e.g., for fertility preservation or poor surgical candidates. Accurate prediction of response to hormonal treatment would allow for personalized and potentially improved recommendations for the treatment of these conditions. In this study, we investigate the feasibility of utilizing weakly supervised deep learning models on whole slide images of endometrial tissue samples for the prediction of patient response to hormonal treatment. We curated a clinical whole-slide-image (WSI) dataset of 112 patients from two clinical sites. We developed an end-to-end machine learning model using WSIs of endometrial specimens for the prediction of hormonal treatment response among women with CAH/EC. The model takes patches extracted from pathologist-annotated CAH/EC regions as input and utilizes an unsupervised deep learning architecture (Autoencoder or ResNet50) to embed the images into a low-dimensional space, followed by fully connected layers for binary prediction. Our autoencoder model yielded an AUC of 0.79 with 95% CI [0.61, 0.98] on a hold-out test set in the task of predicting a patient with CAH/EC as a responder vs non-responder to hormonal treatment. Our results, demonstrate the potential for using weakly supervised machine learning models on WSIs for predicting response to hormonal treatment of CAH/EC patients.

The tumor microenvironment (TME) is comprised of multiple cell types, with their spatial organization having been previously studied to identify associations with disease progression and response to therapy. These works, however, have focused on spatial interactions of a single cell type, ignoring spatial interplay between the remaining cells. Here, we introduce a framework to quantify complex spatial interactions on H&E-stained image between multiple cell families simultaneously within the TME, called spatial connectivity of tumor and associated cells (SpaCell). First, nuclei are segmented and classified into different families (e.g., cancerous cells and lymphocytes) using a combination of image processing and machine learning techniques. Local clusters of proximal nuclei are then built for each family. Next, quantitative metrics are extracted from these clusters to capture inter- and intra-family relationships, namely: density of clusters, area intersected between clusters, diversity of clusters surrounding a cluster, architecture of clusters, among others. When evaluated for predicting risk of recurrence in HPV-associated oropharyngeal squamous cell carcinoma (n=233, 107 vs 126 patients for training vs testing) and non-small cell lung cancer (n=186, 70 vs 116 patients for training vs for testing), SpaCell was able to differentiate between patients at high and low risk of recurrence (p=0.03 and p=0.02, respectively). SpaCell was compared against a deep learning and a state-of-the-art approach that uses single-family cell cluster graphs (CGG). CCG extracted metrics were not prognostic of disease-free survival (DFS) for oropharyngeal (p=0.98) nor lung (p=0.15) cancer, and deep learning was prognostic of DFS for lung (p=0.03) but not for oropharyngeal cancer (p=0.26). SpaCell was not only prognostic for both cancer types but also provides more explainability in terms of tumor biology.

In computational pathology, training and inference of conventional deep convolutional neural networks (CNN) are usually limited to patches of small sizes (e.g., 256 × 256) sampled from whole slide images. In practice, however, diagnostic and prognostic information could lie within the context of tumor microenvironment across multiple regions, far beyond the scope of individual patches. For instance, the spatial relationship of tumor-infiltrating lymphocytes (TIL) across regions of interest might be prognostic for non-small cell lung cancer (NSCLC). This poses a multi-instance learning (MIL) problem, and a single-patch-driven CNN typically fails to learn spatial information and context between multiple patches, especially their spatial relationship. In this work, we present a cell graph-based MIL framework to predict the risk of death for early-stage NSCLC by aggregating feature representation of TIL-enclosing patches according to their spatial relationship. Inspired by PATCHY-SAN, a graph-embedding framework for CNNs, we use graph kernel-based approaches to embed a bag of patches into a sequence with their spatial information encoded into the sequence order. A transformer model was then trained to aggregate patch-level features based on spatial information. We demonstrate the capability of this framework to predict the likelihood of the patient with NSCLC in two cohorts (n=240) to survive for more than 5 years. The training cohort (n=195) comprised hematoxylin and eosin (H&E)-stained whole slide images (WSI), while the testing cohort (n=45) comprised H&E-stained tumor microarrays (TMA). We show that, with the spatial context of multiple patches encoded as an ordered patch sequence, the performance in the testing cohort of our approach achieves an area under the receiver operating characteristic curve (AUC) of 0.836 (p=0.009; HR=5.62), as opposed to a baseline conventional CNN with an AUC of 0.542 (p=0.105; HR=1.66). The results suggest that the Transformer is a generic spatial information aware MIL framework that can learn the spatial relationship of multiple TIL-enclosing patches from the graph representation of immune cells.

Computational pathology, a developing area of primarily deep learning (DL) solutions aiming to aid pathologists at their daily tasks, has shown promising results in research settings. In recent years, uncertainty estimation has gained substantial recognition as having high potential to bring value to DL algorithms for medical applications. But it is not trivial how to incorporate it with a DL system to obtain a real positive impact. In this work we propose a framework to spatially aggregated epistemic uncertainty in order to detect false negatives produced by a segmentation algorithm of breast cancer metastases. We show a strong correlation between the false negative segmentation areas and the aggregated uncertainty values. Furthermore, the results include examples of reducing false negatives, where the uncertainty approach led to detection of some tumour metastases that had been missed.

Most current deep learning models for hematoxylin and eosin (H&E) histopathology image analysis lack the power of generalization to datasets collected from other institutes due to the domain shift in the data. In this research, we study the domain shift problem on two prostate cancer (PCa) datasets collected from the Vancouver Prostate Centre (source dataset) and the University of Colorado (target dataset) and develop a novel center-based H&E color augmentation for cross-center model generalization. While previous work used methods such as random augmentation, color normalization, or learning domain-independent features to improve the robustness of the model to changes in H&E stains, our method first augments the H&E color space of the source dataset to color space of both datasets and then adds random color augmentation. Our method covers the larger range of the color distribution of both institutions resulting in a better generalization. We compared our method with two different State-Of-The-Art (SOTA) un-annotated domain adaptation methods: color normalization and unsupervised domain adversarial neural network (DANN) training, with an ablation study. Our proposed method improves the model performance on both the source and target datasets, and has the best performance on the unlabeled target dataset, showing promise as an approach to learning more generalizable features for histopathology image analysis.

Pathologists regularly use ink markings on histopathology slides to highlight specific areas of interest or orientation, making it an integral part of the workflow. Unfortunately, digitization of these ink-annotated slides hinders any computer-aided analyses, particularly deep learning algorithms, which require clean data free from artifacts. We propose a methodology that can identify and remove the ink markings for the purpose of computational analyses. We propose a two-stage network with a binary classifier for ink filtering and Pix2Pix for ink removal. We trained our network by artificially generating pseudo ink markings using only clean slides, requiring no manual annotation or curation of data. Furthermore, we demonstrate our algorithm’s efficacy over an independent dataset of H&E stained breast carcinoma slides scanned before and after the removal of pen markings. Our quantitative analysis shows promising results, achieving 98.7% accuracy for the binary classifier. For Pix2Pix, we observed a 65.6% increase in structure similarity index, a 21.3% increase in peak signal-to-noise ratio, and a 30% increase in visual information fidelity. As only clean slides are required for training, the pipeline can be adapted to multiple colors of ink markings or new domains, making it easy to deploy over different sets of histopathology slides. Code and trained models are available at: https://github.com/Vishwesh4/Ink-WSI.

Predicting the presence and counting various cell types in the tumor micro-environment is crucial for identifying cell spatial and morphological features in computational pathology. Deep learning (DL) approaches have been used extensively in predicting and counting different cell types in whole slide images (WSIs). However, DL-based probabilistic approaches have limitations in classifying pixels and determining cellular boundaries. In this study, we propose an end-to-end evidential multi-task deep neural network for counting different cell types in hematoxylin and eosin-stained digital pathology images. The aim is to achieve a reasonable accuracy with lower uncertainties than state-of-the-art methods. Our proposed approach consists of one encoder and three task-specific decoders. The encoder utilizes a pre-trained ResNet50 model with ImageNet weights to extract contextual and spatial imaging features. The multi-tasking approach comprises of three tasks: predicting various cell type counts (main task), nuclei semantic segmentation and estimating cell-type density maps (auxiliary tasks). We evaluated the effectiveness of our proposed approach on two open-source datasets, PanNuke and MoNuSAC, which consist of 205,343 and 46,909 nuclei segmentation labels, respectively. Our results were compared with state-of-the-art methods, HoVer-Net, StarDist, and ALBRT. Our proposed approach showed superior performance in cell-type counting compared to HoVer-Net, ALBRT, and StarDist with relative improvements of 11%, 7%, and 5% on the PanNuke dataset test set, and 8.5%, 2%, and 11% on the MoNuSAC dataset test set, respectively, in terms of R². Our experiments suggest that our proposed approach can provide compelling interpretations of diverse cell types and is applicable for various downstream tasks in computational pathology.

Weakly-supervised classification of histopathology slides is a computationally intensive task, with a typical whole slide image (WSI) containing billions of pixels to process. We propose Discriminative Region Active Sampling for Multiple Instance Learning (DRAS-MIL), a computationally efficient slide classification method using attention scores to focus sampling on highly discriminative regions. We apply this to the diagnosis of ovarian cancer histological subtypes, which is an essential part of the patient care pathway as different subtypes have different genetic and molecular profiles, treatment options, and patient outcomes. We use a dataset of 714 WSIs acquired from 147 epithelial ovarian cancer patients at Leeds Teaching Hospitals NHS Trust to distinguish the most common subtype, high-grade serous carcinoma, from the other four subtypes (low-grade serous, endometrioid, clear cell, and mucinous carcinomas) combined. We demonstrate that DRAS-MIL can achieve similar classification performance to exhaustive slide analysis, with a 3-fold cross-validated AUC of 0.8679 compared to 0.8781 with standard attention-based MIL classification. Our approach uses at most 18% as much memory as the standard approach, while taking 33% of the time when evaluating on a GPU and only 14% on a CPU alone. Reducing prediction time and memory requirements may benefit clinical deployment and the democratisation of AI, reducing the extent to which computational hardware limits end-user adoption.

Circulating Tumor Cells, or CTCs, are cancerous cells that shed from a primary tumor and intravasate into the bloodstream. This type of screening, otherwise known as biopsy screening, is effective at determining the early stages of cancer and discussing treatment. Expert cytopathologists have been required to look at these images to screen for cancer. Anything between the numbers of hundreds to thousands of images must be gone through. At a heavy time cost and a basis of work effort, we thus proposed an idea using a U-Net to screen for these CTCs and enumerate them in a more efficient and time-costing method. The Ball-scale transform technique, a filter that allows us to determine the maximum sphericality in a thresholded homogeneity, was introduced into this field of digital pathology alongside our proposed novel deep learning-based (UNet) CTC detection and enumeration approach. We collected 466 images for CTC detection and another 198 images with 323 CTCs for testing CTC enumeration. We investigated two ways to use the Ball-scale image: using B-scale images in the input channel of deep learning and using B-scale images in the output layer by providing high-level information (size and shape) encoded in the B-scale image itself to do the enumeration. We also tested deep learning-based CTC detection by using different labels. Results show that our method is much better than those which utilize thresholding with a missing rate comparison of 0.04 to 0.30. Meanwhile, our method is certainly comparable and competitive with the results in recent publications and may facilitate other types of research.

Annotation of true ground truth is a difficult task for many computational pathology problems. Types of ground truth labels in the field include bounding boxes, text labels, binary class labels, and full tissue maps. The compounding issue is when multiple different pathologists label the same image, and there is disagreement between them. In this work, we investigate multiply reannotated tumor maps for squamous cell carcinoma, and if different annotation fusion methods have an impact on tumor segmentation. We find in this work that tumor label maps with an average annotation similarity of 0.759, do not have a significant quantitative difference in tumor segmentation.

Tumor budding (TB) is defined as a cluster of one to four tumor cells at the tumor invasive front. Though promising as a prognostic factor for colorectal cancer, its routine clinical use is hampered by high inter- and intra- observer disagreement on routine H&E staining. Pan-cytokeratin immunohistochemical staining increases agreement but is costly, non-routine, and may yield false tumor buds (false positives). This makes the development of automatic algorithms to identify TB difficult. Therefore, we propose a weakly-supervised method that does not require strictly accurate tissue level annotations and is resilient to false positives. Our database consists of 29 H&E whole slide images. TB and nontumor ROIs were generated by cropping 512x512 regions around annotated tumor buds and within annotated non-tumor regions, respectively. Attention-based multiple instance learning was applied to identify ROIs containing tumor buds. This resulted in a precision of 0.9477 ± 0.0516, recall of 0.9131 ± 0.0568, and AUC of 0.9482 ± 0.0679 on an external dataset. These results provide preliminary evidence for the feasibility of our method to identify tumor buds accurately.

Papillary thyroid carcinoma (PTC) has the highest incidence rate of all thyroid cancers for the last several decades. Although this particular disease tends to be fairly indolent overall, classical and tall cell variants exhibit more aggressive behavior due to a higher progression and mortality rate. However among classical PTCs, there is a need for improvement of clinical management of patients who are at higher risk of progression/death and who will potentially benefit from more aggressive management, and similarly for lower risk patients who are unnecessarily overtreated. In this study, we aimed to evaluate the prognostic role of nuclear features for disease-specific and disease-free survival within classical and follicular histological subtypes of PTC. A set of features describing the nuclear shape, architecture, and texture of both tumor and lymphocyte cells were used to train a Lasso-Cox regression model with 199 patients. Coefficients of the model were then used to assign a quantitative risk score (QuRiS) per patient. Patients were further subdivided into high and low risk of recurrence to compare survival probabilities within a 15 year study period using Kaplan-Meier estimates (C-index of 0.786 (p = 0.009 on log-rank test). Survival probabilities of both risk groups were computed with a hazard ratio 1.54 and compared using the log-rank test (p = 0.004). The trained model was then validated on the remaining 149 patients, consisting of 3 independent sets of classical (N = 107), follicular (N = 74), and tall cell (N = 16) variants of PTC, with HR=8.83, 4.21, and 0.583, respectively. We then identified specific genes that were differentially expressed between the risk groups identified by the pathomic model. Using gene set enrichment and mutation analysis, we discovered that the signaling pathways TCA cycle and amino acid metabolism are associated with poorer outcome, while mutation in the BRAF oncogene is significantly associated with better survival in cPTC patients. We also discovered that the DTX4 gene was prognostic for disease-specific survival (DSS) and further stratified the risk of disease-related death of patients with high and low expression of DTX4 using the image-based nuclear morphology features.

Whole slide images (WSIs) in histopathology naturally provide multi-scale information. Several previous studies have shown that leveraging such multi-scale information in histopathology image analysis is effective to improve performance. Here, we propose making use of recent advances in contrastive learning and self-attention techniques in multi-scale WSIs for cancer subtype classification using weak labels. The proposed method is based on a Siamese architecture to share a common encoder network for images on different scales to reduce the model size and training cost. In addition, we propose a variant of the self-attention module specifically designed for multi-scale WSIs so that the network can focus on important textural features across different image scales. We assess the efficacy of the proposed method via an ablation study on a real intrahepatic cholangiocarcinoma dataset. The result confirms that our method outperforms conventional multi-scale models with fewer model parameters.

Crohn’s disease (CD) is a debilitating inflammatory bowel disease with no known cure. Computational analysis of hematoxylin and eosin (H&E) stained colon biopsy whole slide images (WSIs) from CD patients provides the opportunity to discover unknown and complex relationships between tissue cellular features and disease severity. While there have been works using cell nuclei-derived features for predicting slide-level traits, this has not been performed on CD H&E WSIs for classifying normal tissue from CD patients vs active CD and assessing slide label-predictive performance while using both separate and combined information from pseudo-segmentation labels of nuclei from neutrophils, eosinophils, epithelial cells, lymphocytes, plasma cells, and connective cells. We used 413 WSIs of CD patient biopsies and calculated normalized histograms of nucleus density for the six cell classes for each WSI. We used a support vector machine to classify the truncated singular value decomposition representations of the normalized histograms as normal or active CD with four-fold cross-validation in rounds where nucleus types were first compared individually, the best was selected, and further types were added each round. We found that neutrophils were the most predictive individual nucleus type, with an AUC of 0.92 ± 0.0003 on the withheld test set. Adding information improved cross-validation performance for the first two rounds and on the withheld test set for the first three rounds, though performance metrics did not increase substantially beyond when neutrophils were used alone.

High-content screening (HCS) has catalyzed drug development through enabling fast, large-scale, and reproducible testing of changes in cellular invitro models in response to different types of perturbations. One HCS approach, known as Cell Painting (CP), can conduct the morphological profiling of images containing cells perturbed with different treatments to quantitatively assess complex biological changes. Profiling stages of macrophage polarization, in particular, enables new drug discovery with disease-relevant conditions. To analyze cell images at single cell level, deep learning algorithms - in addition to classical image segmentation methods - may also be used to conduct single cell cropping for accurate and fast detection of individual cells. While the classical Watershed Segmentation and Gaussian mixture model (GMM) was first implemented for robust single-cell detection in the CP workflow, its performance is sometimes compromised when cells are clumped. A deep learning-based cell segmentation method called Cellpose was introduced and proposed as an alternative means for cell localization, however, coming at the cost of compromised runtime for HCS. In this study, we demonstrate the use of YOLOv5, a fast deep learning object detection algorithm, to yield comparable cell detection performance to the other two methods, while bringing improvements in high cell density regions and a faster runtime. This study demonstrates the use of the YOLOv5 model for performing ~2x faster cell detection with comparable IoU scores on HCS macrophage nuclei images, demonstrating its value in extracting coordinates for single-cell cropping needed in deep learning-based phenotypic profiling in HCS. We compare the accuracy and speed of the model developed using YOLOv5 with those of the current Watershed/GMM method and Cellpose method in macrophage cell detection in the context of investigating drug activity.

In histopathology, whole-mount sections (WMS) can be utilized to capture entire cross-sections of excised tissue, thereby reducing the number of slides per case, preserving tissue context, and minimizing cutting artefacts. Additionally, the use of WMS allows for easier correlation of histology and pre-operative imaging. However, in digital pathology, WMS are not frequently acquired due to the limited availability of whole-slide scanners that can scan double-width glass slides. Moreover, whole-mounts may still be too large to fit on a single double-width glass slide, and archival material is typically stored as tissue fragments. In this work, we present PythoStitcher, an improved version of the AutoStitcher algorithm for automatically stitching multiple tissue fragments and constructing a full-resolution WMS. PythoStitcher is based on a genetic algorithm that iteratively optimizes the affine transformation matrix for each tissue fragment using the Euclidean distance between corner points of neighbouring fragments. We demonstrate the efficiency and generalisability of PythoStitcher in a pilot validation of ten prostatectomy cases with four fragments and two oesophagectomy cases with two fragments. On average, PythoStitcher obtained the final stitching result in 60 seconds (range 51-73) for the prostatectomy cases and 68 seconds (range 43-94) for the oesophagectomy cases. The stitching accuracy was objectively evaluated by computing the average Euclidean distance between all corner points from the edges of neighbouring fragments. For the reconstructed WMS at full resolution, the average residual registration error was 1.6 mm (range 0.5-3.2) for the prostatectomy cases and 2.1 mm (range 0.5-3.8) for the oesophagectomy cases. The key strengths of PythoStitcher include its computational efficiency, the ability to reconstruct the WMS in full resolution, and an open-source implementation in Python.

The effectiveness of untrained convolutional layers for feature extraction in a computational pathology task using real-world data from a necrosis detection dataset is investigated. The study aims to determine whether ImageNet pretrained layers from deep CNNs combined with frozen untrained weights are sufficient for effective necrosis detection in canine Perivascular Wall Tumour (cPWT) whole slide images. Additionally, the authors investigate the impact of pruning CNNs, and whether it can be effective for necrosis detection as this technique can contribute towards reducing memory requirements and improve inference speed in diagnostic settings. The study found that fine-tuning the last (deepest) layers of a pretrained ImageNet model for necrosis detection in cPWT produces the highest test F1-score (0.715) when compared to alternative set ups. This score is further improved to 0.754 when the results are optimised using an optimal threshold predetermined on maximising the validation set F1-score. Resetting weights (untrained) and freezing the last few convolutional layers in the last dense block also demonstrated some capability in necrosis detection with an optimised F1-score of 0.747, still outperforming models trained from scratch as well as an ImageNet pretrained feature extraction model. Pruning the fine-tuned model using lower thresholds also showed the potential to improve performance, however thresholds higher than 0.40 negatively impacted performance.

The current advances in Whole-Slide Imaging (WSI) scanners allow for more and better visualization of histological slides. However, the analysis of histological samples by visual inspection is subjective and could be challenging. State-of-the-art object detection algorithms can be trained for cell spotting in a WSI. In this work, a new framework for the detection of tumor cells in high-resolution and high-detail using both RGB and Hyperspectral (HS) imaging is proposed. The framework introduces techniques to be trained on partially labeled data, since labeling at the cellular level is a time and energy-consuming task. Furthermore, the framework has been developed for working with RGB and HS information reduced to 3 bands. Current results are promising, showcasing in RGB similar performance as reference works (F1-score = 66.2%) and high possibilities for the integration of reduced HS information into current state-of-art deep learning models, with current results improving the mean precision a 6.3% from synthetic RGB images.

Digital pathology Whole Slide Images (WSIs) are large images (∼30 GB/slide uncompressed) of high resolution (0.25 microns per pixel), presenting a significant data storage challenge for hospitals wishing to adopt digital pathology. Lossy compression has been adopted by scanner manufacturers to address this issue - we compare lossy Joint Photographic Experts Group (JPEG) compression for WSIs and investigate the Vector Quantised Variational Autoencoder 2 variant (VQVAE2) as a possible alternative to reduce file size while encoding useful features in the compressed representation. We trained three VQVAE2 models on a Camelyon 2016 subset to the Compression Ratio (CR) of 19.2:1 (CR₁), 9.6:1 (CR₂) and 4.8:1 (CR₃) and tested on a Camelyon 2016 (DS₁) subset; University of California (DS₂) and Internal Validation Set (DS₃). We then compared compression performance to ImageMagick JPEG and JPEG 2000 implementations. Both JPEG and JPEG 2000 compression outperformed the VQVAE2 implementation within the Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics. The trained VQVAE2 models could visually reproduce WSI tissue structure, but used colours from the original training data within the reconstructions on other datasets.

Nowadays, the availability of different types of biomedical digital data offers many opportunities to investigate the relationships between the different modalities and thus develop a more comprehensive understanding of complex diseases such as cancer. In this paper, we propose a multi-modal model, called deep modality association learning (DMAL), that maps immune cell sequencing patterns to morphological tissue features of whole slide imageds (WSIs) in an embedding space. Useful information is extracted from T-cell receptor (TCR) sequences to guide the training process. DMAL maps the TCR features to the morphology features in histopathology images, which in turn enables the model to learn the association features between the two modalities. The discrimination power of the WSI-TCR association features has been assessed by classifying samples with different cancer subtypes. The conducted experiments have shown that DMAL generates more discriminative features compared to features obtained from single-modal data. In addition, DMAL has been utilized to predict TCR information from histopathology image representations without the need to have the actual TCR sequencing data.

The aim of this study is to develop a method to transform hyperspectral images of H&E-stained slides to natural-color RGB histology images for easy visualization. Hyperspectral images were obtained at 40× objective magnification and downsampled by various factors to generate data equivalent to different magnifications. High-resolution digital histologic RGB images were cropped and registered to the corresponding hyperspectral images as the ground truth. A conditional generative adversarial network (cGAN) was trained to output natural color RGB images of the histological tissue samples. The generated synthetic RGBs have similar color and sharpness to real RGBs. Image classification was implemented using the real and synthetic RGBs, respectively, with a pretrained network. The classification of tumor and normal tissue using the HSI-synthesized RGBs yielded a comparable but slightly higher accuracy and AUC than the real RGBs.

Data augmentation is a commonly applied method to increase the number of training samples for histopathology applications. However, data augmentation has yet to be explored for multiple instance learning applications in histopathology. Therefore, we developed a basic methodology to increase the number of bags for training in an application of attention-based multiple instance learning to classify whole slide histopathology images from Camelyon16. Our method applies common image transformations to histopathology patches at random to generate any number of bags for each slide. Across a ten-fold cross-validation with several magnitudes of augmentation, 10x augmentation (i.e. 10 augmented bags per slide plus the original bag) resulted in normal/tumor accuracies and AUC of 0.9463±0.0263, 0.7531±0.0755, and 0.8847±0.0241, respectively, which significantly exceeded baseline performance (i.e. no augmentation) of.0.9300±0.0363, 0.6918±0.1070, and 0.8827±0.0549, respectively. Additionally, our results suggest that greater levels of augmentation increase overall performance metrics.

The tumor microenvironment (TME) plays an important role in driving cancer progression and affecting treatment efficacy. Cellular components of the TME include various immune subsets (tumor infiltrating lymphocytes (TILs) and macrophages), cancer-associated fibroblasts (CAFs) and vascular cells. While immune lineage has been a main focus of intensive research on the TME, CAFs have also been shown to be highly heterogeneous in their molecular phenotype and function. Using a protein marker immunofluorescence multiplexing system (Cell DIVE, Leica Microsystems) and quantitative imaging tools, we investigated the identity of various CAF clusters based on the expression of α-Smooth Muscle Actin (αSMA) and Fibroblast Activation Protein (FAP), and compared their distributions across breast cancer subtypes. We determined the cell counts of various CAF subsets using binary counting and identified the heterogeneous presentations of clusters using K-means clustering and Uniform Manifold Approximation and Projection (UMAP). We found that the abundance of CAF clusters varied among breast cancer subtypes. An integrated analysis of CAF cluster composition in each cancer and the transcriptomic data of CAF-associated genes such as CD29, IL6 and PDGFRβ was performed. We observed increased densities of proliferative, αSMA-positive CAFs in basal-like breast cancers that exhibited a co-expression signature of CAF-associated genes. Finally, an association analysis of CAF cluster composition and gene expression with previously identified radiomic phenotype was performed, but significant correlation was not detected.

Purpose: Identification of cardiomyocyte nuclei is of great importance when assessing heart tissue for cardiovascular disease. However, manual identification of cardiomyocyte nuclei in hematoxylin & eosin (H&E)-stained heart tissue is particularly complex due to extensive heterogeneity in morphology. While expensive and time-consuming staining processes can be utilized to identify cardiomyocyte nuclei, this study proposes to test the feasibility of using a cycle-consistent generative adversarial network (CycleGAN) to automate the identification of cardiomyocyte nuclei in H&E-stained heart tissue without the need for additional staining.

Materials and Methods: 100 H&E-stained heart tissue sample regions from cancer patients were manually labeled with the input from an expert pathologist. Sample regions were processed into 3,584 labeled tiles and 3,584 tiles without labels, and then sorted, augmented, and run through a CycleGAN to facilitate the identification of cardiomyocyte nuclei. Performance was assessed with Fréchet Inception Distance (FID), Fréchet ResNet-50 Distance (FRD), sensitivity, specificity, and accuracy.

Results: Sensitivity and specificity were 61.6% sensitivity, 96.3% specificity with an accuracy of 85.0% for 30 randomly selected samples. FID and FRD reached as low as 64.8 and 2.46, respectively, with downward trend for FID and high volatility with a slight downward trend for FRD. Notably, the best results were achieved when FRD was at its lowest value.

Conclusion: The results suggest cardiomyocyte nuclei can be identified by a CycleGAN. However, targeted improvements are necessary to ensure accurate identification. An enhanced version of this approach may facilitate the automation of identifying cardiomyocyte nuclei in H&E-stained heart tissue, facilitating cardiovascular research.

Studies have shown that the increased presence of tumor-infiltrating lymphocytes (TILs) is associated with better long-term clinical outcomes and survival, which makes TILs a potentially useful quantitative biomarker. In clinics, pathologists’ visual assessment of TILs in biopsies and surgical resections result in a quantitative score (TILs-score). The Tumor-infiltrating lymphocytes in breast cancer (TiGER) challenge is the first public challenge on automated TILs-scoring algorithms using whole slide images of hematoxylin and eosin-stained (H&E) slides of human epidermal growth factor receptor-2 positive (HER2+) and triple-negative breast cancer (TNBC) patients. We participated in the TiGER challenge and developed algorithms for tumor-stroma segmentation, TILs cell detection, and TILs-scoring. The whole slide images in this challenge are from three sources, each with apparent color variations. We hypothesized that color-normalization may improve the cross-source generalizability of our deep learning models. Here, we expand our initial work by implementing a color-normalization technique and investigate its effect on the performance of our segmentation model. We compare the segmentation performance before and after color-normalization by cross validating the models on the three datasets. Our results show a substantial increase in the performance of the segmentation model after color-normalization when trained and tested on different sources. This might potentially improve the model’s generalizability and robustness when applied to the external sequestered test set from the TiGER challenge.

Gliomas are primary brain tumors that arise from neural stem cells or glial precursors. Diagnosis of glioma is based on histological evaluation of pathological cell features and some molecular markers. Gliomas are infiltrated by myeloid cells that accumulate preferentially in malignant tumors and their abundance inversely correlates with survival, which is of interest for cancer immunotherapies. To avoid time-consuming and laborious manual examination of the images, a deep learning approach for automatic multiclass classification of tumor grades was proposed. Moreover, the challenge of the study was given by small sample size of human leukocyte antigen tissue microarrays dataset (HLA-TMA) - total 204 images from 5 classes - and imbalanced data distribution. This has been addressed by images augmentation of the underrepresented classes, as already shown in a similar study about predicting mutations from glioma biopsies.¹ For this glioma multiclass classification task, the architecture of residual neural network has been adapted. The best model produced an accuracy score of 0.7727, and the mean accuracy of the cross-validation iterations was the value of 0.7248 on the validation set. This promising approach can be used as an additional diagnostic tool to improve assessment during intra-operative examination or sub-typing tissues for treatment selection, despite the challenges presented by the difficult dataset.

Machine learning algorithms have made strides in the classification of metastasis in tissue histopathology images. However, one of the major roadblocks faced by these algorithms is the difference in staining of tissue samples taken and scanned at different laboratories. Stain normalization works to standardize the color and intensity of stain patches to a reference image, hence bringing different laboratory stainings to a similar domain. We propose to compare different stain normalization methods in conjunction with color augmentation to evaluate the performance of combinations of techniques for binary classification of metastatic tissue slides taken from lymph nodes. We examine the accuracy, precision, recall, F1 score, and AUROC of a convolutional neural network (CNN) model trained and tested on images that have been normalized using the Macenko and Vahadane methods. Six different configurations that combine color augmentation and stain normalization were analyzed on the PatchCamelyon (PCAM) dataset consisting of over 300K images. Our analysis showed that the Macenko method of stain normalization improves model performance, and similarly, data augmentation shows general improvement, as the increased diversity amongst the data counters overfit in the model. Model accuracy with Macenko and color augmentation improved from baseline by 1.59% and F1 score for Macenko improved from baseline by 2.50%. The best performing combination was color augmentation with the Macenko method.

For many applications in oncology, pathologists inspect 2D tissue sections of resected tumor samples to assess tumor morphology, identify infiltrating lymphocytes, and predict disease progression. While 3D tissue models obtained with confocal or light-sheet microscopes promise added value over 2D scans, data-handling and visualization of whole-slide scans at cell-resolution becomes computationally demanding and sometimes even impossible on consumer-grade hardware.

To enable the simultaneous display of large tissue volumes and individual cell phenotypes with a small memory footprint, we created Cell2Voxel, a light-weight, cell-based 3D tissue model in which every cell is represented by a single voxel on a coarse, regular grid. As an alternative to confocal and light-sheet microscopy, we present a workflow including widefield imaging of multiplex 2D tissue sections, cell segmentation and phenotyping, image registration, conflict-free assignment of cell-voxels to the target grid, and the creation of the final 3D tissue model.

As a showcase, we present 3D models of human colon cancer samples and murine pancreatic islets. We show that Cell2Voxel can reduce the memory footprint compared to other methods by 2-3 orders of magnitude and enables the simultaneous visualization of millions of individual cells in whole-slide tissue volumes on conventional computer hardware. While our final 3D model can be inspected using open-source software on all major operating systems, we additionally present an interactive data viewer tailored to our data format. Cell2Voxel can be created from any 2D widefield image stack, making it an inexpensive, easy-to-use alternative for large-volume 3D tissue visualization.

The common method for histopathology image classification is to sample small patches from large whole slide images and make predictions based on aggregations of patch representations. Transformer models provide a promising alternative with their ability to capture long-range dependencies of patches and their potential to detect representative regions, thanks to their novel self-attention strategy. However, as a sequence-based architecture, transformers are unable to directly capture the two-dimensional nature of images. While it is possible to get around this problem by converting an image into a sequence of patches in raster scan order, the basic transformer architecture is still insensitive to the locations of the patches in the image. The aim of this work is to make the model be aware of the spatial context of the patches as neighboring patches are likely to be part of the same diagnostically relevant structure. We propose a transformer-based whole slide image classification framework that uses space-filling curves to generate patch sequences that are adaptive to the variations in the shapes of the tissue structures. The goal is to preserve the locality of the patches so that neighboring patches in the one-dimensional sequence are closer to each other in the two-dimensional slide. We use positional encodings to capture the spatial arrangements of the patches in these sequences. Experiments using a lung cancer dataset obtained from The Cancer Genome Atlas show that the proposed sequence generation approach that best preserves the locality of the patches achieves 87.6% accuracy, which is higher than baseline models that use raster scan ordering (86.7% accuracy), no ordering (86.3% accuracy), and a model that uses convolutions to relate the neighboring patches (81.7% accuracy).

Whole slide imaging (WSI), also called digital virtual microscopy, is a new imaging modality. It allows for the application of AI and machine learning methods to cancer pathology to help establish a means for the automatic diagnosis of cancer cases. However, designing machine-learning models for WSI is computationally challenging due to its required ultra-high resolution. The current state-of-the-art models use multiple instance learning (MIL). MIL is a weakly-supervised learning method in which the model uses an array of inferences from many smaller instances to make a final classification about the entire set. In the context of WSI, researchers divide the ultra-high-resolution image into many patches. The model then classifies the slide based on an array of inferences from the patches. Among several ways of making the final classification, attention-based mechanisms have resulted in superb accuracy scores. The Transformer, one attention-based algorithm, has reported substantial improvements for WSI comprehension tasks. In this project, we studied and compared several WSI comprehension algorithms. We used the following three datasets: CAMELYON16+17, TCGALung, and TCGA-Kidney. We found that attention-based MIL algorithms performed better than standard MIL algorithms for classifying WSI images, achieving a higher mean accuracy and AUC. However, none of the attention-based algorithms performed significantly better than the others, reporting accuracy scores that varied widely. Presumably, it is due to the limited availability of training samples in the data corpus. Since it is not easy to increase the samples from human subjects, some machine learning techniques like transfer learning could help mitigate this issue.

The recent success of deep neural networks has generated remarkable growth in Artificial Intelligence (AI) research and has received much interest over the past few years. One of the main challenges for the broad adoption of deep learning-based models such as Convolutional Neural Networks (CNN) is the lack of understanding of their decisions. To address this issue, Explainable Artificial Intelligence (XAI) has been proposed to shift toward more transparent AI, resulting in the development of techniques to explain decisions by AI models. This paper aims to explore and develop a multi-scale scheme of LIME (Local Interpretable Model-Agnostic Explanations) applied to image classification to explain decisions made by CNN models through heatmaps of coarse to finer scales. More precisely, when LIME highlights large superpixels from a coarse scale, there may be smaller regions in the corresponding superpixel that influenced the model’s prediction at some finer scale. In the proposed multi-scale scheme, two weighting approaches, one based on Gaussian distribution and another parameter-free framework will be introduced to produce visual explanations observed from different scales. Promising results for multi-scale classification heatmaps of histopathology images are presented. More specifically, we investigated the proposed multi-scale approach on Camelyon16 dataset. The results show that the explanations are faithful to the underlying model, and the visualizations are reasonably interpretable.

Detection, characterization and counting of mitosis is a main biomarker in cancer, allowing diagnosis, histological grading and prognosis. Nonetheless, mitosis identification remains as a challenging task (inter-observer variability up to 20%). Even, the computational support strategies remain limited to include wide visual variability of mitotic patterns, with an inherent bias because labeled observations. This work introduces a semi-supervised scheme that learns to identify mitotic cells from an initial limited amount of labeled data. Then, the initial trained backbone is used to propagate pseudo-labels into training samples. The most challenging samples, i.e., false positive and false negative pseudo-labeled samples, are included in further batches to re-train the model. At each iteration, a set of complementary non-mitotic patches are generated from an auxiliary net. The proposed approach was validated with the public ICPR dataset, achieving competitive results of 0.74 accuracy and 0.78 sensitivity. In addition, the proposed approach achieves an average inference time of 5.21 seconds (on a batch of 240 candidate patches).

Melanoma is the deadliest skin cancer with the fastest rising incidence rate in the United States. The most important predictor of melanoma patient survival is the volume of invasive tumor at the initial biopsy. The appearance of in-situ melanoma in epidermis and invasive melanoma in dermis, which invades the underlying soft tissue and drives mortality, is often visually similar. We propose a novel two-stage method to segment invasive melanoma. The first stage computes two segmentation maps, one for tumor vs non-tumor and one for dermis vs epidermis. These two segmentation prediction maps of tumor and epidermis from the first stage combine to yield invasive melanoma predictions. Our method utilizes multiple resolutions and downscaling to increase information passed to the model and to improve model accuracy. Using an HRNet+OCR model for both epidermis and melanoma segmentation in our proposed two-stage system results in a marked improvement of F1 score (mIoU) to 0.44 (0.64) as compared to the current state-of-the-art of 0.14 (0.53).

Segmentation of histology tissue whole side images is an important step for tissue analysis. Given enough annotated training data, modern neural networks are capable of accurate reproducible segmentation; however, the annotation of training datasets is time consuming. Techniques such as human-in-the-loop annotation attempt to reduce this annotation burden, but still require vast initial annotation. Semi-supervised learning—a technique which leverages both labeled and unlabeled data to learn features—has shown promise for easing the burden of annotation. Towards this goal, we employ a recently published semi-supervised method, datasetGAN, for the segmentation of glomeruli from renal biopsy images. We compare the performance of models trained using datasetGAN and traditional annotation and show that datasetGAN significantly reduces the amount of annotation required to develop a highly performing segmentation model. We also explore the usefulness of datasetGAN for transfer learning and find that this method greatly enhances the performance when a limited number of whole slide images are used for training.

Reference histomorphometric data of healthy human kidneys are lacking due to laborious quantitation requirements. We leveraged deep learning to investigate the relationship of histomorphometry with patient age, sex, and serum creatinine in a multinational set of reference kidney tissue sections.

A panoptic segmentation neural network was developed and used to segment viable and sclerotic glomeruli, cortical and medullary interstitia, tubules, and arteries/arterioles in digitized images of 79 periodic acid-Schiff (PAS)-stained human nephrectomy sections showing minimal pathologic changes. Simple morphometrics (e.g., area, radius, density) were measured from the segmented classes. Regression analysis was used to determine the relationship of histomorphometric parameters with age, sex, and serum creatinine.

The model achieved high segmentation performance for all test compartments. We found that the size and density of nephrons, arteries/arterioles, and the baseline level of interstitium vary significantly among healthy humans, with potentially large differences between subjects from different geographic locations. Nephron size in any region of the kidney was significantly dependent on patient creatinine. Slight differences in renal vasculature and interstitium were observed between sexes. Finally, glomerulosclerosis percentage increased and cortical density of arteries/arterioles decreased as a function of age.

We show that precise measurements of kidney histomorphometric parameters can be automated. Even in reference kidney tissue sections with minimal pathologic changes, several histomorphometric parameters demonstrated significant correlation to patient demographics and serum creatinine. These robust tools support the feasibility of deep learning to increase efficiency and rigor in histomorphometric analysis and pave the way for future large-scale studies.

Cutaneous melanoma is an invasive cancer with a worldwide annual death toll of 57,000 (Arnold et al., JAMA Dermatol 2022). In a metastatic state, surgical interventions are not curative and must be coupled with targeted therapy, or immunotherapy. However, resistance appears almost systematically and late-stage prognosis can remain poor. The complexity to eradicate melanoma stems from its plasticity; these cancer cells continually adapt to the tumor microenvironment, which leads to treatment resistance. Our primary assumption is that therapeutic resistance relies in part on a series of non-genetic transitions including changes in the metabolic states of these cancer cells. The 3D spatial distribution of blood vessels that are sources of nutrition and oxygen that drive this metabolic status is an important variable for understanding zoning aspects of this adaptation process. Using Whole Slide Images (WSI) of melanoma tumors from Patient-Derived Xenograft (PDX) mouse models, we build 3D vascular models to help predict and understand the metabolic states of cancer cells within the tumor. Our 3D reconstruction pipeline was based on PDX tumor samples sectioned over 2mm depth and stained with Hematoxylin and Eosin (H&E). The pipeline involves three primary steps, including 2D vessel segmentation using Deep Learning, intensity- and affine-based image registration, and 3D reconstruction using interpolation and 3D rendering (allowing for better interaction with biologists, pathologists, and clinicians). The originality of our computer-assisted pipeline is its capability to (a) deal with sparse data (i.e., not all tissue sections were readily available), and (b) adapt to a multitude of WSI-related challenges (e.g., epistemic uncertainty, extended processing times due to WSI scale, etc.). We posit both our 3D reconstruction pipeline, quantitative results of the major stages of the process, and a detailed illustration of the challenges faced, presenting resolutions to improve the pipeline’s efficiency.

Recently, high-performance deep learning models have enabled automatic and precise analysis of medical images with high content. In digital histopathology, a challenge lies in analyzing Whole Slide Images (WSI) due to their large size, often requiring splitting them into smaller patches for deep learning models. This leads to the loss of global tissue information and limits the classification or clustering of patients based on tissue characteristics.

In this study, we develop a meta-graph approach for semantic spatial analysis of WSI of human brain tissue containing tau protein aggregates, a hallmark of Alzheimer’s disease (AD) in gray matter. Our pipeline extracts morphological features of tau aggregates, such as forming neuritic plaques, and builds a graph based on Delaunay triangulation at the WSI level to extract topological features. This generates morphological and topological data from WSI for patient classification and clustering. We tested this pipeline on a dataset of 15 WSIs from different AD patients. We aim to identify new insights into AD evolution and provide a generic framework for WSI characterization and analysis.

Cellular resolution of optical coherence tomography (OCT) is vital to achieving precise diagnosis by offering high-quality images of slide-free histology. Currently, the common solution is to apply dynamic focusing to axially translate the focus through the region of interest with a high numerical aperture (N.A.) objective, followed by Z-stacking to rebuild a high-resolution 3D volume. To accelerate the imaging acquisition, this work developed optical metasurface plates to generate multiple foci along the axial direction. Two-/three-/seven-foci had been testified with bead phantom using a scanning OCT. Human skin and human brain samples were imaged with cellular resolution.

High-resolution histopathological images have rich characteristics of cancer tissues and cells. Recent studies have shown that digital pathology analysis can aid clinical decision-making by identifying metastases, subtyping and grading tumors, and predicting clinical outcomes. Still, the analysis of digital histologic images remains challenging due to the imbalance of the training data, the intrinsic complexity of histology characteristics of tumor tissue, and the extremely heavy computation burden for processing extremely high-resolution whole slide imaging (WSI) images. In this study, we developed a new deep learning-based classification framework that addresses these unique challenges to support clinical decision-making. The proposed method is motivated by our recently developed adversarial learning strategy with two major innovations. First, an image pre-processing module was designed to process the high-resolution histology images to reduce computational burden and keep informative features, alleviating the risk of overfitting issues when training the network. Second, recently developed StyleGAN2 with powerful generative capability was employed to recognize complex texture patterns and stain information in histology images and learn deep classification-relevant information, further improving the classification and reconstruction performance of our method. The experimental results on three different histology image datasets for different classification tasks demonstrated superior classification performance compared to traditional deep learning-based methods, and the generality of the proposed method to be applied to various applications.

Recent studies have achieved a great success in medical image segmentation, but do not perform well in the application of pathological image segmentation. In traditional segmentation networks, some important features may be lost during the encoding process. In this paper, an Enhanced Pooling-Convolution (EPC) module is proposed to add weights to the space and channels of features in the encoding process. EPC evaluates the differences and complementarities of features between max pooling, average pooling, and convolution in the pooling process. Channel based attention is further used to weight different channels. VGG16 is used as the backbone in the U-shaped network, and the number of channels for upsampling is reduced during decoding process. It shows that the pooling and convolution block with three consecutive convolution layers can be replaced with the EPC module. Experimental results shows that the average DICE coefficient of our method is 2.55% higher than that of U-Net.

In recent works, more and more attention mechanisms have been used for medical image segmentation, however, attention mechanisms are not very good at distinguishing categories in multi-category medical image segmentation tasks. In this paper, we propose a category feature reconstruction module (CFRM) for multi-category pathological image segmentation of pancreatic adenosquamous carcinoma. Compared with the attention mechanism to enhance the features of the region of interest, the proposed CFRM pays more attention to the reconstruction of category features. The CFRM enables the network not only to highlight the features of the region of interest, but also to increase the discrimination of different categories of features. Based on UNet, the proposed CFRM is added at the top of the encoding path. Compared with other state-of-art methods, both the Dice coefficients and the Iou coefficients of the proposed method have reached the best level on our pancreatic adenosquamous carcinoma segmentation dataset.

Automatic nuclear instance segmentation is a crucial task in computational pathology as this information is required for extracting cell-based features in down-stream analysis. However, instance segmentation is a challenging task due to the nature of histology images which show large variations and irregularities in nuclei appearances and arrangements. Various deep learning-based methods have tried to tackle these challenges, however, most of these methods fail to segment the nuclei instances in crowded scenes accurately, or they are not fast enough for practical usage. In this paper, we propose a double-stage neural network for nuclear instance segmentation which leverages the power of an interactive model, NuClick, for accurate instance segmentation by replacing the user input with a nuclei detection module, YOLOv5. We optimized the proposed method to be lightweight and fast and show that it can achieve promising results when tested on the largest publicly available nuclei dataset.

Light microscopy of tissue slides is an important tool for analyzing human diseases including cancer. In this work, we focus on classifying patches from a immunohistochemically stained tissue microarray (TMA) of vulvar cancer. We propose a novel ensemble-based deep learning technique to classify patches of tissue as cancerous, stroma, both, or none. Our ensemble model consists of a pre-trained data-efficient image transformer (DeiT) module to extract features of patches followed by a transformer block and graph convolutional networks (GCN). Transformer blocks aid the sequential learning of the extracted features from DeiT while the graph convolutional network (GCN) extracts neighborhood information. Our approach combines both methods for classification. In the evaluation, we show that our approach outperforms state-of-the-art architectures for the addressed application. We also show that is applicable when only small amounts of labelled data are available.