This PDF file contains the front matter associated with SPIE Proceedings Volume 12039, including the Title Page, Copyright information, Table of Contents and Conference Committee listings.

This Conference Presentation, “Data-efficient and multimodal computational pathology” was presented at the Digital and Computational Pathology conference at SPIE Medical Imaging 2022.

Anatomic pathology is still the gold standard for tissue-based disease diagnosis, is centered on interpretation of H&E-stained sections on glass slides, a process requires labor-intensive and time-consuming specimen processing. We propose a simple, clinically relevant solution, termed FIBI (fluorescence imitating brightfield imaging), for directly creating diagnostic-quality images from unsectioned, fresh or fixed tissue specimens. FIBI can generate full-color histology-grade images within minutes. We have demonstrated the validity of this method by collecting 50 tissue samples from various organs and pathologies and comparing the diagnosis obtained using FIBI images with those determined from adjacent, conventionally prepared H&E-stained slides.

The liver is a common site for metastases for multiple cancer types – with adenocarcinoma being the most common subtype. Standard pathology assessments fail to identify a site of origin for three to five percent of these patients, who are then treated in an empirical manner.¹ Computational pathology is an ideal technology to improve diagnostic accuracy for metastatic cancers with a significant reduction in costs and increase in tissue preservation for future testing. Recently, Deep Learning (DL)-based algorithms have been developed to automatically identify the origination of metastases using only H&E-stained whole slide images as inputs.² However, immunohistochemical based analysis is time- and resource-intensive and tissue destructive, while the “black box” nature of the DL-based approach makes it difficult if not impossible to explain the biology behind the classifier’s decision. This pilot study demonstrates that an analysis of computer extracted quantitative cellular features from tumor epithelial nuclei, including a tumor nuclei morphological feature and graph feature as well as cytoplasm texture from H&E stained histology slides, can be used to identify the origination of liver metastasis. Since the primary source of liver metastasis is typically the colon, breast, pancreas, or esophagus,³ a data set consisting of 120 patients with primary adenocarcinoma and matched liver metastases from the breast, colon, esophagus, and pancreas was constructed. A combination of supervised classification and unsupervised clustering was applied in order to identify the tumors’ origination using the above mentioned histomorphometric features from the data set. We evaluated the utility of histomorphometric features to identify the origination of the metastatic tumors in the liver using both supervised machine classifiers and unsupervised clustering. A Random Forest classifier achieved AUCs of 0.83, 0.63, 0.82, and 0.64 in classifying the primary or metastatic tumor from colon, esophagus, breast, and pancreas. The top performing features included local cellular diversity as well as cytoplasmic texture feature families. We further verified selected features using unsupervised methods such as UMAP, hierarchical clustering, and content-based image retrieving. Based off similarity measures calculated between image tiles in the primary and metastatic tissue, a heatmap corresponding to the WSIs for the primary tumor was constructed to highlight the sites representing potential points of origination of the metastases. Our preliminary results suggest that the site of origination for metastases in the context of primary breast tumors appears to be within the stroma.

Feature representations of histopathology whole slide images (WSIs) are crucial to the downstream applications for computer-aided cancer diagnosis, including whole slide image classification, region of interest detection, hash retrieval, prognosis analysis, and other high-level inference tasks. State-of-the-art methods for whole slide image feature extraction generally rely on supervised learning algorithms based on fine-grained manual annotations, unsupervised learning algorithms without annotation, or directly use pre-trained features. At present, there is a lack of research on weakly supervised feature learning methods that only utilize WSI-level labeling. In this paper, we propose a weakly supervised framework that learns the feature representations of various lesion areas from histopathology whole slide images. The proposed framework consists of a contrastive learning network as the backbone and a designed contrastive dynamic clustering (CDC) module to embedding the lesion information into the feature representations. The proposed method was evaluated on a large scale endometrial whole slide image dataset. The experimental results have demonstrated that our method can learn discriminative feature representations for histopathology image classification and the quantitative performance of our method is close to the fully-supervision learning methods. The code is available at https://github.com/junl21/cdc.

Machine learning techniques have shown great promise in digital pathology. However, a major bottleneck is the difficulty of annotating necessary amount of tissue to deal with several variability factors, namely chemical fixation, sample slicing, or staining. Usually, models are trained using sets of annotated small image patches, but then, the number of required patches may increase exponentially and yet they must represent such variability. This paper presents a method for automatic sample selection to train a classifier for ovarian cancer by integrating a novel soft clustering strategy. The method starts by classifying a large set of patches with a previously trained classifier and divide patches from the cancer class as highly and moderately confident. An unsupervised selection of moderately confident patches by a Probabilistic Latent Semantic Analysis (PLSA), picks samples from relevant and meaningful groups with maximum within-group variance. A new model is re-trained using the highly confident patches together with patches obtained from the associated PLSA. This strategy outperforms a model trained with a larger set of annotated patches while the training times and the number of samples are much more smaller. The strategy was evaluated in a set of patches from 18 patients with Serous Ovarian Cancer, obtaining a reduction of 54.62% in the training time and 73.66% in the number of samples, while recall rate improved from 0.69 to 0.73.

Active Learning (AL) is an artificial intelligence (AI) training paradigm that improves training efficiency in cases where labeled training is hard to obtain. In AL, unlabeled samples are selected for annotation using a bootstrap classifier to identify samples whose informational content is not represented in the current training set. Given a small number of samples, this optimizes training by focusing annotation on “informative” samples. For computational pathology, identifying the most-informative samples is non-trivial, particularly for segmentation. In this work, we develop a feature-driven approach to identifying informative samples. We use a feature extraction pipeline operating on segmentation results to find “outlier” samples which are likely incorrectly segmented. This process allows us to automatically flag samples for re-annotation based on architecture of segmentation (compared with less robust confidence-based approaches). We apply this process to the problem of segmenting oral cavity cancer (OCC) H&E stained whole-slide images (WSIs), where the architecture of OCC tumor growth is an aggressive pathological indicator. Improving segmentation requires costly annotation of WSIs; thus, we seek to employ an AL approach to improve annotation efficiency. Our results show that, while outlier features alone are not sufficient to flag samples for re-annotation, we can identify some WSIs which fail segmentation.

Tau proteins in the gray matter are widely known to be a part of Alzheimer’s disease symptoms. They can aggregate in three different structures within the brain: neurites, tangles, and neuritic plaques. The morphology and the spatial disposition of these three aggregates are hypothesised to be correlated to the advancement of the disease. In order to establish a behavioural disease model related to the Tau proteins aggregates, it is necessary to develop algorithms to detect and segment them automatically. We present a 5-folded pipeline aiming to perform with clinically operational results. This pipeline is composed of a non-linear colour normalisation, a CNN-based image classifier, an Unet-based image segmentation stage, and a morphological analysis of the segmented objects. The tangle detection and segmentation algorithms improve state-of-the-art performances (75.8% and 91.1% F1- score, respectively), and create a reference for neuritic plaques detection and segmentation (81.3% and 78.2% F1-score, respectively). These results constitute an initial baseline in an area where no prior results exist, as far as we know. The pipeline is complete and based on a promising state-of-the-art architecture. Therefore, we consider this study a handy baseline of an impactful extension to support new advances in Alzheimer’s disease. Moreover, building a fully operational pipeline will be crucial to create a 3D histology map for a deeper understanding of clinico-pathological associations in Alzheimer’s disease and the histology-based evidence of disease stratification among different sub-types.

Tumor budding (TB) is defined as a cluster of one to four tumor cells at the tumor invasive front. Though promising as a prognostic factor for colorectal cancer, its routine clinical use is hampered by high inter- and intra- observer disagreement on routine H&E staining. Pan-cytokeratin immunohistochemical staining increases agreement but is costly, nonroutine, and may yield tumor bud false positives. This makes the development of automatic algorithms to identify TB difficult. Therefore, we sought to develop an automated method to generate ground truth for tumor budding from routine H&E that benefits from pan-cytokeratin staining. Our database consisted of 1) 120 adjacent pan-cytokeratin and H&E tissue sections from which 648 manually-registered high-power fields were extracted and annotated for tumor regions and 2) ten adjacent sections from which 109 fields were extracted and annotated for tumor buds. Swin transformers were applied to segment tumor regions, resulting in 95% and 99% accuracy for tumor and non-tumor regions. This allowed for a narrow margin outside the tumor invasive front to identify TB. Next, we developed a novel method which registers pan-cytokeratin images to adjacent H&E images, transfers automatically-identified positive regions of pan-cytokeratin to H&E, then filters positive regions on H&E using size and number of nuclei. The average precision and recall were 0.3856 and 0.3254 on H&E when compared to pathologists on H&E, which exceeded 0.3470 and 0.1932 when comparing pathologists on H&E to pan-cytokeratin. Results provide preliminary evidence for the feasibility of our method to generate ground truth for TB using H&E.

In our work, we present an approach to regressing breast cancer cellularity in patches extracted from Whole Slide Imagery (WSI) on Hematoxylin and Eosin (H&E) stains using a fully-convolutional neural network which is trained with two heads: one which computes a global average pool for weakly-labeled data (data with a cellularity score of 0- 1.0) and another which enforces pixel-wise activations for strongly-labeled (segmentation) data. Our method was the top-performing algorithm of all submissions to the BreastPathQ challenge, achieving a prediction probability of 0.941.

We present WeakSTIL, an interpretable two-stage weak label deep learning pipeline for scoring the percentage of stromal tumor infiltrating lymphocytes (sTIL%) in H&E-stained whole-slide images (WSIs) of breast cancer tissue. The sTIL% score is a prognostic and predictive biomarker for many solid tumor types. However, due to the high labeling efforts and high intra- and interobserver variability within and between expert annotators, this biomarker is currently not used in routine clinical decision making. WeakSTIL compresses tiles of a WSI using a feature extractor pre-trained with self-supervised learning on unlabeled histopathology data and learns to predict precise sTIL% scores for each tile in the tumor bed by using a multiple instance learning regressor that only requires a weak WSI-level label. By requiring only a weak label, we overcome the large annotation efforts required to train currently existing TIL detection methods. We show that WeakSTIL is at least as good as other TIL detection methods when predicting the WSI-level sTIL% score, reaching a coefficient of determination of 0.45 ± 0.15 when compared to scores generated by an expert pathologist, and an AUC of 0.89 ± 0.05 when treating it as the clinically interesting sTIL-high vs sTIL-low classification task. Additionally, we show that the intermediate tile-level predictions of WeakSTIL are highly interpretable, which suggests that WeakSTIL pays attention to latent features related to the number of TILs and the tissue type. In the future, WeakSTIL may be used to provide consistent and interpretable sTIL% predictions to stratify breast cancer patients into targeted therapy arms.

Human epidermal growth factor (HER2) is a predictive and prognostic biomarker whose degree of presence in breast cancer informs prognosis and therapeutic decision making. In clinical practice, it is routinely assessed using immunohistochemical (IHC) staining. A pathologist assigns a score from 0 to 3+ depending on the intensity and distribution of staining – 0 or 1+ scores are classified as HER2 negative, 3+ scores as HER2 positive, and 2+ as equivocal. Unfortunately, variations in HER2 staining and the subjectivity in scoring can lead to inaccuracies. Therefore, we sought to develop an automated method to predict HER2 scores from HER2 and H&E slide images. Our database consisted of 52 adjacent HER2 and H&E tissue sections. Positive regions on HER2 were segmented using a previously developed method. Using 13-fold cross-validation, a truncated Resnet18 was then trained to classify extracted patches using HER2 score as labels for positive regions and a score of 0 for negative regions. Using the same folds, attentionbased multiple instance learning was used to aggregate learned patch embeddings into overall slide-level embeddings, which were subsequently classified. The preliminary method achieved 88% accuracy on 0/1+ and 85% accuracy on 2+ and 3+. Subsequent preliminary experiments qualitatively demonstrate that identified positive regions from HER2 can successfully be transferred over to H&E via image registration. Furthermore, applying the proposed method to predict HER2 score from H&E demonstrates that attention is paid to HER2 positive regions on H&E. Results provide preliminary evidence that HER2 can be localized and therefore scored using H&E images alone.

Breast cancer is one of the most common occurring cancers in women. Residual cancer burden measures tumor response to the therapy and is shown to be prognostic for long term survival. The percentage of invasive or in situ tumor cellularity is an important component of tumor burden assessment. In the current clinical practice, tumor cellularity is manually estimated by pathologists on hematoxylin and eosin (H&E) stained slides. In this paper, we present a deep ordinal regression framework to automatically assess cellularity from pathological images. We formulate the cellularity assessment as an ordinal regression problem and address by an end-to-end learning approach using deep convolutional neural networks. We evaluated the proposed methods on the SPIE BreastPathQ challenge dataset and achieved significant higher agreement with expert pathologist scoring in terms of intraclass correlation (ICC) of 0.94 (vs. 0.89 of the inter-rater agreements between pathologists).

Hyperspectral (HS) imaging (HSI) is a novel technique that allows a better understanding of materials, being an improvement respect to other imaging modalities in multiple applications. Specifically, HSI technology applied to breast cancer histology, could significantly reduce the time of tumor diagnosis at the histopathology department. First, histological samples from twelve different breast cancer patients have been prepared and examined. Second, they were digitally scanned, using RGB (Red-Green-Blue) whole-slide imaging, and further annotated at cell level. Then, the annotated regions were captured with an HS microscopic acquisition system at 20× magnification, covering the 400-1000 nm spectral range. The HS data was registered (through synthetic RGB images) to the whole-slide images, allowing the transfer of accurate annotations made by pathologists to the HS image and extract each annotated cell from such image. Then, both spectral and spatial-spectral classifications were carried out to automatically detect tumor cells from the rest of the coexisting cells in the breast tissue (fibroblasts and lymphocytes). In this work, different supervised classifiers have been employed, namely kNN (k-Nearest-Neighbors), Random Forest, DNN (Deep Neural Network), Support Vector Machines (SVM) and CNN (Convolutional Neural Network). Test results for tumor cells vs. fibroblast classification show that the kNN performed with the best sensitivity/specificity (64/52%) trade-off and the CNN achieved the best sensitivity and AUC results (96% and 0.91, respectively). Moreover, at the tumor cells vs. lymphocyte classification, kNN also provided the best sensitivity-specificity ratio (58.47/58.86%) and an F1-score of 74.12%. The SVM algorithm also provided a high F-score result (70.38%). In conclusion, several machine learning algorithms provide promising results for cell classification in breast cancer tissue and so, future work must include these discoveries for faster cancer diagnosis.

There has been a long pursuit for precise and reproducible glomerular quantification in the field of renal pathology in both research and clinical practice. Currently, 3D glomerular identification and reconstruction of large-scale glomeruli are labor-intensive tasks, and time-consuming by manual analysis on whole slide imaging (WSI) in 2D serial sectioning representation. The accuracy of serial section analysis is also limited in the 2D serial context. Moreover, there are no approaches to present 3D glomerular visualization for human examination (volume calculation, 3D phenotype analysis, etc.). In this paper, we introduce an end-to-end holistic deep-learning-based method that achieves automatic detection, segmentation and multi-object tracking (MOT) of individual glomeruli with large-scale glomerular-registered assessment in a 3D context on WSIs. The high-resolution WSIs are the inputs, while the outputs are the 3D glomerular reconstruction and volume estimation. This pipeline achieves 81.8 in IDF1 and 69.1 in MOTA as MOT performance, while the proposed volume estimation achieves 0.84 Spearman correlation coefficient with manual annotation. The end-to-end MAP3D+ pipeline provides an approach for extensive 3D glomerular reconstruction and volume quantification from 2D serial section WSIs.

The study is to incorporate polarized hyperspectral imaging (PHSI) with deep learning for automatic detection of head and neck squamous cell carcinoma (SCC) on hematoxylin and eosin (H&E) stained tissue slides. A polarized hyperspectral imaging microscope had been developed in our group. In this paper, we firstly collected the Stokes vector data cubes (S0, S1, S2, and S3) of histologic slides from 17 patients with SCC by the PHSI microscope, under the wavelength range from 467 nm to 750 nm. Secondly, we generated the synthetic RGB images from the original Stokes vector data cubes. Thirdly, we cropped the synthetic RGB images into image patches at the image size of 96x96 pixels, and then set up a ResNet50- based convolutional neural network (CNN) to classify the image patches of the four Stokes vector parameters (S0, S1, S2, and S3) by application of transfer learning. To test the performances of the model, each time we trained the model based on the image patches (S0, S1, S2, and S3) of 16 patients out of 17 patients, and used the trained model to calculate the testing accuracy based on the image patches of the rest 1 patient (S0, S1, S2, and S3). We repeated the process for 6 times and obtained 24 testing accuracies (S0, S1, S2, and S3) from 6 different patients out of the 17 patients. The preliminary results showed that the average testing accuracy (84.2%) on S3 outperformed the average testing accuracy (83.5%) on S0. Furthermore, 4 of 6 testing accuracies of S3 (96.0%, 87.3%, 82.8%, and 86.7%) outperformed the testing accuracies of S0 (93.3%, 85.2%, 80.2%, and 79.0%). The study demonstrated the potential of using polarized hyperspectral imaging and deep learning for automatic detection of head and neck SCC on pathologic slides.

Hyperspectral imaging (HSI), a non-invasive imaging modality, has been successfully used in many different biological and medical applications. One such application is in the field of oncology, where hyperspectral imaging is being used on histologic samples. This study compares the performances of different image classifiers using different imaging modalities as training data. From a database of 33 fixed tissues from head and neck patients with follicular thyroid carcinoma, we produced three different datasets: an RGB image dataset that was acquired from a whole slide image scanner, a hyperspectral (HS) dataset that was acquired with a compact hyperspectral camera, and an HS-synthesized RGB image dataset. Three separate deep learning classifiers were trained using the three datasets. We show that the deep learning classifier trained on HSI data has an area under the receiver operator characteristic curve (AUC-ROC) of 0.966, higher than that of the classifiers trained on RGB and HSI-synthesized RGB data. This study demonstrates that hyperspectral images improve the performance of cancer classification on whole histologic slides. Hyperspectral imaging and deep learning provide an automatic tool for thyroid cancer detection on whole histologic slides.

Cytometry plays essential roles in immunology and oncology. Recent advancements in cellular imaging allow more detailed characterization of cells by labeling each cell with multiple protein markers. The increase of dimensionality makes manual analysis challenging. Clustering algorithms provide a means for phenotyping high-dimensional cell populations in an unsupervised manner for downstream analysis. The choice and usability of the methods are critical in practice. Literature provided comprehensive studies on those topics using publicly available flow cytometry data, which validated cell phenotypes by those methods against manual gated cell populations. In order to extend the knowledge for identification of cell phenotypes including unknown cell populations in our dataset, we conducted an exploratory study using clinical relevant tissue types as reference standard. Using our in-house database of multiplexed immunofluorescence images of breast cancer tissue microarrays (TMAs), we experimented with two commonly used algorithms (PhenoGraph and FlowSOM). Our pipeline includes: 1) cell phenotyping using Phenograph/FlowSOM; 2) clustering TMA cores into four groups using the percentage of each cell phenotypes with the algorithms (PhenoGraph/Spectral/K-means); 3) comparing the tissue groups to clinically relevant subtypes that were manually assigned based on the immunohistochemistry scores of serial sections. We experimented with different hyperparameter settings and input markers. Cell phenotypes using Phenograph with 10 markers and tissue clustering using Spectral yielded the highest mean F-measure (average over four tissue subtypes) of 0.71. In general, our results showed that cell phenotypes by Phenograph yielded better performance with larger variations than FlowSOM, which gives very consistent results.

Multiplex immunofluorescence (MxIF) is an emerging technique that allows for staining multiple cellular and histological markers to stain simultaneously on a single tissue section. However, with multiple rounds of staining and bleaching, it is inevitable that the scarce tissue may be physically depleted. Thus, a digital way of synthesizing such missing tissue would be appealing since it would increase the useable areas for the downstream single-cell analysis. In this work, we investigate the feasibility of employing generative adversarial network (GAN) approaches to synthesize missing tissues using 11 MxIF structural molecular markers (i.e., epithelial and stromal). Briefly, we integrate a multi-channel high-resolution image synthesis approach to synthesize the missing tissue from the remaining markers. The performance of different methods is quantitatively evaluated via the downstream cell membrane segmentation task. Our contribution is that we, for the first time, assess the feasibility of synthesizing missing tissues in MxIF via quantitative segmentation. The proposed synthesis method has comparable reproducibility with the baseline method on performance for the missing tissue region reconstruction only, but it improves 40% on whole tissue synthesis that is crucial for practical application. We conclude that GANs are a promising direction of advancing MxIF imaging with deep image synthesis.

Single cell phenotyping using molecular or protein multiplexing techniques is gaining momentum, especially in the characterization of cancer and the tumor microenvironment. It has proven to be particularly useful in studying the extent of heterogeneity in cancer, and in the profiling of the immune environment to assess whether certain cell subsets could be predictive of treatment response. Using a sequential protein marker labelling system called Multiplex Immunofluorescence (MxIF, GE Research), we have developed quantitative image analysis and computational tools for phenotyping individual immune and cancer cells for various cancer types. The expressions of T cell markers CD3, CD8, macrophage markers CD68, immune checkpoint proteins PD-1 and PD-L1, together with proliferative marker (Ki67) and cancer-specific marker PCK (pan-Cytokeratin) were studied on single 4um sections of formalin-fixed, paraffinembedded (FFPE) ovarian cancer tissue sections. We explored the composition of immune phenotype using t-SNE and quantified cell densities and marker co-expression patterns using binary cell counting. In addition to phenotyping immune cell types, their spatial localizations were analyzed. Neighborhood analysis was conducted using co-occurrence matrices to determine the number of times that a particular cell type is proximal to one another. Cell-to-cell spatial relationship was assessed by quantifying the Euclidean distances between individual cell types. These tools are being applied to specimens from an immunotherapy clinical trial to evaluate the dynamic changes in immune phenotype during the course of immune blockade therapy.

Federated learning (FL) contributed to the popularity of using multi-centric and collaborative integration of decentralized data across multiple institutions to train a robust model. Yet, data heterogeneity among the distributed clients remains a big challenge in FL. Specifically, in the case of digital histology, stain variation is commonplace. In a collaboration setting, color normalization and data augmentation are adopted to alleviate the variation. But they cannot be directly applied in the FL paradigm, as the requirement of data sharing or the availability of proxy samples. To address this issue, we propose a novel personalized federated learning (PFL) approach. Personalization stain transfer layers learn to project stain variant input images into a homogeneous space before being fed to the FL backbone. The proposed method is simple yet efficient. Empirical results on public available large patient cohorts demonstrate an observable classification accuracy improvement on popular neural network architectures including ResNet, Vgg, Wide ResNet.

Collecting substantial amounts of tissue image data is expensive, time consuming and often restricted by sample availability. This data sparsity limits the use of neural networks for image classification and knowledge extraction for many medical and biological applications. In this work, we present a novel method to procedurally generate synthetic fluorescent multiplex immunohistochemistry (fm-IHC) images using human-defined rules and a conditional Generative Adversarial Network (cGAN). Our method consists of three steps: First, we compress high-resolution fm-IHC images using a cell-based compression method (Cell2Grid) in which every cell is represented by just one pixel. We subsequently train a cGAN to reverse this compression. In a second step, we define a procedural algorithm for the generation of synthetic Cell2Grid images. We therefore (1) define rules for the generation of large-scale tissue morphology, (2) define the relative abundance of cell phenotypes in each tissue segment, and (3) define marker distributions for each phenotype. In a third step, we apply the trained cGAN to convert synthetic Cell2Grid images into fm-IHC images. As a showcase, we generated synthetic fm-IHC images of murine pancreatic islets in different stages of insulitis consisting of seven antibody color channels. We derived rules for the procedural algorithm using only 59 real sample images and trained the cGAN on 1,575 unlabeled fm-IHC image crops. We show that our method provides full control over tissue morphology, cell composition, and image ground truth and can support the training of image classification networks.

Hematoxylin- and eosin-stained tissue sections are central to patient diagnosis and management guidance. Direct viewing of these slides using a microscope is slowly but seemingly inevitably giving way to the new world of digital pathology. Having the histology images in digital format facilitates deployment of digital analysis tools, ranging from basic image enhancement to straightforward assessment of quantifiable metrics (area, intensity, molecular marker intensity and distribution) all the way to applications of novel machine learning and artificial intelligence tools. However, to date, virtually all such H&E whole-slide scans have been accomplished only in brightfield mode. We have observed that simple fluorescence imaging of H&E-stained slides can provide a great deal of useful histology content. We have developed a scanner that can acquire pixel-matched brightfield and fluorescence images in area-scanning mode, a process we termed DUET, for DUal-mode Emission and Transmission, i

Hyperspectral imaging (HSI) has many advantages in microscopic applications, including high sensitivity and specificity for cancer detection on histological slides. However, acquiring hyperspectral images of a whole slide with a high image resolution and a high image quality can take a long scanning time and require a very large data storage. One potential solution is to acquire and save low-resolution hyperspectral images and reconstruct the high-resolution ones only when needed. The purpose of this study is to develop a simple yet effective unsupervised super resolution network for hyperspectral histologic imaging with the guidance of RGB digital histology images. High-resolution hyperspectral images of hemoxylin & eosin (H&E) stained slides were obtained at 10× magnification and down-sampled 2×, 4×, and 5× to generate low-resolution hyperspectral data. High-resolution digital histologic RGB images of the same field of view (FOV) were cropped and registered to the corresponding high-resolution hyperspectral images. A neural network based on a modified U-Net architecture, which takes the low-resolution hyperspectral images and high-resolution RGB images as inputs, was trained with unsupervised methods to output high-resolution hyperspectral data. The generated high-resolution hyperspectral images have similar spectral signatures and improved image contrast than the original high-resolution hyperspectral images, which indicates that the super resolution network with RGB guidance can improve the image quality. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will potentially promote the use of hyperspectral imaging technology in digital pathology and many other clinical applications.

Podocyte injury plays a crucial role in the progression of diabetic kidney disease (DKD). Injured podocytes demonstrate variations in nuclear shape and chromatin distribution. These morphometric changes have not yet been quantified in podocytes. Furthermore, the molecular mechanisms underlying these variations are poorly understood. Recent advances in omics have shed new lights into the biological mechanisms behind podocyte injury. However, there currently exists no study analyzing the biological mechanisms underlying podocyte morphometric variations during DKD. First, to study the importance of nuclear morphometrics, we performed morphometric quantification of podocyte nuclei from whole slide images of renal tissue sections obtained from murine models of DKD. Our results indicated that podocyte nuclear textural features demonstrate statistically significant difference in diabetic podocytes when compared to control. Additionally, the morphometric features demonstrated the existence of multiple subpopulations of podocytes suggesting a potential cause for their varying response to injury. Second, to study the underlying pathophysiology, we employed single cell RNA sequencing data from the murine models. Our results again indicated five subpopulations of podocytes in control and diabetic mouse models, validating the morphometrics-based results. Additionally, gene set enrichment analysis revealed epithelial to mesenchymal transition and apoptotic pathways in a subgroup of podocytes exclusive to diabetic mice, suggesting the molecular mechanism behind injury. Lastly, our results highlighted two distinct lineages of podocytes in control and diabetic cases suggesting a phenotypical change in podocytes during DKD. These results suggest that textural variations in podocyte nuclei may be key to understanding the pathophysiology behind podocyte injury.

Glomeruli are clusters of capillaries that are responsible for filtering the blood to form urine, thus excreting waste and maintaining fluid and acid-base balance. The detection and characterization of glomeruli are key elements in diagnostic and experimental nephropathology. Although the field of machine vision has already advanced the detection, classification, and prognostication of diseases in the specialties of radiology and oncology, renal pathology is just entering the digital imaging era. However, developing quantitative machine learning approaches (e.g., self-supervised deep learning) that characterize glomerular lesions (e.g., global glomerulosclerosis (GGS)) from whole slide images (WSIs) typically requires large-scale heterogeneous images, which is resource extensive for individual labs. In this study, we assess the feasibility of leveraging fine-grained GGS characterization via large-scale web image mining (e.g., from journals, search engines, websites) and self-supervised deep learning. Three types of GGS were assessed-solidified (S-GGS, associated with hypertension-related injury), disappearing (D-GGS, a further end result of the SGGS becoming contiguous with fibrotic interstitium), and obsolescent (O-GGS, nonspecific GGS increasing with aging). We employed the SimSiam network as the baseline method of self-supervised contrastive learning. By deploying our previously developed compound figure separation approach, we provided 30,000 unannotated glomerular images via web image mining to train the SimSiam network. From the results, the GGS fine-grained classification model achieved superior performance compared with baseline methods. The segmentation networks evaluated across six different resolutions

The incorporation of automated computational tools has a great amount of potential to positively influence the field of pathology. However, pathologists and regulatory agencies are reluctant to trust the output of complex models such as Convolutional Neural Networks (CNNs) due to their usual implementation as black-box tools. Increasing the interpretability of quantitative analyses is a critical line of research in order to increase the adoption of modern Machine Learning (ML) pipelines in clinical environments. Towards that goal, we present HistoLens, a Graphical User Interface (GUI) designed to facilitate quantitative assessments of datasets of annotated histological compartments. Additionally, we introduce the use of hand-engineered feature visualizations to highlight regions within each structure that contribute to particular feature values. These feature visualizations can then be paired with feature hierarchy determinations in order to view which regions within an image are significant to a particular sub-group within the dataset. As a use case, we analyzed a dataset of old and young mouse kidney sections with glomeruli annotated. We highlight some of the functional components within HistoLens that allow non-computational experts to efficiently navigate a new dataset as well as allowing for easier transition to downstream computational analyses.

Immunotherapy is a novel anti-cancer treatment that shows significant improvements in outcomes for lung cancer patients. However, this treatment has the potential for substantial side effects, and many lung cancer patients do not benefit from it. Programmed-death ligand-1 expression in tumor cells is currently the main biomarker used to identify those who might benefit but it is not very accurate. Tumor mutational burden (TMB) is a promising alternative, with lung cancers having more than 10 mutations/megabase being more likely to respond to immunotherapy. However, the cost and time it takes to obtain TMB makes it difficult to implement in the clinic. In our previous work, we have shown that TMB status can be predicted from digitized hematoxylin and eosin histology slides that are routinely obtained from surgical resection of squamous cell carcinoma from a single center using the deep learning technique of transfer learning. In this study, we have shown that this is possible across different centers and explored various design parameters for such a system using 30 slides from 20 different centers from the Cancer Genome Atlas. The parameters exploration yielded 108 models with a median area under the receiver operator characteristic curve of 0.70, and an interquartile range of 0.30 on the validation set (n=7). The model we selected for further testing had perfect classification. This motivates additional work in this direction to build a system that can be used in the future to inform physicians as to which patients with squamous lung carcinoma would benefit from immunotherapy.

Histological image data and molecular profiles provide context into renal condition. Often, a biopsy is drawn to diagnose or monitor a suspected kidney problem. However, molecular profiles can go beyond a pathologist’s ability to see and diagnose. Using AI, we computationally incorporated urinary proteomic profiles with microstructural morphology from renal biopsy to investigate new and existing molecular links to image phenotypes.

We studied whole slide images of periodic acid-Schiff stained renal biopsies from 56 DN patients matched with 2,038 proteins measured from each patient’s urine. Using Seurat, we identified differentially expressed proteins in patients that developed end-stage renal disease within 2 years of biopsy. Glomeruli, globally sclerotic glomeruli, and tubules were segmented from WSI using our previously published HAIL pipeline. For each glomerulus, 315 handcrafted digital image features were measured, and for tubules, 207 features. We trained fully connected networks to predict urinary protein measurements that were differentially expressed between patients who did/ did not progress to ESRD within 2 years of biopsy. The input to this network was either glomerular or tubular histomorphological features in biopsy. Trained network weights were used as a proxy to rank which morphological features correlated most highly with specific urinary proteins.

We identified significant image feature-protein pairs by ranking network weights by magnitude. We also looked at which features on average were most significant in predicting proteins. For both glomeruli and tubules, RGB color values and variance in PAS+ areas (specifically basement membrane for tubules) were, on average, more predictive of molecular profiles than other features.

There is a strong connection between molecular profile and image phenotype, which can be elucidated through computational methods. These discovered links can provide insight to disease pathways, and discover new factors contributing to incidence and progression.

Stage III Colorectal cancer (CRC) is treated with surgery followed by chemotherapy. Yet >20% of clinically low-risk patients develop recurrence. It is critical to identify high-risk stage III patients who can benefit from closer monitoring and escalation of therapy. Previous studies showed promising results in predicting risk from H&E slides in CRC using deep learning based algorithms. One biomarker which was heavily studied and showed good results in predicting risk is Tumor Infiltrating Lymphocytes (TILs). In CRC, TIL density has been shown to be significantly and independently associated with overall patient survival¹. Furthermore, additional studies have demonstrated that analyzing spatial organization of TILs could be more informative than density alone². Hence, our study aimed to stratify stage III CRC patients into distinct risk groups based on features derived from TILs and to determine if this classification could have independent prognostic significance. The training set (D₁) included 50 patients and validation set (D₂) consisted of 70 patients from an independent site. A survival model was trained to predict the risk of recurrence in stage III CRC patients. First, a deep learning (DL) model was used to segment TILs on WSIs. Next, 1036 features related to spatial architecture (SpaTIL) and density of TILs (DenTIL) were extracted. A Cox proportional hazards regression model in conjunction with the least absolute shrinkage and selection operator (Lasso) was used to find top 5 features and the feature coefficients associated with progression free survival (PFS) and provide risk scores to each patient. The risk scores for the training dataset were computed using the selected features with their respective coefficients. A cut-off value was determined according to these risk scores, above which patients were labelled high-risk and below was low risk. In the validation set, the median PFS for the high-risk group was 15.1mos and in the low-risk group was 27mos. The model was able to accurately predict higher incidence of progression in patients in the high-risk group (HR = 3.76, 95% CI 1.3-10.9, p-value=0.0053, c-index=0.687) in the validation set. Future work will entail additional multi-site, multi-institutional validation of our biomarker to further understand its strengths and applications.

Myelodysplastic syndromes (MDS) are relatively rare blood diseases that vary widely in their severity, symptoms, and tendency to progress to acute myeloid leukemia and therefore require precise diagnosis and classification^1,2 . Flow cytometry immunophenotyping of bone marrow cells could be helpful in making the diagnosis of MDS³. Due to natural properties of the fluorescent dyes used in flow cytometry, raw digital data from the instrument must be compensated to account for the spillover of signal between fluorochromes.

“Ground truth” cell type classification in MDS immunophenotype flow cytometry panel of 14 markers performed on samples from patients with confirmed MDS (n=118), precursor condition (CCUS, n=86), non-clonal idiopathic cytopenia of uncertain significance (ICUS, n=152) and normal controls (n=21) was performed using Infinicyt.

A neural network with an input layer accepting light scatter properties (6 channels) and fluorescent channels (8 channels) for each tube along with a tube indicator (15 total channels) followed by three fully connected hidden layers (64, 128 and 64 nodes) and an output layer including aggregates, basophils, blasts, dendritic cells, debris, granulocytes, hematogones, lymphocytes, mast cells, monocytes, plasma cells, RBCs, and unknown was trained twice on a randomly selected 80% of 353,655,369 unique events, once on uncompensated data and again with the per-tube compensated data. The uncompensated network trained to a cost of 0.16514 in 275 epochs. The compensated network reached a cost of 0.17089 after 1067 epochs. Tested on reserved data, the networks perform essentially identically, providing support to the potential clinical validity of using uncompensated data.

Perineural invasion refers to a process where tumor cells invade, surround, or pass through nerve cells, serving as an indicator of aggressive tumor and related to poor prognosis. Herein, we propose an efficient and effective hybrid computational method for an automated detection of perineural invasion junctions in multi-tissue digitized histology images. The proposed approach conducts the detection of perineural invasion junctions in three stages. The first state identifies candidate regions for perineural invasion. The second stage delineates perineural invasion junctions. The last stage eliminates any false positive regions for perineural invasion. In the first two stages, we exploit an advanced deep neural network. In the last stage, we utilize hand-crafted features and a conventional machine learning algorithm. To evaluate the proposed approach, we employ 150 whole slide images obtained from PAIP2021 Challenge: Perineural Invasion in Multiple Organ Cancer and conduct a five-fold cross-validation. The experimental results show that the proposed hybrid approach could facilitate an automated, accurate identification of perineural invasion in histology images.

Nuclei detection is a key task in Ki67 proliferation index estimation in breast cancer images. Deep learning algorithms have shown strong potential in nuclei detection tasks. However, they face challenges when applied to pathology images with dense medium and overlapping nuclei since _ne details are often diluted or completely lost by early maxpooling layers. This paper introduces an optimized UV-Net architecture, specifically developed to recover nuclear details with high-resolution through feature preservation for Ki67 proliferation index computation. UV-Net achieves an average F1-score of 0.83 on held-out test patch data, while other architectures obtain 0.74- 0.79. On tissue microarrays (unseen) test data obtained from multiple centers, UV-Net's accuracy exceeds other architectures by a wide margin, including 9-42% on Ontario Veterinary College, 7-35% on Protein Atlas and 0.3-3% on University Health Network.

Management of aggressive malignancies, such as glioma, is complicated by a lack of predictive biomarkers that could reliably stratify patients based on treatment outcome. The complex mechanisms driving glioma recurrence and treatment resistance cannot be fully understood without the integration of multiscale factors such as cellular morphology, tissue microenvironment, and macroscopic features of the tumor and the host tissue. We present a weakly-supervised, interpretable, multimodal deep learning-based model fusing histology, radiology, and genomics features for glioma survival predictions. The proposed framework demonstrates the feasibility of multimodal integration for improved survival prediction in glioma patients.

Shaped-based descriptors from Computed Tomography (CT) scans and whole slide digital pathology images were used to differentiate the two major histopathological subtypes of non-small-cell lung cancer (NSCLC). Our two hypotheses are 1) Encoding information on local heterogeneity will augment the model’s classification capabilities 2) Shape-based biomarkers from radiology and pathology can complement each other. Shape features were extracted from the tumor map from pathology and radiology images. In pathology, tumor-microenvironment features were encoded by clustering the tumor map into phenotype maps. These features performed better than the features from whole tumor map. Integration of radio-pathomics performed best, achieving 0.802 AUC.

Glioblastoma (GBM) is a highly aggressive brain tumor and is notoriously known for its intra-tumoral heterogeneity. Diagnosis of GBM is based on histopathology confirmation via tissue samples obtained from intra-cranial biopsies. After surgical intervention, histopathology tissue slides are visually analyzed by neuro-pathologists to identify distinct GBM histological hallmarks. GBMs may be histologically undergraded based on the amount of viable tissue due to sampling errors associated with small tissue samples obtained. Consequently, there is a need for automatic identification of histopathological GBM hallmarks. In this work, we present a hierarchical deep learning strategy to automatically segment distinct GBM niches including necrosis, cellular tumor, and hyperplastic blood-vessels, on H&E digitized histopathology slides. Our approach includes first segmenting necrosis and cellular tumor regions, then identifying hyperplastic blood-vessels within cellular tumor regions.

In our previous work, we have demonstrated that it is possible to use a small bootstrap set of fully annotated regions of interest (ROIs) to generate segmentation results on the WSI scale. In this work, pathologists were asked to edit the previously generated annotations on 150 WSIs, focusing on only the tumor class. Of these re-annotated WSIs, 21 were then sampled from, and used to train a new version of the classifier. Segmentation results were then generated for the remainder of the images. This work demonstrates an improvement in segmentation of the tumor class.

Precise tissue segmentation of histopathology images is often a crucial step in computational pathology pipelines. However, visual scoring by pathologists is sensitive and depends on their experience and perception. Therefore, there is a need for novel automatic systems to improve the accuracy and reproducibility of pathologists’ interpretations. Here, a group affinity weakly supervised segmentation method (GAWS) is proposed to conquer this task, with the following pipeline. First, we create a cluster image by extracting the visual feature of each pixel using CNN and clustering it into different classes. Then, we create a target image by refining this cluster image with the constraints on prior tissue, color, and spatial distribution of pixels. Finally, a backpropagation process with a segmentation loss is considered to evaluate the error signals between cluster and target images and update the network parameters. We validate our method with extracellular mucin-to-tumor area quantification using a colorectal cancer clinical dataset with 163 Hematoxylin Eosin (H&E) whole slide images from 97 patients. Inter-observer agreement between pathologists and the proposed algorithm is excellent (ICC=0.917) and more accurate compared with two state-of-the-art unsupervised segmentation methods. Our results show that the GAWS results in a high average performance and excellent reliability when applied to histopathology images and possibly is a promising method for inclusion into clinical practice. This approach takes advantage of weakly supervised learning without any pre-trained network to have a tumor quantification tool that could improve the pathologist’s workflow.

Image segmentation is one primary area in which deep learning has made a major contribution to medical image analysis. The automatic and precise segmentation of cells in cytopathology, or cytology for short, can significantly reduce the diagnostic work from pathologists. The biomedical image segmentation task routinely employs an encoder-decoder structure, e.g. U-Net, in which the receptive field is often fixed. However, to achieve a better morphological segmentation performance, we empirically found receptive field should be correlated with cell size by differential structures. In this paper, we proposed a novel deep-learning based cytology image segmentation model, namely CellSegNet. This model can dynamically catalog cells by their size, and subsequently fit to their corresponding light-weight structures, characterized with weighted multiple receptive fields to better retrieve feature extraction. The proposed model can outperform other state-of-art biomedical image segmentation networks with observable improvements. Moreover, the high interpretability of the proposed model can be flexibly extended to other cytology datasets. The source code in the experiments and part of our collection of cervical images are publicly available at https://github.com/SJTU-AI-GPU/CellSegNet.

It is important to correlate histopathology images with radiological images to establish disease signatures and radiomic features defined on radiological images. Although modern imaging modalities such as CT and MRI can provide detailed patient anatomy information, the pathological interpretation of these images may be difficult and subject to physician’s experiences. In contrast, histopathology images which provide tissue structure at cellular level is considered the gold standard for cancer diagnosis. Therefore, histopathological images fusion with radiological images helps interpretation and computer-aided disease classification and segmentation. In this study, we propose a new MR volume to histopathology slice image registration method using unsupervised deep learning (DL). The pathology images were first manually rotated to align with the MRI. Histopathology slice correspondence to the MRI slice was then established by calculating image similarity between the two. Because of the endorectal MRI coil, the MRI prostate is often pressed, causing deformation, from the posterior side. Even without endorectal coil, the bladder could cause the shape mismatch between the histology and MRI. Therefore, it is important to model the prostate deformation. In this study, a thin-plate-spline deformation model was calculated from prostate surface difference between the MRI slice and the histopathology. To match the image content, a DL-based method was proposed to deformably register the histopathology image to the MRI. The proposed method utilizes modality independent neighborhood descriptor (MIND) as the image similarity measure during network training. Tested on 10 cases, the SSIM between the two were on average 0.83 and 0.90 before and after registration. Visual inspection suggests good image registration performance of the proposed method.

The past few years digital pathology has been widely adopted. The display system is a crucial component in the overall digital pathology system, since pathologists decide based upon images visualized on the display. Quality of the display can influence clinical performance, but also workflow efficiency and ergonomics.

Performance of radiology display systems has been extensively studied, and this resulted into standardization and clear requirements and guidelines. Digital pathology images and viewing conditions are very different compared to radiology. Fewer effort has gone in understanding what makes a digital pathology display fit for use, and there is no consensus yet in the digital pathology community about ideal specifications for digital pathology displays.

This paper studies specific characteristics of digital pathology display systems, such as luminance, contrast and resolution. Effects of these characteristics on visibility of relevant pathological features is described, and recommendations are made for clinically meaningful levels of luminance, contrast and resolution.

Anonymising medical data for use in machine learning is important to preserve patient privacy and, in many circumstances, is a requirement before data can be made available. One approach to anonymising image data is to train a generative model to produce data that is statistically similar to the input data and then use the output of the model for downstream tasks, such as image classification, instead of the original sensitive data. In digital pathology, it's not yet well understood how using generative models to anonymise histology slide data impacts the performance of downstream tasks. To begin addressing this, we present an evaluation of a histology image classifier trained using patches extracted from the Camelyon 16 dataset and compare it to a classifier trained on the same number of synthetic images generated with a Deep Convolutional Generative Adversarial Network (DCGAN), from the same data. When predicting the class of an image patch as either cancer or normal it's shown that the accuracy reduces from 0.78 for original alone to 0.59 for synthetic alone, and the recall is significantly reduced from 0.70 to 0.44 when training exclusively on the same amount of synthetic data. If retaining a similar accuracy is required for the downstream task, then either the original data must be used or an improved anonymisation strategy must be devised. We conclude that using this DCGAN to anonymise the dataset, degrades the accuracy of the classifier which implies that it has failed to capture the required variation in the original data to generalise and act as a sufficient anonymisation strategy.

Cancer of the brain and central nervous system (CNS) is one of the leading causes of death in the United States. Approximately 85-90% of all primary CNS tumors are brain tumors. Gliomas, the most prevalent kind of malignant brain tumor, contain uncontrollably proliferating cells. Despite the fact that they rarely spread to the spinal cord or other human organs, they grow quickly and can infiltrate healthy tissues. Early diagnosis of glioma subtypes, such as glioblastoma and oligodendroglioma, is clinically challenging; more importantly, it is critical due to the differences in treatment options, therapeutic responsiveness, and patient survival. Histopathological study of biopsy specimens is used to diagnose and classify brain tumors. The existing procedure is time-consuming, labor-intensive, and prone to human error. These drawbacks emphasize the importance of developing a fully automated technique for brain tumor categorization. To ensure diagnostic accuracy, efficiency and reduce the required time, the use of automated brain tumor grading systems is being increasingly explored. Development of automated techniques can assist neuropathologists in streamlining the clinical diagnostic tasks. In this study we propose a two stage attention based network to locate diagnostically relevant regions of interest and then to accurately categorize a cohort of brain tumor (glioma) histopathology images (N=203) into three sub-types: glioblastoma, oligodendroglioma, and astrocytoma. Unlike traditional methods in histopathology image analysis, which assume that each extracted patch from a whole-slide image has the same label regardless of whether or not all patches are tumorous, our technique determines the region of interest in an weakly supervised manner and uses the discovered regions for downstream analysis. Our proposed method outperforms a single-stage attention network, achieving balanced accuracy, F1-Micro, and Cohen Kappa score of 0.73, 0.67, and 0.82, respectively, on a held out test set (N=27 cases) as compared to 0.59, 0.66 and 0.43, respectively, for the single-stage network.

Eosinophilic esophagitis (EoE) is an immune-mediated, clinicopathologic disease of the esophagus. EoE is histologically characterized by the accretion of eosinophils in the esophageal epithelium. The current practice involving manual identification of the small-scale histologic features of EoE relative to the size of the esophageal biopsies can be burdensome and prone to interpreter errors. The existing automatic, computer-assisted EoE identification approaches are typically designed as a train-from-scratch setting, which is prone to overfitting. In this study, we propose to use transfer deep-learning via both the ImageNet pre-trained ResNet50 as well as the more recent Big Transfer (BiT) model to achieve automated EoE feature identification on whole slide images. As opposed to existing deep-learning-based approaches that typically focus on a single pathological phenotype, our study investigates five EoE-relevant histologic features including basal zone hyperplasia, dilated intercellular spaces, eosinophils, lamina propria fibrosis, and normal lamina propria simultaneously. From the results, the model achieved a promising testing balanced accuracy of 61.9%, which is better than that of its trained-from-scratch counterparts.

When using different cameras and displays in the shot and display of one subject, different color images are often showed. To solve this problem, we have developed a color chart in which the constituent colors dyed with dyes are evenly distributed in the color space. We have also developed a tool that creates an International Color Consortium (ICC) profile from the captured image of this color chart. In this report, we will describe the color difference correction accuracy when our method is applied to actual stained pathological specimens taken with multiple whole-slide imaging (WSI). We confirmed color correction accuracy of Lab values in major parts such as the cell nucleus. The Lab value of the specimen itself measured by a spectrocolorimeter was compared with that of the captured image. As a result, the color difference ΔE in H&E-stained cell nucleus was improved from 32.2 to 6.4 for Nanozoomer and from 13.5 to 7.2 for ultra-fast scanner (UFS) by our color correction. The results of the evaluations for other areas and other stain methods (PAS, EVG, MT, and PAM) were good. In the future, high-accuracy color correction of teacher data/evaluation data in AI diagnosis using pathological images will be important.

Lung cancer is the leading cause of cancer-related deaths responsible for over 130,000 deaths each year in the US. Early diagnosis and prompt treatment is crucial for prolonging survival. Many studies have been performed with regard to detecting lung cancer using computed tomography images, or classifying the type of a cancer from pathology images. Some research focuses on survival time prediction instead, which consists of ranking patients according to their expected survival time. This has proven to be a difficult task, and most approaches offer only a slight improvement over random guessing. Instead of ranking patients according to survival time, we propose to predict whether a patient falls into a long- or short-term survival group. We show that this approach outperforms regression-based approaches when predicting precise survival time is not necessary. In addition to that, we show that it is possible to predict short-term and long-term survival from lung cancer histopathology images without ROI annotations. We have obtained an 0.81 AUC when predicting whether a patient would fall into the short-term survival group (less than 12 months) or long-term survival group (greater than 60 months). Furthermore, we show that our model is capable of classifying patients into long and short-term survival even when their survival time falls outside of our chosen ranges.

Tumor mutation burden (TMB) is an important biomarker for the prediction of response to anti-PD-1 immunotherapies. Studies have shown that higher level of TMB (TMB-H) is associated with higher response rate to immunotherapies in patients with various types of advanced solid tumors. However, the measurement of TMB depends on whole exome sequencing (WES) which is an expensive assay and not always available in standard clinical oncology settings. In this work, we assess the feasibility of predicting TMB-H based upon hematoxylin and eosin (H&E)-stained histopathology images, which is a routinely conducted assay in clinical oncology. Using an Inception-V3 convolutional neural network (CNN) as a baseline feature extractor, we compare adding a multi-layer perceptron (MLP) and a squeeze-and-excitation (SE) network on top of the baseline CNN. Training from random initialization and tuning with pretrained weights are also compared. Experiments are conducted on the H&E whole-slide images (WSI) of the melanoma dataset of The Cancer Genome Atlas (TCGA). Results from a 4-fold cross-validation show that the highest average area under the receiver operating characteristic curve (AUC) is 0.589, which implies that the prediction of TMB based on H&E WSI for melanoma remains a challenging problem that will warrant further investigations.

Digital pathology involves the digitization of high quality tissue biopsies on microscope slides to be used by physicians for patient diagnosis and prognosis. These slides have become exciting avenues for deep learning applications to improve care. Despite this, labels are difficult to produce and thus remain rare. In this work, we create a sparse capsule network with a spatial broadcast decoder to perform representation learning on segmented nuclei patches extracted from the BreastPathQ dataset. This was able to produce disentangled latent space for categories such as rotations, and logistic regression classifiers trained on the latent space performed well.

Pancreatic cancer will soon be the second leading cause of cancer-related death in Western society. Imaging techniques such as CT, MRI and ultrasound typically help providing the initial diagnosis, but histopathological assessment is still the gold standard for final confirmation of disease presence and prognosis. In recent years machine learning approaches and pathomics pipelines have shown potential in improving diagnostics and prognostics in other cancerous entities, such as breast and prostate cancer. A crucial first step in these pipelines is typically identification and segmentation of the tumour area. Ideally this step is done automatically to prevent time consuming manual annotation. In this paper, we propose a multi-task convolutional neural network to balance disease detection and segmentation accuracy. We validated our approach on a dataset of 29 patients (for a total of 58 slides) at different resolutions. The best single task segmentation network achieved a median Dice of 0.885 (0.122) IQR at a resolution of 15.56 μm. Our multi-task network improved on that with a median Dice score of 0.934 (0.077) IQR.

One of the strongest prognostic predictors of chronic kidney disease is interstitial fibrosis and tubular atrophy (IFTA). The ultimate goal of IFTA calculation is an estimation of the functional nephritic area. However, the clinical gold standard of estimation by pathologist is imprecise, primarily due to the overwhelming number of tubules sampled in a standard kidney biopsy. Artificial intelligence algorithms could provide significant benefit in this aspect as their high-throughput could identify and quantitatively measure thousands of tubules in mere minutes. Towards this goal, we use a custom panoptic convolutional network similar to Panoptic-DeepLab to detect tubules from 87 WSIs of biopsies from native diabetic kidneys and transplant kidneys. We measure 206 features on each tubule, including commonly understood features like tubular basement membrane thickness and tubular diameter. Finally, we have developed a tool which allows a user to select a range of tubule morphometric features to be highlighted in corresponding WSIs. The tool can also highlight tubules in WSI leveraging multiple morphometric features through selection of regions-of-interest in a uniform manifold approximation and projection plot.

Evaluation of cancer cell and immune cell distribution in tumor microenvironment (TME) is one of the most important factors for guiding cancer immunotherapy and assessing therapeutic response. Multiplexed immunohistochemistry (mIHC) is often used to obtain the different types of cellular biomarker expression and distribution information in TME, but mIHC is limited by time-consuming and cost-intensive, and pathologists’ objectives etc. In this work, we proposed a deep learning-based modified U-Net (m-Unet), by replacing the original convolution sub-module with a modified block to predict the distribution of several typical cellular biomarkers’ expression and distribution information in TME. We have demonstrated that our model can be trained in both fully supervised and semi-supervised manners. The model can extract segmentation information from Hematoxylin and Eosin (H&E) images, and predict the cellular biomarker distributions including panCK for colon cancer cells, CD3 and CD20 for tumor infiltrating lymphocytes (TILs) and DAPI for nucleus. We have demonstrated that our model can be trained in both fully supervised and semi-supervised manners and. the performance of the m-Unet is better than the U-Net in this work. The optimal prediction accuracy of m-Unet is 88.3% on the test dataset. In general, this model possesses the potential to assist the clinical TME analysis.

Computational tools in pathology become more and more widespread, however, development of such tools usually needs large amounts of data. Furthermore, producing the required annotations can be tedious for pathologists. Previous approaches were able to omit the need for pixel-wise annotations and instead rely on global slide labels. Furthermore, a smart selection of relevant tiles within whole slide images reduces the amount of data needed for training. Such technique is feasible for end-to-end learning. In this paper, a weakly supervised learning algorithm was trained on 668 whole slide images of lymph nodes from lung cancer patients with a nodal disease stage of either N1 or N2. Systematic experiments were designed to explore less complex deep learning models. We evaluated our study on different numbers of representative tiles for each slide. The best performing model scored 84% and 0.903 on accuracy and AUC respectively on a small amount of training data.

It is commonly known that diverse datasets of WSIs are beneficial when training convolutional neural networks, however sharing medical data between institutions is often hindered by regulatory concerns. We have developed a cloud-based tool for federated WSI segmentation, allowing collaboration between institutions without the need to directly share data. To show the feasibility of federated learning on pathology data in the real world, We demonstrate this tool by segmenting IFTA from three institutions and show that keeping the three datasets separate does not hinder segmentation performance. This pipeline is deployed in the cloud for easy access for data viewing and annotation by each site’s respective constituents.

Histopathological examination of animal tissue by pathologists forms a crucial part of preclinical drug development. Machine learning techniques have been utilized to develop increasingly reliable and accurate analytical solutions in tissue imaging in recent years. Accurate assessment of the estrous cycle in rats and mice is important in evaluating pathogenesis and identifying potential test article related toxicity in toxicologic studies. In this paper, we present a Deep-Learning based framework for the classification of different stages of the Estrous cycle using whole slide images (WSI) of Hematoxylin & Eosin (H&E) stained sections from the Wistar rat vagina. We present an encoder-decoder convolution neural network, RAE-Net, based on three key criteria: Residual blocks for the decoder, Attention gate in the skip connection, and EfficientNetB4 for the encoder backbone. We show that the architecture achieves significant performance improvement over state-of-the-art segmentation architecture. The proposed estrous staging system could advance the pathology workflow in female preclinical reproductive toxicology studies.

The detection of gastric ulcers is commonly carried out during clinical interventions. It poses many challenges, such as extended time of diagnosis, requirement for clinical expertise, and background noise elimination, especially in early ulcer detection. We adopt a transfer learning-based approach to automatically detect ulcers from wireless-capsule endoscopic (WCE) images. We employed DenseNet121, a convolutional neural network-based pre-trained model for ulcer classification, and realized an optimal performance matrix for test data after training and validation. The DenseNet121 pre-trained model achieved 99.94% classification accuracy with 100% precision, 97.67% recall, and 98.82% F1-score for the test dataset that demonstrates the efficacy of the adopted deep learning model for fast and accurate gastric ulcer screening.

In recent years, deep convolutional neural networks (CNNs) have shown tremendous success in solving many biomedical tasks. However, the development of deep convolutional networks requires access to large quantities of high-quality annotated images for training and evaluation. As image annotation is a tedious task for biomedical experts, recruiting non-expert crowd workers is an economical and efficient way to provide a rich dataset of annotated images. We present an approach to improve the accuracy of segmenting nuclei in Hematoxylin and Eosin (H&E) slides by hiring crowd workers. We first present a crowdsourcing framework that enables fast and efficient acquisition of nuclei-segmented masks from the crowd by providing manual and semi-automatic annotation methods. Then, we present CrowdDeep, a novel technique to improve the accuracy of deep learning models trained on expert annotation by efficiently hiring crowd-annotated data. CrowdDeep consists of two sub-networks: Crowd-Subnet, and Expert-Subnet. The Crowd-Subnet is trained on the crowd-annotated images to extract crowd-related features from the crowd-annotated masks, while the Expert-Subnet is trained on the expert-driven annotations to extract expert-related features from the expert-annotated masks. Then, it calculates the final segmentation mask from the generated segmentation masks by two sub-networks. The results show that CrowdDeep outperforms a CNN model trained on solely expert-derived annotations in terms of F1-Score, IOU, and Pixel Accuracy. This approach is multi-organ and generalizes across different organs, staining, and disease states and is easily expandable by crowdsourcing images with an assortment of nuclei shapes and sizes from any desirable body tissue.

Whole Slide Image (WSI) analysis is a powerful method to facilitate the diagnosis of cancer in tissue samples. Automating this diagnosis poses various issues, most notably caused by the immense image resolution and limited annotations. WSI’s commonly exhibit resolutions of 100, 000 × 100, 000 pixels. Annotating cancerous areas in WSI’s on the pixel-level is prohibitively labor-intensive and requires a high level of expert knowledge. Multiple instance learning (MIL) alleviates the need for expensive pixel-level annotations. In MIL, learning is performed on slide-level labels, in which a pathologist provides information about whether a slide includes cancerous tissue. Here, we propose Self-ViT-MIL, a novel approach for classifying and localizing cancerous areas based on slide-level annotations, eliminating the need for pixel-wise annotated training data. Self-ViTMIL is pre-trained in a self-supervised setting to learn rich feature representation without relying on any labels. The recent Vision Transformer (ViT) architecture builds the feature extractor of Self-ViT-MIL. For localizing cancerous regions, a MIL aggregator with global attention is utilized. To the best of our knowledge, Self-ViTMIL is the first approach to introduce self-supervised ViT’s in MIL-based WSI analysis tasks. We showcase the effectiveness of our approach on the common Camelyon16 dataset. Self-ViT-MIL surpasses existing stateof-the-art MIL-based approaches in terms of accuracy and area under the curve (AUC). Code is available at https://github.com/gokberkgul/self-learning-transformer-mil

Non-Small Cell Lung Cancer(NSCLC) is the most common cause of cancer death in Canada and worldwide. As the immune component of the NSCLC tumor microenvironment(TME) is highly prognostic of patient outcome, further understanding of TME and the spatial organization of the immune cells within the TME is needed for better patient prognosis and treatment planning. Current immunohistochemistry techniques quantify immune cell counts and density, but generally cannot asses the spatial relationship between tumour and immune cells. We have developed a multiplexed Immunohistochemistry(mIHC) procedure combining multiple labels per round with several rounds, enabling analysis of immune cell populations on a slide through consecutive cycles of staining, destaining & hyperspectral imaging. By integrating serial imaging, sequential labeling & image registration, we are able to spatially map the TME. Robust, accurate, segmentation of cell nuclei for overlapping nuclei is one of the most significant unsolved issues in digital pathology. We have trained a deep learning segmentation method to accurately segment individual cell nuclei within overlapping clusters of nuclei. By combining a mIHC technique which enables the detection of multiple markers with deep learning segmentation methods to segment every individual cell nuclei in tissue sections with an accuracy comparable to human annotation, we can analyze the cell-cell interactions between immune and tumour cells, enhancing our ability to perform molecularly based single cell analysis of multiple cell types simultaneously within the tissue. These two techniques joined can be scaled up to the entire tissue section level, improving our understanding of the biological aggressiveness of NSCLC’s.