Digital pathology and artificial intelligence

doi:10.1016/S1470-2045(19)30154-8

The Lancet Oncology

Volume 20, Issue 5, May 2019, Pages e253-e261

https://doi.org/10.1016/S1470-2045(19)30154-8 Get rights and content

Summary

In modern clinical practice, digital pathology has a crucial role and is increasingly a technological requirement in the scientific laboratory environment. The advent of whole-slide imaging, availability of faster networks, and cheaper storage solutions has made it easier for pathologists to manage digital slide images and share them for clinical use. In parallel, unprecedented advances in machine learning have enabled the synergy of artificial intelligence and digital pathology, which offers image-based diagnosis possibilities that were once limited only to radiology and cardiology. Integration of digital slides into the pathology workflow, advanced algorithms, and computer-aided diagnostic techniques extend the frontiers of the pathologist's view beyond a microscopic slide and enable true utilisation and integration of knowledge that is beyond human limits and boundaries, and we believe there is clear potential for artificial intelligence breakthroughs in the pathology setting. In this Review, we discuss advancements in digital slide-based image diagnosis for cancer along with some challenges and opportunities for artificial intelligence in digital pathology.

Introduction

Digital pathology has a crucial role in modern clinical practice and is increasingly a technology requirement within the laboratory environment.1, 2 Advances in computing power, faster networks, and cheaper storage have enabled pathologists to manage digital slide images with more ease and flexibility than they could in the past and has enabled pathologists to share images for telepathology and clinical use. In the last two decades, digital imaging in pathology has seen the inception and evolution of whole-slide imaging, which allows entire slides to be imaged and permanently stored at high resolution.

In particular, whole-slide imaging serves as an enabling platform for the application of artificial intelligence (AI) in digital pathology. Until now, AI has been mostly used for image-based diagnosis in radiology and cardiology. Its application to pathology is an expanding field of active research with several research groups and dedicated companies. Images produced by whole-slide imaging are a great source of information; complexity is higher than in many other imaging modalities because of their large size (a resolution of 100k × 100k is common), presence of colour information (haematoxylin and eosin and immunohistochemistry), no apparent anatomical orientation as in radiology, availability of information at multiple scales (eg, ×4, ×20), and multiple z-stack levels (each slice contains a finite thickness and, depending on the plane of focus, will generate different images). Clearly, this kind of visual information cannot be extracted as easily by a human reader.

With integration of digital slides into the pathology workflow, advanced algorithms and computer-aided diagnostic techniques extend the frontiers of the pathologist's view beyond a microscopic slide and enable true utilisation and integration of knowledge beyond human limits and boundaries.² AI already enables pathologists to identify unique imaging markers associated with disease processes with the goal of improving early detection, determining prognosis, and selecting treatments most likely to be effective. This allows pathologists to serve more patients while maintaining diagnostic and prognostic accuracy. This integration is especially important considering the increasing number of ageing patients, and that less than 2% of medical graduates go into pathology because of the information management, integration, and digital media accessibility aspect of the review and case sign-out process.³

Digital pathology and AI can have immense potential for oncology and precision medicine. Much like the evolution of efficiency and effectiveness in radiology, the pressure on pathologists to reduce turnaround time and develop more efficient workflows is trending towards digitalisation. This digital innovation has potential to change the way cancer diagnoses occur, with added benefits of shared images and data, increased efficiency and integrated diagnostics, modernisation of pathology workflows to improve patient care and safety, increased collaboration through multidisciplinary and disease-specific patient care conferences, improved accountability (on behalf of the physician, who makes the final clinical decision), and cost savings by optimising staff performance. By using AI algorithms many of the tasks that are manual and subjective can become more automated and standardised.

Although AI is slated to benefit many areas of clinical health sciences (eg, oncology and drug development), the focus of this Review is to highlight its use in digital pathology and whole-slide imaging, including education (eg, digital slide teaching sets), quality assurance (eg, second opinions, proficiency testing, and archiving), and clinical diagnosis (ie, telepathology). First, we explore how AI has advanced these areas of digital pathology, as well as specific use cases and applications of AI in research, image analysis, and computer-aided diagnosis; and discuss the techniques used, challenges, and barriers.¹ Second, we discuss the ultimate goal of AI and whole-slide imaging—integration of pathological image information with clinical data— and its limitations. With whole-slide imaging as an enabling platform for AI, digital pathology will have meaningful and measurable effects on both clinical and research components of pathology workflow.2, 3

Section snippets

AI and education

Whole-slide imaging is already used for teaching at conferences, virtual workshops, presentations, and tumour boards.² Equipped with whole-slide imaging, AI tools can help further training of the next generation of pathologists by providing on demand, standardised, and interactive digital slides that can be shared with multiple users anywhere, at any time.2, 3 Additionally, AI tools can provide automated annotations in the form of quizzes for trainees. With the help of these interactive tools

AI and quality assurance

The development of automated, high-speed, and high-resolution whole-slide imaging has had a substantial effect on quality assurance. Digitised slides that are readily available to pathologists in the laboratory information system or on the intranet can be used for several quality assurance tasks, including teleconsultation, gauging inter-observer and intra-observer variance, proficiency testing, and archiving of slides. For example, the College of American Pathologists optionally sends

AI for clinical diagnosis

Rendering routine pathological diagnoses using whole-slide imaging is a feasible approach. Several studies6, 7 have been published comparing diagnostic interpretation using digital slides to diagnoses rendered using glass slides and a conventional light microscope. These studies show a range of concordance from 89% to 99% when comparing diagnostic interpretation using digital slides to diagnoses rendered using glass slides and a conventional light microscope. The range is wide and encompasses

AI and image analysis

Image analysis tools can automate and quantify with greater consistency and accuracy than light microscopy. Computer-aided diagnosis is widely used for oestrogen receptor, progesterone receptor, and HER2/neu assessments in breast cancer, Ki67 assessment in carcinoid tumours, as well as multiple other clinical and research stains. The reliability of these methods requires the standardisation of the image acquisition step, which has been discussed previously. The development of whole-slide

Integration of AI with other clinical data

Histopathological image analysis is not only limited to visual analysis; several other sources of data need to be included coming from omics, clinical records, and patient demographic information.51, 52 Clinical data (eg, demographic information, medical history, and laboratory and clinical results) are mostly in unstructured free-text reports. Natural language processing technologies can be used to extract relevant information and tie those to the information in histopathological slides.⁵³

Perception and limitations of AI

Many AI approaches, particularly systems based on deep learning, are criticised for not being able to explain how they arrive at their decisions, hence the label black boxes. Although these algorithms will still offer benefits, clinical, legal, and regulatory issues need to be sorted out going forward. At the same time, there is active research to make the algorithms easier to interpret by humans and provide insight on how they work (eg, by providing some of the features that the algorithm is

Conclusion

Pathology is rapidly transitioning to digital methods with new developments in AI. Therefore, bringing the disciplines of pathology and AI together could result in exciting changes to health care, although a large number of technical, ethical, and legal questions still need to be answered.63, 64 Combining digital pathology and AI will lead to an improved workflow and advanced diagnostics, enabling researchers and clinical teams to share and review images instantly, and use computational

Search strategy and selection criteria

We used Google Scholar and PubMed to find relevant manuscripts. We restricted our search to papers published in English between Jan 1, 2013, and Feb 25, 2019. We used the following terms in different combinations: “WSI”, “deep learning”, “AI”, “digital pathology”, “GAN”, “histopathologic image analysis”, “nuclei segmentation”, “whole slide imaging”, “artificial Intelligence and digital pathology”, “digital pathology and deep learning”, “histopathology and deep learning”, “GAN and

References (64)

HR Tizhoosh et al.
Artificial intelligence and digital pathology: challenges and opportunities
J Pathol Inform
(2018)
S Thorstenson et al.
Implementation of large-scale routine diagnostics using whole slide imaging in Sweden: Digital pathology experiences 2006-2013
J Pathol Inform
(2014)
P Shrestha et al.
A quantitative approach to evaluate image quality of whole slide imaging scanners
J Pathol Inform
(2016)
D Komura et al.
Machine learning methods for histopathological image analysis
Comput Struct Biotechnol J
(2018)
A Tremeau et al.
A region growing and merging algorithm to color segmentation
Pattern Recognition
(1997)
C Li et al.
Deep mitosis: mitosis detection via deep detection, verification and segmentation networks
Medical image analysis
(2018)
MKK Niazi et al.
MP58–06 automated staging of T1 bladder cancer using digital pathologic H&E images: a deep learning approach
J Urol
(2018)
SE Schaeffer
Graph clustering
Comp Sci Rev
(2007)
A Heindl et al.
Mapping spatial heterogeneity in the tumor microenvironment: a new era for digital pathology
Lab Invest
(2015)
N Farahani et al.
Whole slide imaging in pathology: advantages, limitations, and emerging perspectives
Pathol Lab Med Int
(2015)

MD Zarella et al.

A practical guide to whole slide imaging: a white paper from the digital pathology association

Arch Pathol Lab Med

(2019)

MKK Niazi et al.

Nuclear IHC enumeration: a digital phantom to evaluate the performance of automated algorithms in digital pathology

PloS One

(2018)

M Mirza et al.

Conditional generative adversarial nets

K Tabata et al.

Whole-slide imaging at primary pathological diagnosis: validation of whole-slide imaging-based primary pathological diagnosis at twelve Japanese academic institutes

Pathol Int

(2017)

MB Loughrey et al.

Digital slide viewing for primary reporting in gastrointestinal pathology: a validation study

Virchows Arch

(2015)

Lloyd M, Kellough D, Shanks T, et al. How to acquire over 500 000 whole slides images a year: creating a massive novel...

C Senaras et al.

DeepFocus: detection of out-of-focus regions in whole slide digital images using deep learning

PloS One

(2018)

XM Lopez et al.

An automated blur detection method for histological whole slide imaging

PloS One

(2013)

MT Shaban et al.

StainGAN: stain style transfer for digital histological images

FG Zanjani et al.

Stain normalization of histopathology images using generative adversarial networks

Generative adversarial nets

A Bentaieb et al.

Adversarial stain transfer for histopathology image analysis

IEEE Trans Med Imaging

(2018)

S Kothari et al.

Removing batch effects from histopathological images for enhanced cancer diagnosis

IEEE Trans Med Imaging

(2014)

J-Y Zhu et al.

Unpaired image-to-image translation using cycle-consistent adversarial networks

N Kumar et al.

A dataset and a technique for generalized nuclear segmentation for computational pathology

IEEE Trans Med Imaging

(2017)

F Xing et al.

Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: a comprehensive review

IEEE Rev Biomed Eng

(2016)

Y Song et al.

Accurate segmentation of cervical cytoplasm and nuclei based on multiscale convolutional network and graph partitioning

IEEE Rev Biomed Eng

(2015)

F Xing et al.

An automatic learning-based framework for robust nucleus segmentation

IEEE Trans Med Imaging

(2016)

JN Weinstein et al.

The cancer genome atlas pan-cancer analysis project

Nature genetics

(2013)

F Mahmood et al.

Deep adversarial training for multi-organ nuclei segmentation in histopathology images

S Yousefi et al.

Transfer learning from nucleus detection to classification in histopathology images

BioRxiv

(2019)

Cited by (644)

Number of intraepithelial lymphocytes and presence of a subepithelial band in normal colonic mucosa differs according to stainings and evaluation method
2024, Journal of Pathology Informatics
Chronic watery diarrhea is a frequent symptom. In approximately 10% of the patients, a diagnosis of microscopic colitis (MC) is established. The diagnosis relies on specific, but sometimes subtle, histopathological findings. As the histology of normal intestinal mucosa vary, discriminating subtle features of MC from normal tissue can be challenging and therefore auxiliary stainings are increasingly used. The aim of this study was to determine the variance in number of intraepithelial lymphocytes (IELs) and presence of a subepithelial band in normal ileum and colonic mucosa, according to different stains and digital assessment. Sixty-one patients without diarrhea referred to screening colonoscopy due to a positive feacal blood test and presenting with endoscopically normal mucosa were included. Basic histological features, number of IELs, and thickness of a subepithelial band was manually evaluated and a deep learning-based algorithm was developed to digitally determine the number of IELs in each of the two compartments; surface epithelium and cryptal epithelium, and the density of lymphocytes in the lamina propria compartment. The number of IELs was significantly higher on CD3-stained slides compared with slides stained with Hematoxylin-and-Eosin (HE) (p<0.001), and even higher numbers were reached using digital analysis. No significant difference between right and left colon in IELs or density of CD3-positive lymphocytes in lamina propria was found. No subepithelial band was present in HE-stained slides while a thin band was visualized on special stains. Conclusively, in this cohort of prospectively collected ileum and colonic biopsies from asymptomatic patients, the range of IELs and detection of a subepithelial collagenous band varied depending on the stain and method used for assessment. As assessment of biopsies from patients with diarrhea constitute a considerable workload in the pathology departments digital image analysis is highly desired. Knowledge provided by the present study highlight important differences that should be considered before introducing this method in the clinic.
Computational pathology: A survey review and the way forward
2024, Journal of Pathology Informatics
Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field’s future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.
Artificial intelligence (AI) for tumor microenvironment (TME) and tumor budding (TB) identification in colorectal cancer (CRC) patients: A systematic review
2024, Journal of Pathology Informatics
Evaluation of the parameters such as tumor microenvironment (TME) and tumor budding (TB) is one of the most important steps in colorectal cancer (CRC) diagnosis and cancer development prognosis. In recent years, artificial intelligence (AI) has been successfully used to solve such problems. In this paper, we summarize the latest data on the use of artificial intelligence to predict tumor microenvironment and tumor budding in histological scans of patients with colorectal cancer. We performed a systematic literature search using 2 databases (Medline and Scopus) with the following search terms: ("tumor microenvironment" OR "tumor budding") AND ("colorectal cancer" OR CRC) AND ("artificial intelligence" OR "machine learning " OR "deep learning"). During the analysis, we gathered from the articles performance scores such as sensitivity, specificity, and accuracy of identifying TME and TB using artificial intelligence. The systematic review showed that machine learning and deep learning successfully cope with the prediction of these parameters. The highest accuracy values in TB and TME prediction were 97.7% and 97.3%, respectively. This review led us to the conclusion that AI platforms can already be used as diagnostic aids, which will greatly facilitate the work of pathologists in detection and estimation of TB and TME as instruments and second-opinion services. A key limitation in writing this systematic review was the heterogeneous use of performance metrics for machine learning models by different authors, as well as relatively small datasets used in some studies.
Computational pathology in the identification of HER2-low breast cancer: Opportunities and challenges
2024, Journal of Pathology Informatics
For the past 2 decades, pathologists have been accustomed to reporting the HER2 status of breast cancer as either positive or negative, based on HER2 IHC. Today, however, there is a clinical imperative to employ a 3-tier approach to interpreting HER2 IHC that can also identify tumours categorised as HER2-low. Meeting this need for a finer degree of discrimination may be challenging, and in this article, we consider the potential for the integration of computational approaches to support pathologists in achieving accurate and reproducible HER2 IHC scoring as well as outlining some of the practicalities involved.
ProDiv: Prototype-driven consistent pseudo-bag division for whole-slide image classification
2024, Computer Methods and Programs in Biomedicine
Pathology image classification is one of the most essential auxiliary processes in cancer diagnosis. To overcome the problem of inadequate Whole-Slide Image (WSI) samples with weak labels, pseudo-bag-based multiple instance learning (MIL) methods have attracted wide attention in pathology image classification. In this type of method, the division scheme of pseudo-bags is usually a primary factor affecting classification performance. In order to improve the division of WSI pseudo-bags on existing random/clustering approaches, this paper proposes a new Prototype-driven Division (ProDiv) scheme for the pseudo-bag-based MIL classification framework on pathology images.
This scheme first designs an attention-based method to generate a bag prototype for each slide. On this basis, it further groups WSI patch instances into a series of instance clusters according to the feature similarities between the prototype and patches. Finally, pseudo-bags are obtained by randomly combining the non-overlapping patch instances of different instance clusters. Moreover, the design scheme of our ProDiv considers practicality, and it could be smoothly assembled with almost all the MIL-based WSI classification methods in recent years.
Empirical results show that our ProDiv, when integrated with several existing methods, can deliver classification AUC improvements of up to 7.3% and 10.3%, respectively on two public WSI datasets.
ProDiv could almost always bring obvious performance improvements to compared MIL models on typical metrics, which suggests the effectiveness of our scheme. Experimental visualization also visually interprets the correctness of the proposed ProDiv.
A review: Exploring the role of ChatGPT in the diagnosis and treatment of oral pathologies
2024, Oral Oncology Reports
This review article delves into the evolving landscape of artificial intelligence (AI), specifically focusing on the integration of ChatGPT in the domain of oral pathologies. ChatGPT, a language model powered by deep learning, has demonstrated significant potential in providing insights and support in various fields. Seeking the capabilities of ChatGPT, several researches have been explored in the context of ChatGPT's effectiveness in oral pathologies in this review article. In the context of oral health, the model's capabilities and limitations are scrutinized, offering a critical analysis of its current applications and future prospects in different sections of this review article.

View all citing articles on Scopus

View full text

SeriesDigital pathology and artificial intelligence

Summary

Introduction

Section snippets

AI and education

AI and quality assurance

AI for clinical diagnosis

AI and image analysis

Integration of AI with other clinical data

Perception and limitations of AI

Conclusion

Search strategy and selection criteria

J Pathol Inform

J Pathol Inform

J Pathol Inform

Comput Struct Biotechnol J

Pattern Recognition

Medical image analysis

J Urol

Comp Sci Rev

Lab Invest

Whole slide imaging in pathology: advantages, limitations, and emerging perspectives

Pathol Lab Med Int

A practical guide to whole slide imaging: a white paper from the digital pathology association

Arch Pathol Lab Med

Nuclear IHC enumeration: a digital phantom to evaluate the performance of automated algorithms in digital pathology

PloS One

Conditional generative adversarial nets

Whole-slide imaging at primary pathological diagnosis: validation of whole-slide imaging-based primary pathological diagnosis at twelve Japanese academic institutes

Pathol Int

Digital slide viewing for primary reporting in gastrointestinal pathology: a validation study

Virchows Arch

DeepFocus: detection of out-of-focus regions in whole slide digital images using deep learning

PloS One

An automated blur detection method for histological whole slide imaging

PloS One

StainGAN: stain style transfer for digital histological images

Stain normalization of histopathology images using generative adversarial networks

Generative adversarial nets

Adversarial stain transfer for histopathology image analysis

IEEE Trans Med Imaging

Removing batch effects from histopathological images for enhanced cancer diagnosis

IEEE Trans Med Imaging

Unpaired image-to-image translation using cycle-consistent adversarial networks

A dataset and a technique for generalized nuclear segmentation for computational pathology

IEEE Trans Med Imaging

Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: a comprehensive review

IEEE Rev Biomed Eng

Accurate segmentation of cervical cytoplasm and nuclei based on multiscale convolutional network and graph partitioning

IEEE Rev Biomed Eng

An automatic learning-based framework for robust nucleus segmentation

IEEE Trans Med Imaging

The cancer genome atlas pan-cancer analysis project

Nature genetics

Deep adversarial training for multi-organ nuclei segmentation in histopathology images

Transfer learning from nucleus detection to classification in histopathology images

BioRxiv

Series
Digital pathology and artificial intelligence