JP7686201B2 - System and method for processing images of slides for digital pathology - Patents.com - Google Patents

Description

(Related Applications)
This application claims priority to U.S. Provisional Application No. 62/897,745, filed September 9, 2019, the entire disclosure of which is incorporated herein by reference in its entirety.

FIELD OF THE DISCLOSURE
Various embodiments of the present disclosure relate generally to image-based sample analysis and related image processing methods. More specifically, certain embodiments of the present disclosure relate to systems and methods for identifying sample properties and providing an integrated pathology workflow based on processing images of tissue samples.

(background)
In order to use digital pathology images within a hospital or research environment, it may be important to identify and categorize the tissue type of the sample, the nature of the sample's acquisition (e.g., prostate needle biopsy, breast biopsy, mastectomy, etc.), and other relevant properties of the sample or image.

A method for providing an integrated pathology workflow based on processing images of tissue samples is desired. The following disclosure is directed to systems and methods for providing user interfaces and artificial intelligence (AI) tools that can be integrated into the workflow to facilitate and improve the pathologist's work solution.

The foregoing general description and the following detailed description are exemplary and explanatory only and are not limitations of the present disclosure. The background provided herein is generally for the purpose of providing a context for the present disclosure. Unless otherwise indicated herein, the matter described in this section is not, by inclusion in this section, made prior art to any claim of this application, nor is it an admission of prior art or an indication of prior art.

(summary)
According to an aspect of the present disclosure, a system and method for identifying sample properties and providing an integrated pathology workflow based on processing of images of tissue samples is disclosed.

A computer-implemented method for analyzing an electronic image corresponding to a sample includes receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and/or at least one characteristic of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting the at least one characteristic, the training images comprising human images and/or algorithmically generated images; and outputting the target electronic image that identifies an area of interest based on the at least one characteristic of the target sample and/or the at least one characteristic of the target electronic image.

A system for analyzing an electronic image corresponding to a sample includes a memory storing instructions and a processor executing the instructions to perform a process including receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample, applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and/or at least one characteristic of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting the at least one characteristic, the training images comprising images of human tissue and/or algorithmically generated images, and outputting a target electronic image that identifies an area of interest based on the at least one characteristic of the target sample and/or the at least one characteristic of the target electronic image.

A non-transitory computer readable medium stores instructions that, when executed by a processor, cause the processor to perform a method for analyzing an image corresponding to a sample, the method including: receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample; applying a machine learning system to the target electronic image to determine at least one characteristic of the target sample and/or at least one characteristic of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting the at least one characteristic, the training images comprising human images and/or algorithmically generated images; and outputting the target electronic image that identifies an area of interest based on the at least one characteristic of the target sample and/or the at least one characteristic of the target electronic image.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments as claimed.
The present specification also provides, for example, the following items:
(Item 1)
1. A computer-implemented method for analyzing an electronic image corresponding to a sample, the method comprising:
receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample;
applying a machine learning system to the target electronic image to determine at least one property of the target sample and/or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and/or algorithmically generated images;
outputting the target electronic image identifying an area of interest based on at least one characteristic of the target specimen and/or at least one characteristic of the target electronic image;
4. A computer-implemented method comprising:
(Item 2)
2. The computer-implemented method of claim 1, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image.
(Item 3)
2. The computer-implemented method of claim 1, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly.
(Item 4)
identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly;
2. The computer-implemented method of claim 1, wherein the heat map overlay is transparent or semi-transparent.
(Item 5)
displaying a magnification window over at least a portion of the target electronic image;
presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image;
2. The computer-implemented method of claim 1, further comprising:
(Item 6)
displaying a magnification window over at least a portion of the target electronic image;
presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image;
Further comprising:
2. The computer-implemented method of claim 1, wherein the magnification window comprises a selectable icon for toggling a heatmap overlay on the magnified image.
(Item 7)
displaying a slide tray tool over the target electronic image that identifies an outline of the target sample;
applying the machine learning system to the target electronic image to determine whether a portion of the target sample contains an anomaly; and
in response to determining that the portion contains an anomaly, presenting an indicator of the anomaly within the slide tray tool.
2. The computer-implemented method of claim 1, further comprising:
(Item 8)
10. The computer-implemented method of claim 1, further comprising displaying an annotation log comprising an indicator identifying the area of interest and a consultation request associated with the area of interest.
(Item 9)
receiving a second target electronic image corresponding to the target sample;
determining a first portion of the target specimen associated with the target electronic image;
determining a second portion of the target specimen associated with the second target electronic image;
identifying whether the first portion and the second portion are the same or overlap;
displaying the first representation of the target electronic image and the second representation of the second target electronic image in a predetermined proximity to one another in response to identifying that the first portion and the second portion are the same or overlapping;
determining whether there is an area of interest associated with the target electronic image and/or an area of interest associated with the second target electronic image;
displaying an indicator associated with a first representation of the target electronic image in response to determining that the area of interest is associated with the target electronic image;
displaying an indicator associated with a second representation of the second target electronic image in response to determining that the area of interest is associated with the second target electronic image.
2. The computer-implemented method of claim 1, further comprising:
(Item 10)
1. A system for analyzing an electronic image corresponding to a specimen, the system comprising:
at least one memory for storing instructions;
At least one processor;
Equipped with
The at least one processor executes the instructions;
receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample;
applying a machine learning system to the target electronic image to determine at least one property of the target sample and/or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and/or algorithmically generated images;
outputting the target electronic image identifying an area of interest based on at least one characteristic of the target specimen and/or at least one characteristic of the target electronic image;
A system that performs a process including
(Item 11)
11. The system of claim 10, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image.
(Item 12)
11. The system of claim 10, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly.
(Item 13)
identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly;
11. The system of claim 10, wherein the heat map overlay is transparent or semi-transparent.
(Item 14)
displaying a magnification window over at least a portion of the target electronic image;
presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image;
Item 11. The system of item 10, further comprising:
(Item 15)
displaying a magnification window over at least a portion of the target electronic image;
presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image;
Further comprising:
11. The system of claim 10, wherein the magnification window comprises a selectable icon for toggling a heatmap overlay on the magnified image.
(Item 16)
displaying a slide tray tool over the target electronic image that identifies an outline of the target sample;
applying the machine learning system to the target electronic image to determine whether a portion of the target sample contains an anomaly; and
in response to determining that the portion contains an anomaly, presenting an indicator of the anomaly within the slide tray tool.
Item 11. The system of item 10, further comprising:
(Item 17)
11. The system of claim 10, further comprising displaying an annotation log comprising an indicator identifying the area of interest and a consultation request associated with the area of interest.
(Item 18)
receiving a second target electronic image corresponding to the target sample;
determining a first portion of the target specimen associated with the target electronic image;
determining a second portion of the target specimen associated with the second target electronic image;
identifying whether the first portion and the second portion are the same or overlap;
displaying the first representation of the target electronic image and the second representation of the second target electronic image in a predetermined proximity to one another in response to identifying that the first portion and the second portion are the same or overlapping;
determining whether there is an area of interest associated with the target electronic image and/or an area of interest associated with the second target electronic image;
displaying an indicator associated with a first representation of the target electronic image in response to determining that the area of interest is associated with the target electronic image;
displaying an indicator associated with a second representation of the second target electronic image in response to determining that the area of interest is associated with the second target electronic image.
Item 11. The system of item 10, further comprising:
(Item 19)
A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing an electronic image corresponding to a sample, the method comprising:
receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample;
applying a machine learning system to the target electronic image to determine at least one property of the target sample and/or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and/or algorithmically generated images;
outputting the target electronic image identifying an area of interest based on at least one characteristic of the target specimen and/or at least one characteristic of the target electronic image;
16. A non-transitory computer readable medium comprising:
(Item 20)
20. The non-transitory computer-readable medium of claim 19, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

FIG. 1A illustrates an example block diagram of a system and network for determining sample property or image property information for a digital pathology image, according to an example embodiment of the present disclosure.

FIG. 1B illustrates an exemplary block diagram of a disease detection platform 100 according to an exemplary embodiment of the present disclosure.

FIG. 1C illustrates an example block diagram of a viewing application tool 101 according to an example embodiment of the present disclosure.

FIG. 1D illustrates an example block diagram of a viewing application tool 101 according to an example embodiment of the present disclosure.

FIG. 2 is a flowchart illustrating an example method for identifying an area of interest and outputting a target image, in accordance with one or more exemplary embodiments of the present disclosure.

FIG. 3 illustrates an example output of the viewing application tool 101 according to an example embodiment of the present disclosure.

FIG. 4 illustrates an example output of an overlay tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

FIG. 5 illustrates an example output of a work list tool of the viewing application tool 101 in accordance with an example embodiment of the present disclosure.

FIG. 6 illustrates an example output of a slide tray tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

FIG. 7 illustrates an example output of a slide sharing tool of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

FIG. 8 illustrates an example output of an annotation tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

FIG. 9 illustrates an example output of an annotation tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

FIG. 10 illustrates an example output of an annotation tool of the viewing application tool 101 according to an example embodiment of the present disclosure.

FIG. 11 illustrates an example output of a research tool of the browsing application tool 101 according to an exemplary embodiment of the present disclosure.

FIG. 12 illustrates an example output of a research tool of the browsing application tool 101 according to an example embodiment of the present disclosure.

FIG. 13 illustrates an example workflow of the viewing application tool 101 according to an example embodiment of the present disclosure.

14A, 14B, and 14C illustrate example outputs for a slide view of the viewing application tool 101 according to an exemplary embodiment of the present disclosure. 14A, 14B, and 14C illustrate example outputs for a slide view of the viewing application tool 101 according to an exemplary embodiment of the present disclosure. 14A, 14B, and 14C illustrate example outputs for a slide view of the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

FIG. 15 illustrates an example workflow for creating an account for a new user and requesting a consultation according to an example embodiment of the present disclosure. FIG. 15 illustrates an example workflow for creating an account for a new user and requesting a consultation according to an example embodiment of the present disclosure.

16A and 16B illustrate an example output for a pathology consultation dashboard, according to an example embodiment of the present disclosure. 16A and 16B illustrate an example output for a pathology consultation dashboard, according to an example embodiment of the present disclosure.

FIG. 17 illustrates an example output of a case view for the viewing application tool 101 according to an exemplary embodiment of the present disclosure.

FIG. 18 depicts an example system that can implement the techniques presented herein.

(Description of the embodiment)
Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

The systems, devices, and methods disclosed herein are described in detail, by way of example, with reference to the figures. The examples discussed herein are examples only and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be construed as essential for any particular implementation of any of these devices, systems, or methods, unless specifically designated as essential.

In addition, with respect to any method described, whether or not the method is described in conjunction with a flow diagram, unless otherwise specified or required by context, it should be understood that any explicit or implicit ordering of steps performed in the execution of the method does not imply that those steps must be performed in the order presented, but may instead be performed in a different order or in parallel.

As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal." Additionally, the terms "a" and "an" are not used herein to denote a limitation of quantity, but rather to denote the presence of one or more of the referenced items.

Pathology refers to the study of disease. More specifically, pathology refers to performing tests and analyses used to diagnose disease. For example, a tissue sample may be placed on a slide to be viewed under a microscope by a pathologist (e.g., a medical doctor who is an expert who analyzes tissue samples and determines if any abnormalities are present). That is, a pathology sample may be cut into multiple pieces, stained, and prepared as a slide for the pathologist to examine and give a diagnosis. When the diagnostic findings on the slide are uncertain, the pathologist may order additional cutting levels, stains, or other tests to gather more information from the tissue. A technician may then create a new slide, which may contain additional information for the pathologist to use in making a diagnosis. This process of creating additional slides may be time consuming, not only because it may involve retrieving a block of tissue, cutting it, making a new slide, and then staining the slide, but also because it may be batched for multiple orders. This may significantly delay the final diagnosis that the pathologist gives. Additionally, even after a delay, there may still be no guarantee that the new slides will have enough information to render a diagnosis.

A computer can be used to analyze images of tissue samples and quickly identify whether additional information about a particular tissue sample may be required and/or highlight areas that may need to be examined more closely by a pathologist. Thus, the process of obtaining additional stained slides and tests can be done automatically before being reviewed by a pathologist. When paired with automated slide sectioning and staining machinery, this can provide a fully automated slide preparation pipeline. This automation has the advantages of at least (1) minimizing the amount of time spent by a pathologist determining that a slide is insufficient to make a diagnosis; (2) minimizing the (average total) time from sample acquisition to diagnosis by avoiding additional time between the time additional tests are ordered and the time they are produced; (3) reducing the amount of time and amount of material wasted per re-cut by allowing re-cutting to be performed while the tissue block (e.g., pathology sample) is in the cutting desk; (4) reducing the amount of tissue material required during slide preparation; (5) reducing the cost of slide preparation by partially or fully automating the procedure; (6) enabling automated customized cutting and staining of slides that will result in more representative/informative slides from the sample; (7) allowing a larger number of slides to be generated per tissue block by reducing the overhead of requiring additional tests on the pathologist, contributing to a more informative/precise diagnosis; and/or (8) identifying or matching the correct nature (e.g., with respect to sample type) of the digital pathology image, etc.

The process of using computers to assist pathologists is known as computational pathology. Computing methods used for computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI may include, but are not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. Using computational pathology may save lives by helping pathologists improve their diagnostic accuracy, reliability, efficiency, and accessibility. For example, computational pathology may be used to help detect slides that are suspicious for cancer, thereby allowing pathologists to check and confirm their initial assessment before giving a final diagnosis.

Histopathology refers to the study of specimens that are placed on slides. For example, a digital pathology image may consist of a digitized image of a microscope slide containing a specimen (e.g., a smear). One method a pathologist may use is to analyze the image on the slide to identify the nuclei and classify whether they are normal (e.g., benign) or abnormal (e.g., malignant). To assist the pathologist in identifying and classifying the nuclei, histological stains may be used to visualize the cells. Many dye-based staining systems have been developed, including periodic acid-Schiff reaction, Masson's trichrome, Nissl and methylene blue, and hematoxylin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method in which hematoxylin stains cell nuclei blue, eosin stains cytoplasm and extracellular matrix pink, and other tissue regions take on variations of these colors. However, in many cases, histological preparations stained with H&E do not provide enough information for a pathologist to visually identify biomarkers that may aid diagnosis or guide treatment. In this situation, techniques such as immunohistochemistry (IHC), immunofluorescence, in situ hybridization (ISH), or fluorescence in situ hybridization (FISH) may be used. IHC and immunofluorescence involve the use of antibodies that, for example, bind to specific antigens in tissues and allow visual detection of cells expressing specific proteins of interest, which may reveal biomarkers that are not reliably identifiable to a trained pathologist based on the analysis of H&E stained slides. ISH and FISH may be employed to assess the number of copies of a gene or the abundance of a specific RNA molecule, depending on the type of probe employed (e.g., DNA probes for gene copy number and RNA probes for assessment of RNA expression). If these methods also fail to provide sufficient information to detect some biomarkers, genetic testing of tissues may be used to confirm whether a biomarker is present (e.g., overexpression of a specific protein or gene product in a tumor, amplification of a given gene in a cancer).

Digitized images may be prepared to show stained microscope slides, which may allow a pathologist to manually view the images on the slides and estimate the number of stained abnormal cells in the images. However, this process may be time consuming and may lead to errors in identifying abnormalities, as some abnormalities are difficult to detect. Computer processes and devices may be used to assist pathologists in detecting abnormalities that may otherwise be difficult to detect. For example, AI may be used to predict biomarkers (such as overexpression of proteins and/or gene products, amplification or mutation of specific genes, etc.) from prominent areas in digital images of tissues stained using H&E and other dye-based methods. Images of tissues may be whole slide images (WSIs), images of tissue cores in microarrays, or selected areas of interest in tissue sections. Using staining methods such as H&E, these biomarkers may be difficult for humans to visually detect or quantify without the aid of additional testing. Using AI to infer these biomarkers from digital images of tissue has the potential to improve patient care while also being faster and cheaper.

Only the detected biomarkers or images can then be used to recommend a specific cancer drug or drug combination therapy to treat the patient, where the AI can identify drugs or drug combinations that are more likely to be successful by correlating the detected biomarkers with a database of treatment options. This can be used to facilitate automated recommendation of immunotherapy drugs to target the patient's specific cancer. Furthermore, this can be used to enable personalized cancer treatment for specific subsets of patients and/or rarer cancer types.

In today's field of pathology, it can be difficult to provide systematic quality control ("QC") for pathology sample preparation and quality assurance ("QA") for quality of diagnosis throughout the histopathology workflow. Systematic quality assurance is difficult because it can require duplicated efforts by two pathologists and is resource and time intensive. Some methods for quality assurance include (1) a second review of the original diagnosed cancer cases, (2) periodic review of discordant or changed diagnoses by a quality assurance committee, and (3) random review of a subset of cases. These are non-comprehensive, primarily retrospective, and manual. With automated and systematic QC and QA mechanisms, quality can be ensured on a case-by-case basis and throughout the workflow. Laboratory quality control and digital pathology quality control can be critical to the success of patient sample collection, processing, diagnosis, and archiving. Manual and sampling approaches to QC and QA offer substantial advantages. Systematic QC and QA have the potential to provide efficiencies and improve diagnostic quality.

As described above, the computational pathology process and device of the present disclosure may provide an integrated platform while integrating with a Laboratory Information System (LIS) to enable a fully automated process including data collection, processing, and viewing of digital pathology images via a web browser or other user interface. Additionally, clinical information may be aggregated using cloud-based data analytics of patient data. Data may originate from hospitals, clinics, field researchers, etc. and may be analyzed by machine learning, computer vision, natural language processing, and/or statistical algorithms to provide real-time monitoring and prediction of health patterns at multiple levels of geographic specificity.

The present disclosure presents an integrated workflow to facilitate disease detection and/or cancer diagnosis by providing a workflow that integrates, for example, slide evaluation, tasks, image analysis and cancer detection AI, annotation, consultation, and recommendations in one workstation. The present disclosure describes various exemplary user interfaces available within the workflow, as well as AI tools that may be integrated into the workflow to facilitate and assist the pathologist's work.

FIG. 1A illustrates a block diagram of a system and network for providing a workflow for determining and outputting sample property or image property information for digital pathology images using machine learning, according to an exemplary embodiment of the present disclosure.

Specifically, FIG. 1A illustrates an electronic network 120 that may be connected to servers in a hospital, laboratory, and/or doctor's office, etc. For example, a doctor server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and/or a laboratory information system 125, etc. may each be connected to the electronic network 120, such as the Internet, through one or more computers, servers, and/or handheld mobile devices. According to an exemplary embodiment of the present application, the electronic network 120 may also be connected to a server system 127, which may include a processing device configured to implement a disease detection platform 100 according to an exemplary embodiment of the present disclosure, which includes a viewing application tool 101 for determining and outputting sample property or image property information for digital pathology images using machine learning.

The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125 may create or otherwise obtain images of one or more patient cytology samples, histopathology samples, slides of cytology samples, digitized images of histopathology sample slides, or any combination thereof. The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, previous biopsy or cytology information, etc. The physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125 may transmit the digitized slide images and/or patient-specific information to the server system 127 via the electronic network 120. The server system 127 may include one or more storage devices 126 for storing images and data received from at least one of the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125. The server system 127 may also include a processing device for processing the images and data stored in the storage device 126. The server system 127 may further include one or more machine learning tools or capabilities. For example, the processing device may include machine learning tools for the disease detection platform 100 according to one embodiment. Alternatively or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

Physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and/or laboratory information system 125 refer to systems used by pathologists to review images of slides. In a hospital setting, tissue type information may be stored in LIS 125. However, the correct tissue classification information is not always paired with image content. In addition, even if a LIS is used to access the sample type for digital pathology images, this indication may not be correct due to the fact that many components of the LIS are manually entered and may leave a large margin of error. According to exemplary embodiments of the present disclosure, the sample type and/or other sample information may be identified without the need to access LIS 125, or potentially identified to the correct LIS 125. For example, a third party may be given anonymized access to image content without the corresponding sample type indication stored in the LIS. In addition, access to LIS content may be limited due to its sensitive content.

FIG. 1B illustrates an exemplary block diagram of a disease detection platform 100 for determining and outputting sample property or image property information for digital pathology images using machine learning.

Specifically, FIG. 1B depicts components of a disease detection platform 100, according to one embodiment. For example, the disease detection platform 100 may include a viewing application tool 101, a data collection tool 102, a slide capture tool 103, a slide scanner 104, a slide manager 105, and/or a storage device 106.

The viewing application tool 101, as described below, may refer to processes and systems for providing a user (e.g., a pathologist) with sample property and/or image property information regarding a digital pathology image, according to an exemplary embodiment. The information may be provided through various output interfaces (e.g., a screen, a monitor, a storage device, and/or a web browser, etc.).

Data collection tools 102 refer to processes and systems for facilitating the transfer of digital pathology images to various tools, modules, components, and devices used to classify and process the digital pathology images, according to an exemplary embodiment.

Slide capture tool 103 refers to a process and system for scanning pathology images and converting them into digital form, according to an exemplary embodiment. Slides may be scanned using slide scanner 104, and slide manager 105 may process the images on the slides into digitized pathology images and store the digitized images in storage device 106.

The viewing application tool 101 and its components may each transmit and/or receive digitized slide images and/or patient information to and/or from the server system 127, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125 via the network 120. Additionally, the server system 127 may include a storage device for storing images and data received from at least one of the viewing application tool 101, the data collection tool 102, the slide capture tool 103, the slide scanner 104, and/or the slide manager 105. The server system 127 may also include a processing device for processing images and data stored in the storage device. The server system 127 may further include one or more machine learning tools or capabilities, for example, due to the processing device. Alternatively or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

Any of the above devices, tools, and modules may be located on a device that may be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and/or handheld mobile devices.

1C illustrates an exemplary block diagram of a viewing application tool 101 according to an exemplary embodiment of the present disclosure. The viewing application tool 101 may include a work list tool 107, a slide tray tool 108, a slide share tool 109, and an annotation tool 110, an investigation tool 111, and/or an overlay tool 112. The work list tool 107 may provide an overview of the end-to-end workflow for slide viewing and/or case management. The slide tray tool 108 may organize the slides of a case into parts and provide high level case information including case number, demographic information, etc. The slide share tool 109 may provide the user with the ability to share various slides and include and/or write brief annotations about the nature of the sharing. The annotation tools 110 may include a brush tool, an auto brush tool, a trajectory tool, a pinning tool, an arrow tool, a text field tool, a detailed area tool, a ROI tool, a prediction tool, a measurement tool, a multi-measurement tool, an annotation tool, and/or a screenshot tool. The investigation tool 111 may provide an investigation window featuring a magnified view of an area of interest on the target image. The overlay tool may provide a heat map overlay on the magnified view of the area of interest on the target image to identify the area of interest on the tissue sample in the magnified view of the target image.

FIG. 1D illustrates an example block diagram of a viewing application tool 101 according to an example embodiment of the present disclosure. The viewing application tool 101 may include a training image platform 131 and/or a target image platform 135.

In one embodiment, the training image platform 131 may include a training image capture module 132 and/or an image analysis module 133.

The training image platform 131, according to one embodiment, may create or receive training images that are used to train the machine learning system to effectively analyze and classify digital pathology images. For example, the training images may be received from any one or any combination of the server system 127, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125. The images used for training may be from real sources (e.g., humans, animals, etc.) or may be from synthetic sources (e.g., graphic rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains such as (but not limited to) H&E, hematoxylin only, IHC, molecular pathology, etc., and/or (b) digitized tissue samples from 3D imaging devices such as microCT.

The training image capture module 132 may create or receive a data set comprising one or more training images corresponding to one or both of images of human tissue and graphically/synthetically rendered images. For example, the training images may be received from any one or any combination of the server system 127, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125. The data set may be kept on a digital storage device. The image analysis module 133 may analyze the images and identify sample properties and/or image property information (e.g., sample type, overall quality of the cut of the sample, overall quality of the glass pathology slide itself, and/or tissue morphology characteristics).

According to one embodiment, the target image platform 135 may include a target image capture module 136, a sample detection module 137, and an output interface 138. The target image platform 135 may receive the target image and apply a machine learning system to the received target image to determine characteristics of the target sample. For example, the target image may be received from any one or any combination of the server system 127, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125. The target image capture module 136 may receive a target image corresponding to the target sample. The sample detection module 137 may apply a machine learning system to the target image to determine characteristics of the target sample and/or characteristics of the target image. For example, the sample detection module 137 may detect a sample type of the target sample. Additionally, the sample detection module 137 may apply a machine learning system to determine whether an area of the sample includes one or more anomalies.

The output interface 138 may be used to output the target image and information about the target sample (e.g., to a screen, monitor, storage device, web browser, etc.).

2 is a flow chart illustrating an exemplary method of providing a viewing application tool 101 for identifying sample information and providing a user interface for viewing the sample information, according to an exemplary embodiment of the present disclosure. For example, the exemplary method 200 (e.g., steps 202-206) may be performed by the viewing application tool 101 automatically or in response to a request from a user (e.g., a physician, pathologist, technician, etc.).

According to one embodiment, an exemplary method 200 for identifying sample information and providing a user interface for viewing the sample information may include one or more of the following steps: In step 202, the method may include receiving a target image corresponding to the target sample, the target sample comprising a patient tissue sample. For example, the target image may be received from any one or any combination of server system 127, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and/or laboratory information system 125.

In step 204, the method may include applying a machine learning system to the target image to determine at least one characteristic of the target sample and/or at least one characteristic of the target image. Determining the characteristic of the target sample may include determining sample information for the target sample. For example, determining the characteristic may include determining whether an anomaly is present in the target sample.

The machine learning system may be generated by processing a plurality of training images and predicting at least one characteristic, where the training images may include images of human tissue and/or algorithmically/synthetically generated images. The machine learning system may be implemented using machine learning methods for classification and regression. The training inputs may include real or synthetic images. The training inputs may or may not be augmented (e.g., adding noise or creating variants of the inputs by inversion/distortion). Exemplary machine learning systems may include, but are not limited to, any one or any combination of neural networks, convolutional neural networks, random forests, logistic regression, and nearest neighbor methods. Convolutional neural networks can directly learn the image feature representations required to discriminate characteristics, which can work very well when there is a large amount of data to train on per sample, while other methods can be used in conjunction with either traditional computer vision features, such as speed-up robust features (SURF) or scale-invariant feature transform (SIFT), or learned embeddings (e.g., descriptors) generated by trained convolutional neural networks, which can provide advantages when there is only a small amount of data to train on. Training images may be received from any one or any combination of the server system 127, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125. This data set may be kept on a digital storage device. Images used for training may come from real sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphic rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains such as (but not limited to) H&E, IHC, molecular pathology, and/or (b) digitized tissue samples from 3D imaging devices such as microCT.

In step 206, the method may include outputting a target image that identifies an area of interest based on at least one characteristic of the target sample and/or at least one characteristic of the target image.

Different methods for implementing machine learning algorithms and/or architectures include, but are not limited to, (1) CNN (Convolutional Neural Networks), (2) MIL (Multiple Instance Learning), (3) RNN (Recurrent Neural Networks), (4) feature aggregation via CNN, and/or (5) feature extraction followed by ensemble methods (e.g., Random Forests), linear/nonlinear classifiers (e.g., SVM (Support Vector Machines), MLP (Multilayer Perceptrons), and/or dimensionality reduction techniques (e.g., PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis)). Exemplary features may include vector embeddings from a CNN, single/multi-class output from a CNN, and/or multi-dimensional output from a CNN (e.g., mask overlay of the original image). A CNN may learn feature representations for classification tasks directly from pixels, which may lead to better diagnostic performance. When detailed annotations for regions or per-pixel labels are available, and there is a large amount of labeled data, a CNN may be trained directly. However, when labels are available at the global slide level and/or at the local pixel level, a CNN may be trained directly. When the sigma is only over a set of slides in a group (which in pathology may be called a "part"), MIL may be used to train a CNN or another neural network classifier, leading to the ability to learn image regions that are diagnostic for classification tasks, without global annotation. RNNs may be used on features extracted from multiple image regions (e.g., tiles), which then perform processing and make predictions. Other machine learning methods, such as random forests, SVMs, and many others, may be used in conjunction with any of the features learned by CNN, CNN with MIL, or by using hand-crafted image features (e.g., SIFT or SURF) to perform classification tasks, but they may perform poorly when trained directly from pixels. These methods may perform poorly compared to CNN-based systems when there is a large amount of annotated training data available. Dimensionality reduction techniques may be used as a preprocessing step before using any of the mentioned classifiers, which may be useful when there is little data available.

According to one or more embodiments, any of the above algorithms, architectures, methodologies, attributes, and/or features may be combined with any or all of the other algorithms, architectures, methodologies, attributes, and/or features. For example, any of the machine learning algorithms and/or architectures (e.g., neural network methods, convolutional neural networks (CNN), recurrent neural networks (RNN), etc.) may be trained using any of the training methodologies (e.g., multiple instance learning, reinforcement learning, activation learning, etc.).

The explanations of terms below are merely illustrative and are not intended to limit the terms in any way.

The labels may refer to information about the inputs to a machine learning algorithm that the algorithm attempts to predict.

For a given image size of NxM, the segmentation may be another image of size NxM that assigns to each pixel in the original image a number that describes the class or type of that pixel. For example, in WSI, elements in the mask may categorize each pixel in the input image as belonging to, for example, background, tissue, and/or unknown classes.

Slide-level information may refer to information about a slide in general, but not necessarily to the specific location of that information within the slide.

A heuristic may refer to a logical rule or function that deterministically produces an output given an input. For example, a prediction that a slide has an anomaly outputs 1 if it does, and 0 if it does not.

Embedding may refer to a conceptual high-dimensional numerical representation of low-dimensional data. For example, if a WSI is passed through a CNN training to classify tissue type, the numbers on the last layer of the network may provide a set of numbers (e.g., on the order of thousands) that contain information about the slide (e.g., information about the type of tissue).

Slide-level predictions may refer to concrete predictions about the slide as a whole. For example, a slide-level prediction may be that the slide contains an anomaly. Additionally, slide-level predictions may refer to individual probability predictions across a defined set of classes.

A classifier can refer to a model and/or system that is trained to take input data and associate it with a category.

According to one or more embodiments, the machine learning model and/or system may be trained in different ways. For example, training of the machine learning model may be performed by any one or any combination of supervised training, semi-supervised training, unsupervised training, classifier training, mixed training, and/or uncertainty estimation. The type of training used may depend on the amount of data, the type of data, and/or the quality of the data. Table 1 below describes a non-limiting list of some types of training and corresponding features.

Supervised training may be used in conjunction with small amounts of data to provide a seed for a machine learning model. In supervised training, the machine learning model may look for specific items (e.g., bubbles, tissue folds, etc.), flag the slide, and quantify the amount of the specific items present in the slide.

According to one embodiment, an exemplary fully supervised training may take as input the WSI and may include an indication of the segmentation. A pipeline for fully supervised training may include (1) 1, (2) 1, heuristic, (3) 1,4, heuristic, (4) 1,4,5, heuristic, and/or (5) 1,5, heuristic. Advantages of fully supervised training may be that (1) it may require fewer slides and/or (2) the output is interpretable because the area of the image that contributed to the diagnosis may be known. Disadvantages of using fully supervised training may be that it may require a large amount of segmentation, which may be difficult to obtain.

According to one embodiment, an exemplary semi-supervised (e.g., weakly supervised) training may take as input the WSI and may include indicators of slide-level information. A pipeline for semi-supervised training may include (1) 2, (2) 2, heuristic, (3) 2,4, heuristic, (4) 2,4,5, heuristic, and/or (5) 2,5, heuristic. An advantage of using semi-supervised training may be that (1) the type of indicator required may be present in many hospital records, and (2) the output is interpretable because the area of the image that contributed most to the diagnosis may be known. A disadvantage of using semi-supervised training is that it may be difficult to train. For example, the system may need to use training schemes such as multiple instance learning, activation learning, and/or distributed training to account for the fact that there is limited information about the locations in the slide where the information that should lead to a decision is located.

According to one embodiment, an exemplary unsupervised training may take WSI as input and may not require any labels. A pipeline for unsupervised training may include (1) 3,4, and/or (2) 3,4, heuristics. An advantage of unsupervised training may be that it does not require any labels. A disadvantage of using unsupervised training may be that (1) it may be difficult to train. For example, using training schemes such as multiple instance learning, activation learning, and/or distributed training, one may need to take into account the fact that there is limited information about where in the slide the information that should lead to the decision is located, (2) it may require additional slides, and/or (3) it may not be very interpretable because it may output predictions and probabilities without explaining why the predictions were made.

According to one embodiment, the exemplary mixed training includes training any of the exemplary pipelines described above with respect to fully supervised training, semi-supervised training, and/or unsupervised training, and then the resulting model may be used as an initial point for any of the training methods. Advantages of mixed training may be that (1) it may require less data, (2) it may have improved performance, and/or (3) it may allow mixing of different levels of labels (e.g., segmentation, slide-level information, no information). Disadvantages of mixed training may be that (1) training may be more complex and/or expensive, and/or (2) it may require more code, which may increase the number and complexity of potential bugs.

According to one embodiment, the exemplary uncertainty estimation may include training any of the exemplary pipelines described above with respect to fully supervised training, semi-supervised training, and/or unsupervised training for any task involving slide data with uncertainty estimation at the end of the pipeline. Additionally, heuristics or classifiers may be used to predict whether a slide has an anomaly based on the amount of uncertainty in the prediction of the test. An advantage of the uncertainty estimation may be that it is robust to out-of-distribution data. For example, when presented with unfamiliar data, it may still correctly predict uncertainty. A disadvantage of the uncertainty estimation may be that (1) more data may be required, (2) it may have poor overall performance, and/or (3) it may not be very interpretable since the model may not necessarily identify the extent to which a slide or slide embedding is anomalous.

According to one embodiment, ensemble training may include simultaneously launching models generated by any of the exemplary pipelines described above and combining the outputs by heuristics or classifiers to generate robust and accurate results. The advantages of ensemble training may be that (1) it is robust to out-of-distribution data, and/or (2) it may combine advantages and disadvantages of other models, resulting in minimization of disadvantages (e.g., a supervised training model combined with an uncertainty estimation model, and a heuristic using a supervised model when the incoming data is in-distribution and an uncertainty model when the data is out-of-distribution, etc.). The disadvantages of ensemble training may be that (1) it may be more complex, and/or (2) it may be expensive to train and launch.

The training techniques discussed herein may also be incremental, with images with more annotations being used for training initially, which may allow for more effective subsequent training using less supervised slides with fewer annotations.

Training may begin using the most completely annotated slides compared to all training slide images that may be used. For example, training may begin using supervised learning. A first set of slide images may be received or determined with associated annotations. Each slide may have marked and/or masked regions and may include information such as whether the slide has an anomaly. The first set of slides may be provided to a training algorithm, e.g., a CNN, which may determine correlations between the first set of slides and their associated annotations.

After training with the first set of images is completed, a second set of slide images may be received or determined with fewer annotations than the first set, e.g., with partial annotations. In one embodiment, the annotations may only indicate that the slide has a diagnosis or quality issue associated with it, and may not specify the disease that may be found or its location, etc. The second set of slide images may be trained using a different training algorithm than the first, e.g., multiple instance learning. The first set of training data may be used to partially train the system, making the second training process more effective at generating accurate algorithms.

In this manner, training may proceed in any number of stages based on the quality and type of training slide images using any number of algorithms. These techniques may be utilized in situations where multiple sets of training images are received, which may be of varying quality, annotation levels, and/or annotation types.

3 illustrates an exemplary output 300 of the viewing application tool 101, according to an exemplary embodiment. As illustrated in FIG. 3, the viewing application tool 101 may include a navigation menu 301 for navigating to a work list view for the work list tool 107, a slide view for the slide tray tool 108 and slide share tool 109, an annotation tool 110, an investigation tool 111, and/or an overlay toggle for the overlay tool 112. The navigation menu 301 may also include a view mode input for adjusting the view (e.g., splitting the screen and viewing multiple slides and/or stains at once). The zoom menu 302 may be used to quickly adjust the zoom level of the target image 303. The target image may be displayed in any color. For example, a dark neutral color scheme may be used to display details of the target image 303. The viewing application tool 101 may include a positive/negative indicator as to whether a possible disease is present at the (x,y) coordinate in the image. Additionally, the positive/negative indicators may be viewed and/or edited by the user.

Slides may be prioritized using prioritization features based on characteristics identified in the target image. Slides may be organized using a folder system, including default and custom systems. The viewing application tool 101 may include a case view, which may include results, notes, attachments, patient information, and status icons. Slides within a case may be organized by type and color coded. The case sharing feature may be designed to be secure and reliable. Cases may be archived to provide the necessary storage space. There may be functionality to switch between masses within a single slide. The viewing application tool 101 may provide notifications for new messages and/or new slides. Status icons may indicate whether a case and/or slide is new and whether it has been shared and/or ordered. There may be functionality to search for cases, patients, and/or projects. The viewing application tool 101 may include a patient queue and/or a consulting queue.

The viewing application tools 101 may include viewing options, which may include viewing modes, windows, rotation, scale bars, and/or overlays. Exemplary outputs of the viewing application tools 101 may include patient information, slide information (e.g., image overview), and/or file information (e.g., slide map). Viewing modes may include a default view and viewing modes for different ratios. The viewing application tools 101 may include a fluorescence mode, a magnifier view, a scale bar, pan and zoom functionality, image rotation functionality, focus of interest, and/or other slide actions (e.g., ordering a new stain). The viewing application may analyze the images and determine the patient's viability.

The viewing application tools 101 may include functionality for ordering slides and levels and for recommending slides. For example, the viewing application tools 101 may include options for selecting a slide type, submitting a new slide order, receiving a slide order confirmation, a slide ordering indicator, viewing the new slide, and/or comparing the new slide with a previous slide. The viewing application tools 101 may also include functionality for recommending levels and selecting a level.

The viewing application tool 101 may include functionality for searching for similar cases using the viewing application tool 101, according to an example embodiment. For example, areas of interest may be identified, the areas of interest may be rotated and quantified, and cases with similar areas of interest may be identified and linked to the patient case.

The viewing application tool 101 may include functionality to provide mitotic counts. For example, areas of interest may be identified, the areas of interest may be moved/rotated and quantified, and cases with similar areas of interest may be identified and linked to patient cases. The viewing application tool 101 may display the count results, including the number of mitoses and/or visual markers. Identified areas may be magnified and the count results may be reviewed and/or edited by the user. Results may be shared across the system.

4 illustrates an exemplary output 400 of the overlay tool 112 of the viewing application tool 101, according to an exemplary embodiment. The overlay tool 112 may provide a heatmap overlay 401 on the target image 303. The heatmap overlay may identify areas of interest on the tissue sample of the target image. For example, the heatmap overlay may identify areas in which the AI system predicts that an abnormality may be present in the tissue sample. The predictive heatmap overlay visualization may be toggled on and off using the navigation menu 301. For example, a user may select an overlay icon on the navigation menu and toggle the heatmap overlay on or off. The predictive heatmap interface may indicate to a user (e.g., a pathologist) one or more areas on the tissue that the user should investigate. The heatmap overlay may be transparent and/or semi-transparent, so that the user may see the underlying tissue while viewing the heatmap overlay. The heatmap overlay may include different colors and/or shading to indicate the severity of the detected disease. Other types of overlays may also be available in context to the type of tissue being viewed. For example, FIG. 4 illustrates a prostate biopsy. However, other diseases may require different visualizations or AI systems associated with them.

5 illustrates an exemplary output 500 of the worklist tool 107 of the viewing application tool 101, according to an exemplary embodiment. As illustrated in FIG. 5, the viewing application tool 101 may include a worklist 501, which may be opened by selecting a worklist icon in the navigation menu 301. The main navigation menu 301 may provide an overview of the end-to-end workflow for slide viewing and/or case management. For example, the overview may include a worklist 501, which may display cases for all users (e.g., pathologists). This may be accessed from the viewer user interface (UI) by opening a worklist panel. Within the worklist panel, the pathologist may view the patient's medical record number (MRN), patient ID number, patient name, suspected disease type, case status, surgery date, and select from various case actions, such as view case details, share case, and/or search.

6 illustrates an exemplary output 600 of the slide tray tool 108 of the viewing application tool 101, according to an exemplary embodiment. As illustrated in FIG. 6, a slide icon in the navigation menu 301 may open a slide tray 601 as a drawer/dropdown from the menu on the left of the screen. The menu may present high level case information, including case number, demographic information, etc. The slides of the case may be organized into parts and represented in the slide tray 601 as parent tabs in dark gray. Each part may expand to display the slides therein. Thus, slides determined to be associated with the same part, which may be a predefined area of a sample or a predefined sample area of a patient, may be displayed within a predefined proximity to each other. A red dot 602, which may be a different color or shape indicator, may indicate on a part that the AI system has found a possible disease (e.g., cancer) at a location within that part. A red dot 603 on a slide may indicate that the AI system has found a possible disease at a location on that slide. Those parts and slides where the AI system finds possible disease may be brought to the top of the list for immediate viewing by a user (e.g., a pathologist). This workflow may help pathologists identify diseases more quickly.

7 illustrates an exemplary output 700 of the slide sharing tool 109 of the view application tool 101, according to an exemplary embodiment. From the slide panel, a user may select the "Share" button and select single, multiple, and/or all slides to share. The slide sharing panel 701 may be located to the right of the slide tray 601 and may allow a user to type in the recipients to share with, select various slides to include, and/or write a brief commentary about the nature of the share. Once the send button is selected, the slides and brief commentary may be sent to the recipients, who may receive a notification and view these slides and/or commentary within their view application. The interaction between users may be captured in a log, which is described below.

8 illustrates an exemplary output 800 of the annotation tool 110 of the viewing application tool 101, according to an exemplary embodiment. For each annotation made, the annotation log 801 may capture the appearance of the actual annotation made, the type of annotation, any measurements, the annotation, the user who made the annotation, and/or the time of the annotation. The annotation log 801 may serve as a centralized view for consultation and information sharing to be reviewed within the context of the annotation or area of interest. The user requesting the consultation (e.g., a pathologist) and the consulting pathologist may read annotations, make annotations, and/or have ongoing dialogue specific to each annotation, each with a timestamp. The user may also select a thumbnail in the annotation log 801 to view that area of interest at large scale in the main viewer window.

FIG. 9 illustrates an exemplary output of the annotation tool 110 of the viewing application tool 101, according to an exemplary embodiment. For example, the annotation log illustrated in FIG. 9 includes a dialogue between pathologists to discuss a slide. The pathologists may submit notes along with the annotated image for quick consultation on any areas of interest on the slide.

10 illustrates an exemplary output 1000 of the annotation tool 110 of the viewing application tool 101, according to an exemplary embodiment. The viewing application may include various annotation tools, illustrated in the annotation menu 1001. For example, the annotation tools may include a brush tool, an auto brush tool, a trajectory tool, a pinning tool, an arrow tool, a text field tool, a detailed area tool, an ROI tool, a prediction tool, a measurement tool, a multi-measurement tool, a lesion area drawing tool, a region of interest tool, an add tag tool, an annotation tool, and/or a screenshot tool. Annotations may be used for research purposes and/or in clinical settings. For example, a pathologist may make annotations, write annotations, and/or share them with a colleague to get a second opinion. Annotations may also be used in a final diagnosis report as supporting evidence of a diagnosis (e.g., the tumor is x length, has y number of mitotic counts, etc.)

11 illustrates an exemplary output 1100 of an investigation tool 111 of a viewing application tool 101, according to an exemplary embodiment. The investigation tool 111 may include an investigation window 1101, featuring a magnified view of an area of interest for a target image. Based on user input, the investigation window may be dragged across an image to quickly examine the image. Thus, a pathologist may be able to quickly move across a slide in a manner similar to the speed at which a slide is currently being moved within a microscopy. A user may quickly switch or toggle a predictive heat map overlay on and off while browsing through the investigation tool 111. A user may take screenshots of tissue viewable within the investigation tool and quickly share the screenshots. Different annotation tools, such as measurement tools and/or area highlighting, may also be available with the investigation tool. Additionally, a user may increase and decrease the magnification within the investigation tool independent of the main slide magnification level.

12 illustrates an example output 1200 of the investigation tool 111 of the viewing application tool 101, according to an example embodiment. As illustrated in FIG. 12, the investigation window 1201 may display AI output in a magnified view for verification and assistance in diagnosis. The investigation window 1201 may include a heat map overlay that displays characteristics of the tissue sample predicted by the AI system. For example, the heat map overlay may identify areas in which the AI system predicts that anomalies may exist in the tissue sample.

FIG. 13 illustrates an example workflow of the viewing application tool 101, according to an example embodiment. The workflow may include a work list, a case view, and/or a slide view as modes within a user's (e.g., a pathologist's) workflow. As illustrated in FIG. 13, a diagnostic report may be output from the case view, which may include information required for digital signature.

14A, 14B, and 14C illustrate exemplary outputs for a slide view of the viewing application tool 101, according to an exemplary embodiment. For example, as illustrated at the bottom of FIG. 14A, a slide view toolbar 1401 may be located at the bottom of the display with a vertical orientation of the slides. Slides may be grouped by parts and may display related case information within that context. As illustrated in FIG. 14B, the toolbar 1401 may be located on the left side of the display, within which the toolbar may include menu items such as tools, zoom functions, and thumbnail slides. As illustrated in FIG. 14C, two vertical toolbars 1401 and 1402 may be located on the left and right sides of the screen, which may allow easy access to various functions for touch screen devices (e.g., tools and/or views).

15 illustrates an exemplary workflow for a new user (e.g., patient, physician, etc.) to create an account for requesting and receiving consultations, according to an exemplary embodiment. A request to create a new account may be submitted through the hospital website and/or through the viewing application tool 101. Once an account is created, a pathologist may send notes along with annotated images for quick consultation of any areas of interest on a slide. A patient may also receive consultation information from a pathologist and/or billing information related to their case. If the user is a physician, billing options may differ based on whether the party being billed is a referring physician, a patient, and/or an insurance company. If the user is a patient, the patient may be billed directly or through an insurance company.

16A and 16B illustrate example outputs for a pathology consultation dashboard, according to an example embodiment. As illustrated in FIG. 16A, new requests and registered consultations may include patient name, referring physician, external ID, internal ID, facility, requested date, status, updated date, and/or links to additional details. The consultation dashboard may include functionality to select slides to send, a send slides dialogue box, consulting request notes, consulting attachments, and/or a consultation log. As illustrated in FIG. 16B, consultation requests may include patient name, referring physician, facility, requested date, status, updated date, and/or links to further details about the request.

FIG. 17 illustrates an example output of a case view for the viewing application tool 101, according to an example embodiment. The case view may include all of the necessary information for a digital seal workflow. In addition, as illustrated in FIG. 17, an annotation log may be integrated into the case view.

As shown in FIG. 18, device 1800 may include a central processing unit (CPU) 1820. CPU 1820 may be any type of processor device, including, for example, any type of special purpose or general purpose microprocessor device. As will be appreciated by those skilled in the art, CPU 1820 may also be a single processor in a multi-core/multi-processor system operating alone or in a cluster of computing devices operating in a cluster or server farm. CPU 1820 may be connected to a data communications infrastructure 1810, for example, a bus, a message queue, a network, or a multi-core message passing scheme.

The device 1800 may also include a main memory 1840, e.g., random access memory (RAM), and may also include a secondary memory 1830. The secondary memory 1830, e.g., read only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may comprise, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, or the like. The removable storage drive, in this embodiment, reads from and/or writes to a removable storage unit in a well-known manner. The removable storage unit may comprise a floppy disk, a magnetic tape, an optical disk, or the like, which is read and written by the removable storage drive. As will be appreciated by those skilled in the art, such removable storage units generally include computer usable storage media having computer software and/or data stored therein.

In alternative implementations, secondary memory 1830 may include other similar means for allowing computer programs or other instructions to be loaded into device 1800. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from removable storage units to device 1800.

Device 1800 may also include a communications interface ("COM") 1860. Communications interface 1860 allows software and data to be transferred between device 1800 and external devices. Communications interface 1860 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via communications interface 1860 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1860. These signals may be provided to communications interface 1860 via a communications path in device 1800, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

The hardware elements, operating systems, and programming languages of such equipment are conventional in nature and are assumed to be fully familiar to those skilled in the art. The device 1800 may also include input and output ports 1850 for connecting with input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, the various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming of one computer hardware platform.

Throughout this disclosure, references to components or modules generally refer to items that may be logically grouped together to perform a function or group of related functions. Like reference numbers are generally intended to refer to the same or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

The tools, modules, and functions described above may be implemented by one or more processors. A "storage" type medium may include any or all of the tangible memory of a computer, processor, or equivalent, or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, which may provide optional non-transitory storage for software programming.

The software may be communicated over the Internet, a cloud service provider, or other telecommunications network. For example, the communication may enable the software to be loaded from one computer or processor into another. As used herein, unless limited to non-transient, the term tangible "storage" media, such as computer or machine "readable media," refers to any medium involved in providing instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only and is not a limitation of the present disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as illustrative only.

Claims (21)

1. A computer-implemented method for analyzing an electronic image corresponding to a sample, the method comprising:
receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample;
applying a machine learning system to the target electronic image to determine at least one property of the target sample and/or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and/or algorithmically generated images;
outputting the target electronic image identifying an area of interest based on at least one characteristic of the target sample and/or at least one characteristic of the target electronic image ;
receiving a second target electronic image corresponding to the target sample;
determining a first portion of the target specimen associated with the target electronic image;
determining a second portion of the target specimen associated with the second target electronic image;
identifying whether the first portion and the second portion are the same or overlap;
displaying the first representation of the target electronic image and the second representation of the second target electronic image in a predetermined proximity to one another in response to identifying that the first portion and the second portion are the same or overlapping.
4. A computer-implemented method comprising: determining whether there is an area of interest associated with the target electronic image and/or an area of interest associated with the second target electronic image;
displaying an indicator associated with a first representation of the target electronic image in response to determining that the area of interest is associated with the target electronic image;
2. The computer-implemented method of claim 1, further comprising: in response to determining that the area of interest is associated with the second target electronic image, displaying an indicator associated with a second representation of the second target electronic image. The computer-implemented method of claim 1 or 2 , wherein identifying the area of interest comprises displaying a heat map overlay on the target electronic image. 3. The computer-implemented method of claim 1 or 2, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly . identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly;
The computer-implemented method of claim 1 or 2 , wherein the heat map overlay is transparent or semi-transparent. displaying a magnification window over at least a portion of the target electronic image;
The computer-implemented method of claim 1 or 2 , further comprising: presenting a magnified image of the target specimen within the magnification window at a magnification level different from a magnification level of the target electronic image. displaying a magnification window over at least a portion of the target electronic image;
presenting a magnified image of the target sample within the magnification window at a magnification level different from a magnification level of the target electronic image;
The computer-implemented method of claim 1 or 2 , wherein the magnification window comprises a selectable icon for toggling a heatmap overlay on the magnified image. displaying a slide tray tool over the target electronic image that identifies an outline of the target sample;
applying the machine learning system to the target electronic image to determine whether a portion of the target sample contains an anomaly; and
3. The computer-implemented method of claim 1, further comprising: in response to determining that the portion contains an anomaly, presenting an indicator of the anomaly within the slide tray tool. The computer-implemented method of claim 1 or 2 , further comprising displaying an annotation log comprising an indicator identifying the area of interest and a consultation request associated with the area of interest. 1. A system for analyzing an electronic image corresponding to a specimen, the system comprising:
at least one memory for storing instructions;
At least one processor;
The at least one processor executes the instructions;
receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample;
applying a machine learning system to the target electronic image to determine at least one property of the target sample and/or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and/or algorithmically generated images;
outputting the target electronic image identifying an area of interest based on at least one characteristic of the target sample and/or at least one characteristic of the target electronic image ;
receiving a second target electronic image corresponding to the target sample;
determining a first portion of the target specimen associated with the target electronic image;
determining a second portion of the target specimen associated with the second target electronic image;
identifying whether the first portion and the second portion are the same or overlap;
displaying the first representation of the target electronic image and the second representation of the second target electronic image in a predetermined proximity to one another in response to identifying that the first portion and the second portion are the same or overlapping.
A system that performs a process including determining whether there is an area of interest associated with the target electronic image and/or an area of interest associated with the second target electronic image;
displaying an indicator associated with a first representation of the target electronic image in response to determining that the area of interest is associated with the target electronic image;
11. The system of claim 10, further comprising: in response to determining that the area of interest is associated with the second target electronic image, displaying an indicator associated with a second representation of the second target electronic image. The system of claim 10 or 11 , wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image. 12. The system of claim 10 or 11, wherein identifying the area of interest includes displaying a heatmap overlay on the target electronic image, the heatmap overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly. identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly;
The system of claim 10 or 11 , wherein the heatmap overlay is transparent or semi-transparent. displaying a magnification window over at least a portion of the target electronic image;
and presenting a magnified image of the target sample within the magnification window at a magnification level different from a magnification level of the target electronic image. displaying a magnification window over at least a portion of the target electronic image;
presenting a magnified image of the target sample within the magnification window at a magnification level different from a magnification level of the target electronic image;
The system of claim 10 or 11 , wherein the magnification window comprises a selectable icon for toggling a heatmap overlay on the magnified image. displaying a slide tray tool over the target electronic image that identifies an outline of the target sample;
applying the machine learning system to the target electronic image to determine whether a portion of the target sample contains an anomaly; and
12. The system of claim 10 or 11 , further comprising: in response to determining that the portion contains an anomaly, presenting an indicator of the anomaly within the slide tray tool. The system of claim 10 or 11 , further comprising displaying an annotation log comprising an indicator identifying the area of interest and a consultation request associated with the area of interest. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for analyzing an electronic image corresponding to a sample, the method comprising:
receiving a target electronic image corresponding to a target sample, the target sample comprising a patient tissue sample;
applying a machine learning system to the target electronic image to determine at least one property of the target sample and/or at least one property of the target electronic image, the machine learning system having been generated by processing a plurality of training images and predicting at least one property, the training images comprising images of human tissue and/or algorithmically generated images;
outputting the target electronic image identifying an area of interest based on at least one characteristic of the target sample and/or at least one characteristic of the target electronic image ;
receiving a second target electronic image corresponding to the target sample;
determining a first portion of the target specimen associated with the target electronic image;
determining a second portion of the target specimen associated with the second target electronic image;
identifying whether the first portion and the second portion are the same or overlap;
displaying the first representation of the target electronic image and the second representation of the second target electronic image in a predetermined proximity to one another in response to identifying that the first portion and the second portion are the same or overlapping.
16. A non-transitory computer readable medium comprising: The method comprises:
determining whether there is an area of interest associated with the target electronic image and/or an area of interest associated with the second target electronic image;
displaying an indicator associated with a first representation of the target electronic image in response to determining that the area of interest is associated with the target electronic image;
displaying an indicator associated with a second representation of the second target electronic image in response to determining that the area of interest is associated with the second target electronic image.
20. The non-transitory computer readable medium of claim 19, further comprising: 21. The non-transitory computer readable medium of claim 19 or 20, wherein identifying the area of interest includes displaying a heat map overlay on the target electronic image, the heat map overlay comprising shading and/or coloring based on a predicted likelihood that a location contains an anomaly.