JP7769732B2 - Image analysis method, image analysis device, and program - Google Patents

Description

This specification discloses an image analysis method, an image analysis device, a program, a method for manufacturing a trained deep learning algorithm, and a trained deep learning algorithm. More specifically, this specification discloses an image analysis method, an image analysis device, a program, a method for manufacturing a trained deep learning algorithm, a trained deep learning algorithm, etc., which include generating data indicating whether an area included in analysis data for an image of tissue to be analyzed is a tumorous area.

Patent Document 1 discloses an image diagnosis support device that classifies and assesses tissue images in pathological tissue images into four groups: normal, benign tumor, precancerous state, and cancerous state. The image classification means extracts a region of interest from the image data, calculates feature quantities that indicate the characteristics of the region of interest, and classifies the group based on the calculated feature quantities. The feature quantities include the density of clumps per unit area in the cell nucleus, the density of the clump area, the clump area, the clump thickness, and the clump length. The image assessment means learns the relationship between these feature quantities and assessment results, and makes assessments based on the learned learning parameters. Learning is performed using machine learning algorithms such as support vector machines.

JP 2010-203949 A

When making a definitive diagnosis of whether a tumor is malignant, a histopathological diagnosis is performed using a pathological tissue specimen. Histopathological diagnosis is also often performed as intraoperative rapid diagnosis to determine the area of tissue containing a malignant tumor to be removed during surgery. Intraoperative rapid diagnosis involves having the patient wait with the affected area incised during surgery, and using histopathological diagnosis to determine whether the tumor is malignant, whether any tumor remains at the margins of the resected tissue, whether there is lymph node metastasis, and so on. The results of the intraoperative rapid diagnosis determine the direction of subsequent surgery for the patient.

Pathological tissue diagnoses are made by doctors, particularly pathologists, who observe tissue specimens using a microscope or other device. However, to be able to make an accurate, definitive diagnosis through the observation of tissue specimens, a long period of time is required for repeated observation of tissue specimens from various cases under the supervision of an experienced pathologist, and the training of pathologists also takes an enormous amount of time.

The shortage of pathologists is serious, and as a result, there are concerns that a definitive diagnosis of a patient's malignant tumor will be delayed, leading to delayed treatment, or that treatment will begin without waiting for a definitive diagnosis. Furthermore, because both regular tissue diagnosis and rapid intraoperative diagnosis are concentrated among a small number of pathologists, the workload for each pathologist is enormous, and the working conditions of the pathologists themselves are also becoming an issue. However, no solution to this problem has yet been found.

Therefore, the ability of devices to support pathological tissue diagnosis, especially to the extent that the diagnosis is close to that made by the human eye, is expected to make a significant contribution to resolving the shortage of pathologists and improving their working conditions.

In terms of devices that support pathological tissue diagnosis, the invention described in the aforementioned Patent Document 1 performs pathological judgment of specimen tissue based on image analysis using machine learning. This method requires that features be created manually. The problem with manually creating features is that the skill of the person performing the manual creation of features has a significant impact on the performance of the image analysis. Furthermore, the invention described in Patent Document 1 determines whether each cell nucleus in an image of a tissue specimen magnified under a microscope is a cancer cell nucleus. Therefore, there is a problem in that analyzing the entire specimen requires a long analysis time.

Therefore, before a pathologist or machine learning algorithm determines whether an individual cell is a cancer cell nucleus, observing the entire tissue contained in a single specimen and screening which parts of the specimen contain tumor tissue can help shorten the time required for a pathological diagnosis of a single specimen.

The present invention aims to provide an image analysis method, image analysis device, and program that generate data indicating whether an area of the tissue to be analyzed, included in the analysis data, is a tumor area.

In order to solve the above-mentioned problems, one embodiment of the image analysis method of the present invention is an image analysis method that analyzes a tissue image using a deep learning algorithm (60) with a neural network structure, and includes selecting a deep learning algorithm (60) corresponding to the type of tissue to be analyzed from a plurality of deep learning algorithms (60) generated according to the type of tissue, generating analysis data (80) from an analysis target image (78) including the tissue to be analyzed, inputting the analysis data (80) into the selected deep learning algorithm (60), and displaying an image of the tissue to be analyzed in a manner that allows differentiation between tumor regions and other regions based on analysis results output from the deep learning algorithm (60) to which the analysis data (80) has been input.

The plurality of deep learning algorithms (60) may be stored in a database (105), and the deep learning algorithm (60) corresponding to the type of tissue to be analyzed may be selected from the plurality of deep learning algorithms (60) stored in the database (105).

Each of the plurality of deep learning algorithms (60) may be associated with a tissue type and stored in the database (105).

At least one of the plurality of deep learning algorithms (60) may be stored in the database (105) in association with one tissue type.

The deep learning algorithm (60) corresponding to the type of tissue to be analyzed may be selected based on user operation via the input unit (26).

The image analysis method may further include performing a rendering process on the analysis target image (78) to distinguish between tumor areas and other areas based on the analysis results output from the deep learning algorithm (60).

The drawing process may include drawing the tumor area and other areas in different colors.

The image to be analyzed (78) may be an image of a tissue diagnostic specimen, and the image to be analyzed (78) may include hues that combine two or more primary colors.

The image (78) to be analyzed may be an image of the tissue to be analyzed captured at a magnification of 3 to 20 times.

The analysis data (80) corresponds to an image cut out from the analysis target image, and multiple pieces of the analysis data (80) may be generated from the analysis target image (78).

The image to be analyzed (78) may be an image of a tissue diagnostic specimen, which may be a stained specimen, and the image to be analyzed (78) may be an image of the stained specimen captured under a bright field microscope.

The training data used to train the deep learning algorithm (60) may be generated based on bright-field images of a tissue specimen containing a tumor area taken from an individual, stained for bright-field observation, and then captured under a bright-field microscope.

The output layer (50b) of the neural network (50) may be a node that uses a softmax function as an activation function.

In order to solve the above problems, an image analysis device (200) according to one embodiment of the present invention is an image analysis device that analyzes an image of tissue using a deep learning algorithm (60) with a neural network structure, and includes a display unit (27) and a control unit (20) that selects a deep learning algorithm (60) corresponding to the type of tissue to be analyzed from a plurality of deep learning algorithms (60) generated according to the type of tissue, generates analysis data (80) from an analysis target image (78) containing the tissue to be analyzed, inputs the analysis data (80) to the selected deep learning algorithm (60), and displays an image of the tissue to be analyzed on the display unit (27) so that tumor areas and other areas can be distinguished based on the analysis results output from the deep learning algorithm (60) to which the analysis data (80) has been input.

The image analysis device (200) may further include a memory unit (23) that stores the plurality of deep learning algorithms (60), and the control unit (20) may select the deep learning algorithm (60) corresponding to the type of tissue to be analyzed from the plurality of deep learning algorithms (60) stored in the memory unit (23).

The memory unit (23) may store each of the plurality of deep learning algorithms (60) in association with a tissue type.

The memory unit (23) may store at least one of the plurality of deep learning algorithms (60) in association with one tissue type.

The image analysis device (200) may further include an input unit (26) operated by a user, and the control unit (20) may select the deep learning algorithm (60) corresponding to the type of tissue to be analyzed based on operation via the input unit (26).

The control unit (20) may perform a drawing process on the analysis target image (78) based on the generated analysis data to display the tumor area and other areas in a distinguishable manner.

The drawing process may include drawing the tumor area and other areas in different colors.

The analysis data (80) corresponds to an image cut out from the analysis target image, and multiple pieces of the analysis data (80) may be generated from the analysis target image (78).

In order to solve the above problem, one embodiment of the present invention provides a computer program that analyzes tissue images using a deep learning algorithm (60) with a neural network structure, and causes a computer to execute the following processes: selecting a deep learning algorithm (60) corresponding to the type of tissue to be analyzed from multiple deep learning algorithms (60) generated according to the type of tissue; generating analysis data (80) from an analysis target image (78) containing the tissue to be analyzed; inputting the analysis data (80) into the selected deep learning algorithm (60); and displaying an image of the tissue to be analyzed in a manner that allows differentiation between tumor regions and other regions based on the analysis results output from the deep learning algorithm (60) to which the analysis data (80) has been input.

It is possible to observe the entire tissue contained in a specimen and generate data to screen which parts of the specimen contain tumor tissue.

FIG. 1 is a schematic diagram for explaining an overview of a deep learning method. FIG. 1 is a schematic diagram for explaining an overview of a deep learning method. FIG. 1 is a schematic diagram for explaining an overview of a deep learning method. FIG. 10 is a schematic diagram for explaining details of training data. FIG. 10 is a schematic diagram for explaining details of training data. FIG. 1 is a schematic diagram for explaining an outline of an image analysis method. 1 is a schematic configuration diagram of an image analysis system according to a first embodiment. FIG. 2 is a block diagram showing the hardware configuration of a vendor-side device 100. FIG. 2 is a block diagram showing the hardware configuration of a user device 200. FIG. 2 is a block diagram for explaining the functions of the deep learning device 100A according to the first embodiment. 1 is a flowchart showing the procedure of deep learning processing. FIG. 10 is a schematic diagram for explaining details of learning by a neural network. FIG. 2 is a block diagram for explaining functions of an image analysis device 200A according to the first embodiment. 10 is a flowchart showing the procedure of an image analysis process. FIG. 10 is a schematic configuration diagram of an image analysis system according to a second embodiment. FIG. 10 is a block diagram for explaining functions of an integrated image analyzing device 200B according to a second embodiment. FIG. 10 is a schematic configuration diagram of an image analysis system according to a third embodiment. FIG. 10 is a block diagram for explaining functions of an integrated image analyzing device 100B according to a third embodiment. 19 shows the analysis results of a tissue specimen obtained from gastric cancer tissue. (a) is a whole slide image of a bright-field image of gastric cancer tissue stained with HE, and is the image to be analyzed. (b) is an image of the whole slide image in which the pathologist has designated the tumor region and enclosed it with a solid line. (c) shows the analysis results using training images and analysis images magnified by 1x. (d) shows the analysis results using training images and analysis images magnified by 5x. (e) shows the analysis results using training images and analysis images magnified by 40x. In Figures 19(c) to (e), the tumor region obtained by the analysis process is displayed in white, the non-tumor region is displayed in gray, and the non-tissue region is displayed in black. 19 shows the analysis results of a tissue specimen obtained from gastric cancer tissue. (a) is a whole slide image of a bright-field image of gastric cancer tissue stained with HE, and is the image to be analyzed. (b) is an image of the whole slide image in which the tumor region is designated by a pathologist and surrounded by a solid line. (c) shows the analysis results using training images and analysis images magnified 5x. (d) shows the analysis results using training images and analysis images magnified 16x. In Figures 19(c) and (d), the tumor region obtained by the analysis process is displayed in white, the non-tumor region is displayed in gray, and the non-tissue region is displayed in black. 1 shows the difference in discrimination accuracy depending on the magnification of the tissue specimen and the number of pixels in the window size. (a) shows the sensitivity at each magnification, (b) shows the positive predictive value at each magnification, (c) shows the change in sensitivity due to differences in the number of pixels in the window size, and (d) shows the change in the positive predictive value.

The following provides a detailed description of the overview and embodiments of the present invention, with reference to the accompanying drawings. Note that in the following description and drawings, the same reference numerals will indicate the same or similar components, and therefore, descriptions of the same or similar components will be omitted.

The image analysis method analyzes an image of tissue. The image analysis method uses a deep learning algorithm with a neural network structure, preferably a convolutional neural network structure. The image analysis method includes generating data indicating whether an area included in the analysis data is a tumor area.

In the present invention, a tissue or cell image is an image obtained from a tissue sample specimen or a cell-containing sample specimen. The tissue sample specimen or cell-containing sample specimen is collected from an individual. The individual is not particularly limited, but is preferably a mammal, more preferably a human. The individual may be alive or dead when the sample is collected from the individual. There are no limitations on the tissue, as long as it is present within the individual. Examples of tissue collected from the individual include surgically resected tissue and biopsy tissue. The tumor may be either epithelial or non-epithelial. The tumor is preferably a malignant epithelial tumor. The malignant tumor is not particularly limited, but examples of the malignant tumor include respiratory system malignant tumors arising from the trachea, bronchi, lungs, etc.; digestive system malignant tumors arising from the nasopharynx, esophagus, stomach, duodenum, jejunum, ileum, cecum, appendix, ascending colon, transverse colon, sigmoid colon, rectum, anus, etc.; liver cancer; pancreatic cancer; urinary system malignant tumors arising from the bladder, ureter, or kidney; female reproductive system malignant tumors arising from the ovaries, fallopian tubes, uterus, etc.; breast cancer; prostate cancer; skin cancer; endocrine system malignant tumors such as the hypothalamus, pituitary gland, thyroid gland, parathyroid gland, and adrenal gland; central nervous system malignant tumors; and solid tumors such as malignant tumors arising from bone and soft tissue. More preferred are respiratory epithelial malignant tumors such as lung cancer (squamous cell carcinoma, small cell carcinoma, large cell carcinoma, adenocarcinoma); digestive epithelial malignant tumors such as gastric cancer, duodenal cancer, and colon cancer (sigmoid colon cancer, rectal cancer, etc.); liver cancer; pancreatic cancer; bladder cancer; thyroid cancer; ovarian cancer; breast cancer; and prostate cancer. Gastric cancer is the most preferred.

The specimen is intended to be a state in which the tissue has been processed so that it can be observed under a microscope, for example, such as a slide. The specimen can be prepared according to known methods. For example, in the case of a tissue specimen, after collecting the tissue from the individual, the tissue is fixed in a specified fixative (formalin fixation, etc.), the fixed tissue is embedded in paraffin, and the paraffin-embedded tissue is thinly sliced. The thin slices are placed on a glass slide. The slide with the slices on it is stained for observation under an optical microscope, i.e., for brightfield observation, and the specimen is completed after a specified mounting process. A typical example of a tissue specimen is a tissue diagnostic specimen (pathological specimen), and the stain is hematoxylin-eosin (HE) staining.

Hematoxylin is the nuclear stain used in HE staining. Hematoxylin is widely used as a nuclear stain in tissue and cell staining (e.g., immunostaining, lectin staining, sugar staining, fat staining, collagen fiber staining, etc.). Therefore, the present invention can be applied to all specimens in which hematoxylin is used as a nuclear stain.

The image analysis uses a deep learning algorithm trained using training images. The image analysis generates analysis data from an analysis target image obtained from the specimen and containing the tissue to be analyzed. The analysis data is input into the deep learning algorithm, which generates data indicating whether a region of tissue present in the analysis target image included in the analysis data is a tumorous region. The data indicating whether a region included in the analysis data is a tumorous region is data indicating the possibility that a region including multiple pixels of interest contained in the tissue to be distinguished contains tumor tissue and/or is a non-tumorous region. The data indicating whether a region included in the analysis data is a tumorous region may be a label value, display, etc. that can distinguish between including a tumorous region and a non-tumorous region.

The training images are obtained from multiple regions of the specimen to be trained. Preferably, the training images are obtained from a region containing a tumor region, a region containing a non-tumor region, and a region containing no tissue for each of the multiple layer structures contained in the tissue to be trained.

For example, the first training image 70R1 is an image acquired from a region containing a tumor region in tissue collected from an individual. This image is acquired from a specimen stained so that the tissue structure can be recognized by microscopic observation. The staining is not limited as long as the tissue structure can be recognized, but is preferably a staining for bright-field observation. The staining for bright-field observation is not limited as long as it can stain at least cell nuclei and areas other than cell nuclei in a distinguishable hue. If the specimen is a mammalian tissue specimen, for example, HE staining can be used.

First training data 74R1 and first preliminary training data 70R1L are generated from the first training image 70R1. The first training data 74R1 is information regarding single-color images in which the hues contained in the first training image 70R1 are separated into primary colors. The first preliminary training data 70R1L is generated as binarized data (label values) indicating that the tissue region contained in the first training image 70R1 includes a tumor region. Whether the tissue region contained in the first training image 70R1 includes a tumor region is determined, for example, by specimen observation by a pathologist or the like. Here, the tissue region contained in the training image may contain a mixture of tumor cells, non-tumor cells, and tissue regions other than cells. In this case, when the area of the entire tissue region included in the training image is taken as 100%, the tissue region can be determined to contain a tumor region if tumor cells occupy at least about 10%, at least about 20%, at least about 30%, at least about 40%, at least about 50%, at least about 60%, at least about 70%, at least about 80%, or at least about 90% of the total area. Preferably, the tissue region can be determined to contain a tumor region if it occupies at least about 40%, at least about 50%, at least about 60%, at least about 70%, at least about 80%, or at least about 90%. The percentage of the total tissue area occupied by tumor cells may be determined by subtracting the percentage of non-tumor cells and tissue other than cells (non-tumor region) from the total area (100%) of the tissue region included in the training image.

The second training image 70R2 included in the training images is an image acquired from a tissue region including a non-tumor region (preferably a region including non-tumor cells and tissue regions other than cells). This image is acquired from a specimen stained so that the tissue structure can be recognized by microscopic observation. The staining is not limited as long as the tissue structure can be recognized, but is preferably a staining for brightfield observation. The staining for brightfield observation is not limited as long as it can stain at least cell nuclei and regions other than cell nuclei in a distinguishable hue. When the specimen is a mammalian tissue specimen, for example, HE staining can be used. The staining is preferably the same as that used for the specimen from which the first training image 70R1 was acquired.

Second training data 74R2 and second preliminary training data 70R2L are generated from the second training image 70R2. The second training data 74R2 is information regarding single-color images in which the hues contained in the second training image 70R2 are separated into primary colors. The second preliminary training data 70R2L is binarized data (label values) indicating that the tissue region contained in the second training image 70R2 includes a non-tumor region. Whether the tissue region contained in the second training image 70R2 includes a non-tumor region is determined, for example, by specimen observation by a pathologist, etc. Here, the tissue region contained in the training image may contain a mixture of tumor cells, non-tumor cells, and tissue regions other than cells. In this case, the tissue area included in the training image can be determined to contain a non-tumorous region if, for example, the area of the entire tissue area included in the training image is taken as 100%, and the area of non-tumor cells and tissue other than cells (non-tumor region) occupies approximately 10% or more, approximately 20% or more, approximately 30% or more, approximately 40% or more, approximately 50% or more, approximately 60% or more, approximately 70% or more, approximately 80% or more, approximately 90% or more, or 100% of the area of the entire tissue. Preferably, the tissue area can be determined to contain a non-tumorous region if the area of non-tumor cells and tissue other than cells occupies approximately 40% or more, approximately 50% or more, approximately 60% or more, approximately 70% or more, approximately 80% or more, approximately 90% or more, or 100%. The percentage of the area of the entire tissue area included in the second training image 70R2 (100%) minus the percentage of the tumorous region may be determined as the percentage of the area of the entire tissue area.

The third training image 70R3 included in the training images is an image obtained from a region that does not contain tissue (also called a non-tissue region) in the specimen from which the first training image 70R1 and/or the second training image 70R2 were obtained.

From the third training image, third training data 74R3 and third preliminary training data 70R3L are generated. The third training data 74R3 is information relating to a single-color image in which the hues contained in the third training image 70R3 are separated into primary colors. The third preliminary training data 70R2L is binarized data (label values) indicating that the areas contained in the third training image 70R3 are areas that do not contain tissue. The areas contained in the third training image 70R3 are determined, for example, by specimen observation by a pathologist or the like. Here, the training image may contain a mixture of tissue areas and background areas (e.g., the glass portion of a slide slide). In this case, when the area of the background region included in the training image is, for example, 100% when the area of the entire area included in one training image is taken as 100%, if the background region occupies approximately 90% or more, approximately 91% or more, approximately 92% or more, approximately 93% or more, approximately 94% or more, approximately 95% or more, approximately 96% or more, approximately 97% or more, approximately 98% or more, or approximately 99% or more, or if the background region occupies 100%, the region can be determined to be tissue-free. Preferably, if the background region occupies approximately 95% or more, approximately 96% or more, approximately 97% or more, approximately 98% or more, or approximately 99% or more, or if the background region occupies 100%, the region can be determined to be tissue-free.

Each training image is preferably acquired as an image of the training tissue specimen enlarged, for example, by 5 to 20 times. From the standpoint of discrimination accuracy, images enlarged by 30 times or more, and especially 40 times or more, are unsuitable as training images.

Tumor region training data is generated from the first training data 74R1 and the first preliminary training data 70R1L, non-tumor region training data is generated from the second training data 74R2 and the second preliminary training data 70R2L, and non-tissue region training data is generated from the third training data 74R3 and the third preliminary training data 70R2L. The tumor region training data, non-tumor region training data, and non-tissue region training data are sometimes collectively referred to as training data.

In the overview and embodiments of the present invention, we will explain, as an example, a case in which a deep learning algorithm is used to determine whether the tissue contained in an image of an HE-stained tissue specimen includes tumorous regions, non-tumorous regions, and non-tissue regions.

[Overview of deep learning and image analysis methods]
First, we explain the deep learning method.

Overview of the Deep Learning Method: An overview of the first training data 74R1, the first preliminary training data 70R1L, and the tumor region training data 75R1 will be described using Figure 1. Figure 1 shows an example of training data input to a neural network using a whole slide image 70W1 of a tissue specimen acquired by a slide image scanner. The whole slide image 70W1 is a slide image of a tissue specimen prepared by HE staining for bright-field observation and captured in bright-field. The whole slide image 70W2 is an image designated by a pathologist by enclosing a portion of the whole slide image 70W1 containing the tumor region with a solid line. The whole slide image 70W3 is an image obtained by dividing the whole slide image 70W2 (e.g., 512 divisions) as a training image. The area enclosed by a square frame marked with the symbol R1 in the whole slide image 70W3 indicates the area used as the first training image 70R1 shown in Figure 1. The rectangular area indicated by the symbol R2 indicates the area used as the second training image 70R2 shown in Figure 2. The rectangular area indicated by the symbol R3 indicates the area used as the third training image 70R3 shown in Figure 3. The fact that the first training image 70R1 contains a tumor region, the second training image 70R2 contains a non-tumor region, and the third training image 70R3 is a region that does not contain tissue may be determined before or after each training image is acquired.

In Figure 1, the first training image 70R1 is an HE-stained specimen captured under bright-field observation, for example as a color image, and therefore includes multiple hues.

The first training image 70R1 can be acquired in advance using an image acquisition device such as a known optical microscope, fluorescent microscope, or virtual slide scanner. For example, in this embodiment, the color image acquired from the image acquisition device preferably has a color space of 24-bit RGB color. In 24-bit RGB color, the intensity (color density) of each of the red, green, and blue colors is preferably represented by 8-bit (256 levels) gradation. The first training image 70R1 may be any image that includes one or more primary colors. First training data 74R1 is generated corresponding to the first training image 70R1.

In the present invention, hue is illustratively defined as a combination of the three primary colors of light or the three primary colors of color. The first training data 74R1 is generated from the first training image 70R1 by separating the hues appearing in the first training image 70R1 into individual primary colors, and data is generated for each primary color, expressed as a code corresponding to its density. In Figure 1, images 72R1R, 72R1G, and 72R1B are obtained, which are shown in shades of a single color separated into the three primary colors of light: red (R), green (G), and blue (B) (hereinafter also referred to as "single-color images").

The color density of each pixel in the single-color images 72R1R, 72R1G, and 72R1B is encoded, and coded maps (hereinafter referred to as "color density coded maps") 72R1r, 72R1g, and 72R1b corresponding to the color density of each pixel are generated for each R, G, and B image. Color density may be coded using a numerical value representing 256 levels of each color. Furthermore, the numerical values representing 256 levels of color density may be further preprocessed to code the color density of each pixel using a number representing eight levels, for example, from 0 to 7. For ease of explanation, the color density coded maps 72R1r, 72R1g, and 72R1b generated from the R, G, and B single-color images shown in Figure 1 represent the color density of each pixel using eight levels of codes, from 0 to 7. First training data 74R1 is generated from the color density encoding charts 72R1r, 72R1g, and 72R1b shown in FIG. 1, combining R, G, and B color density values for each pixel. Codes indicating color density are also referred to as color density values in this specification. Alternatively, a matrix of color density values corresponding to each pixel may be generated as the first training data 74R1 instead of the color density encoding chart.

In FIG. 1, the first preliminary training data 70R1L is data indicating that the tissue contained in the first training image 70R1 includes a tumor region. For example, since the entire area of the first training image 70R1 has been determined by a pathologist to be a tumor region, the first training image 70R1 is assigned the same numerical value, for example, "1," as a label value indicating that each area of a predetermined number of pixels described below contains a tumor region, and this becomes the first preliminary training data 70R1L.

In the deep learning method, tumor region training data 75R1 is generated from the first training data 74R1 and first preliminary training data 70R1L shown in Figure 1. Then, a neural network 50 is trained using color density value data 76 (obtained from the first training data 74R1) of each pixel in the tumor region training data 75R1 as the input layer 50a and label values 77R1 (obtained from the first preliminary training data 70R1L) as the output layer 50b.

Next, an overview of the second training data 74R2, the second preliminary training data 70R2L, and the non-tumor region training data 75R2 will be provided using Figure 2. In Figure 2, hole slide image 70W1, hole slide image 70W2, and hole slide image 70W3 are the same as those in Figure 1, so the second training image 70R2 contains multiple hues, just like the first training image 70R1.

The second training images 70R2 are acquired in the same manner as the first training images 70R1, except that tissue containing non-tumorous regions is used instead of tissue containing tumorous regions. Second training data 74R2 is generated corresponding to the second training images 70R2.

The second training data 74R2 is generated in the same manner as the first training data 74R1, except that tissue containing non-tumorous regions is used instead of tissue containing tumorous regions. The second training data 74R2 is generated by combining R, G, and B color density values for each pixel from the color density encoding diagrams 72R2r, 72R2g, and 72R2b shown in Figure 2.

In Figure 2, second preliminary training data 70R2L is data indicating that second training image 70R2 contains a non-tumorous region. Since all regions of second training image 70R2 have been determined by a pathologist to be non-tumorous, the second training image 70R2 is assigned the same numerical value, for example, "2," as a label value indicating that it contains a non-tumorous region for each region of a predetermined number of pixels (described below), thereby becoming second preliminary training data 70R2L. The label value indicating that a region contains a non-tumorous region is not limited as long as it can be distinguished from the label values indicating regions containing tumorous regions and regions not containing tissue.

In the deep learning method, non-tumor region training data 75R2 is generated from the second training data 74R2 and second preliminary training data 70R2L shown in Figure 2. Then, a neural network 50 is trained using color density value data 76 (obtained from the second training data 74R2) of each pixel in the non-tumor region training data 75R2 as the input layer 50a and label values 77R2 (obtained from the second preliminary training data 70R2L) as the output layer 50b.

Next, an overview of the third training data 74R3, the third preliminary training data 70R3L, and the non-tissue region training data 75R3 will be provided using Figure 3. In Figure 3, hole slide image 70W1, hole slide image 70W2, and hole slide image 70W3 are the same as those in Figure 1, so the third training image 70R3 contains multiple hues, just like the first training image 70R1.

The third training image 70R3 is acquired in the same manner as the first training image 70R1, except that a tissue-free region is used instead of the region containing the tumor region. Third training data 74R3 is generated corresponding to the third training image 70R3.

The third training data 74R3 is generated in the same manner as the first training data 74R1, except that a tissue-free region is used instead of a region containing a tumor region. The third training data 74R3 is generated by combining R, G, and B color density values for each pixel from the color density encoding diagrams 72R3r, 72R3g, and 72R3b shown in Figure 3.

In Figure 3, the third preliminary training data 70R3L is data indicating that the areas included in the third training image 70R3 are areas that do not contain tissue. Since all areas of the third training image 70R3 have been determined by a pathologist to be non-tissue areas, the third training image 70R3 is assigned the same numerical value, for example "0," as a label value indicating that each area contains no tissue for each region of a predetermined number of pixels (described below), thereby forming the third preliminary training data 70R3L. The label value indicating that each area does not contain tissue is distinguished from the numerical value indicating an area containing a tumorous region and an area containing a non-tumorous region.

In the deep learning method, non-tissue region training data 75R3 is generated from the third training data 74R3 and third preliminary training data 70R3L shown in Figure 3. Then, the non-tissue region training data 75R3 is trained on a neural network 50, which uses color density value data 76 of each pixel of the non-tissue region training data 75R3 (obtained from the third training data 74R3) as the input layer 50a and label values 77R3 (obtained from the third preliminary training data 70R3L) as the output layer 50b.

A method for generating tumor region training data 75R1 will be described with reference to Figures 4(a) and (b). The first training data 74R1 shown in Figure 4(a) is data combining the color density values of the color density encoding maps 72R1r, 72R1g, and 72R1b of tumor regions with a predetermined number of pixels. For convenience, the position of each pixel in the first training data 74R1 is represented by columns _l1 , _l2 ... _l1 from left to right, and rows _m1 , _m2 ... _mj from top to bottom. In Figure 4(a), the image size (size per training data item) of the first training data 74R1 is simplified for ease of explanation, and the first training data 74R1 has a total of 81 pixels, nine vertically and nine horizontally. That is, the position of each pixel is represented by columns _l1 , _l2 ... _l9 from left to right, and rows _m1 , _m2 ... _m9 from top to bottom. The three values shown in FIG. 4A are the color density values of the R, G, and B colors at each pixel. For example, the three values are stored in the order of red (R), green (G), and blue (B) from left to right. For convenience, the color density values of each pixel in the first training data 74R1 are shown in eight levels, from 0 to 7. This is an example of image preprocessing, in which the brightness of each color image 72R1R, 72R1G, and 72R1B, which is represented in 256 levels when captured, is converted into eight levels of color density values. For example, the color density value is assigned such that the lowest brightness (a group of gradations with low brightness values when expressed in the 256-level RGB color system) is assigned a color density value of 0, and gradually higher values are assigned as the brightness increases, with the highest brightness (a group of gradations with high brightness values when expressed in the 256-level RGB color system) being assigned a color density value of 7.

1 and 4(b), the tumor region training data 75R1 is obtained by extracting a region consisting of a predetermined number of pixels (hereinafter referred to as the "window size") from the first training data 74R1 shown in Fig. 4(a ₎ and assigning a label value 77R1 corresponding to the label value of the first preliminary training data 70R1L to the extracted data. For ease of explanation, the window-sized tumor region training data 75R1 is also shown simplified to 3 x 3 pixels (l1: _l3 , _m1 : _m3 ), but for convenience, the position of each pixel is represented by _l1 , _l2 ... _l1 from left to right for columns and _m1 , _m2 ... _mj for rows from top to bottom. In terms of learning efficiency, preferred window sizes are, for example, approximately 125 x 125 pixels (i = 125, j = 125), 150 x 150 pixels (i = 150, j = 150), 166 x 166 pixels (i = 166, j = 166), 180 x 180 pixels (i = 180, j = 180), 200 x 200 pixels (i = 200, j = 200), 220 x 220 pixels (i = 220, j = 220), and 250 x 250 pixels (i = 250, j = 250). More preferably, the window size is approximately 200 x 200 pixels. The window size is, for example, in the range of 60 μm x 60 μm to 500 μm x 500 μm. Preferably, the window size is 60 μm × 60 μm, 100 μm × 100 μm, 150 μm × 150 μm, 200 μm × 200 μm, 250 μm × 250 μm, 300 μm × 300 μm, 350 μm × 350 μm, 400 μm × 400 μm, 450 μm × 450 μm, or 500 μm × 500 μm. More preferably, the window size is 200 μm × 200 μm to 400 μm × 400 μm. For example, as shown in FIG. 4B, a 3 × 3 pixel window W1 (l ₁ :l ₃ , m ₁ :m ₃ ) is set from the left end of the first training data 74R1, and the window W1 is moved from left to right relative to the first training data 74R1. After extracting tumor region training data 75R1 of ( _l1 : _l3 , _m1 : _m3 ) from window W1, label values 77R1 corresponding to the label values of first preliminary training data 70R1L are assigned to the tumor region training data 75R1. Window W1 then moves to the next window ( _l4 : _l6 , _m4 : _m6 ) of tumor region training data 74R1, indicated by the dotted line, and extracts ( _l4 : _l6 , _m4 : _m6 ) as window-sized tumor region training data 75R1. Label values 77R1 corresponding to the label values of first preliminary training data 70R1L are assigned to the newly extracted tumor region training data 75R1. This operation is repeated to extract tumor region training data 75R1 of multiple window sizes from the first training data 74R1. The tumor region training data 75R1 of the extracted window size is assigned a label value 77R1 corresponding to the label value of the first preliminary training data 70R1L and is used for training the neural network 50 shown in FIG.

As shown in Figure 1, the number of nodes in the input layer 50a of the neural network 50 corresponds to the product of the number of pixels in the input window-sized tumor region training data 75R1 and the number of primary colors contained in the image (for example, R, G, and B for the three primary colors of light). The color density value data 76 of each pixel of the window-sized tumor region training data 75R1 is used as the input layer 50a of the neural network, and the label value 77R1 corresponding to the first preliminary training data 70R1L is used as the output layer 50b of the neural network, and the neural network 50 is trained. The color density value data 76 of each pixel is a collection of color density values of the R, G, and B colors of each pixel of the tumor region training data 75R1. For example, if the window size of the tumor region training data 75R1 is 3 x 3 pixels, each pixel is assigned one color density value for each of R, G, and B, so the number of color density values in the color density value data 76 is 27 (3 x 3 x 3 = 27), and the number of nodes in the input layer 50a of the neural network 50 is also 27.

In this way, the window-sized tumor region training data 75R1 to be input to the neural network 50 can be automatically created by the computer without the user having to create it. This promotes efficient deep learning of the neural network 50.

As shown in Figure 4(b), in the initial state, window W1 is located in the upper left corner of first training data 74R1. Thereafter, window-sized tumor region training data 75R1 is extracted using window W1, and the position of window W1 is moved each time neural network 50 is trained. Specifically, window W1 is moved so that it scans, for example, all pixels of first training data 74R1. As a result, window-sized tumor region training data 75R1 extracted from all pixels of first training data 74R1 is used for training neural network 50. This improves the level of learning of neural network 50, and as a result of deep learning, a deep learning algorithm having the neural network 60 structure shown in Figure 6 is obtained.

Figure 5(a) shows second training data 74R2. The method for generating non-tumor region training data 75R2 from second training data 74R2 is similar to that for tumor region training data 75R1, except that second training data 74R2 and second preliminary training data 70R2L are used instead of first training data 74R1 and first preliminary training data 70R1L. Label values 77R2 corresponding to the label values of second preliminary training data 70R2L assigned to non-tumor region training data 75R2 are input to output layer 50b shown in Figure 2.

Figure 5(b) shows third training data 74R3. The method for generating non-tissue region training data 75R3 from third training data 74R3 is similar to that for tumor region training data 75R1, except that third training data 74R3 and third preliminary training data 70R3L are used instead of first training data 74R1 and first preliminary training data 70R1L. Label value 77R3, which corresponds to the label value of third preliminary training data 70R3L assigned to non-tissue region training data 75R3, is input to output layer 50b shown in Figure 3.

It is preferable that training data for each region be generated using 10 or more, 20 or more, 50 or more, or 100 or more training images for each region.

Overview of Image Analysis Method: As shown in FIG. 6 , in the image analysis method, analysis data 80 is generated from an analysis target image (bright-field image) 78, which is an image of a specimen containing the target tissue. The specimen is preferably stained in the same manner as the first training image. The analysis target image 78 can also be acquired, for example, as a color image using, for example, a known microscope or virtual slide scanner. The analysis target image 78 is preferably acquired at the same magnification as the training image. The analysis target image (bright-field image) 78 may be any image containing one or more primary colors. By encoding the color density values of R, G, and B for each pixel of the color analysis target image 78, the entire image can be represented as an encoded diagram of the color density values for each pixel of R, G, and B (analysis color density encoded diagrams 79r, 79g, and 79b). Analysis target data (not shown) is generated by combining the R, G, and B color density values for each pixel from the analysis color density encoded diagrams 79r, 79g, and 79b. 6, color density coding charts 79r, 79g, and 79b showing the codes of color density in the single-color images of R, G, and B, respectively, display color density values expressed in eight levels of codes from value 0 to value 7, instead of the images 79R, 79G, and 79B of the three primary colors. For convenience, in this specification, the position of each pixel in the data to be analyzed will be represented by _l1 , _l2 , ... _li from left to right for columns and _m1 , _m2 , ... _mj for rows from top to bottom.

The analysis data 80 is data obtained by cutting the analysis target data into a region of a predetermined number of pixels (i.e., the same window size as the above-mentioned region training data), and includes the color density values of the tissues contained in the analysis target image 78. In Figure 6, the window-sized analysis data 80 is simplified to 3 x 3 pixels for ease of explanation, as with the tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. However, the actual preferred window size is the same as that of the tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. For example, a 3 x 3 pixel window W2 is set and moved relative to the analysis target data. Window W2 includes a predetermined number of pixels in the analysis target data. By cutting the analysis target data using the window W2, for example, indicated by a 3 x 3 pixel black frame, in the same manner as with the region training data, the window-sized analysis data 80 is obtained. In a similar manner to the region training data, multiple analysis data 80 are generated from the analysis target data for each region of a predetermined number of pixels corresponding to the window size. In the analysis target data and window size analysis data 80, color density values for each pixel are stored in the order of red (R), green (G), and blue (B), just like in tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3.

In the image analysis method, analysis data 80 is processed using a deep learning algorithm 60 having a neural network trained using tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3, each having the window sizes shown in Figures 1 to 3. By processing the analysis data 80, data 83 is generated that indicates whether an area included in the analysis data in the tissue to be analyzed is a tumor region.

Referring back to Figure 6, analysis data 80 extracted from the analysis target data generated based on the R, G, and B color density coding maps 79r, 79g, and 79b are input to a neural network 60, which constitutes a deep learning algorithm. The number of nodes in the input layer 60a of the neural network 60 corresponds to the product of the number of input pixels and the number of primary colors contained in the image. When color density value data 81 for each pixel of the analysis data 80 is input to the neural network 60, an estimated value 82 (ternary value) of the analysis data 80 is output from the output layer 60b. For example, an estimated value of 1 indicates that a region of a predetermined number of pixels (pixel region) constituting the analysis data 80 contains a tumor region; an estimated value of 2 indicates that the pixel region contains a non-tumor region; and an estimated value of 0 indicates that the pixel region is a non-tissue region. In other words, the estimated value 82 output from the output layer 60b of the neural network 60 is a label value generated for each pixel region of the analysis target image and indicates whether the region of the analysis target image contained in the analysis data is a tumor region. The estimated values 82 are also referred to as classes in the description of neural networks below. For the input analysis data 80, the neural network 60 generates a label value indicating the inclusion of a tumorous region, a label value indicating the inclusion of a non-tumorous region, or a label value indicating a non-tissue region based on the color density value data 81 of a predetermined number of pixels contained in the analysis data 80. In other words, the neural network 60 classifies the analysis data 80 into classes indicating the state of the pixel region of interest contained in the image to be analyzed (tumorous, non-tumorous, or non-tissue). Here, the color density value data 81 of each pixel is a collection of color density values of the R, G, and B colors of each pixel in the analysis data 80.

Subsequently, the analysis data 80 is extracted by the window size while moving the window W2 in units of a predetermined number of regions so that the window W2 scans all pixels of the analysis target data. The extracted analysis data 80 is input to the neural network 60. This obtains a label value 83 based on data indicating whether a region included in the analysis data of tissue in the analysis target image is a tumor region. In the example shown in Figure 6, a region-enhanced image 84 indicating the region is obtained by further performing region detection processing on the label value 83. Specifically, the region detection processing is a process of detecting each region according to the estimated value 82, for example, and actually classifying pixel regions with an estimated value 82 of "1" as tumor regions, pixel regions with an estimated value 82 of "2" as non-tumor regions, and pixel regions with an estimated value 82 of "0" as non-tissue regions. The region-enhanced image 84 is a diagram of the regions detected by the image analysis processing, with the label value 83 represented by color (tumor regions in white, non-tumor regions in gray, and non-tissue regions in black). Furthermore, after each region is identified, a process may be performed to display the tumor region and other regions (i.e., non-tumor regions and/or non-tissue regions) in a distinguishable manner on the display device. For example, the tumor region may be filled in with color, or a line may be drawn between the tumor region and other regions, and these may be displayed in a distinguishable manner on the display device. A process may also be performed to display the non-tumor region and/or non-tissue region in a distinguishable manner on the display device.

First Embodiment
In the first embodiment, a specific description will be given of the configuration of a system that implements the deep learning method and image analysis method outlined above.

[Configuration Overview]
Referring to FIG. 7 , the image analysis system according to the first embodiment includes a deep learning device 100A and an image analysis device 200A. The vendor-side device 100 operates as the deep learning device 100A, and the user-side device 200 operates as the image analysis device 200A. The deep learning device 100A trains a neural network 50 using training data and provides the deep learning algorithm 60 trained by the training data to the user. The deep learning algorithm configured from the trained neural network 60 is provided from the deep learning device 100A to the image analysis device 200A via a storage medium 98 or a network 99. The image analysis device 200A analyzes the image to be analyzed using the deep learning algorithm configured from the trained neural network 60.

Deep learning device 100A is configured, for example, by a general-purpose computer, and performs deep learning processing based on the flowchart described below. Image analysis device 200A is configured, for example, by a general-purpose computer, and performs image analysis processing based on the flowchart described below. Storage medium 98 is a computer-readable, non-transitory, tangible storage medium, such as a DVD-ROM or USB memory.

The deep learning device 100A is connected to the imaging device 300. The imaging device 300 includes an image sensor 301 and a fluorescence microscope 302, and captures bright-field images and fluorescent images of a training specimen 308 set on a stage 309. The training specimen 308 has been stained as described above. The deep learning device 100A acquires the training images captured by the imaging device 300.

The image analysis device 200A is connected to the imaging device 400. The imaging device 400 includes an imaging element 401 and a fluorescence microscope 402, and captures a bright-field image of a specimen 408 to be analyzed that is set on a stage 409. The specimen 408 to be analyzed has been stained in advance as described above. The image analysis device 200A acquires the image 78 to be analyzed captured by the imaging device 400.

The imaging devices 300 and 400 can be well-known fluorescent microscopes or virtual slide scanners that have the function of capturing images of specimens. The imaging device 400 can also be an optical microscope, as long as it has the function of capturing images of specimens.

[Hardware configuration]
Referring to FIG. 8, the vendor-side device 100 (100A, 100B) includes a processing unit 10 (10A, 10B), an input unit 16, and an output unit 17.

The processing unit 10 includes a CPU (Central Processing Unit) 11 that performs data processing (described below), a memory 12 used as a work area for data processing, a storage unit 13 that stores programs and processing data (described below), a bus 14 that transmits data between each unit, an interface unit 15 that inputs and outputs data to and from external devices, and a GPU (Graphics Processing Unit) 19. An input unit 16 and an output unit 17 are connected to the processing unit 10. Illustratively, the input unit 16 is an input device such as a keyboard or mouse, and the output unit 17 is a display device such as an LCD display. The GPU 19 functions as an accelerator that assists the arithmetic processing (e.g., parallel arithmetic processing) performed by the CPU 11. In other words, in the following description, processing performed by the CPU 11 also includes processing performed by the CPU 11 using the GPU 19 as an accelerator.

In order to perform the processing of each step described below in FIG. 10, the processing unit 10 pre-stores the program according to the present invention and the pre-trained neural network 50 in, for example, an executable format in the storage unit 13. The executable format is, for example, a format generated by conversion from a programming language using a compiler. The processing unit 10 performs processing using the program and pre-trained neural network 50 stored in the storage unit 13.

In the following description, unless otherwise specified, the processing performed by the processing unit 10 refers to the processing performed by the CPU 11 based on the programs and neural network 50 stored in the storage unit 13 or memory 12. The CPU 11 uses the memory 12 as a working area to temporarily store necessary data (such as intermediate data during processing), and stores calculation results and other data to be stored long-term in the storage unit 13 as appropriate.

Referring to FIG. 9, the user device 200 (200A, 200B, 200C) includes a processing unit 20 (20A, 20B, 20C), an input unit 26, and an output unit 27.

The processing unit 20 includes a CPU (Central Processing Unit) 21 that performs data processing (described below), a memory 22 used as a work area for data processing, a storage unit 23 that stores programs and processing data (described below), a bus 24 that transmits data between each unit, an interface unit 25 that inputs and outputs data to and from external devices, and a GPU (Graphics Processing Unit) 29. An input unit 26 and an output unit 27 are connected to the processing unit 20. Illustratively, the input unit 26 is an input device such as a keyboard or mouse, and the output unit 27 is a display device such as an LCD display. The GPU 29 functions as an accelerator that assists the arithmetic processing (e.g., parallel arithmetic processing) performed by the CPU 21. In other words, in the following description, processing performed by the CPU 21 also includes processing performed by the CPU 21 using the GPU 29 as an accelerator.

In addition, in order to perform the processing of each step described below in Figure 14, the processing unit 20 pre-stores the program of the present invention and a deep learning algorithm 60 with a trained neural network structure in the memory unit 23, for example, in executable format. The executable format is, for example, a format generated by conversion from a programming language using a compiler. The processing unit 20 performs processing using the program and deep learning algorithm 60 stored in the memory unit 23.

In the following description, unless otherwise specified, the processing performed by the processing unit 20 actually means processing performed by the CPU 21 of the processing unit 20 based on the program and deep learning algorithm 60 stored in the storage unit 23 or memory 22. The CPU 21 uses the memory 22 as a working area to temporarily store necessary data (such as intermediate data during processing), and stores calculation results and other data to be stored long-term in the storage unit 23 as appropriate.

[Function blocks and processing procedures]
10 , the processing unit 10A of the deep learning device 100A according to the first embodiment includes a training data generation unit 101, a training data input unit 102, and an algorithm update unit 103. These functional blocks are implemented by installing a program that causes a computer to execute deep learning processing in the storage unit 13 or memory 12 of the processing unit 10A and executing this program by the CPU 11. A window size database 104 and an algorithm database 105 are stored in the storage unit 13 or memory 12 of the processing unit 10A.

The first training image 70R1, second training image 70R2, and third training image 70R3 of the learning sample are assumed to have been captured in advance by the imaging device 300 and stored in advance in the storage unit 13 or memory 12 of the processing unit 10A. The neural network 50 is pre-stored in the algorithm database 105, for example, in association with the type of tissue (e.g., tissue name) from which the sample to be analyzed is derived.

The processing unit 10A of the deep learning device 100A performs the processing shown in Figure 11. Explaining this using the functional blocks shown in Figure 10, the processing of steps S10 to S12, S17, and S18 is performed by the training data generation unit 101. The processing of step S14 is performed by the training data input unit 102. The processing of steps S15 and S16 is performed by the algorithm update unit 103.

Steps S10 to S18 described below describe deep learning processing using the first training image 70R1, the second training image 70R2, and the third training image 70R3.

Processing unit 10A performs deep learning processing to generate tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3 in accordance with the method described in the Overview of Deep Learning Methods. Processing unit 10A displays an image of a large region (whole slide image 70W1) including the region of first training image 70R1 on output unit 17, for example, in response to operation by a pathologist or other user via input unit 16. The pathologist or other user making the assessment visually confirms the image of whole slide image 70W1 displayed on output unit 17. The pathologist or other user specifies, via input unit 16, an area in whole slide image 70W1 determined to contain a tumor region, and encircles the area in whole slide image 70W1 with a solid line, for example, red. Similarly, the pathologist or other user encircles the non-tumor region and non-tissue region in whole slide image 70W1 with a solid line, for example, blue or green, different from red. Instead of displaying the whole slide image 70W1 on the output unit 17 and having the pathologist make a judgment, the processing unit 10A may acquire the judged whole slide image 70W2 via the I/F unit 15, for example, via the network 99. The pathologist or the like may divide the whole slide image 70W2 into pixels that are suitable for use as a training image.

The processing unit 10A, based on instructions from the pathologist or other person using the input unit 16, cuts out a first training image 70R1 obtained from the area surrounded by a solid red line in the whole slide image 70W2, so that it contains a predetermined number of pixels. Similarly, it cuts out a second training image 70R2 and a third training image 70R3 from the areas determined to be non-tumorous and non-tissue regions, so that they each contain a predetermined number of pixel areas.

In step S10, the processing unit 10A generates color density coding maps 72R1r, 72R1g, and 72R1b for each of the R, G, and B colors from the extracted first training image 70R1. The color density coding maps 72R1r, 72R1g, and 72R1b are generated by assigning to each pixel of the first training image 70R1 a code that represents the color density value of each of the R, G, and B colors of that pixel in a stepwise manner. In this embodiment, the color density values are represented in 256 gradations from 0 to 255, and color density coding maps 72R1r, 72R1g, and 72R1b are generated for each of the R, G, and B gradation images. The color density values are assigned, for example, by assigning a color density value of 0 to the lowest brightness and gradually increasing values as the brightness level increases, with the highest brightness being a color density value of 255. The processing unit 10A generates first training data 74R1 by assigning R, G, and B color density values to each pixel from the color density encoding charts 72R1r, 72R1g, and 72R1b. The processing unit 10A generates first preliminary training data 70R1L by associating each pixel of the first training image 70R1 with a label value of "1" indicating a tumorous region.

In step S11, the processing unit 10A generates second training data 74R2 from the second training image 70R2 using a method similar to step S10. The processing unit 10A generates second preliminary training data 70R2L in which each pixel of the second training image 70R2 is assigned a label value of "2," indicating a non-tumorous region.

In step S12, the processing unit 10A generates third training data 74R3 from the third training image 70R3 using a method similar to step S10. The processing unit 10A generates third preliminary training data 70R3L in which each pixel of the third training image 70R3 is assigned a label value of "0," indicating a non-tissue region.

Steps S10 to S12 may be performed simultaneously or in any order.

In step S13, processing unit 10A receives input of the type of tissue to be used for learning from the user of deep learning device 100A via input unit 16. Based on the input type of tissue, processing unit 10A sets the window size by referencing window size database 104 (window size DB104) and sets the neural network 50 to be used for learning by referencing algorithm database 105 (algorithm DB105). The window size is, for example, 200 x 200 pixels. The window size is the unit of training data input to the neural network 50 in one input, and the product of the number of pixels of the training data corresponding to each region of the window size and the number of primary colors included in the image corresponds to the number of nodes in the input layer 50a. The window size is associated with the type of tissue and pre-stored in window size database 104.

In step S14, processing unit 10A generates window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3 from the training data corresponding to each region. Specifically, as described above in the "Outline of the Deep Learning Method" with reference to Figures 4(a) and (b), processing unit 10A generates window-sized tumor region training data 75R1 using window W1 from first training data 74R1 and first preliminary training data 70R1L. Processing unit 10A generates window-sized non-tumor region training data 75R2 using second training data 74R2 and second preliminary training data 70R2L. Specifically, as described above in the "Outline of the Deep Learning Method" with reference to Figures 4(a), (b), and 5(a), processing unit 10A generates window-sized non-tumor region training data 75R2 using window W1 from second training data 74R2 and second preliminary training data 70R2L. Processing unit 10A generates window-sized non-tissue region training data 75R3 from third training data 74R3 and third preliminary training data 70R3L. Specifically, as described above in the "Overview of Deep Learning Method" with reference to Figures 4(a), (b), and 5(b), processing unit 10A creates window-sized non-tissue region training data 75R3 using window W1 from third training data 74R3 and third preliminary training data 70R3L.

In step S15 shown in FIG. 11, processing unit 10A trains neural network 50 using window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3. The training results of neural network 50 are accumulated each time neural network 50 is trained using window-sized tumor region training data 75R1, non-tumor region training data 75R2, and non-tissue region training data 75R3.

The image analysis method according to the embodiment uses a convolutional neural network and employs the stochastic gradient descent method. Therefore, in step S16, the processing unit 10A determines whether learning results for a predetermined number of trials have been accumulated. If learning results for the predetermined number of trials have been accumulated, the processing unit 10A performs the processing of step S17. If learning results for the predetermined number of trials have not been accumulated, the processing unit 10A performs the processing of step S18.

If learning results have been accumulated for a predetermined number of trials, in step S17, processing unit 10A updates the connection weights w of neural network 50 using the learning results accumulated in step S15. Since the image analysis method according to the embodiment uses stochastic gradient descent, the connection weights w of neural network 50 are updated once learning results for a predetermined number of trials have been accumulated. Specifically, the process of updating the connection weights w is a process of performing calculations using the gradient descent method shown in (Equation 11) and (Equation 12) described below.

In step S17, the processing unit 10A determines whether a specified number of pixels in each area training data for each of the first training data 74R1, second training data 74R2, and third training data 74R3 have been processed. If the series of processes from step S15 to step S17 have been performed for the specified number of pixels in each area training data, the deep learning process is terminated. Neural network learning does not necessarily have to be performed on all pixels in each training data; the processing unit 10A can process and learn only a portion of the pixels in each training data, i.e., a specified number of pixels. The specified number of pixels may be all pixels in each area training data.

If the specified number of pixels in each region training data has not been processed, in step S19, the processing unit 10A moves the window position within the first training data 74R1, the second training data 74R2, or the third training data 74R3, as shown in FIG. 4(b). The processing unit 10A then performs a series of processes from step S14 to step S16 at the new window position after the movement. That is, in step S14, the processing unit 10A extracts region training data 75R1, 75R2, and 75R3 of the window size for each of the first training data 74R1, the second training data 74R2, and the third training data 74R3 at the new window position after the movement, and assigns a label value indicating each region. Subsequently, in step S15, the processing unit 10A trains the neural network 50 using the region training data 75R1, 75R2, and 75R3 of the newly extracted window size. In step S16, if learning results for a predetermined number of trials have been accumulated, processing unit 10A updates the connection weight w of neural network 50 in step S17. This type of learning of neural network 50 for each window size is performed on a specified number of pixels for each of the first training data 74R1, second training data 74R2, and third training data 74R3.

The deep learning process of steps S10 to S19 described above is repeated for the first training image 70R1, the second training image 70R2, and the third training image 70R3, which are acquired separately, thereby improving the level of learning of the neural network 50. This results in the deep learning algorithm 60 with the neural network structure shown in Figure 5.

Neural Network Structure As shown in Fig. 12(a), the first embodiment uses a deep learning type neural network. The deep learning type neural network includes an input layer 50a, an output layer 50b, and an intermediate layer 50c between the input layer 50a and the output layer 50b, as in the neural network 50 shown in Fig. 12, and the intermediate layer 50c is composed of multiple layers. The number of layers constituting the intermediate layer 50c can be, for example, five or more.

In the neural network 50, multiple nodes 89 are arranged in layers and connected between layers. This allows information to propagate in only one direction, as indicated by arrow D in the figure, from the input layer 50a to the output layer 50b. In this embodiment, the number of nodes in the input layer 50a corresponds to the product of the number of pixels in the input image, i.e., the number of pixels in the window W1 shown in Figure 3(c), and the number of primary colors contained in each pixel. Since image pixel data (color density values) can be input to the input layer 50a, the user can input an input image to the input layer 50a without having to separately calculate feature values from the input image.

12(b) is a schematic diagram showing the operations at each node. Each node 89 receives multiple inputs and calculates one output (z). In the example shown in FIG. 12(b), node 89 receives four inputs. The total input (u) received by node 89 is expressed by the following (Equation 1):

Each input is multiplied by a different weight. In (Equation 1), b is a value called the bias. The output (z) of the node is the output of a predetermined function f for the total input (u) expressed in (Equation 1), and is expressed in (Equation 2) below. The function f is called the activation function.

FIG. 12(c) is a schematic diagram showing the operations between nodes. In the neural network 50, nodes that output a result (z) expressed by (Equation 2) in response to a total input (u) expressed by (Equation 1) are arranged in layers. The output of a node in the previous layer becomes the input of a node in the next layer. In the example shown in FIG. 12(c), the output of node 89a in the left layer becomes the input of node 89b in the right layer. Each node 89b in the right layer receives the output from node 89a in the left layer. A different weight is applied to each connection between each node 89a in the left layer and each node 89b in the right layer. If the outputs of each of the multiple nodes 89a in the left layer are _x1 to _x4 , the inputs to each of the three nodes 89b in the right layer are expressed by the following (Equation 3-1) to (Equation 3-3).

Generalizing these (Equation 3-1) to (Equation 3-3), we get (Equation 3-4), where i = 1, ... I, j = 1, ... J.

Applying (Equation 3-4) to the activation function gives the output, which is expressed as (Equation 4) below.

Activation Function In the image analysis method according to the embodiment, a rectified linear unit function is used as the activation function. The rectified linear unit function is expressed by the following (Equation 5).

(Equation 5) is a linear function of z = u, where the part of u<0 is set to u=0. In the example shown in Figure 12(c), the output of node j=1 is expressed by (Equation 5) as follows:

Neural Network Training If a function expressed using a neural network is defined as y(x:w), the function y(x:w) changes when the parameter w of the neural network is changed. Adjusting the function y(x:w) so that the neural network selects a parameter w that is more suitable for the input x is called neural network training. Suppose multiple pairs of input and output of a function expressed using a neural network are given. If the desired output for a certain input x is d, the input/output pairs are given as {( _x1 , _d1 ), ( _x2 , _d2 ), ..., ( _xn , _dn )}. The set of pairs represented by (x, d) is called training data. Specifically, the set of pairs of color density values and true value image labels for each pixel in the single-color images of R, G, and B shown in Figure 3(b) is the training data shown in Figure 3(a).

Learning a neural network means adjusting the weights w so that the output y( _xn :w) of the neural network when given an input _xn is as close as possible to the output _dn for any input/output pair ( _xn , _dn ). The error function is the degree of closeness between the function expressed using a neural network and the training data.

It is a measure of the error function. The error function is also called a loss function. The error function E(w) used in the image analysis method according to the embodiment is expressed by the following (Equation 6). (Equation 6) is called cross entropy.

A method for calculating the cross entropy of (Equation 6) will now be described. In the output layer 50b of the neural network 50 used in the image analysis method according to the embodiment, i.e., in the final layer of the neural network, an activation function is used to classify the input x into a finite number of classes according to its content. The activation function is called a softmax function, and is expressed by the following (Equation 7). It is assumed that the output layer 50b has the same number of nodes as the number of classes k. The total input u of each node k (k = 1, ..., K) in the output layer L is given by u _k ^(L) from the output of the previous layer L-1. As a result, the output of the kth node in the output layer is expressed by the following (Equation 7).

(Equation 7) is the softmax function. The sum of the outputs y ₁ , . . . , y _K determined by (Equation 7) is always 1.

If each class is represented as C ₁ , ..., C _K , the output y _k (i.e., u _k ^(L) ) of node k in the output layer L represents the probability that a given input x belongs to class C _k . See Equation 8 below. The input x is classified into the class that maximizes the probability represented by Equation 8.

When training a neural network, the function represented by the neural network is considered to be a model of the posterior probability of each class, and under such a probability model, the likelihood of the weight w for the training data is evaluated, and the weight w that maximizes the likelihood is selected.

The target output _dn by the softmax function in (Equation 7) is set to 1 only if the output is the correct class, and to 0 otherwise. If the target output is expressed in a vector format as _dn = [ _dn1 , ..., _dnK ], for example, if the correct class of the input _xn is _C3 , only the target output _dn3 will be 1, and the other target outputs will be 0. When encoded in this way, the posterior distribution is expressed by the following (Equation 9).

The likelihood L(w) of weight w for training data {(x _n , d _n )} (n=1, ..., N) is expressed by the following (Equation 10): Taking the logarithm of the likelihood L(w) and inverting the sign derives the error function of (Equation 6).

Learning means minimizing the error function E(w) calculated based on training data with respect to the neural network parameter w. In the image analysis method according to the embodiment, the error function E(w) is expressed by (Equation 6).

Minimizing the error function E(w) with respect to the parameter w is equivalent to finding a local minimum of the function E(w). The parameter w is the weight of the connection between nodes. The minimum of the weight w is found by iterative calculations that start with an arbitrary initial value and repeatedly update the parameter w. An example of such calculations is the gradient descent method.

The gradient descent method uses the vector expressed in the following (Equation 11).

In gradient descent, the process of moving the current value of parameter w in the negative gradient direction (i.e., -∇E) is repeated many times. If the current weight is w ^(t) and the weight after the movement is w ^(t+1) , the calculation using gradient descent is expressed by the following (Equation 12). The value t represents the number of times the parameter w has been moved.

symbol
is a constant that determines the magnitude of the update amount of the parameter w, and is called the learning coefficient. By repeating the calculation expressed by (Equation 12), the error function E(w ^(t) ) decreases as the value t increases, and the parameter w reaches a minimum point.

Note that the calculation using (Equation 12) may be performed on all training data (n = 1, ..., N), or on only a portion of the training data. Gradient descent performed on only a portion of the training data is called stochastic gradient descent. The image analysis method according to the embodiment uses stochastic gradient descent.

13 , the processing unit 20A of the image analysis device 200A according to the first embodiment includes an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and an area detection unit 204. These functional blocks are realized by installing a program for causing a computer according to the present invention to execute image analysis processing in the storage unit 23 or memory 22 of the processing unit 20A and executing this program by the CPU 21. The window size database 104 and the algorithm database 105 are provided from the deep learning device 100A via the storage medium 98 or the network 99 and stored in the storage unit 23 or memory 22 of the processing unit 20A.

An analysis target image 78 of the tissue to be analyzed is assumed to have been captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or memory 22 of the processing unit 20A. The deep learning algorithm 60, including the learned connection weights w, is stored in the algorithm database 105 in association with the tissue type from which the tissue sample to be analyzed is derived, and functions as a program module that is part of a program that causes a computer to execute image analysis processing. That is, the deep learning algorithm 60 is used in a computer equipped with a CPU and memory, and causes the computer to perform calculations or processing of specific information according to its intended use, such as outputting data indicating each region in the tissue to be analyzed. Specifically, the CPU 21 of the processing unit 20A performs calculations of the neural network 60 based on the learned connection weights w in accordance with the algorithm specified in the deep learning algorithm 60 stored in the storage unit 23 or memory 22. The CPU 21 of the processing unit 20A performs calculations on the analysis target image 78, which is an image of the tissue to be analyzed and is input to the input layer 60a, and outputs a ternary image 83, which is data indicating each region in the tissue to be analyzed, from the output layer 60b.

Referring to Figure 14, the processing unit 20A of the image analysis device 200A performs the processing shown in Figure 13. Explaining this using the functional blocks shown in Figure 13, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The processing of step S29 is performed by the area detection unit 204.

In step S21, the processing unit 20A generates color density coded maps 79r, 79g, and 79b for each of the R, G, and B colors from the input analysis target image 78 in response to a command to start processing input, for example, by a user or the like, via the input unit 26. Analysis target data that combines the color density values of each of the R, G, and B colors for each pixel is generated from the color density coded maps 79r, 79g, and 79b. The method for generating the color density coded maps 79r, 79g, and 79b is the same as the method for generating them in step S10 during the deep learning process shown in FIG. 11.

In step S22 shown in FIG. 14, processing unit 20A receives input of tissue type from the user of image analysis device 200A via input unit 26 as an analysis condition. Based on the input tissue type, processing unit 20A references window size database 104 and algorithm database 105 to set a window size to be used for analysis and acquires a deep learning algorithm 60 to be used for analysis. The window size is the unit of analysis data input to neural network 60 at one time, and the product of the number of pixels of analysis data 80 in the window size and the number of primary colors contained in the image corresponds to the number of nodes in input layer 60a. Window sizes are associated with tissue types and pre-stored in window size database 104. The window size is, for example, 3 x 3 pixels, as in window W2 shown in FIG. 6. The deep learning algorithm 60 is also associated with tissue types and pre-stored in algorithm database 105 shown in FIG. 13.

In step S23 shown in FIG. 14, the processing unit 20A generates analysis target data by combining the color density values of R, G, and B for each pixel from the color density encoding diagrams 79r, 79g, and 79b, and further generates analysis data 80 for the window size.

In step S24, the processing unit 20A inputs the analysis data 80 shown in Figure 5 to the deep learning algorithm 60. As in step S14 during deep learning processing, the initial position of the window is, for example, a position where the pixel located within a 3x3 pixel area within the window corresponds to the upper left corner of the data to be analyzed. When the processing unit 20A inputs data 81 of a total of 27 color density values (3x3 pixels x 3 primary colors) contained in the window-sized analysis data 80 to the input layer 60a, the deep learning algorithm 60 outputs a discrimination result 82 to the output layer 60b.

In step S25 shown in FIG. 14, the processing unit 20A stores the discrimination result 82 output to the output layer 60b shown in FIG. 5. The discrimination result 82 is an estimated value (one of three values: 0, 1, or 2) for each pixel region of the data to be analyzed.

In step S26 shown in FIG. 14, the processing unit 20A determines whether all pixels in the input image have been processed. The input image is the color density encoding charts 79r, 79g, and 79b shown in FIG. 6, and if the series of processes from step S23 to step S25 shown in FIG. 14 have been performed for all pixels in the data to be analyzed, the processing unit 20A performs the process of step S28.

If all pixels in the analysis target data have not been processed, in step S27, the processing unit 20A moves the window W2 within the color density encoding diagrams 79r, 79g, and 79b shown in FIG. 5 by a predetermined number of pixels, similar to step S20 during deep learning processing. The processing unit 20A then performs a series of processes from step S23 to step S25 at the new position of the moved window W2. In step S25, the processing unit 20A stores the discrimination result 82 corresponding to the new window position after the movement. By storing the discrimination result 82 for each window size in this manner for all pixels in the analysis target image, a ternary image 83 of the analysis result is obtained. The image size of the ternary image 83 of the analysis result is the same as the image size of the analysis target image. Here, the ternary image 83 may be numerical data in which estimated values 2, 1, and 0 are assigned to each pixel, or may be an image in which, instead of estimated values 2, 1, and 0, display colors corresponding to values 2, 1, and 0, respectively, are used.

In step S28 shown in FIG. 14, the processing unit 20A outputs a ternary image 83 of the analysis results to the output unit 27.

In step S29, following step S28, the processing unit 20A further performs region detection processing on the ternary image 83 of the analysis results. In the ternary image 83, the tumor region, non-tumor region, and non-tissue region are distinguished and represented by three values.

Optionally, the processing unit 20A creates a region-enhanced image 84 by overlaying each of the obtained regions on the analysis target image 78 of the analysis target. The processing unit 20A outputs the created region-enhanced image 84 to the output unit 27, and ends the image analysis process.

Optionally, processing unit 20A may calculate the extent to which the tissue contained in analysis target image 78 contains tumorous regions. For example, the percentage of pixels determined to be tumorous regions can be calculated by taking the number of pixels corresponding to the tissue contained in analysis target image 78, i.e., the sum of the number of pixels determined to be tumorous regions and non-tumorous regions (total number of pixels in the tissue region), as 100%. The percentage of non-tumorous regions can also be calculated by subtracting the number of pixels determined to be tumorous regions from the total number of pixels, or by calculating the percentage of pixels determined to be non-tumorous regions when the total number of pixels in the tissue region is taken as 100%. The ratio between the percentage of tumorous regions and the percentage of non-tumorous regions can then be calculated. The calculated value may be output to output unit 27. The calculated value may also be output to output unit 27 along with region-enhanced image 84.

Calculating the tumor area content in this way is useful for screening whether tissue used for genetic analysis of cancer tissue contains tumor tissue suitable for testing.

As described above, a user of image analysis device 200A can input analysis target image 78 of the tissue to be analyzed into image analysis device 200A to obtain ternary image 83 as the analysis result. ternary image 83 represents the tumor region, non-tumor region, and non-tissue region in the specimen to be analyzed, allowing the user to distinguish between each region in the specimen to be analyzed.

Furthermore, the user of the image analysis device 200A can obtain a region-enhanced image 84 as the analysis result. The region-enhanced image 84 is generated, for example, by filling in each region with color in the analysis target image 78 of the analysis target. In another aspect, it is generated by overlapping the boundaries of each region. This allows the user to grasp each region in the tissue being analyzed at a glance.

Indicating tumor and/or non-tumor regions in the specimen to be analyzed helps to screen for tumor regions in the tissue to be analyzed prior to tissue diagnosis and improves specimen observation efficiency.

Second Embodiment
The image analysis system according to the second embodiment will be described below, focusing on the differences from the image analysis system according to the first embodiment.

[Configuration Overview]
Referring to FIG. 15 , the image analysis system according to the second embodiment includes a user-side device 200, which operates as an integrated image analysis device 200B. The image analysis device 200B is configured, for example, as a general-purpose computer and performs both the deep learning process and the image analysis process described in the first embodiment. In other words, the image analysis system according to the second embodiment is a standalone system in which deep learning and image analysis are performed on the user side. The image analysis system according to the second embodiment differs from the image analysis system according to the first embodiment in that the integrated image analysis device 200B installed on the user side performs the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment.

Image analysis device 200B is connected to imaging device 400. During deep learning processing, imaging device 400 acquires training images of tissue for learning, and during image analysis processing, acquires analysis target images 78 of the tissue to be analyzed.

[Hardware configuration]
The hardware configuration of the image analyzing device 200B is the same as the hardware configuration of the user device 200 shown in FIG.

[Function blocks and processing procedures]
16 , the processing unit 20B of the image analysis device 200B according to the second embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a region detection unit 204. These functional blocks are implemented by installing a program for causing a computer to execute deep learning processing and image analysis processing in the storage unit 23 or memory 22 of the processing unit 20B and executing the program by the CPU 21. The window size database 104 and the algorithm database 105 are stored in the storage unit 23 or memory 22 of the processing unit 20B and are both used during deep learning and image analysis processing. A trained neural network 60 is associated with a tissue type or a cell-containing sample type and pre-stored in the algorithm database 105. The connection weights w are updated by the deep learning processing and stored in the algorithm database 105 as a deep learning algorithm 60. It is assumed that the first training image 70R1, the second training image 70R2, and the third training image 70R3, which are training images, are captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or the memory 22 of the processing unit 20B. It is also assumed that the analysis target image 78 of the specimen to be analyzed is captured in advance by the imaging device 400 and stored in advance in the storage unit 23 or the memory 22 of the processing unit 20B.

The processing unit 20B of the image analysis device 200B performs the processing shown in FIG. 11 during deep learning processing, and performs the processing shown in FIG. 14 during image analysis processing. Using the functional blocks shown in FIG. 16, during deep learning processing, the processing of steps S10 to S12, S14, S18, and S19 is performed by the training data generation unit 101. The processing of step S13 is performed by the training data input unit 102. The processing of steps S16 and S17 is performed by the algorithm update unit 103. During image analysis processing, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The processing of step S29 is performed by the area detection unit 204.

The deep learning processing procedures and image analysis processing procedures performed by the image analysis device 200B according to the second embodiment are similar to the procedures performed by the deep learning device 100A and image analysis device 200A according to the first embodiment, respectively. Note that the image analysis device 200B according to the second embodiment differs from the deep learning device 100A and image analysis device 200A according to the first embodiment in the following respects:

In step S13 during deep learning processing, processing unit 20B accepts input of the type of tissue to be used for learning from the user of image analysis device 200B via input unit 26. Based on the input type of tissue, processing unit 20B references window size database 104 to set the window size, and references algorithm database 105 to set the neural network 50 to be used for learning.

As described above, by inputting the analysis target image 78 into the image analysis device 200B, the user of the image analysis device 200B can obtain a ternary image 83 as the analysis result. Furthermore, the user of the image analysis device 200B can obtain a region-enhanced image 84 as the analysis result.

The image analysis device 200B according to the second embodiment allows the user to use the type of tissue they select as the tissue for learning. This means that the learning of the neural network 50 is not left to the vendor, but the user can improve the level of learning of the neural network 50 themselves.

Third Embodiment
The image analysis system according to the third embodiment will be described below, focusing on the differences from the image analysis system according to the second embodiment.

[Configuration Overview]
17 , the image analysis system according to the third embodiment includes a vendor-side device 100 and a user-side device 200. The vendor-side device 100 operates as an integrated image analysis device 100B, and the user-side device 200 operates as a terminal device 200C. The image analysis device 100B is configured, for example, as a general-purpose computer, and is a cloud server-side device that performs both the deep learning processing and the image analysis processing described in the first embodiment. The terminal device 200C is configured, for example, as a general-purpose computer, and is a user-side terminal device that transmits images to be analyzed to the image analysis device 100B via a network 99 and receives images of the analysis results from the image analysis device 100B via the network 99.

The image analysis system according to the third embodiment is similar to the image analysis system according to the second embodiment in that an integrated image analysis device 100B installed on the vendor side performs the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment. On the other hand, the image analysis system according to the third embodiment differs from the image analysis system according to the second embodiment in that it includes a terminal device 200C, which provides an input interface for images to be analyzed and an output interface for images of the analysis results to the user-side terminal device 200C. In other words, the image analysis system according to the third embodiment is a cloud service-type system in which the vendor side, which performs deep learning processing and image analysis processing, provides the user side with input/output interfaces for images to be analyzed and images of the analysis results.

The image analysis device 100B is connected to the imaging device 300 and acquires training images of the tissue to be studied, which are captured by the imaging device 300.

The terminal device 200C is connected to the imaging device 400 and acquires an analysis target image 78 of the tissue to be analyzed, which is captured by the imaging device 400.

[Hardware configuration]
The hardware configuration of the image analyzing device 100B is similar to that of the vendor-side device 100 shown in Fig. 7. The hardware configuration of the terminal device 200C is similar to that of the user-side device 200 shown in Fig. 9.

[Function blocks and processing procedures]
18 , the processing unit 10B of the image analysis device 100B according to the third embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a region detection unit 204. These functional blocks are implemented by installing a program for causing a computer to execute deep learning processing and image analysis processing in the storage unit 13 or memory 12 of the processing unit 10B and executing the program by the CPU 11. The window size database 104 and the algorithm database 105 are stored in the storage unit 13 or memory 12 of the processing unit 10B, and both are used during deep learning and image analysis processing. The neural network 50 is associated with the tissue type and pre-stored in the algorithm database 105. The connection weight w is updated through deep learning processing and stored in the algorithm database 105 as a deep learning algorithm 60.

The training images, first training image 70R1, second training image 70R2, and third training image 70R3, are assumed to have been captured in advance by imaging device 300 and stored in advance in storage unit 13 or memory 12 of processing unit 10B. The analysis target image 78 of the tissue to be analyzed is also assumed to have been captured in advance by imaging device 400 and stored in advance in storage unit 23 or memory 22 of processing unit 20C of terminal device 200C.

The processing unit 10B of the image analysis device 100B performs the processing shown in FIG. 11 during deep learning processing, and performs the processing shown in FIG. 14 during image analysis processing. Using the functional blocks shown in FIG. 18, during deep learning processing, the processing of steps S10 to S12, S14, S18, and S19 is performed by the training data generation unit 101. The processing of step S13 is performed by the training data input unit 102. The processing of steps S16 and S17 is performed by the algorithm update unit 103. During image analysis processing, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The processing of step S29 is performed by the area detection unit 204.

The deep learning processing procedure and image analysis processing procedure performed by the image analysis device 100B according to the third embodiment are similar to the procedures performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment, respectively. Note that the image analysis device 100B according to the third embodiment differs from the deep learning device 100A and the image analysis device 200A according to the first embodiment in the following four points:

In step S21 of the image analysis process shown in FIG. 14, the processing unit 10B receives an analysis target image 78 of the tissue to be analyzed from the user's terminal device 200C, and generates color density coded maps 79r, 79g, and 79b for each of the colors R, G, and B from the received analysis target image 78. The method for generating the color density coded maps 79r, 79g, and 79b and the analysis target data is the same as the method for generating them in step S10 of the deep learning process shown in FIG. 11.

In step S22 of the image analysis process shown in FIG. 14, the processing unit 10B receives input of the tissue type from the user of the terminal device 200C as an analysis condition via the input unit 26 of the terminal device 200C. Based on the input tissue type, the processing unit 10B references the window size database 104 and the algorithm database 105 to set the window size to be used in the analysis and obtain the deep learning algorithm 60 to be used in the analysis.

In step S28 during image analysis processing, the processing unit 10B transmits the ternary image 83 of the analysis results to the user's terminal device 200C. In the user's terminal device 200C, the processing unit 20C outputs the received ternary image 83 of the analysis results to the output unit 27.

In step S29 of the image analysis process, following step S28, the processing unit 10B further performs a nuclear region detection process on the ternary image 83 of the analysis results. As an optional step, the processing unit 10B creates a region-enhanced image 84 by overlaying each of the obtained regions on the analysis target image 78 of the analysis target. The processing unit 10B transmits the created region-enhanced image 84 to the user-side terminal device 200C. In the user-side terminal device 200C, the processing unit 20C outputs the received region-enhanced image 84 to the output unit 27, thereby completing the image analysis process.

As described above, the user of terminal device 200C can obtain a ternary image 83 as the analysis result by sending an analysis target image 78 of the tissue to be analyzed to image analysis device 100B. Furthermore, the user of terminal device 200C can obtain a region-enhanced image 84 as the analysis result.

The image analysis device 100B according to the third embodiment allows a user to enjoy the results of image analysis processing without acquiring the window size database 104 and algorithm database 105 from the deep learning device 100A. This makes it possible to provide a cloud service that analyzes the tissue to be analyzed, distinguishing between tumorous and/or non-tumorous regions and screening for tumorous regions in each tissue.

[Computer Program]
Embodiments of the present invention include a computer program and a product thereof that causes processing units 10A, 20B, and 10B to execute steps S10 to S20 and generate a trained deep learning algorithm.Embodiments of the present invention also include a computer program and a product thereof that causes processing units 10A, 20B, and 10B to execute steps S21 to S29 and cause a computer to function to analyze images of tissue collected from an individual.

[Other forms]
Although the present invention has been described above with reference to the outline and specific embodiments, the present invention is not limited to the outline and each embodiment described above.

In the first to third embodiments described above, in step S13, processing units 10A, 20B, and 10B refer to the window size database 104 to set the number of pixels for the window size, but the operator or user may set the window size directly. In this case, the window size database 104 is not necessary.

In the first to third embodiments described above, in step S13, processing units 10A, 20B, and 10B set the number of pixels in the window size based on the input tissue type. However, instead of inputting the tissue type, the tissue size may be input. Processing units 10A, 20B, and 10B may set the number of pixels in the window size by referencing window size database 104 based on the input tissue size. In step S22, as in step S13, instead of inputting the tissue type, the tissue size may be input. Processing units 20A, 20B, and 10B may set the number of pixels in the window size by referencing window size database 104 and algorithm database 105 based on the input tissue size, and acquire the neural network 60.

When entering the size of the tissue, the size may be entered directly as a numerical value, or the input user interface may be a pull-down menu, allowing the user to select a predetermined numerical range that corresponds to the size they wish to enter.

Furthermore, in steps S13 and S22, in addition to the tissue type or tissue size, the imaging magnification used when capturing the training image and the analysis target image 78 may be input. The imaging magnification may be input directly as a numerical value, or, for example, the input user interface may be a pull-down menu, allowing the user to select a predetermined numerical range corresponding to the magnification they wish to input.

In the first to third embodiments described above, the window size is set to 3x3 pixels during deep learning processing and image analysis processing for ease of explanation, but the number of pixels in the window size is not limited to this. The window size may be set according to the type of tissue, for example. In this case, the product of the number of pixels in the window size and the number of primary colors contained in the image should correspond to the number of nodes in the input layers 50a, 60a of the neural networks 50, 60.

In the first to third embodiments described above, in step S17, the processing units 10A, 20B, and 10B store the deep learning algorithms 60 in the algorithm database 105 in one-to-one correspondence with the organization type. Alternatively, in step S17, the processing units 10A, 20B, and 10B may store one deep learning algorithm 60 in the algorithm database 105 in correspondence with multiple organization types.

In the first to third embodiments, the hue is defined as a combination of the three primary colors of light or the three primary colors of color, but the number of hues is not limited to three. The number of hues may be four, i.e., red (R), green (G), blue (B), and yellow (Y), or two, i.e., two hues, obtained by removing one hue from the three primary colors of red (R), green (G), and blue (B). Alternatively, the hue may be a single primary color, i.e., only one of the three primary colors of red (R), green (G), and blue (B) (e.g., green (G)). For example, the training images 70R1, 70R2, and 70R3 and the analysis target image 78, acquired using a known microscope or virtual slide scanner, are not limited to color images of the three primary colors of red (R), green (G), and blue (B). They may be color images of two primary colors, or may be images containing one or more primary colors.

In the first to third embodiments described above, in step S10, processing units 10A, 20B, and 10B generate each density coding map of the training image (density coding maps 70R1r, 70R1g, and 70R1b; density coding maps 70R2r, 70R2g, and 70R2b; and density coding maps 70R3r, 70R3g, and 70R3b) as an 8-bit (256-level) single-color image of each primary color. However, the gradation of the primary colors used to create each density coding map is not limited to 8 bits. The gradation of each density coding map may depend on the imaging conditions of the slide image scanner. For example, the gradation may be 1 bit or more, such as 2 bits, 4 bits, 16 bits, or 24 bits. Similarly, in step S21, processing units 20A, 20B, and 10B generate color density coding maps 79r, 79g, and 79b as single-color images of each primary color. However, the number of gradations of the primary colors used to create the color density coding maps is not limited to three gradations. The number of gradations of the primary colors used to create the color density coding maps is not limited to 8 bits. The gradations of the color density coding maps 79r, 79g, and 79b may depend on the imaging conditions of the slide image scanner. For example, they may be 1 bit or more, such as 2 bits, 4 bits, 16 bits, or 24 bits. It is preferable that the gradations of each color density coding map and color density coding maps 79r, 79g, and 79b of the training images are all the same.

In the first to third embodiments described above, in step S10, the processing units 10A, 20B, and 10B generate each color density coding map from the training image. However, a matrix table in which each pixel is pre-coded according to gradation may be used as the training image. The processing units 10A, 20B, and 10B may acquire the training image as each color density coding map directly, for example, from a virtual slide scanner. Similarly, in step S21, the processing units 20A, 20B, and 10B generate color density coding maps 79r, 79g, and 79b for each of the R, G, and B colors from the analysis target image 78. However, a matrix table in which each pixel is pre-coded according to gradation may be used as the analysis image. That is, the processing units 20A, 20B, and 10B may acquire the color density coding maps 79r, 79g, and 79b directly, for example, from a virtual slide scanner.

In the first to third embodiments, RGB is used as the color space when generating each color density coding map from the training images, but the color space is not limited to RGB. Various color spaces such as YUV, CMY, and CIE L ^* a ^* b ^* can be used instead of RGB.

In the first to third embodiments described above, the density values for each pixel in the area training data and analysis data 80 are stored in the order of red (R), green (G), and blue (B), but the order in which the density values are stored and handled is not limited to this. For example, the density values may be stored in the order of blue (B), green (G), and red (R), as long as the order of the density values in the area training data and the analysis data 80 are the same.

In the first to third embodiments described above, processing units 10A and 10B are realized as an integrated device, but processing units 10A and 10B do not need to be an integrated device; the CPU 11, memory 12, storage unit 13, etc. may be located in separate locations and connected via a network. Processing units 10A and 10B, input unit 16, and output unit 17 also do not necessarily need to be located in the same place; they may be located in separate locations and connected to each other via a network so that they can communicate with each other. The same applies to processing units 20A, 20B, and 20C.

In the first to third embodiments described above, the functional blocks of the training data generation unit 101, training data input unit 102, algorithm update unit 103, analysis data generation unit 201, analysis data input unit 202, analysis unit 203, and area detection unit 204 are executed by a single CPU 11 or a single CPU 21, but these functional blocks do not necessarily have to be executed by a single CPU and may be executed in a distributed manner across multiple CPUs. Furthermore, these functional blocks may be executed in a distributed manner across multiple GPUs, or in a distributed manner across multiple CPUs and multiple GPUs.

In the second and third embodiments described above, the programs for performing the processing of each step described in Figures 11 and 14 are stored in advance in the storage units 13 and 23. Alternatively, the programs may be installed in the processing units 10B and 20B from a computer-readable, non-transitory, tangible storage medium 98, such as a DVD-ROM or USB memory. Alternatively, the processing units 10B and 20B may be connected to a network 99, and the programs may be downloaded and installed via the network 99 from, for example, an external server (not shown).

In the first to third embodiments described above, the input units 16, 26 are input devices such as a keyboard or a mouse, and the output units 17, 27 are realized as display devices such as a liquid crystal display. Alternatively, the input units 16, 26 and the output units 17, 27 may be integrated into a touch panel display device. Alternatively, the output units 17, 27 may be configured as a printer or the like, and the ternary image 83 or region-enhanced image 84 of the analysis results may be printed and output.

In the first to third embodiments described above, imaging device 300 is directly connected to deep learning device 100A or image analysis device 100B, but imaging device 300 may be connected to deep learning device 100A or image analysis device 100B via network 99. Similarly, imaging device 400 is directly connected to image analysis device 200A or image analysis device 200B, but imaging device 400 may be connected to image analysis device 200A or image analysis device 200B via network 99.

<Verification of trained deep learning algorithms>
Deep learning processing and image analysis processing were performed using the standalone system described in the second embodiment. Human stomach cancer tissue, including tumor and normal regions, was used as the target tissue for learning. To create the trained deep learning algorithm, training images were captured at magnifications of 1x, 3x, 5x, 10x, 20x, and 40x for the tissue specimen. The neural network was trained using the training data using the training images at different magnifications, and the accuracy of the analysis results was confirmed.
Details of the training data and analysis data are as follows:
Training data: 106 hole slide images Verification analysis data: 45 hole slide images

[Creating training data and learning]
Bright-field whole-slide images of HE-stained gastric cancer tissue were scanned in color using a virtual slide scanner (NanoZoomer-XR, Hamamatsu Photonics; scanning resolution: 0.46 μm/pixel in 20x mode, 0.23 μm/pixel in 40x mode). The imaging magnifications were 1x, 3x, 5x, 10x, 20x, and 40x. A pathologist designated tumor, non-tumor, and non-tissue regions on the whole-slide images containing the target tissue. The whole-slide images with designated regions were divided according to the above criteria and used as training images. Each training image was determined to be tumor, non-tumor, or non-tissue. For tissue regions containing both tumor and non-tumor regions, a region was considered tumor if 50% or more of the pixels designated as tumor were included in the tissue region in the training image. Furthermore, if the proportion of pixels designated as non-tumorous regions among the number of pixels corresponding to tissue regions in the training image was 100%, the region was designated as non-tumorous. For training images containing a mixture of non-tissue and tissue regions, if the tissue region accounted for 0% of the number of pixels in the training image, the region was designated as non-tissue, and otherwise the region was designated as tissue. Each pixel in each training image determined to be a tumor region, non-tumor region, or non-tissue region was assigned a label value (tumor region: "1", non-tumor region: "2", non-tissue region: "0") to distinguish between the regions, and first preliminary training data 70R1L, second preliminary training data 70R2L, and third preliminary training data 70R2L were generated. The training image determined to be a tumor region was designated as the first training image, the training image determined to be a non-tumor region was designated as the second training image 70R2, and the training image determined to be a non-tissue region was designated as the third training image 70R3. For each training image, the color density values of each of the R, G, and B colors were gradated using 8 bits to generate color density encoding maps for each of the R, G, and B colors, and the created color density encoding maps for each of the R, G, and B colors were combined to generate the first training data 74R1, the second training data 74R2, and the third training data 74R3, respectively.

Tumor region training data was then created by combining the first training data 74R1 and the first preliminary training data 70R1L. Non-tumor region training data was created by combining the second training data 74R2 and the second preliminary training data 70R2L. Non-tissue region training data was created by combining the third training data 74R3 and the third preliminary training data 70R2L. Each region training data created was divided into windows of 200 x 200 pixels, and the neural network was trained using the training data of the divided window sizes as the input layer.

Training data for each region was generated for each magnification of the tissue specimen, and the neural network was trained for each magnification.

[Creating an image to be analyzed]
Similar to the training data, whole-slide images of HE-stained human gastric cancer tissues were captured in bright field using a virtual slide scanner to obtain the analysis images. The magnifications were 1x, 3x, 5x, 10x, 20x, and 40x. R, G, and B color density coded maps were then created based on the captured analysis images, and the resulting R, G, and B color density coded maps were combined to generate the analysis data. The analysis data was generated for each magnification.

[Analysis results]
Analysis data with a window size of 200 x 200 pixels was created from the data to be analyzed, and the analysis data was input into a trained neural network corresponding to the magnification. Based on the analysis results output from the neural network, the data was classified into tumor regions, non-tumor regions, and non-tissue regions (background). The tumor regions obtained by the analysis process were displayed in white, non-tumor regions in gray, and non-tissue regions in black. The analysis results are shown in Figure 19.

In Figure 19, (a) is a whole slide image of a bright-field image of HE-stained gastric cancer tissue, and is the image to be analyzed. (b) is an image of the whole slide image in which the pathologist designated the tumor area and enclosed by a solid line. (c) shows the analysis results using training images and analysis images magnified by 1x. It was not possible to accurately identify each of the areas designated by the pathologist. (d) shows the analysis results using training images and analysis images magnified by 5x. The tumor area designated by the pathologist was generally able to be identified as tumor area. In addition, the non-tumor area designated by the pathologist was generally able to be identified as non-tumor area. The non-tissue area designated by the pathologist was generally able to be identified as non-tissue area. (e) shows the analysis results using training images and analysis images magnified by 40x. The tumor area designated by the pathologist was generally able to be identified as tumor area. However, among areas designated by pathologists as non-tumorous, the number of areas judged to be tumorous was increasing, resulting in a high rate of non-tumorous areas being mistakenly identified as tumorous.

These results show that using magnifications that are too low or too high when acquiring training images and analysis images can lead to incorrect classification.

In Figure 20, (a) is a whole slide image of a bright-field image of HE-stained gastric cancer tissue, different from that shown in Figure 19, and is the image to be analyzed. (b) is an image of the whole slide image in which the pathologist designated the tumor region and circled it with a solid line. (c) shows the analysis results using training images and analysis images at 5x magnification. The tumor region designated by the pathologist was generally able to be distinguished as tumor region. Additionally, the non-tumor region designated by the pathologist was generally able to be distinguished as non-tumor region. The non-tissue region designated by the pathologist was generally able to be distinguished as non-tissue region. (d) shows the analysis results using training images and analysis images at 20x magnification. As with the 5x magnification, each region designated by the pathologist was generally able to be distinguished.

For the analysis images at 5x magnification, the tumor area content in the tissue area present in one whole slide image was calculated according to the following formula:

Tumor area content (%) = {[Number of pixels identified as tumor area] / [(Number of pixels identified as tumor area) + (Number of pixels identified as non-tumor area)]} x 100

The tumor area content was 54.5%.

Figure 21 also shows the sensitivity (a) and positive predictive value (b) for each magnification. From these results, it was determined that a magnification of 5 to 20 times is appropriate for the tissue specimen when acquiring training images and analysis images. Figure 21 also shows the change in sensitivity (c) and positive predictive value (d) due to differences in the number of pixels in the window size. The analysis images were captured at a magnification of 5 times. A window size of 125 x 125 to 200 x 200 pixels was considered preferable.

In this study, the analysis time for one whole slide image was approximately several minutes to 30 minutes.

<Additional Notes>
One embodiment relates to an image analysis method for analyzing a tissue image using a deep learning algorithm (60) with a neural network structure. The image analysis method may generate analysis data (80) from an analysis target image (78) including the tissue to be analyzed, input the analysis data (80) to a deep learning algorithm (60), and generate data indicating whether a region included in the analysis data is a tumor region using the deep learning algorithm (60). According to the present invention, data indicating whether a region included in the analysis data in the tissue to be analyzed is a tumor region can be generated.

In the above embodiment, preferably, the data indicating whether an area included in the analysis data is a tumor area is data indicating the area of the tumor area. This embodiment allows the user to grasp the tumor area in the tissue to be analyzed at a glance.

In the above embodiment, the image to be analyzed is preferably an image of a tissue diagnostic specimen and includes hues that combine two or more primary colors. This embodiment can aid in pathological diagnosis.

In the above-described embodiments, the image to be analyzed is preferably an image of the tissue to be analyzed captured at a magnification of 3 to 20 times. In the above-described embodiments, the size of the area included in the image to be analyzed is 200 μm x 200 μm or more and 400 μm x 400 μm or less. These embodiments enable more accurate determination.

In the above embodiment, data is preferably generated to distinguish between tumor cell regions and other regions based on data indicating whether a region included in the analysis data is a tumor region. This embodiment allows the user to grasp the tumor region in the tissue being analyzed at a glance.

In the above embodiment, data indicating the boundary between tumor cell regions and other regions is preferably generated based on data indicating whether a region included in the analysis data is a tumor region. This embodiment allows the user to grasp at a glance the boundary between tumor regions and non-tumor regions in the tissue being analyzed.

In the above embodiment, data indicating the percentage of tumor regions in the tissue to be analyzed is preferably generated based on data indicating whether or not a region included in the analysis data is a tumor region. This embodiment allows the user to grasp the percentage of tumor regions in the tissue to be analyzed.

In the above embodiment, data indicating the ratio of tumorous to non-tumorous areas in the tissue to be analyzed is preferably generated based on data indicating whether an area included in the analysis data is a tumorous area. This embodiment allows the user to understand the proportion of tumorous areas in the tissue to be analyzed. It also allows the user to know whether samples to be used for genetic testing, etc., have been collected appropriately.

In this embodiment, data is preferably generated for distinguishing and presenting tumorous regions, non-tumorous regions, and regions that do not contain tissue for a single image to be analyzed. This embodiment allows the user to grasp at a glance whether or not there is a tumorous region in the tissue to be analyzed.

In the above embodiment, preferably, multiple pieces of analysis data are generated for each region of a predetermined number of pixels for one image to be analyzed. This embodiment enables accurate analysis of a wide range of tissue to be analyzed.

In the above embodiment, preferably, the deep learning algorithm (60) generates a label indicating that the region of the specified number of pixels contains at least one selected from a tumor region and a non-tumor region for the input analysis data.

In the above embodiment, the number of nodes in the input layer (50a) of the neural network (50) preferably corresponds to the product of the predetermined number of pixels in the analysis data (80) and the number of combined primary colors. This embodiment enables highly accurate analysis.

In the above embodiment, preferably, the specimen is a stained specimen, and the image to be analyzed is an image of the stained specimen captured under a bright field microscope. This embodiment can aid in pathological diagnosis.

In the above embodiment, preferably, the training data used to train the deep learning algorithm is generated based on bright-field images of tissue specimens, including tumor regions, extracted from an individual and stained for bright-field observation, captured under a bright-field microscope. This embodiment can aid in tissue diagnosis.

In the above embodiment, preferably, the staining for brightfield observation uses hematoxylin for nuclear staining. More preferably, the staining for brightfield observation is hematoxylin-eosin staining (HE staining). This embodiment enables a wide range of highly versatile tests to be performed in tissue diagnosis.

In the above embodiment, the training data preferably includes a label value indicating a tumor region determined from the bright-field image. More preferably, the training data includes the label value for each region of a predetermined number of pixels in the bright-field image. This embodiment makes it possible to generate training data for tumor regions.

In the above embodiment, the training data is preferably generated for each region of a predetermined number of pixels in the bright-field image. This aspect improves the learning efficiency of the deep learning algorithm (50).

In the above embodiment, preferably, the deep learning algorithm classifies the analysis data into a class indicating that the tissue included in the image to be analyzed is a tumor region.

In the above embodiment, preferably, the output layer (50b) of the neural network (50) is a node that uses a softmax function as an activation function. This embodiment can improve learning efficiency and analysis accuracy.

In the above embodiment, preferably, the deep learning algorithm generates data indicating that the tissue included in the image to be analyzed is a tumor region for each region of a predetermined number of pixels each time analysis data (78) is input. This embodiment can improve analysis efficiency.

In the above embodiment, preferably, a deep learning algorithm (60) is generated according to the type of tissue. Furthermore, preferably, the analysis data is processed using a deep learning algorithm corresponding to the type of tissue to be analyzed, selected from among a plurality of deep learning algorithms according to the type of tissue. This embodiment can improve the accuracy of analysis.

One embodiment relates to an image analysis device (100) that analyzes tissue images using a deep learning algorithm (60) with a neural network structure. The analysis device may include a processing unit (10) that generates analysis data (80) from an analysis target image (78) that includes the tissue to be analyzed, inputs the analysis data (80) to a deep learning algorithm (50), and uses the deep learning algorithm (50) to generate data indicating whether a region included in the analysis data is a tumor region.

One embodiment relates to a computer program that analyzes tissue images using a deep learning algorithm (60) with a neural network structure. The computer may be configured to perform the following steps: generate analysis data (80) from an analysis target image (78) containing the tissue to be analyzed; input the analysis data (80) into the deep learning algorithm (60); and generate, using the deep learning algorithm (60), data indicating whether a region included in the analysis data is a tumor region.

One embodiment includes a step of generating training data used for training a deep learning algorithm (50) based on training images acquired from a training tissue specimen, and a step of training the neural network (50) to learn the training data, wherein the generating step includes a first acquisition step of acquiring first training data (74R1) corresponding to a first training image (70R1) obtained by imaging a tumor region, a second acquisition step of acquiring second training data (74R2) corresponding to a second training image (70R2) obtained by imaging a non-tumor region, and a third acquisition step of acquiring a third training image (74R3) obtained by imaging a region not containing tissue. and a third acquisition step of acquiring third training data (74R3) corresponding to the first training data (74R1) that corresponds to the first training data (74R2) that corresponds to the second training data (74R3), wherein the learning step includes a first learning step of making a neural network (50) learn that the first training data (74R1) is a tumor region, a second learning step of making the neural network (50) learn that the second training data (74R2) is a non-tumor region, and a third learning step of making the neural network (50) learn that the third training data (74R3) is a region that does not contain tissue. Preferably, the first training data (74R1), the second training data (74R2), and the third training data (74R3) are used as an input layer (50a) of the neural network (50), and a tumor region, a non-tumor region, and a region not containing tissue are used as an output layer (50b) of the neural network (50) corresponding to the first training data (74R1), the second training data (74R2), and the third training data (74R3), respectively. More preferably, the method may further include the steps of generating the first training data (74R1) from the first training image (70R1) before the first acquisition step, generating the second training data (74R2) from the second training image (70R2) before the second acquisition step, and generating the third training data (74R3) from the third training image (70R3) before the third acquisition step. According to this embodiment, a trained deep learning algorithm can be generated that determines whether the tissue to be analyzed includes at least one selected from a tumorous region and a non-tumorous region.

In the above generation method, the training images are preferably bright-field images obtained by capturing stained images of specimens prepared by staining tissue collected from an individual with staining agents for bright-field observation under a bright-field microscope.

In one embodiment, first training data (74R1), second training data (74R2), and third training data (74R3) are used as the input layer (50a) of a neural network (50), and tumor regions, non-tumor regions, and regions not containing tissue are learned as the output layer (50b) of the neural network (50) corresponding to the first training data (74R1), second training data (74R2), and third training data (74R3), respectively. The trained deep learning algorithm (60) may be a trained deep learning algorithm, wherein the first training data (74R1) is generated from a first training image (70R1) capturing an image of a tumorous region of training tissue, the second training data (74R2) is generated from a second training image (70R2) capturing an image of a non-tumorous region of training tissue, and the third training data (74R3) is generated from a third training image (70R3) capturing an image of a region not containing tissue.

10 (10A, 10B) Processing unit 20 (20A, 20B, 20C) Processing units 11, 21 CPU
12, 22 Memory 13, 23 Recording unit 14, 24 Bus 15, 25 Interface unit 16, 26 Input unit 17, 27 Output unit 19, 29 GPU
50 Neural Networks (Deep Learning Algorithms)
50a Input layer 50b Output layer 50c Hidden layer 60 Trained neural network (trained deep learning algorithm)
60a Input layer 60b Output layer 60c Hidden layer 70W1 Hole slide image 70W2 Hole slide image 70W3 Hole slide image 70R1L First preliminary training data 70R2L Second preliminary training data 70R3L Third preliminary training data 72R1r, 72R1g, 72R1b Color density coded map 72R2r, 72R2g, 72R2b Color density coded map 72R3r, 72R3g, 72R3b Color density coded map 74R1, 74R2, 74R3 Training data 75R1, 75R2, 75R3 Window size training data 76 Color density values 77R1, 77R2, 77R3 Label value of true value image 78 Bright-field images to be analyzed 79r, 79g, 79b Color density coded map to be analyzed 80 Analysis data 81 Color density values 82 Discrimination result (pixel estimate)
83 Ternary image of analysis result 84 Region-enhanced image 89 (89a, 89b) Node 98 Recording medium 99 Network 100 Vendor-side device 100A Deep learning device 100B Integrated image analysis device 101 Training data generation unit 102 Training data input unit 103 Algorithm update unit 104 Window size database 105 Algorithm database 200 User-side device 200A Image analysis device 200B Integrated image analysis device 200C Terminal device 201 Analysis data generation unit 202 Analysis data input unit 203 Analysis unit 204 Region detection unit 300, 400 Imaging device 301, 401 Imaging element 308, 408 Sample structure 309, 409 Stage W1 Window W2 Window

Claims (20)

An image analysis method for analyzing tissue images using a deep learning algorithm with a neural network structure, comprising:
Accepts input of the type of tissue to be analyzed,
By referring to an algorithm database that stores a plurality of deep learning algorithms generated according to the type of tissue, and a window size database that stores the number of pixels of the window size of an image that is to be input to the deep learning algorithm at one time, the deep learning algorithm and the number of pixels corresponding to the received type of tissue to be analyzed are set;
generating analysis data from the analysis target image including the tissue to be analyzed according to the set number of pixels;
inputting the analysis data into the configured deep learning algorithm;
and displaying an image of the tissue to be analyzed in a manner that allows a tumor region to be distinguished from other regions, based on data that indicates whether a pixel region consisting of a plurality of pixels included in the analysis data is a tumor region, the data being output from the deep learning algorithm to which the analysis data has been input.
Image analysis methods. The image analysis method of claim 1, wherein each of the plurality of deep learning algorithms is associated with a tissue type and stored in the algorithm database. The image analysis method described in claim 2, wherein at least one of the plurality of deep learning algorithms is stored in the algorithm database in association with one tissue type. The image analysis method described in any one of claims 1 to 3, wherein the deep learning algorithm corresponding to the type of tissue to be analyzed is selected based on a user's operation via an input unit. The image analysis method of any one of claims 1 to 4, further comprising: applying a rendering process to the image to be analyzed based on the data to display tumor areas and other areas in a distinguishable manner. The image analysis method of claim 5, wherein the drawing process includes drawing the tumor region and other regions in different colors. The image analysis method according to any one of claims 1 to 6, wherein the image to be analyzed is an image of a tissue diagnostic specimen, and the image to be analyzed includes a hue that is a combination of two or more primary colors. The image analysis method according to any one of claims 1 to 7, wherein the image to be analyzed is an image of the tissue to be analyzed captured at a magnification of 3x to 20x. An image analysis method according to any one of claims 1 to 8, wherein the analysis data corresponds to an image cut out from the image to be analyzed, and a plurality of pieces of analysis data are generated from the image to be analyzed. The image analysis method according to any one of claims 1 to 9, wherein the image to be analyzed is an image of a tissue diagnostic specimen, the tissue diagnostic specimen being a stained specimen, and the image to be analyzed is an image of the stained specimen captured under bright field conditions with a microscope. The image analysis method of claim 10, wherein the training data used to train the deep learning algorithm is generated based on bright-field images of a tissue specimen containing a tumor region taken from an individual, stained for bright-field observation, and then captured under bright-field microscope conditions. The image analysis method described in any one of claims 1 to 11, wherein the output layer of the neural network is a node that uses a softmax function as an activation function. An image analysis device that analyzes tissue images using a deep learning algorithm with a neural network structure,
A display unit;
a control unit that receives an input of a type of tissue to be analyzed, and references an algorithm database that stores a plurality of deep learning algorithms generated according to the type of tissue, and a window size database that stores the number of pixels of a window size of an image that is to be input to the deep learning algorithm at one time, to set the deep learning algorithm and the number of pixels corresponding to the received type of tissue to be analyzed, generates analysis data from the analysis target image including the tissue to be analyzed according to the set number of pixels, inputs the analysis data to the set deep learning algorithm, and displays the image of the tissue to be analyzed on the display unit so that a tumor region and other regions can be distinguished based on data that indicates whether a pixel region consisting of a plurality of pixels included in the analysis data is a tumor region, the data being output from the deep learning algorithm to which the analysis data has been input.
Image analysis device. The image analysis device of claim 13, wherein the algorithm database stores each of the plurality of deep learning algorithms in association with a tissue type. The image analysis device of claim 14, wherein the algorithm database stores at least one of the plurality of deep learning algorithms in association with one tissue type. further comprising an input unit operated by a user,
The image analysis device according to claim 13 , wherein the control unit selects the deep learning algorithm corresponding to the type of tissue to be analyzed based on an operation via the input unit. The image analysis device of any one of claims 13 to 16, wherein the control unit performs a drawing process on the image to be analyzed based on the data to display a tumorous region and other regions in a distinguishable manner. The image analysis device of claim 17, wherein the drawing process includes drawing the tumor region and other regions in different colors. An image analysis device according to any one of claims 13 to 18, wherein the analysis data corresponds to an image cut out from the image to be analyzed, and a plurality of pieces of analysis data are generated from the image to be analyzed. A computer program for analyzing tissue images using a deep learning algorithm with a neural network structure, comprising:
On the computer,
receiving input of the type of tissue to be analyzed;
a process of setting the deep learning algorithm and the number of pixels corresponding to the received type of tissue to be analyzed by referring to an algorithm database that stores multiple deep learning algorithms generated according to the type of tissue and a window size database that stores the number of pixels of the window size of an image that is to be input to the deep learning algorithm at one time;
generating analysis data from the analysis target image including the tissue to be analyzed according to the set number of pixels;
inputting the analysis data into the set deep learning algorithm;
and displaying an image of the tissue to be analyzed in a manner that allows a tumor region to be distinguished from other regions, based on data indicating whether a pixel region consisting of a plurality of pixels included in the analysis data is a tumor region, the data being output from the deep learning algorithm to which the analysis data has been input.
Computer program.