JP7736065B2 - Cell image analysis method - Google Patents

Description

This invention relates to a cell image analysis method, and in particular to a cell analysis method that analyzes cells using a trained model.

Conventionally, cell analysis methods that analyze cells using a trained model are known. Such cell analysis methods are disclosed, for example, in International Publication No. 2019/171546.

WO 2019/171546 discloses a cell image analysis method for analyzing cell images captured by an imaging device. Specifically, WO 2019/171546 discloses a configuration for acquiring cell images by photographing cells cultured in a culture plate with an imaging device such as a microscope. The cell image analysis method disclosed in WO 2019/171546 uses the analysis results of a trained model to classify cells depicted in a cell image as normal or abnormal. WO 2019/171546 also discloses a configuration for classifying cells through a segmentation process that determines which category each pixel belongs to in a cell image.

International Publication No. 2019/171546

Although not disclosed in WO 2019/171546, when classifying cells in a cell image using the analysis results of a trained model, classification is performed for each pixel using the highest value of the analysis results for that pixel. However, when classifying cells using the highest value, the same classification result will be obtained regardless of whether the difference between the highest value and the next highest value is small. In other words, when classifying cells in a cell image, the same classification result will be obtained as long as the analysis result has the largest value, regardless of whether the classification accuracy is high or low. Therefore, it may be difficult for an operator to grasp the classification accuracy of cells in a cell image at a glance. Therefore, a cell image analysis method that allows an operator to easily grasp the classification accuracy of cells in a cell image is desired.

This invention has been made to solve the above-mentioned problems, and one object of this invention is to provide a cell image analysis method that makes it possible to easily grasp the accuracy of the classification of cells shown in a cell image.

In order to achieve the above-mentioned object, a cell image analysis method according to one aspect of the present invention comprises the steps of acquiring a cell image containing a cell, inputting the cell image into a trained model that has been trained to classify cells into two or more types, acquiring an index value representing the accuracy of which of the two or more types the cell depicted in the cell image belongs to based on the analysis results of each pixel of the cell image output by the trained model, and displaying the acquired index value, wherein the trained model is trained to output, as the analysis result, a probability value that is an estimate of the classification of each pixel of the cell image, and in the step of acquiring the index value, one representative value that is one of the mean, median, maximum, minimum, and mode of the probability values for each pixel of the cell image output by the trained model is acquired as one index value representing the accuracy of classification for one cell image .

The cell image analysis method according to the above aspect includes the steps of acquiring an index value representing the accuracy of which of two or more classifications the cell image belongs to, based on the analysis results of each pixel of the cell image output by the trained model, and displaying the acquired index value. This allows the operator to easily grasp the accuracy of the classification of the cells depicted in the cell image by checking the index value, as the index value representing the accuracy of which of two or more classifications the cells depicted in the cell image belong to is displayed. As a result, a cell image analysis method can be provided that makes it possible to easily grasp the accuracy of the classification of the cells depicted in the cell image.

1 is a schematic diagram showing the overall configuration of a cell image analyzer according to an embodiment. FIG. 1 is a schematic diagram for explaining a cell image. FIG. 2 is a schematic diagram illustrating cells cultured in a culture vessel. A schematic diagram for explaining a method for learning a learning model according to one embodiment and a method for analyzing cell images using the learned model. 1A to 1C are schematic diagrams for explaining differences in cell images due to different photographing conditions. FIG. 2 is a schematic diagram for explaining a configuration in which an image processing unit according to an embodiment generates a probability distribution image. 1A to 1C are schematic diagrams for explaining a superimposed cell image generated by a superimposed cell image generating unit according to one embodiment. FIG. 10 is a schematic diagram for explaining differences in numerical data of representative values of probability values due to differences in focus of a cell image. FIG. 10 is a schematic diagram for explaining a configuration for displaying a frequency distribution of probability values when a cell image is in focus. FIG. 10 is a schematic diagram for explaining a configuration for displaying a frequency distribution of probability values when a cell image is out of focus. 1 is a schematic diagram for explaining a configuration in which a cell analysis device according to one embodiment displays a superimposed cell image, a representative value of probability values, and a frequency distribution. 1A and 1B are schematic diagrams for explaining the difference in superimposed cell images due to differences in coating agents of culture vessels in which cells are cultured. FIG. 10 is a schematic diagram for explaining differences in numerical data of representative values of probability values due to differences in coating agents. 1A to 1D are schematic diagrams illustrating the difference in superimposed cell images due to differences in the number of days of cell culture. FIG. 10 is a schematic diagram for explaining differences in the numerical data of representative values of probability values due to differences in the number of days of cell culture. 10 is a flowchart for explaining a process in which a cell image analyzing device according to one embodiment displays an index value, a frequency distribution, and a superimposed cell image. 10 is a flowchart illustrating a process for generating a trained model by a cell image analysis apparatus according to an embodiment. 10 is a flowchart for classifying cell images by a cell image analysis apparatus according to one embodiment.

Below, an embodiment of the present invention is described based on the drawings.

Referring to Figure 1, the configuration of a cell image analysis device 100 according to one embodiment will be described.

(Configuration of cell image analyzer)
As shown in FIG. 1, the cell image analyzer 100 includes an image acquisition unit 1, a processor 2, a storage unit 3, a display unit 4, and an input reception unit 5.

The image acquisition unit 1 is configured to acquire a cell image 10. The cell image 10 is an image of a cell 90 (see Figure 2). Specifically, the cell image 10 is an image of a cultured cell 90 cultured in a culture solution 81 (see Figure 3) filled in a culture vessel 80 (see Figure 3). In this embodiment, the image acquisition unit 1 is configured to acquire the cell image 10 from a device that captures the cell image 10, such as a microscope 8 equipped with an imaging device. The image acquisition unit 1 includes, for example, an input/output interface.

The processor 2 is configured to analyze the acquired cell image 10. The processor 2 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array) configured for image processing. The processor 2, which is comprised of hardware such as a CPU, also includes software (program) functional blocks: a control unit 2a, an image analysis unit 2b, an image processing unit 2c, and a superimposed cell image generation unit 2d. By executing the programs stored in the memory unit 3, the processor 2 functions as the control unit 2a, image analysis unit 2b, image processing unit 2c, and superimposed cell image generation unit 2d. The control unit 2a, image analysis unit 2b, image processing unit 2c, and superimposed cell image generation unit 2d may be individually configured by hardware with dedicated processors (processing circuits).

The control unit 2a is configured to control the cell image analysis device 100. The control unit 2a is also configured to acquire an index value 20 that represents the degree of certainty that a cell 90 depicted in a cell image 10 is classified into one of two or more types. Specifically, the control unit 2a is configured to acquire, as the index value 20, a representative value 20a of the probability value 21 acquired based on the probability value 21 (see FIG. 4) output by the trained model 6. The index value 20 is a real number that represents the degree of certainty that a cell 90 depicted in a cell image 10 is classified into one of two or more types. In this embodiment, the index value 20 is a numerical value ranging from 0 to 100. In this embodiment, the control unit 2a outputs one index value 20 for one cell image 10.

More specifically, the control unit 2a is configured to acquire at least one of the following index values 20: whether the cell image 10 was in focus when captured, whether the coating agent on the culture vessel 80 (see Figure 3) is appropriate, and whether the number of culture days is appropriate. The probability value 21 is an estimated value of the classification output by the trained model 6 as the analysis result. The trained model 6 outputs the probability value 21 for each pixel of the cell image 10 as the analysis result.

The control unit 2a is also configured to control the display of the superimposed cell image 50 on the display unit 4. Details of the configuration by which the control unit 2a acquires the index value 20 and the superimposed cell image 50 will be described later.

In this embodiment, the image analysis unit 2b classifies cells 90 (see Figure 2) into two or more types. Specifically, it is configured to classify cells 90 shown in the cell image 10 into two or more types using a trained model 6 that has learned to classify cells 90 into two or more types. The trained model 6 also includes a first trained model 6a, a second trained model 6b, and a third trained model 6c that perform classification based on the imaging conditions and culture conditions described below. Details of normal cells, abnormal cells, the first trained model 6a, the second trained model 6b, and the third trained model 6c will be described later.

The image processing unit 2c is configured to generate a probability distribution image 12 (see Figure 6) described below. The image processing unit 2c is also configured to acquire a cell region, which is the region of a cell 90 depicted in the cell image 10, based on the probability distribution image 12. Details of the configuration of the image processing unit 2c for generating the probability distribution image 12 and the configuration for acquiring the cell region will be described later.

The superimposed cell image generating unit 2d is configured to generate a superimposed cell image 50 in which the distribution of probability values 21 (see Figure 4) is superimposed on the cell image 10. Details of the configuration by which the superimposed cell image generating unit 2d generates the superimposed cell image 50 will be described later.

The memory unit 3 is configured to store the cell image 10, the first trained model 6a, the second trained model 6b, and the third trained model 6c. The memory unit 3 is also configured to store various programs executed by the processor 2. The memory unit 3 includes a storage device such as a hard disk drive (HDD) or a solid state drive (SSD).

The display unit 4 is configured to display the superimposed cell image 50, index value 20, frequency distribution 22, etc. generated by the superimposed cell image generation unit 2d. The display unit 4 includes, for example, a display device such as an LCD monitor.

The input reception unit 5 is configured to be able to receive operational input from an operator. The input reception unit 5 includes input devices such as a mouse and a keyboard.

(Cell image)
A cell image 10 will be described with reference to FIG. 2 . The cell image 10 is an image of a cultured cell 90. In this embodiment, the cell image 10 is a microscopic image captured by a microscope 8 equipped with an imaging device. The cell image 10 is an image of a cell 90 having differentiation potential as the cultured cell 90. For example, the cell 90 includes an iPS cell (induced pluripotent stem cell), an ES cell (embryonic stem cell), etc. An undifferentiated cell is a cell that has differentiation potential. An undifferentiated deviation cell is a cell that has begun to differentiate into a specific cell but no longer has differentiation potential. In this embodiment, an undifferentiated cell is considered to be a normal cell. An undifferentiated deviation cell is considered to be an abnormal cell.

(Cultured cells cultivated in a culture vessel)
Next, the cultured cells 90 cultured in the culture vessel 80 will be described with reference to FIG.

As shown in FIG. 3, cells 90 are cultured cells cultured in a culture solution 81 filled in a culture vessel 80. In this embodiment, cell image 10 is an image including cultured cells 90 cultured in the culture vessel 80. A coating agent for culturing cells 90 is applied to the bottom surface 80a of the culture vessel 80. The coating agent contains proteins and the like that are necessary for cells 90 to settle in the culture vessel 80.

(Image analysis method)
Next, a method for analyzing a cell image 10 using the cell image analysis method according to this embodiment will be described with reference to FIG. 4 . In this embodiment, a configuration will be described in which a cell image analysis device 100 (see FIG. 1 ) analyzes the cell image 10 to classify cells 90 depicted in the cell image 10 into two or more types. In this embodiment, the cell image analysis device 100 analyzes the cell image 10 using a trained model 6 (see FIG. 1 ) to determine which of the two or more types the cell 90 depicted in the cell image 10 belongs to. When the cell image 10 is input, the trained model 6 outputs a probability value 21 for each pixel of the cell image 10. The probability value 21 is an estimated value of the classification.

Figure 4 is a block diagram showing the flow of image processing according to this embodiment. As shown in Figure 4, in this embodiment, the cell image analysis method broadly includes an image analysis method 101 and a method 102 for generating a trained model 6 (see Figure 1).

(Generation of learning model)
A method 102 for generating a trained model 6 according to this embodiment generates the trained model 6 by training the training model 7 using a cell image 10. Specifically, the trained model 6 is generated by training the training model 7 to output a probability value 21 for each pixel of the cell image 10 as an analysis result. As shown in FIG. 4 , the method 102 for generating a trained model 6 includes a step 102a of inputting a teacher cell image 30 into the training model 7 and a step 102b of training the training model 7 to output a teacher correct answer image 31. The trained model 6 is, for example, a convolutional neural network (CNN) shown in FIG. 4 or includes a convolutional neural network as part of the CNN. The trained model 6 generated by training the training model 7 is stored in the memory unit 3 ( FIG. 1 ) of the cell image analysis device 100.

In this embodiment, the trained model 6 is created by training the cell image 10 to classify at least one of the following: whether the cell image 10 was in focus when captured, whether the coating agent on the culture vessel 80 was appropriate, and whether the number of days of culture was appropriate. The method 102 for generating the trained model 6 generates the trained model 6 using a teacher cell image 30, which is the cell image 10, and a teacher correct answer image 31 to which the cell image 10 is assigned label values for at least two types of capture conditions corresponding to the classification, or to which the cell image 10 is assigned label values for at least two types of culture conditions corresponding to the classification.

In this embodiment, the trained model 6 includes a first trained model 6a, a second trained model 6b, and a third trained model 6c. The first trained model 6a is a trained model that has learned to classify, based on a cell image 10, under which type of imaging condition a cell 90 depicted in the cell image 10 was captured, among images captured under two or more types of imaging conditions. In other words, the teacher cell images 30 used to generate the first trained model 6a are cell images 10 captured under different imaging conditions.

The teacher-specific answer image 31 is an image in which different label values are assigned to each pixel depending on the shooting conditions. Specifically, the teacher-specific answer image 31 is an image in which label values for two or more types of shooting conditions are assigned to each pixel. The shooting conditions are whether the cell image 10 (teacher cell image 30) is in focus or not. Therefore, the teacher-specific answer image 31 is an image in which each pixel is assigned a label value indicating that the cell image 10 is in focus and a label value indicating that the cell image 10 is out of focus when photographing it. In other words, the teacher-specific answer image 31 is an image divided into two classes: an in-focus class and an out-of-focus class. This allows the learning model 7 to learn to classify each pixel of the input image into one of two or more types under the shooting conditions, thereby generating the first trained model 6a.

The second trained model 6b and the third trained model 6c are trained models that have learned to classify, based on the cell image 10, whether the cell 90 depicted in the cell image 10 is an image of a cell 90 cultured under two or more types of culture conditions. Specifically, when generating the second trained model 6b and the third trained model 6c, cell images 10 with different culture conditions are used as the teacher cell images 30. Furthermore, an image in which different label values are assigned to each pixel depending on the culture conditions is used as the teacher correct image 31. Specifically, the teacher correct image 31 is an image in which label values for two or more types of culture conditions are assigned to each pixel. Note that the culture conditions include differences in the coating agent of the culture vessel 80 (see Figure 3) in which the cells 90 are cultured and differences in the number of days the cells 90 have been cultured.

That is, when generating the second trained model 6b, a teacher correct answer image 31 is used that has at least two types of label values associated with the coating agent of the culture vessel 80 in which the cells 90 are cultured. Specifically, when generating the second trained model 6b, an image in which each pixel is assigned a label value indicating that the coating agent of the culture vessel 80 in which the cells 90 are cultured is coating agent A and a label value indicating that the coating agent is not coating agent A is used as the teacher correct answer image 31. That is, the teacher correct answer image 31 is an image divided into two classes: a class in which the coating agent is A and a class in which the coating agent is B.

Furthermore, when generating the third trained model 6c, a teacher correct answer image 31 is used that is assigned at least two types of label values related to the number of days the cells 90 have been cultured. Specifically, when generating the third trained model 6c, an image in which each pixel is assigned a label value indicating that the number of days the cells 90 have been cultured for a predetermined number of days and a label value indicating that the number of days the cells have been cultured for is not the predetermined number of days is used as the teacher correct answer image 31. In this embodiment, the predetermined number of days of culture is, for example, five days. In other words, the teacher correct answer image 31 is an image divided into two classes: a class for which the number of days the cells have been cultured for five days and a class for which the number of days the cells have been cultured for other than five days.

In this embodiment, the trained model 6 is created by learning to classify cells 90 into two or more types based on the imaging conditions or culture conditions, thereby determining whether they are suitable for analysis as normal cells or abnormal cells. In this embodiment, the trained model 6 is also created by learning to classify cells 90 of the same type into whether they are suitable for analysis as normal cells or abnormal cells.

(Image analysis method)
The image analysis method 101 according to this embodiment is an image analysis method for classifying a cell 90 depicted in a cell image 10 acquired by an image acquisition unit 1 from a microscope 8 (see FIG. 1 ) or the like as one of two or more types. The image analysis method 101 according to this embodiment includes the steps of acquiring a cell image 10 depicting a cell 90 (see FIG. 2 ), inputting the cell image 10 to a trained model 6, acquiring an index value 20 representing the likelihood of the cell 90 depicted in the cell image 10 being classified into one of two or more types based on the analysis results of each pixel of the cell image 10 output by the trained model 6, and displaying the acquired index value 20. Detailed processing of each step of the image analysis method 101 will be described later.

In this embodiment, the step of acquiring a cell image 10 is performed by the image acquisition unit 1. The image acquisition unit 1 acquires the cell image 10 from an image capture device such as a microscope 8 (see Figure 1). The image acquisition unit 1 also outputs the acquired cell image 10 to the image analysis unit 2b. The image acquisition unit 1 also outputs the acquired cell image 10 to the superimposed cell image generation unit 2d.

In this embodiment, the step of analyzing the cell image 10 is performed by the image analysis unit 2b. The image analysis unit 2b inputs the cell image 10 to the trained model 6 to obtain the index value 20. Specifically, the image analysis unit 2b inputs the cell image 10 to any of the first trained model 6a, the second trained model 6b, and the third trained model 6c to obtain the index value 20. The control unit 2a determines whether the image analysis unit 2b will perform the analysis using the first trained model 6a, the second trained model 6b, or the third trained model 6c. The image analysis unit 2b outputs the obtained index value 20 to the control unit 2a and the superimposed cell image generation unit 2d. Specifically, the image analysis unit 2b outputs a probability value 21 as the index value 20 to the control unit 2a and the superimposed cell image generation unit 2d.

The control unit 2a determines, based on the operator's operational input, whether to perform analysis using the first trained model 6a, the second trained model 6b, or the third trained model 6c. Specifically, the control unit 2a determines, based on the operational input regarding the conditions under which the cell image 10 will be analyzed, whether to perform analysis using the first trained model 6a, the second trained model 6b, or the third trained model 6c.

The control unit 2a also acquires a representative value 20a of the probability values 21 based on the probability values 21. In this embodiment, the control unit 2a acquires one representative value 20a for one cell image 10 based on the probability values 21 acquired for each pixel of the cell image 10. In this embodiment, the control unit 2a is configured to acquire the average value of the probability values 21 as the representative value 20a.

The control unit 2a also acquires a frequency distribution 22 of the probability value 21 based on the probability value 21. The control unit 2a also causes the acquired representative value 20a and frequency distribution 22 to be displayed on the display unit 4. Details of the configuration by which the control unit 2a acquires the representative value 20a and frequency distribution 22 will be described later.

The superimposed cell image generation unit 2d generates a superimposed cell image 50 based on the cell image 10 and the index value 20. The superimposed cell image generation unit 2d also displays the generated superimposed cell image 50 on the display unit 4.

(Differences in cell images due to differences in imaging conditions)
Next, differences in the cell image 10 due to differences in imaging conditions will be described with reference to Figure 5. In this embodiment, the difference in imaging conditions is whether or not the cell image 10 is in focus when it is captured. Note that cell images 10a to 10c shown in Figures 5(A) to 5(C) were captured at the same location in the culture vessel 80 (see Figure 3) with different focal positions.

The cell image 10a shown in Figure 5(A) is an in-focus cell image 10. That is, the cell image 10a is an image in which the contrast of the cells 90 is high. In other words, the cell image 10a is an image in which the outlines of the cells 90 are clear. Note that an in-focus image does not mean that all of the cells 90 depicted in the cell image 10 are in focus, but rather that the center of the cell image 10 is in focus. In other words, the focus of the cells 90 depicted in the cell image 10a is not uniform, and the degree of defocus increases as the cells move away from the center of the image. In other words, there may be out-of-focus cells 90 in the cell image 10a.

The cell image 10b shown in Figure 5(B) is an out-of-focus cell image 10. That is, the cell image 10b shown in Figure 5(B) is an image in which the contrast of the cell 90 is low. In other words, the cell image 10b is an image in which the outline of the cell 90 is unclear. Furthermore, the cell image 10b is an image in which the degree of out-of-focus (degree of defocus) is smaller than that of the cell image 10c shown in Figure 5(C). Note that the "Defocus -1" marking in Figure 5(B) indicates that the degree of defocus of the cell image 10b is smaller than that of the cell image 10c shown in Figure 5(C). Furthermore, in the cell image 10b shown in Figure 5(B), the outline of the cell 90 is shown with a dashed line to indicate that it is out of focus. Furthermore, in the cell image 10b shown in Figure 5(B), the focus of the cell 90 in the image is not uniform, and the degree of defocus increases as the image moves away from the center of the image.

The cell image 10c shown in Figure 5(C) is an out-of-focus cell image 10. That is, the cell image 10c shown in Figure 5(C) is an image in which the contrast of the cell 90 is low. In other words, the cell image 10c is an image in which the outline of the cell 90 is unclear. Furthermore, the cell image 10c is an image in which the degree of out-of-focus (degree of defocus) is greater than that of cell image 10b. That is, the outline of the cell 90 is less clear in cell image 10c. Note that the "Defocus -2" label in Figure 5(C) indicates that the degree of defocus of cell image 10c is greater than that of cell image 10b shown in Figure 5(B). Furthermore, the cell image 10c shown in Figure 5(C) does not show the outline of the cell 90, indicating that it is more out-of-focus than cell image 10b. Also in the cell image 10b shown in FIG. 5B, the cells 90 in the image are not uniformly focused, and the degree of defocus increases as the image moves away from the center.

(Probability distribution image)
In this embodiment, the image analysis unit 2b uses the first trained model 6a to classify the cells 90 shown in the cell images 10a to 10c into two or more types.

Specifically, the image analysis unit 2b inputs cell images 10a to 10c into the first trained model 6a and generates a probability distribution image 12 based on the probability value 21 output from the first trained model 6a.

In the example shown in FIG. 6, the image analysis unit 2b obtains a probability value 21 by inputting an in-focus cell image 10a into the first trained model 6a. That is, in the example shown in FIG. 6, the image analysis unit 2b obtains a probability value 21 for each pixel of the cell image 10a. The image analysis unit 2b also outputs the obtained probability value 21 to the image processing unit 2c. In the example shown in FIG. 6, the image analysis unit 2b obtains, as the probability value 21, an estimated value of the in-focus class of each pixel of the cell image 10.

As shown in FIG. 6, the image processing unit 2c generates a probability distribution image 12, which is an image showing the distribution of probability values 21. The probability distribution image 12 is an image in which probability values 21, which are estimated values of classification, are distributed as pixel values. Note that the probability distribution image 12 shown in FIG. 6 is an image that represents the distribution of probability values 21, which are estimated values of the class in which each pixel of the cell image 10 is in focus. Note that in the example shown in FIG. 6, differences in probability values 21 are represented by differences in hatching. Note that, as shown in legend 8, the probability value 21 increases in the order of black, dark hatching, and light hatching. Also, as shown in legend 8, one hatching does not represent one probability value 21, but rather one hatching is added for each probability value 21 within a predetermined range.

Although not shown in Figure 6, the image analysis unit 2b also obtains a probability distribution image 12 showing the distribution of probability values 21, which are estimated values of the out-of-focus class, by inputting a cell image 10 into the first trained model 6a.

In addition, the image processing unit 2c similarly obtains a probability distribution image 12 of the in-focus class and a probability distribution image 12 of the out-of-focus class for the out-of-focus cell image 10b (see Figure 5) and cell image 10c (see Figure 5).

(Superimposed cell images and differences in superimposed cell images due to differences in imaging conditions)
Next, the superimposed cell image 50 (see FIG. 1) and the differences in the superimposed cell image 50 due to differences in imaging conditions will be described with reference to FIG. 7. The superimposed cell image generator 2d generates the superimposed cell image 50 based on the cell image 10 and the probability distribution image 12. Specifically, the superimposed cell image generator 2d generates the superimposed cell image 50 using the cell image 10 and the probability distribution image 12 acquired for at least two types of label values.

Specifically, the superimposed cell image generation unit 2d generates the superimposed cell image 50 by superimposing, on the cell image 10, markers that allow for identification of differences in the probability values 21 based on the probability distribution image 12. In this embodiment, the superimposed cell image generation unit 2d superimposes markers that allow for identification of the probability values 21 of label values for two or more types of classifications. Specifically, the superimposed cell image generation unit 2d superimposes markers that allow for identification of the probability values 21 of label values for two or more types of imaging conditions. More specifically, the superimposed cell image generation unit 2d superimposes markers that allow for identification of the probability values 21 of in-focus label values and the probability values 21 of out-of-focus label values on the cell image 10. For example, the superimposed cell image generation unit 2d superimposes a blue marker 51 on the probability values 21 of in-focus label values. Furthermore, the superimposed cell image generation unit 2d superimposes a red marker 52 on the probability values 21 of out-of-focus label values. In the example shown in Fig. 7, the blue marker 51 is represented by hatching with the narrowest intervals, as shown in legend 9. Also, in the example shown in Fig. 7, the red marker 52 is represented by hatching with the widest intervals, as shown in legend 9.

The superimposed cell image 50a shown in Figure 7(A) is an image in which the distribution of probability values 21 obtained by inputting the in-focus cell image 10a (see Figure 5(A)) into the first trained model 6a is superimposed on the in-focus cell image 10a (see Figure 5(A)). The superimposed cell image 50b shown in Figure 7(B) is an image in which the distribution of probability values 21 obtained by inputting the out-of-focus cell image 10b (see Figure 5(B)) into the first trained model 6a is superimposed on the out-of-focus cell image 10c (see Figure 5(C)). The example shown in Figure 7 shows a blue marker 51 superimposed on the in-focus probability values 21. Furthermore, a red marker 52 is superimposed on the out-of-focus probability value 21. Therefore, in the example shown in Fig. 7, in the area where the in-focus probability value 21 and the out-of-focus probability value 21 coexist, a blue and red gradation marker 53 is displayed as if it were superimposed. Note that in the example shown in Fig. 7, as shown in legend 9, the blue and red gradation marker 53 is represented by hatching with medium intervals.

In the superimposed cell image 50a, which is in focus, there are many areas where blue markers 51, which indicate an in-focus probability value of 21, are superimposed. In the superimposed cell image 50c, which is the most out-of-focus, there are many areas where red markers 52, which indicate a high out-of-focus probability value of 21, are superimposed. In the superimposed cell image 50b, which is less out-of-focus than the superimposed cell image 50c, there are the most areas where blue markers 51 are superimposed, followed by areas where blue-red gradation markers 53 are superimposed. In the superimposed cell image 50b, there are also areas where red markers 52 are superimposed.

(Typical values and differences in typical values due to differences in shooting conditions)
In this embodiment, as shown in FIG. 8 , the control unit 2a is configured to acquire a representative value 20a of the probability values 21. Specifically, the control unit 2a is configured to acquire numerical data of the representative value 20a of the probability values 21. That is, in this embodiment, the control unit 2a is configured to acquire one representative value 20a from the probability values 21 acquired for each pixel of the cell image 10a. Also, in this embodiment, the control unit 2a is configured to acquire a representative value 20a of the probability values 21 in a cell region as the representative value 20a of the probability values 21. The cell region is acquired by the image processing unit 2c. Specifically, the image processing unit 2c adds together probability distribution images 12 of at least two types of label values, and acquires, as the cell region, a region in the probability distribution image 12 after addition that has a predetermined probability value 21 or more.

In this embodiment, the control unit 2a acquires the representative value 20a based on the probability value 21 of the label value of one of the classifications of two or more types of imaging conditions. Specifically, the control unit 2a acquires the representative value 20a based on the probability value 21 of the label value that is in focus. In other words, the control unit 2a acquires the representative value 20a based on the probability value 21 of the label value that is suitable for analyzing whether the cell is normal or abnormal.

In this embodiment, the control unit 2a also acquires a graph that collectively displays the numerical data of multiple representative values 20a, as shown in graph 40a. In graph 40a, the horizontal axis indicates the defocus for each cell image 10, and the vertical axis indicates the representative value 20a. That is, the horizontal axis "0" in graph 40a represents an in-focus cell image 10a. The horizontal axis "-1" in graph 40a represents an out-of-focus cell image 10b. The horizontal axis "-2" in graph 40a represents an out-of-focus cell image 10c. As shown in graph 40a, the representative value 20a decreases as the focus increases.

(Frequency distribution and differences in frequency distribution due to differences in shooting conditions)
Next, the frequency distribution 22 (see FIG. 4) acquired by the control unit 2a and the difference in the frequency distribution 22 (see FIG. 4) due to differences in the shooting conditions will be described with reference to FIGS. 9 and 10. FIG.

The frequency distribution 22a shown in Figure 9 is a frequency distribution obtained based on the probability values 21 of the in-focus cell image 10a. In the frequency distribution 22a, the horizontal axis represents the probability value 21 and the vertical axis represents the frequency. That is, the frequency distribution 22a is a graph of the frequency of the probability values 21 for each pixel of the cell image 10a (see Figure 5). In addition, in the frequency distribution 22a, the probability values 21 of the first type of label value among the two or more types are hatched. That is, in the frequency distribution 22a, the probability values 21 of the in-focus class are hatched. In addition, in the frequency distribution 22a, the probability values 21 of the second type of label value, which is different from the first type among the two or more types, are not hatched and are shown in white. That is, in the frequency distribution 22a, the probability values 21 of the out-of-focus class are shown in white.

As shown in Figure 9, the frequency distribution 22a of the in-focus cell image 10a has a high frequency of pixels with a high probability value 21 of the in-focus class, so many pixels are distributed on the right side of the frequency distribution 22a. Furthermore, the frequency distribution 22a of the in-focus cell image 10a also has a high frequency of pixels with a low probability value 21 of the out-of-focus class, so many pixels are also distributed on the left side of the frequency distribution 22a.

The frequency distribution 22b shown in Figure 10 is a frequency distribution obtained based on the probability values 21 of the out-of-focus cell image 10b. In the frequency distribution 22b, the horizontal axis represents the probability value 21, and the vertical axis represents the frequency. In other words, the frequency distribution 22b is a graph of the frequency of the probability value 21 for each pixel of the cell image 10b (see Figure 5). Also, in the frequency distribution 22b, the probability values 21 of the in-focus class are hatched, and the probability values 21 of the out-of-focus class are not hatched and are shown in white.

As shown in Figure 10, the frequency distribution 22b of the out-of-focus cell image 10b has a lower frequency of pixels with a high probability value 21 for the in-focus class and a higher frequency of pixels with a low probability value 21 for the in-focus class compared to the frequency distribution 22a of the in-focus cell image 10a (see Figure 9). Therefore, the distribution of pixels with a high probability value 21 for the in-focus class is distributed across almost the entire image, rather than being biased to the right of the frequency distribution 22a. Similarly, the frequency distribution 22a of the in-focus cell image 10a has a lower frequency of pixels with a low probability value 21 for the out-of-focus class and a higher frequency of pixels with a low probability value 21 for the out-of-focus class, so that the distribution of pixels with a high probability value 21 for the in-focus class is distributed across almost the entire image, rather than being biased to the left of the frequency distribution 22a. In other words, by simply looking at the shape of the frequency distribution 22, one can easily determine which of two or more types of cells 90 appear in the cell image 10 under the imaging conditions belong to.

(Display of superimposed cell images, representative values, and frequency distribution)
In this embodiment, the control unit 2a (see FIG. 1) displays, on the display unit 4, numerical data of the representative value 20a (see FIG. 4) of the probability value 21 (see FIG. 4), and a superimposed cell image 50 in which the distribution of the probability value 21 is superimposed on the cell image 10, as shown in FIG. 11 . In this embodiment, the control unit 2a displays, on the display unit 4, the frequency distribution 22 of the probability value 21 (see FIG. 4), along with the numerical data of the representative value 20a of the probability value 21 and the superimposed cell image 50. In the example shown in FIG. 11 , the control unit 2a displays superimposed cell images 50a to 50c as the superimposed cell image 50. In addition, in the example shown in FIG. 11 , the control unit 2a displays a graph 40a as the numerical data of the representative value 20a. In addition, in the example shown in FIG. 11 , the control unit 2a displays a frequency distribution 22a as the frequency distribution 22.

(Differences in superimposed cell images due to differences in coating agents)
Next, with reference to FIG. 12 , differences in the superimposed cell image 50 (see FIG. 1 ) due to different coating agents will be described. The superimposed cell image 50d shown in FIG. 12(A) is an image generated based on the cell image 10 and a probability distribution image 12 generated based on the index value 20 obtained by analyzing the cell image 10 using the second trained model 6b. Specifically, the superimposed cell image 50d is an image generated based on the cell image 10, which is an image of a cell 90 cultured in a culture vessel 80 whose bottom surface 80a is coated with coating agent A. Note that the configuration for generating the superimposed cell image 50d is similar to the configuration for generating the superimposed cell images 50a to 50c using the first trained model 6a, except that the second trained model 6b is used instead of the first trained model 6a, and therefore a detailed description thereof will be omitted.

The superimposed cell image 50e shown in FIG. 12(B) is an image generated based on the cell image 10 and a probability distribution image 12 generated based on index values 20 obtained by analyzing the cell image 10 using the second trained model 6b. Specifically, the superimposed cell image 50e is an image generated based on a cell image 10 captured by photographing a cell 90 cultured in a culture vessel 80 in which coating agent B has been applied to the bottom surface 80a of the culture vessel 80. In this embodiment, the second trained model 6b is generated by training the model to output the probability that the coating agent applied to the bottom surface 80a of the culture vessel 80 is coating agent A as a probability value 21. Therefore, the superimposed cell image 50d shown in FIG. 12(A) has many areas in which blue markers 51 are superimposed (areas with narrowest hatching). The superimposed cell image 50e shown in FIG. 12(B) has many areas in which red markers 52 are superimposed (areas with widest hatching).

(Representative values vary depending on the coating agent)
The control unit 2a acquires a representative value 20a for each cell image 10 based on the probability value 21 output by the second trained model 6b. The control unit 2a also acquires a graph that collectively displays multiple representative values 20a, such as graph 40b shown in FIG. 13 . Graph 40b is a graph showing differences in representative values 20a due to different coating agents. In graph 40b, the horizontal axis represents the type of coating agent, and the vertical axis represents the representative value 20a. Note that the configuration by which the control unit 2a generates graph 40b is similar to the configuration by which the control unit 2a generates graph 40a, except that the probability value 21 output by the second trained model 6b is used instead of the probability value 21 output by the first trained model 6a, and therefore detailed description thereof will be omitted.

As shown in graph 40b, the representative value 20a of the cell image 10 taken of the cells 90 cultured in the culture vessel 80 coated with coating agent A is larger than the representative value 20a of the cell image 10 taken of the cells 90 cultured in the culture vessel 80 coated with coating agent B.

(Differences in superimposed cell images due to differences in culture days)
Next, with reference to FIG. 14 , differences in the superimposed cell image 50 (see FIG. 1 ) due to differences in the number of days of culture will be described. The superimposed cell image 50f shown in FIG. 14(A) is an image generated based on a cell image 10 captured from a cell 90 that has been cultured for five days. Specifically, the superimposed cell image 50f is an image generated based on the cell image 10 and a probability distribution image 12 generated based on an index value 20 obtained by analyzing the cell image 10a using the third trained model 6c. Note that the configuration for generating the superimposed cell image 50f is similar to the configuration for generating the superimposed cell images 50a to 50c using the first trained model 6a, except that the third trained model 6c is used instead of the first trained model 6a, and therefore a detailed description thereof will be omitted.

Moreover, the superimposed cell image 50g shown in Figure 14(B) is an image generated based on the cell image 10 obtained by photographing a cell 90 that has been in culture for four days. Furthermore, the superimposed cell image 50h shown in Figure 14(C) is an image generated based on the cell image 10 obtained by photographing a cell 90 that has been in culture for six days. Furthermore, the superimposed cell image 50i shown in Figure 14(D) is an image generated based on the cell image 10 obtained by photographing a cell 90 that has been in culture for seven days.

The third trained model 6c is generated by training the model to output a probability value 21 indicating whether the number of culture days is 5 days or not. Therefore, in the superimposed cell image 50f shown in FIG. 14(A), there are many areas where blue markers 51 are superimposed (areas with narrowest hatching). Furthermore, in the superimposed cell image 50g shown in FIG. 14(B), compared to the superimposed cell image 50g shown in FIG. 14(A), there are more areas where red markers 52 are superimposed (areas with widest hatching). Furthermore, as shown in FIGS. 14(C) and 14(D), as the number of culture days increases, the area where red markers 52 are superimposed becomes more prominent, and the area where blue-red gradation markers 53 are superimposed (areas with medium-spaced hatching) increases.

(Representative values vary depending on the number of days cultured)
The control unit 2a acquires a representative value 20a for each cell image 10 based on the probability value 21 output by the third trained model 6c. The control unit 2a also acquires a graph that collectively displays multiple representative values 20a, such as graph 40c shown in FIG. 15. Graph 40c is a graph that shows differences in representative values 20a due to differences in the number of culture days. In graph 40c, the horizontal axis represents the number of culture days, and the vertical axis represents the representative value 20a. Note that the configuration by which the control unit 2a generates graph 40c is similar to the configuration by which the control unit 2a generates graph 40a, except that the probability value 21 output by the third trained model 6c is used instead of the probability value 21 output by the first trained model 6a, and therefore detailed description thereof will be omitted.

As shown in graph 40c, the representative value 20a of the cell image 10 taken of cells 90 that have been in culture for five days is the highest. It can also be seen that the representative value 20a of the cell image 10 taken of cells 90 that have been in culture for other than five days is smaller than the representative value 20a of the cell image 10 taken of cells 90 that have been in culture for five days. Furthermore, when comparing the representative value 20a of the cell image 10 taken of cells 90 that have been in culture for four days with the representative value 20a of the cell image 10 taken of cells 90 that have been in culture for six days, it can be seen that the representative value 20a of the cell image 10 taken of cells 90 that have been in culture for six days is higher.

(Threshold processing of representative values)
In this embodiment, as shown in FIG. 11 , the display unit 4 displays multiple superimposed cell images 50, numerical data (graph) of the representative values 20a, and a frequency distribution 22a. Therefore, the operator can determine which of two or more types of classifications the cell 90 depicted in the cell image 10 belongs to based on the multiple superimposed cell images 50, the numerical data (graph) of the representative values 20a, and the frequency distribution 22a displayed on the display unit 4. That is, the operator can determine the classification of the cell 90 depicted in the cell image 10 that is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell under two or more types of imaging conditions or two or more types of culture conditions. Therefore, the operator can determine whether the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell. In this case, it is possible to set a threshold for determining whether the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell based on the numerical data of the representative values 20a. Therefore, in this embodiment, the control unit 2a is configured to determine whether the index value 20 is greater than a threshold value. Specifically, the control unit 2a determines whether the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell by determining whether the representative value 20a of the probability values 21 is greater than a threshold value set by the operator. For example, if the representative value 20a is greater than 50%, the control unit 2a determines that the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell. Furthermore, for example, if the representative value 20a is less than 50%, the control unit 2a determines that the cell image 10 is not suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell.

(Display processing of superimposed cell images, representative values, and frequency distribution)
Next, with reference to FIG. 16, a process in which the cell-image analyzing device 100 displays the superimposed cell image 50, the representative value 20a, and the frequency distribution 22 will be described.

In step 200, the image acquisition unit 1 acquires a cell image 10 containing a cell 90.

In step 201, the image analysis unit 2b inputs a cell image 10 into a trained model 6 that has been trained to classify cells 90 into two or more types.

In step 202, the image processing unit 2c acquires a cell area, which is the area of the cell 90 depicted in the cell image 10. In this embodiment, the image processing unit 2c acquires the cell area based on the probability distribution image 12 (see Figure 6).

In step 203, the control unit 2a acquires an index value 20 representing the degree of certainty that the cell 90 depicted in the cell image 10 is classified into one of two or more types based on the analysis results of each pixel of the cell image 10 output by the trained model 6. In this embodiment, in the processing of step 203, the control unit 2a acquires, as the index value 20, a representative value 20a of the probability value 21 acquired based on the probability value 21 output by the trained model 6. Specifically, the control unit 2a acquires at least one index value 20 representing whether the cell image 10 was in focus when captured, whether the coating agent on the culture vessel 80 is appropriate, and whether the number of days of culture is appropriate. Also, in the processing of step 203, the control unit 2a acquires, as the index value 20, a value representing the degree of certainty that the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell, based on the probability value 21.

Furthermore, in this embodiment, in the processing of step 203, the control unit 2a acquires a representative value 20a of the probability values 21 within the cell region as a representative value 20a of the probability values 21. Specifically, the control unit 2a acquires the average value of the probability values 21 as the representative value 20a.

In step 204, the control unit 2a acquires the frequency distribution 22. Specifically, the control unit 2a acquires the frequency distribution 22 based on the probability value 21 output by the trained model 6.

In step 205, the superimposed cell image generation unit 2d generates the superimposed cell image 50. Specifically, the superimposed cell image generation unit 2d generates the superimposed cell image 50 based on the cell image 10 and the probability distribution image 12 (see Figure 6) obtained based on the probability value 21.

In step 206, the control unit 2a displays the acquired index value 20. In this embodiment, in the processing of step 206, the control unit 2a displays the numerical data of the representative value 20a of the probability value 21 and a superimposed cell image 50 in which the distribution of the probability value 21 is superimposed on the cell image 10. In this embodiment, the control unit 2a displays the frequency distribution 22 of the probability value 21 together with the numerical data of the representative value 20a of the probability value 21 and the superimposed cell image 50. The processing then ends.

Note that either the processing of step 204 or the processing of step 205 may be performed first.

(Processing for generating trained models)
Next, the generation process of the trained model 6 will be described with reference to FIG. 17 .

In step 300, the image acquisition unit 1 acquires a teacher cell image 30. The teacher cell image 30 is a cell image 10.

In step 301, the image acquisition unit 1 acquires a teacher-use correct answer image 31. The teacher-use correct answer image 31 is a labeled image to which the cell image 10 is assigned label values for at least two types of shooting conditions corresponding to the classification, or to which label values for at least two types of culture conditions corresponding to the classification are assigned.

In this embodiment, when generating the first trained model 6a, in the processing of step 301, a cell image 10 to which label values for at least two types of imaging conditions corresponding to the classification are assigned is acquired as the teacher's correct answer image 31. Specifically, a cell image 10 to which, as label values related to the imaging conditions, a focused label value and an out-of-focus label value when the cell image 10 was captured are assigned for each pixel is acquired as the teacher's correct answer image 31. Note that the out-of-focus label value includes multiple label values depending on the degree of out-of-focus. In this embodiment, the out-of-focus label value includes two label values.

Furthermore, when generating the second trained model 6b, in the processing of step 301, the image acquisition unit 1 acquires, as the teacher correct answer image 31, a cell image 10 to which label values for at least two types of culture conditions corresponding to the classification have been assigned. Specifically, the image acquisition unit 1 acquires, as the teacher correct answer image 31, a cell image 10 to which at least two types of label values for the coating agent of the culture vessel 80 in which the cells 90 are cultured have been assigned. In this embodiment, the image acquisition unit 1 acquires the teacher correct answer image 31 to which two label values, a label value for coating agent A and a label value for coating agent B, have been assigned for each pixel as label values for the coating agent.

Furthermore, when generating the third trained model 6c, in the processing of step 301, the image acquisition unit 1 acquires, as the teacher correct answer image 31, a cell image 10 to which label values for at least two types of culture conditions corresponding to the classification have been assigned. Specifically, the image acquisition unit 1 acquires, as the teacher correct answer image 31, a cell image 10 to which at least two types of label values for the number of days the cells 90 have been cultured have been assigned. In this embodiment, the image acquisition unit 1 acquires the teacher correct answer image 31 in which, as label values for the number of days of culture, a label value indicating that the number of days is 5 days and a label value indicating that the number of days is other than 5 days are assigned to each pixel.

In step 302, the image processing unit 2c creates a trained model 6 using a teacher cell image 30, which is a cell image 10, and a teacher correct answer image 31 to which the cell image 10 is assigned label values for at least two types of shooting conditions corresponding to the classification, or to which label values for at least two types of culture conditions corresponding to the classification have been assigned.

In this embodiment, in the processing of step 302, the image processing unit 2c creates a trained model 6 using a training correct answer image 31 to which at least two label values related to either the imaging conditions, i.e., whether or not the cell image 10 was in focus when it was captured, or the culture conditions, i.e., the coating agent of the culture vessel 80 in which the cells 90 are cultured, and the number of days of culture, have been assigned as label values related to the imaging conditions. The processing then ends.

(Classification of cell images)
Next, the process of classifying the cell image 10 by the cell-image analyzing device 100 will be described with reference to FIG.

In step 400, the control unit 2a acquires the index value 20. In this embodiment, the control unit 2a acquires the index value 20 acquired by the image analysis unit 2b using the cell image 10 and the trained model 6.

In step 401, the control unit 2a acquires a threshold value. Specifically, the control unit 2a acquires a threshold value that has been set in advance by the operator and stored in the memory unit 3.

In step 402, the control unit 2a determines whether the index value 20 is greater than the threshold value. That is, the control unit 2a determines, for each cell image 10, whether the index value 20 is greater than the threshold value. If the index value 20 is greater than the threshold value, the process proceeds to step 403. If the index value 20 is less than the threshold value, the process proceeds to step 404.

In step 403, the control unit 2a classifies the cell image 10 into an image suitable for analyzing whether the cell 90 is a normal cell or an abnormal cell. The control unit 2a also stores the cell image 10 classified into an image suitable for analyzing whether the cell 90 is a normal cell or an abnormal cell in the memory unit 3. The processing then ends.

Also, if the process proceeds from step 402 to step 404, in step 404 the control unit 2a classifies the cell image 10 as an image that is not suitable for analyzing whether the cell 90 is a normal cell or an abnormal cell. In this case, the control unit 2a does not store the cell image 10 in the memory unit 3. The process then ends.

(Effects of this embodiment)
In this embodiment, the following effects can be obtained.

In this embodiment, as described above, the cell image analysis method includes the steps of acquiring a cell image 10 containing a cell 90, inputting the cell image 10 into a trained model 6 that has been trained to classify the cell 90 into two or more types, acquiring an index value 20 that indicates the likelihood that the cell 90 shown in the cell image 10 is one of the two or more types based on the analysis results of each pixel of the cell image 10 output by the trained model 6, and displaying the acquired index value 20.

This displays an index value 20 that indicates the accuracy of which of two or more classifications the cell 90 depicted in the cell image 10 belongs to, allowing the operator to easily grasp the accuracy of the classification of the cell 90 depicted in the cell image 10 by checking the index value 20. As a result, a cell image analysis method can be provided that makes it possible to easily grasp the accuracy of the classification of the cell 90 depicted in the cell image 10.

In addition, in the above embodiment, the following additional effects are obtained by configuring as follows:

In other words, in this embodiment, as described above, the trained model 6 is trained to output probability values 21, which are estimated values of classification, as analysis results, and in the step of acquiring index values 20, a representative value 20a of the probability values 21 acquired based on the probability values 21 output by the trained model 6 is acquired as the index value 20. As a result, unlike a configuration in which the probability value 21 is displayed for each pixel of the cell image 10, the accuracy of classification of the cell 90 depicted in the cell image 10 can be easily grasped for each cell image 10 by the representative value 20a of the probability value 21.

In addition, in this embodiment, as described above, the cell image 10 is an image containing cultured cells 90 cultured in a culture vessel 80, and the trained model 6 is created by training the model to classify at least one of the following: whether the cell image 10 was in focus when captured, whether the coating agent on the culture vessel 80 is appropriate, and whether the number of days of culture is appropriate. In the step of acquiring the index value 20, at least one of the index values 20 is acquired: whether the cell image 10 was in focus when captured, whether the coating agent on the culture vessel 80 is appropriate, and whether the number of days of culture is appropriate. This allows the operator to easily determine at least one of the following by checking the index value 20: whether the cell image 10 was in focus when captured, whether the coating agent on the culture vessel 80 is appropriate, and whether the number of days of culture is appropriate.

Furthermore, in this embodiment, as described above, the trained model 6 is created by learning to classify whether a cell 90 is suitable for analyzing whether it is a normal cell or an abnormal cell. In the step of acquiring the index value 20, a value representing the degree of certainty that the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell is acquired as the index value 20 based on the probability value 21. This displays the index value 20 representing the degree of certainty that the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell. Therefore, by checking the index value 20, the operator can easily determine whether the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell. As a result, a cell image analysis method can be provided that makes it possible to easily determine whether the cell image 10 is suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell.

Furthermore, in this embodiment, as described above, the trained model 6 is created by learning to classify cells 90 of the same type as to whether they are suitable for analysis as normal cells or abnormal cells. As a result, by analyzing cell images 10 using the trained model 6, it is possible to classify images of cells 90 of the same type as to whether they are suitable for analysis as normal cells or abnormal cells.

In addition, as described above, this embodiment further includes a step of acquiring a cell region, which is the region of the cell 90 depicted in the cell image 10, and in the step of acquiring a representative value 20a of the probability values 21, a representative value 20a of the probability values 21 within the cell region is acquired as the representative value 20a of the probability values 21. This makes it possible to suppress an increase in processing load compared to a configuration in which a representative value 20a is acquired based on the probability values 21 of all pixels in the cell image 10.

Furthermore, in this embodiment, as described above, in the step of displaying the representative values 20a of the probability values 21, the numerical data of the representative values 20a of the probability values 21 and the superimposed cell image 50 in which the distribution of the probability values 21 is superimposed on the cell image 10 are displayed. As a result, the representative values 20a of the probability values 21 are displayed, and the numerical data of the representative values 20a of the probability values 21 makes it possible to easily grasp the accuracy of classification of the cells 90 depicted in the cell image 10 for each cell image 10. Furthermore, as the superimposed cell image 50 is displayed, the accuracy of classification for each of the cells 90 depicted in the cell image 10 can be grasped from the superimposed cell image 50.

Furthermore, in this embodiment, as described above, in the step of displaying the representative value 20a of the probability value 21, the frequency distribution 22 of the probability value 21 is displayed together with the numerical data of the representative value 20a of the probability value 21 and the superimposed cell image 50. As a result, by checking the frequency distribution 22 together with the numerical data of the representative value 20a of the probability value 21 and the superimposed cell image 50, the accuracy of the classification of the cell 90 depicted in the cell image 10 can be grasped from multiple perspectives for each cell image 10.

Furthermore, in this embodiment, as described above, in the step of acquiring the representative value 20a of the probability values 21, the average value of the probability values 21 is acquired as the representative value 20a. Here, compared to a configuration in which the median of the probability values 21 is acquired as the representative value 20a, for example, when the cell image 10 includes an area that is small in area but has a very high probability (probability value 21) of being classified as the first type of the two or more types, the representative value 20a becomes the value of the first type classification. In this case, even if the cell 90 depicted in the cell image 10 is classified as the second type, which is different from the first type as a whole, the cell 90 depicted in the cell image 10 is classified as the first type of the two or more types due to the probability value 21 of a portion of the cell image 10. Therefore, by obtaining the average value of the probability values 21 as the representative value 20a as described above, when classifying the cell images 10 by classifying the cells 90 depicted in the cell images 10, it is possible to prevent the cell images 10 from being classified as one of two or more first types due to the probability values 21 of a portion of the cell images 10. As a result, by classifying the cells 90 depicted in the cell images 10, it is possible to prevent a decrease in classification accuracy when classifying the cell images 10.

In addition, this embodiment further includes a step of creating a trained model 6 using a teacher cell image 30, which is a cell image 10, and a teacher correct answer image 31 to which the cell image 10 is assigned label values for at least two types of imaging conditions corresponding to the classification, or to which the cell image 10 is assigned label values for at least two types of culture conditions corresponding to the classification. By using the teacher correct answer image 31 to which the cell image 10 is assigned label values for at least two types of imaging conditions corresponding to the classification, a trained model 6 can be generated that can be used to classify the cell image 10 as to under which of two or more imaging conditions a cell 90 depicted in the cell image 10 was photographed. Furthermore, by using the teacher correct answer image 31 to which the cell image 10 is assigned label values for at least two types of culture conditions corresponding to the classification, a trained model 6 can be generated that can be used to classify the cell image 10 as to under which of two or more culture conditions a cell 90 depicted in the cell image 10 was cultured.

Furthermore, in this embodiment, as described above, in the step of creating the trained model 6, the trained model 6 is created using a training correct answer image 31 to which two types of label values related to the imaging conditions, namely, whether or not the cell image 10 was in focus when it was captured, are assigned, or to which at least two types of label values related to the culture conditions, namely, the coating agent of the culture vessel 80 in which the cells 90 are cultured and the number of culture days, are assigned. This makes it possible to generate a trained model 6 that can be used to classify images into two or more types of categories based on the conditions of whether or not the cell image 10 is in focus, whether or not the coating agent of the culture vessel 80 is appropriate, and whether or not the number of culture days is appropriate.

Furthermore, as described above, this embodiment further includes a step of determining whether the index value 20 is greater than a threshold value. This allows a cell image 10 whose index value 20 is greater than the threshold value to be classified as an image suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell. Furthermore, a cell image 10 whose index value 20 is equal to or less than the threshold value to be classified as an image unsuitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell. Therefore, when analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell using multiple cell images 10, the analysis can be performed using only the cell images 10 suitable for analyzing whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell. As a result, a decrease in the accuracy of the analysis of whether the cell 90 depicted in the cell image 10 is a normal cell or an abnormal cell can be suppressed.

[Modification]
The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not by the description of the above embodiments, and further includes all modifications (variations) within the meaning and scope of the claims.

For example, in the above embodiment, an example of a configuration in which the control unit 2a acquires the average value of the probability values 21 as the representative value 20a was described, but the present invention is not limited to this. For example, the control unit 2a may be configured to acquire any one of the median, maximum, minimum, and mode of the probability values 21 as the representative value 20a.

In addition, in the above embodiment, an example of a configuration was shown in which the trained model 6 was created by learning to classify, as an imaging condition, whether or not the image is in focus, or whether or not the coating agent on the culture vessel 80 is appropriate, and/or whether or not the number of days the cells 90 have been cultured is appropriate. However, the present invention is not limited to this. For example, the trained model 6 may be created by learning to classify, as an imaging condition, conditions other than whether or not the image is in focus. For example, the trained model 6 may be created by learning to classify, as an imaging condition, whether or not the type of imaging device is appropriate. Furthermore, the trained model 6 may be created by learning to classify based on culture conditions other than whether or not the coating agent on the culture vessel 80 is appropriate, or whether or not the number of days the cells 90 have been cultured is appropriate. For example, the trained model 6 may be created by learning to classify based on culture conditions other than whether or not the type of culture device is appropriate. The imaging conditions and culture conditions classified by the trained model 6 may be any.

In addition, while the above embodiment illustrates an example of a configuration in which the control unit 2a acquires a representative value 20a of the probability values 21 in the cell region, the present invention is not limited to this. For example, the control unit 2a may be configured to acquire the representative value 20a based on the probability values 21 of all pixels included in the cell image 10. However, if the control unit 2a is configured to acquire a representative value 20a of the probability values 21 of all pixels included in the cell image 10, the processing load for acquiring the representative value 20a increases. Therefore, it is preferable that the control unit 2a be configured to acquire a representative value 20a of the probability values 21 in the cell region.

In addition, in the above embodiment, an example of a configuration in which the control unit 2a displays the superimposed cell image 50, the numerical data of the representative value 20a, and the frequency distribution 22 on the display unit 4 was shown, but the present invention is not limited to this. For example, the control unit 2a may be configured to display only the numerical data of the representative value 20a on the display unit 4. Furthermore, the control unit 2a may be configured to display the numerical data of the representative value 20a and the superimposed cell image 50 on the display unit 4.

In addition, in the above embodiment, when the control unit 2a displays the numerical data of the representative value 20a on the display unit 4, an example of a configuration in which the control unit 2a displays one of graphs 40a, 40b, and 40c, which collectively display the numerical data of multiple representative values 20a, was shown. However, the present invention is not limited to this. For example, when displaying the numerical data of the representative value 20a, the control unit 2a may be configured to display the numerical value of the representative value 20a itself, rather than displaying it as a graph.

In addition, while the above embodiment illustrates an example of a configuration in which the cell-image analysis device 100 generates a trained model 6, the present invention is not limited to this. For example, the cell-image analysis device 100 may be configured to use a trained model 6 generated by an image analysis device other than the cell-image analysis device 100.

In addition, in the above embodiment, an example of a configuration was shown in which the superimposed cell image generation unit 2d superimposes blue markers 51 on the probability values 21 of the label values of a first type of classification among two or more types, and superimposes red markers 52 on the probability values 21 of the label values of a second type of classification different from the first type, to generate a superimposed cell image 50, but the present invention is not limited to this. As long as the differences in the probability values 21 are distinguishable, the superimposed cell image generation unit 2d may superimpose markers of any color on the probability values 21 of each label value of two or more types of classification.

In addition, in the above embodiment, an example of a configuration in which the image processing unit 2c uses a training correct image 31 to which two types of label values, whether or not it is in focus, are assigned when generating the first trained model 6a is shown, but the present invention is not limited to this. For example, the image processing unit 2c may be configured to train the learning model 7 using a training correct image to which three or more types of label values are assigned depending on the degree of focus coincidence.

In addition, in the above embodiment, an example was shown in which the image processing unit 2c uses a training correct image 31 with two label values, indicating whether the type of coating agent is coating agent A or not, when generating the second trained model 6b, but the present invention is not limited to this. For example, the image processing unit 2c may be configured to train the learning model 7 using a training correct image with three or more label values, depending on the type of coating agent.

In addition, in the above embodiment, an example of a configuration in which the image processing unit 2c uses teacher correct images 31 with two types of label values, indicating whether the number of culture days is 5 days or not, when generating the third trained model 6c is shown, but the present invention is not limited to this. For example, the image processing unit 2c may be configured to train the learning model 7 using teacher correct images with three or more types of label values, depending on the number of culture days.

In addition, in the above embodiment, an example of a configuration in which the image acquisition unit 1 acquires the cell image 10 as the processing of step 201 was shown, but the present invention is not limited to this. For example, the image processing unit 2c may be configured to acquire the cell image 10 that was previously acquired by the image acquisition unit 1 and stored in the memory unit 3.

In addition, in the above embodiment, an example of a configuration was shown in which the control unit 2a performs processing to determine whether the index value 20 is greater than a threshold value, but the present invention is not limited to this. For example, the control unit 2a does not need to perform processing to determine whether the index value 20 is greater than a threshold value.

[Aspects]
It will be appreciated by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

(Item 1)
acquiring a cell image showing the cell;
inputting the cell image into a trained model that has been trained to classify the cells into two or more types;
acquiring an index value representing the likelihood that the cell depicted in the cell image is classified into one of two or more types based on the analysis result of each pixel of the cell image output by the trained model;
and displaying the acquired index value.

(Item 2)
the trained model is trained to output, as the analysis result, a probability value that is an estimate of the classification;
2. The cell image analysis method according to item 1, wherein in the step of acquiring the index value, a representative value of the probability values acquired based on the probability values output by the trained model is acquired as the index value.

(Item 3)
the cell image is an image including cultured cells cultured in a culture vessel,
The trained model is created by learning to classify at least one of whether the cell image is in focus when captured, whether the coating agent of the culture vessel is appropriate, and whether the number of days of culture is appropriate;
3. The cell image analysis method according to item 2, wherein in the step of acquiring the index value, at least one of the index values acquired is whether or not the cell image was in focus when captured, whether or not a coating agent for the culture vessel is appropriate, and whether or not the number of days of culture is appropriate.

(Item 4)
the trained model is created by learning to classify whether the cell is suitable for analysis as a normal cell or an abnormal cell;
4. The image analysis method according to item 2 or 3, wherein in the step of acquiring the index value, a value representing a degree of certainty that the cell image is suitable for analyzing whether the cell depicted in the cell image is the normal cell or the abnormal cell is acquired as the index value based on the probability value.

(Item 5)
5. The cell image analysis method according to item 4, wherein the trained model is created by training the model to classify cells of the same type as to whether they are suitable for analysis as normal cells or abnormal cells.

(Item 6)
further comprising a step of acquiring a cell area that is an area of the cell that appears in the cell image,
6. The cell image analysis method according to any one of items 2 to 5, wherein in the step of acquiring the representative value of the probability values, the representative value of the probability values within the cell region is acquired as the representative value of the probability values.

(Item 7)
7. The cell image analysis method according to any one of items 2 to 6, wherein in the step of displaying the representative value of the probability values, numerical data of the representative value of the probability values and a superimposed cell image in which a distribution of the probability values is superimposed on the cell image are displayed.

(Item 8)
8. The cell image analysis method according to item 7, wherein in the step of displaying the representative value of the probability values, a frequency distribution of the probability values is displayed together with the numerical data of the representative value of the probability values and the superimposed cell image.

(Item 9)
9. The cell image analysis method according to any one of items 2 to 8, wherein in the step of acquiring the representative value of the probability values, an average value of the probability values is acquired as the representative value.

(Item 10)
10. The cell image analysis method according to any one of items 1 to 9, further comprising a step of creating the trained model using a teacher cell image that is the cell image and a teacher correct answer image to which label values for at least two types of imaging conditions corresponding to the classification are assigned, or to which label values for at least two types of culture conditions corresponding to the classification are assigned.

(Item 11)
Item 11. The cell image analysis method according to Item 10, wherein in the step of creating the trained model, two types of label values related to whether the cell image was in focus when captured are assigned as the label values related to the imaging conditions, or the trained model is created using the correct training image to which at least two types of label values related to either a coating agent for a culture vessel in which the cells are cultured and the number of days of culture are assigned as the label values related to the culture conditions.

(Item 12)
12. The cell image analysis method according to any one of items 1 to 11, further comprising a step of determining whether the index value is greater than a threshold value.

6 Trained model 10 Cell image 20 Index value 20a Representative value (average value)
21 Probability value 22, 22a, 22b Frequency distribution 80 Culture vessel 81 Culture solution 90 Cells (cultured cells)

Claims (11)

acquiring a cell image showing the cell;
inputting the cell image into a trained model that has been trained to classify the cells into two or more types;
acquiring an index value representing the likelihood that the cell depicted in the cell image belongs to one of two or more types of classifications based on the analysis results of each pixel of the cell image output by the trained model;
and displaying the acquired index value ;
the trained model is trained to output, as the analysis result, a probability value that is an estimate of the classification of each pixel of the cell image;
A cell image analysis method, in which, in the step of acquiring the index value, one representative value, which is any of the mean, median, maximum, minimum, and mode of the probability values for each pixel of the cell image output by the trained model, is acquired as one index value representing the accuracy of the classification for one cell image. the cell image is an image including cultured cells cultured in a culture vessel,
The trained model is created by learning to classify at least one of whether the cell image is in focus when captured, whether the coating agent of the culture vessel is appropriate, and whether the number of days of culture is appropriate;
2. The cell image analysis method according to claim 1, wherein in the step of acquiring the index value, at least one of the index values acquired is whether the cell image was in focus when captured, whether the coating agent on the culture vessel is appropriate, and whether the number of days of culture is appropriate. the trained model is created by learning to classify whether the cell is suitable for analysis as an undifferentiated cell or an undifferentiated deviation cell ;
The cell image analysis method of claim 1, wherein in the step of acquiring the index value, a value representing the degree to which the cell image is suitable for analyzing whether the cell depicted in the cell image is the undifferentiated cell or the undifferentiated deviation cell is acquired as the index value based on the probability value. The cell image analysis method of claim 3, wherein the trained model is created by training the model to classify cells of the same type as to whether they are suitable for analysis as undifferentiated cells or undifferentiated deviation cells . further comprising a step of acquiring a cell area that is an area of the cell that appears in the cell image,
The cell image analysis method according to claim 1 , wherein in the step of acquiring the representative value of the probability values, the representative value of the probability values within the cell region is acquired as the representative value of the probability values. The cell image analysis method of claim 1 , wherein in the step of displaying the representative value of the probability values, the numerical data of the representative value of the probability values and a superimposed cell image in which the distribution of the probability values is superimposed on the cell image are displayed. The cell image analysis method according to claim 6 , wherein in the step of displaying the representative value of the probability values, a frequency distribution of the probability values is displayed together with the numerical data of the representative value of the probability values and the superimposed cell image. The cell image analysis method according to claim 1 , wherein in the step of acquiring the representative value of the probability values, an average value of the probability values is acquired as the representative value. The cell image analysis method of claim 1 further comprises a step of creating the trained model using a teacher cell image that is the cell image and a teacher correct answer image to which label values for at least two types of imaging conditions corresponding to the classification are assigned, or to which label values for at least two types of culture conditions corresponding to the classification are assigned. 10. The cell image analysis method according to claim 9, wherein in the step of creating the trained model, two types of label values, one relating to whether the cell image was in focus when it was captured or the other relating to the culture conditions, are assigned as the label values related to the imaging conditions, and the trained model is created using the correct training image to which at least two types of label values, one relating to a coating agent for a culture vessel in which the cells are cultured and the number of days of culture, are assigned as the label values related to the culture conditions . The cell image analysis method of claim 1, further comprising a step of determining whether the index value is greater than a threshold value.