JP7500945B2 - Computer program, cell counting method and cell counting device - Google Patents

Description

The present disclosure relates to a computer program, a cell counting method, and a cell counting device.

Cultured cells derived from animals, including humans, are widely used in applied fields such as regenerative medicine and drug discovery research. When using cultured cells, it is necessary to accurately determine the state of the living cells. Phase-contrast microscopes are used as a means of non-invasively observing cultured cells. Observations and recording are done using microscopic images obtained using a phase-contrast microscope.

Patent Document 1 discloses a cell analysis device that measures the number of cells in a culture vessel by image analysis. The cell analysis device of Patent Document 1 includes a cell image acquisition unit that acquires an image of cells in a culture vessel in which cells are cultured, a smoothing processing unit that smoothes the brightness values of the acquired image, a minimum value detection unit that detects minimum values of the smoothed brightness values, a minimum minimum value extraction unit that extracts the minimum minimum value that is the smallest within a range according to the size of the cells from the detected minimum values, a counting unit that counts the number of extracted minimum minimum values, and a cell number calculation unit that calculates the number of cells in the culture vessel based on the number of counted minimum values.

International Publication No. 2016/121065

The cell analysis device in Patent Document 1 has a technical problem in that it cannot accurately measure the number of cells contained in a microscopic image. When cells are seeded to perform a required test, the cells may overlap on the microscopic image. In the cell analysis device in Patent Document 1, the overlapping cells are ignored when counting. For this reason, a large number of microscopic images are required until the number of cells reaches a specified number.

The objective of the present disclosure is to provide a computer program, a cell counting method, a cell counting device, an image display processing device, and an image display processing method that can accurately measure the number of cells contained in an image obtained by capturing the image, including cells that overlap each other.

The computer according to the present embodiment is caused to perform a process of acquiring a first image obtained by imaging a plurality of cells, and a process of detecting nuclear portions of cells contained in the first image by inputting the acquired first image to a trained neural network that has been machine-learned to output a second image showing the nuclear portions of each of the one or more cells when an image including one or more cells is input, and a process of calculating the number of detected nuclear portions as the number of cells.

The cell counting method according to the present aspect obtains a first image obtained by imaging a plurality of cells, and inputs the obtained first image into a trained neural network that has been machine-learned so that when an image including one or more cells is input, a second image showing the nuclei of each of the one or more cells is output, thereby detecting the nuclei of cells included in the first image, and calculating the number of detected nuclei as the cell count.

The cell counting device according to this embodiment includes an image acquisition unit that acquires a first image obtained by imaging a plurality of cells, a nuclear portion detection unit that detects nuclear portions of cells included in the first image by inputting the acquired first image to a trained neural network that has been machine-learned to output a second image showing the nuclear portions of each of the one or more cells when an image including one or more cells is input, and a counting unit that calculates the number of detected nuclear portions as the number of cells.

The computer program according to this embodiment causes a computer to execute a process of displaying an image indicating the number of cells on a first image obtained by imaging a plurality of cells, in association with the plurality of cells, and, if there is a single cell in the first image that is not overlapped with other cells, a process of displaying an image indicating that the number of the cell is single, in association with the single cell, on the first image.

In the image display processing method according to this aspect, when a first image obtained by imaging a plurality of cells contains a clump of overlapping cells, an image indicating the number of the cells is displayed on the first image in association with the plurality of cells, and when a single cell that is not overlapping with other cells is present in the first image, an image indicating that the number of the cell is singular is superimposed on the first image in association with the single cell.

The image display processing device according to this embodiment includes a control unit that performs image processing on a first image obtained by capturing an image of multiple cells, and the control unit executes a process in which, if a clump of multiple overlapping cells is present in the first image, an image indicating the number of the cells is superimposed on the first image in association with the multiple cells, and, if a single cell that is not overlapping with other cells is present in the first image, an image indicating that the number of the cell is single is superimposed on the first image in association with the single cell.

The trained neural network generating method according to this aspect includes preparing an untrained neural network having an input layer, a convolutional layer, a convolutional layer, and an output layer, preparing training data including a first partial image including one cell and a first image showing a nuclear portion of the cell, and a second partial image including two or more cells and a second image showing a nuclear portion of the cell, and deep-training the untrained neural network so that when a partial image of the training data is input to the input layer of the untrained neural network, a corresponding image is output from the output layer.

The trained neural network generating method according to this aspect includes preparing an untrained neural network having an input layer, a convolutional layer, a convolutional layer, and an output layer, preparing training data including a first partial image including one cell and a first image showing the nuclear portion of the cell, a second partial image including two or more cells and a second image showing the nuclear portion of the cell, a third partial image including an object other than a cell, and a third image not including the nuclear portion, and deep-training the untrained neural network so that when a partial image of the training data is input to the input layer of the untrained neural network, a corresponding image is output from the output layer.

As a result of the above, it is possible to measure with high accuracy the number of cells contained in the captured image, including cells that overlap each other.

FIG. 1 is a block diagram illustrating an example of the configuration of a cell counting system according to a first embodiment. FIG. 1 is a functional block diagram of a cell counting device according to a first embodiment. FIG. 2 is an explanatory diagram showing the configuration of a trained neural network. FIG. 2 is an explanatory diagram showing the operation of a trained neural network. 5 is a flowchart showing a processing procedure of a control unit according to the first embodiment. FIG. 11 is an explanatory diagram showing an example of a microscope image. 13 is a flowchart showing a procedure of a cell region extraction process. FIG. 11 is an explanatory diagram showing an example of a mask image. FIG. 11 is an explanatory diagram showing a method for extracting a partial image relating to a cellular region. FIG. 11 is an explanatory diagram showing an example of extraction of a partial image. 13 is a flowchart showing a core portion detection process procedure. FIG. 13 is an explanatory diagram showing an example of detection of a core portion. 13 is a flowchart showing a cell counting process procedure. 13 is a flowchart showing a procedure for displaying cell count results. 13 is a flowchart showing a procedure for displaying cell count results. FIG. 13 is an explanatory diagram showing a microscope image on which the cell count and cell contour images are superimposed. FIG. 13 is an explanatory diagram showing an example of display of cell counting results. FIG. 13 is an explanatory diagram showing an enlarged partial image; FIG. 13 is an explanatory diagram showing another example of an image showing the cell count. 10 is a flowchart showing a display process procedure according to the second embodiment. FIG. 13 is an explanatory diagram showing an example of display of cell counting results according to the third embodiment. FIG. 11 is a block diagram showing a cell counting system according to a fourth embodiment. 11 is a flowchart showing a processing procedure for re-learning a trained neural network.

Specific examples of a computer program, a cell counting method, and a cell counting device according to embodiments of the present disclosure are described below with reference to the drawings. Note that the present disclosure is not limited to these examples, but is indicated by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

Hereinafter, the present disclosure will be specifically described with reference to the drawings showing the embodiments thereof.
(Embodiment 1)
1 is a block diagram illustrating an example of the configuration of a cell counting system according to embodiment 1. The cell counting system according to embodiment 1 includes a cell counting device 1, an imaging device 2, a display device 3, and an input device 4.

The cell counting device 1 is a computer. The cell counting device 1 functions as an image display processing device according to this embodiment. The cell counting device 1 includes a control unit 11 that controls the operation of each component of the computer. The control unit 11 is connected to a memory unit 12, an image input unit 13, an image output unit 14, and an input I/F 15 via a bus.

The imaging device 2 is provided in a phase-contrast microscope capable of observing a plurality of cells seeded in a culture vessel A. The imaging device 2 captures an image including a plurality of cells observed with the phase-contrast microscope, and outputs a microscopic image (first image) P0 obtained by capturing the image. Note that, in this embodiment 1, a microscopic image P0 captured using a microscope is described as an example, but the image is not necessarily limited to an image captured using a microscope. Any image that includes identifiable cells will suffice.

The storage unit 12 is a non-volatile memory such as a hard disk, an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The storage unit 12 stores a computer program 5a for causing a computer to execute a cell counting method capable of measuring the number of cells included in a microscopic image P0 with high accuracy even when some cells overlap, and an image display processing method capable of effectively displaying the cell counting results. The computer program 5a includes a trained neural network 6 (see FIGS. 3 and 4) for detecting the nuclei of cells from the microscopic image P0.
The computer program 5a according to the present embodiment may be recorded in a computer-readable manner on the recording medium 5. The storage unit 12 stores the computer program 5a read from the recording medium 5 by a reading device (not shown). The recording medium 5 is a semiconductor memory such as a flash memory. The recording medium 5 may also be an optical disk such as a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, or a BD (Blu-ray (registered trademark) Disc). The recording medium 5 may also be a magnetic disk such as a flexible disk or a hard disk, a magnetic optical disk, or the like. Furthermore, the computer program 5a according to the present embodiment may be downloaded from an external communication server (not shown) (not shown) connected to a communication network (not shown) and stored in the storage unit 12.

The control unit 11 has processors such as a CPU (Central Processing Unit), a multi-core CPU, a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), a TPU (Tensor Processing Unit), etc., and internal storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The control unit 11 is connected to a storage unit 12, an image input unit 13, an image output unit 14, and an input I/F 15 via a bus. The control unit 11 executes a computer program 5a stored in the storage unit 12, thereby performing a process of counting the cells contained in the microscopic image P0 and a process of displaying the cell counting results.

The image input unit 13 is connected to the imaging device 2. The image input unit 13 receives the microscopic image P0 captured by the imaging device 2. The control unit 11 acquires the microscopic image P0 input to the image input unit 13 and stores it in the memory unit 12.

The display device 3 is connected to the image output unit 14. The display device 3 is a liquid crystal display, an organic EL display, a CRT display, etc. The control unit 11 outputs an image showing the cell counting results to the display device 3 via the image output unit 14. The display device 3 displays the image showing the cell counting results.

An input device 4, such as a keyboard or a mouse, is connected to the input I/F 15.

Figure 2 is a functional block diagram of the cell counting device 1 according to the first embodiment. The cell counting device 1 includes, as functional units, an image acquisition unit 1a, a cell region extraction unit 1b, a nucleus portion detection unit 1c, a counting unit 1d, and a display processing unit 1e.

The image acquisition unit 1a is a functional unit that acquires the microscopic image P0 from the outside (see FIG. 6). For example, the image acquisition unit 1a acquires the microscopic image P0 from the imaging device 2. Generally, the microscopic image P0 includes some cells that are overlapping. In the following, in this embodiment, the microscopic image P0 will be described as including cells that are not overlapping and cells that are overlapping. Note that the image acquisition unit 1a may acquire the microscopic image P0 by reading the microscopic image P0 from an external storage device in which the microscopic image P0 is stored.

The cell region extraction unit 1b is a functional unit that extracts a partial image P1 of a cell region contained in the acquired microscopic image P0 (see Figures 9 and 10). For a cell that does not overlap with other cells, the cell region extraction unit 1b extracts a partial image P1 of the region that contains the single cell. For a cell in which multiple cells overlap, the cell region extraction unit 1b extracts a partial image P1 of the region that contains the multiple cells.

The nuclear part detection unit 1c is a functional unit that detects nuclear parts of cells included in the extracted partial image P1 (see FIG. 12). The nuclear part detection device according to this embodiment uses a trained neural network 6 to detect nuclear parts included in the partial image P1. The trained neural network 6 is a neural network that has been machine-trained so that when an image including one or more cells is input, a nuclear image (second image) P2 showing the nuclear parts of each of the one or more cells is output, even when multiple cells overlap. The nuclear part detection unit 1c detects nuclear parts by inputting the partial image P1 to the trained neural network 6 and having it output the nuclear image P2.

The counting unit 1d is a functional unit that counts the nuclear portions detected by the nuclear portion detection unit 1c as the number of cells contained in the partial image P1. Even if multiple cells overlap, if the nuclei do not overlap, the number of cells can be calculated with higher accuracy by counting the nuclear portions.

The display processing unit 1e is a functional unit that outputs an image showing the cell counting results to the display device 3 (see Figs. 16 and 17). For example, the display processing unit 1e displays a symbol image C1 showing the number of cells in symbols for each partial image P1 extracted as a cell region, superimposed near the cell image. In addition, the display processing unit 1e displays an image showing the contours of the extracted cells, superimposed on the microscope image P0.

Figure 3 is an explanatory diagram showing the configuration of the trained neural network 6, and Figure 4 is an explanatory diagram showing the operation of the trained neural network 6. The trained neural network 6 is, for example, a U-Net. U-Net is a type of autoencoder. With U-Net, it is possible to perform semantic segmentation, that is, labeling of objects contained in an image on an image-by-image basis. In this embodiment, when an image of a cell is input to the trained neural network 6, a nuclear image P2 is output, which is a segmentation map image that represents the nuclear part of the cell and the background on a pixel-by-pixel basis using different pixel values.

The trained neural network 6 includes an input layer 61, a convolution layer 62, a deconvolution layer 63, and an output layer 64. A partial image P1, that is, an image of a cellular region including an image of one or more cells, is input to the input layer 61.
The convolution layer 62 is a layer that performs dimensional compression of image data and extracts image features. For example, as shown in FIG. 4, the data of the input partial image P1 is subjected to convolution processing and pooling processing multiple times to perform dimensional compression. In FIG. 4, the white rightward arrow indicates that the convolution processing is being performed, and the black downward arrow indicates that the pooling processing is being performed. The thin horizontal arrow indicates that the data of the image to be convolution processed is being passed from the convolution layer 62 to the deconvolution layer 63 of the same layer.
The deconvolution layer 63 is a layer that restores the data dimensionally compressed by the convolution layer 62 to the original dimension. As shown in Fig. 4, the dimensionally compressed data is subjected to unpooling and convolution processes multiple times to restore the data to the original dimension. Through this restoration, the input partial image P1 is restored as a core image P2 that represents the characteristics of the partial image P1. In Fig. 4, a white rightward arrow indicates that a convolution process is being performed, and a black upward arrow indicates that an unpooling process is being performed.
The output layer 64 outputs the nuclear image P2 from which features have been extracted in the convolution layer 62 and the deconvolution layer 63.

Nuclear image P2 is an image in which the pixels corresponding to the cells are of class "1" (white image) and the pixels corresponding to the background image are of class "0" (black image) (see Figure 12).

The trained neural network 6 is generated as follows.
First, learning data is prepared. The learning data is a set of a partial image P1 as shown in the upper part of FIG. 12 and a nuclear image P2 (teacher data) as shown in the lower part of FIG. 12. The learning data includes a partial image P1 including one cell and a nuclear image P2 created based on the partial image P1. The learning data also includes a partial image P1 including a plurality of cells and a nuclear image P2 created based on the partial image P1. The learning data also includes a partial image P1 including an object other than a cell and a nuclear image P2 created based on the partial image P1.
Then, the partial image P1 for learning is input to an untrained neural network, and the neural network is subjected to deep learning so that a core image P2 is output from the neural network.

Figure 5 is a flowchart showing the processing procedure of the control unit 11 according to the first embodiment. The control unit 11 acquires a microscopic image P0 from the imaging device 2 (step S1).

Figure 6 is an explanatory diagram showing an example of a microscopic image P0. The microscopic image P0 includes images of multiple cells. As described above, the microscopic image P0 includes individual cells that are not overlapping with other cells, and images of multiple cells that are overlapping with each other.

Next, the control unit 11 executes a cell region extraction process to extract a partial image P1 of the cell region from the microscopic image P0 (step S2).Then, the control unit 11 executes a nucleus portion detection process (step S3) and executes a cell number counting process (step S4).The control unit 11 executes a display process of the cell counting result (step S5) and ends the process.
The details of the nucleation process are described below.

Figure 7 is a flowchart showing the cell region extraction process procedure, Figure 8 is an explanatory diagram showing an example of a mask image M, Figure 9 is an explanatory diagram showing a method for extracting a partial image P1 relating to a cell region, and Figure 10 is an explanatory diagram showing an example of extraction of the partial image P1.

The control unit 11 performs edge detection, binarization, and morphology processing on the microscope image P0 to generate a mask image M showing a cellular region (step S11). The generated mask images M are labeled with numerical values. As shown in FIG. 8, the mask image M is an image in which the parts corresponding to the cells are "1" (white) and the background part is "0" (black).
8 shows a mask image M of a single detail, but the control unit 11 also uses similar processing to generate a mask image M for an overlapping mass of multiple cells. In the mask image M, multiple cells are not distinguished, and the entire mass of overlapping cells becomes "1" (white).
Furthermore, at the stage of generating the mask image M, the control unit 11 does not determine whether or not an object is a cell, so a mask image M showing an object other than a cell may also be generated.

Next, the control unit 11 assigns 1 to the variable N (step S12). The variable N is a numerical value indicating each of the multiple mask images M. As shown in FIG. 9, the control unit 11 cuts out an image of the area corresponding to the Nth mask image M from the microscope image P0 (step S13), and creates a partial image P1 by converting the area outside the mask area to black (step S14).

Then, the control unit 11 determines whether the variable N is the total number of mask images M (step S15). That is, it determines whether partial images P1 of all cell regions have been extracted. If it is determined that the variable N is not the total number of mask images M (step S15: NO), the control unit 11 adds 1 to N (step S16) and returns the process to step S13. If it is determined that the variable N is the total number of mask images M (step S15: YES), the control unit 11 ends the cell region extraction process.

By the series of processes shown in FIG. 7, various partial images P1 contained in the microscopic image P0 are extracted. For example, as shown in FIG. 10, partial image P11 containing two overlapping cells, partial image P12 containing three overlapping cells, partial image P13 of one cell, partial image P14 containing objects other than cells, etc. are extracted.

Figure 11 is a flowchart showing the core part detection process, and Figure 12 is an explanatory diagram showing an example of core part detection.

The control unit 11 assigns 1 to the variable N (step S31). The variable N is a numerical value indicating the total number of partial images P1. The control unit 11 selects the Nth partial image P1 (cell region) (step S32). The control unit 11 determines whether the partial image P1 is equal to or larger than a predetermined size (step S33). The predetermined size is, for example, the upper limit of the size of an image that can be input to the trained neural network 6.

If it is determined that the partial image P1 is equal to or larger than the predetermined size (step S33: YES), the control unit 11 stores information indicating that the nuclear image P2 is undetectable in association with the partial image P1 (step S36).

When it is determined that the partial image P1 is smaller than the predetermined size (step S33: NO), the control unit 11 inputs the partial image P1 to the trained neural network 6 as shown in FIG. 12 (step S34). The control unit 11 stores the nuclear image P2 obtained by the arithmetic processing by the trained neural network 6, that is, the nuclear image P2 representing the nuclear portion, in association with the partial image P1 (step S35).

The control unit 11 determines whether the variable N is the total number of partial images P1 (step S37). That is, it determines whether the core portion detection process has been completed for all partial images P1. If it is determined that the variable N is not the total number of partial images P1 (step S37: NO), the control unit 11 adds 1 to N (step S38) and returns the process to step S32. If it is determined that the variable N is the total number of partial images P1 (step S37: YES), the control unit 11 ends the core portion detection process.

By performing the series of processes shown in FIG. 11, the nuclear portion of the cell contained in each partial image P1 is extracted, except for cases where detection is not possible.

Figure 13 is a flowchart showing the cell counting process procedure. The control unit 11 assigns 1 to the variable N (step S51). The variable N is a numerical value indicating the nuclear image P2. The control unit 11 selects the Nth nuclear image P2 (step S52). The control unit 11 calculates the number of nuclear portions contained in the nuclear image P2 (step S53). Next, the control unit 11 stores the number of cells in association with the nuclear image P2 and the partial image P1 (step S54).

The control unit 11 determines whether the variable N is the total number of partial images P1 (step S55). In other words, it determines whether the calculation process for the number of cells has been completed for all nuclear images P2. If it is determined that the variable N is not the total number of partial images P1 (step S55: NO), the control unit 11 adds 1 to N (step S56) and returns the process to step S52. If it is determined that the variable N is the total number of partial images P1 (step S55: YES), the control unit 11 ends the cell counting process.

Figures 14 and 15 are flowcharts showing the display process procedure for cell counting results, Figure 16 is an explanatory diagram showing a microscope image P0 on which the cell count and cell outline image B are superimposed, Figure 17 is an explanatory diagram showing an example of the display of cell counting results, and Figure 18 is an explanatory diagram showing an enlarged partial image P1.

The control unit 11 assigns 1 to the variable N (step S71). The variable N is a variable that indicates each of the multiple partial images P1. The control unit 11 selects the Nth partial image P1 (step S72) and creates a contour image B that indicates the contour of the cell contained in the partial image P1 (step S73). The contour image B is a line image that indicates the boundary between the cell and the background image. For example, the control unit 11 can generate the contour image B by performing an expansion or contraction process on the mask image M and taking the difference between the mask image M after expansion and the mask image M before expansion.

Then, the control unit 11 superimposes the contour image B on the microscopic image P0 as shown in FIG. 16 (step S74). The control unit 11 determines whether or not the number of cells corresponding to the partial image P1 has been stored (step S75). If it is determined that the number of cells has been stored (step S75: YES), the control unit 11 superimposes a symbol image C1 indicating the number of cells with a symbol at the position of the microscopic image P0 corresponding to the partial image P1 (step S76). The symbol image C1 indicating the number of cells is, for example, an image of Arabic numerals.

For example, the control unit 11 is preferably configured to display the partial image P1 at an image position in the microscope image P0 that corresponds to a predetermined position, such as a corner or the center, of the partial image P1.
The control unit 11 can identify the center of gravity of the cell by the mask image M. Since there is a high possibility that a nucleus is present at the center of gravity of the cell, it is preferable to superimpose the symbol image C1 indicating the number of cells while avoiding the center of gravity.
Since the control unit 11 can identify the position of the nuclear portion in the partial image P1 based on the nuclear image P2, it is preferable to superimpose the symbol image C1 indicating the cell count while avoiding the position of the nuclear portion.

Fig. 19 is an explanatory diagram showing another example of the symbol image C1 indicating the number of cells. Instead of the symbol image C1 indicating the number of cells, the control unit 11 may display a graphic image C2 indicating the number of cells in a graphic form on the microscope image P0, as shown in Fig. 19A. The graphic image C2 includes, for example, the same number of block images as the number of cells.
Furthermore, the control unit 11 may display a color image C3, the color of which varies depending on the number of cells, on the microscope image P0, as shown in FIG. 19B.

If it is determined in step S75 that the cell count is not stored (step S75: NO), the control unit 11 superimposes a predetermined image C0 indicating that the nuclear image P2 could not be detected at an appropriate location on the partial image P1 (step S77). The predetermined image C0 is, for example, an alphabetic character indicating "no." As with the symbol image C1 that indicates the cell count with a symbol, it is preferable that the predetermined image C0 is superimposed while avoiding the center of gravity and nuclear portion of the cells.

Next, the control unit 11 determines whether the variable N is the total number of partial images P1 (step S78). If it is determined that the variable N is not the total number of partial images P1 (step S78: NO), the control unit 11 adds 1 to the variable N (step S79) and returns the process to step S72.

When it is determined that the variable N is the total number of partial images P1 (step S78: YES), the control unit 11 displays the microscopic image P0 and the superimposed image P0' side by side as shown in FIG. 17 (step S80). The superimposed image P0' is an image obtained by superimposing the contour image B and the cell image, etc. on the microscopic image P0 by the above-mentioned processing. Note that the orientation in which the microscopic image P0 and the superimposed image P0' are arranged is not limited to the horizontal direction, and they may be arranged vertically. The microscopic image P0 and the superimposed image P0' may be different sizes, but it is preferable that they are the same size.

Then, the control unit 11 accepts the enlarged display area on the input device 4 (step S81). The user can select the area in the microscope image P0 that the user wants to enlarge by operating the input device 4. For example, the user can specify the area to be enlarged by double-clicking the cell part with the mouse, dragging and dropping the cell area, selecting with a frame, etc. The control unit 11 accepts the enlarged display area specified by the user.

18, the control unit 11 extracts the received enlarged display area from the microscope image P0 and the superimposed image P0' and displays it in an enlarged manner (step S82). When enlarging the display, it is preferable to display the enlarged microscope image P3 extracted from the microscope image P0 and enlarged, and the enlarged superimposed image P3' extracted from the superimposed image P0' and enlarged, side by side. It is also preferable to display the enlarged image P3 side by side at the same magnification.
In this embodiment, when the control unit 11 displays the enlarged microscope image P3 and the enlarged superimposed image P3', it displays the number of cells contained in the enlarged image, as well as an increase button 71 and a decrease button 72 for correcting the number of cells, and also displays an end button 73 for ending the correction.

Next, the control unit 11 accepts a correction to the number of cells via the input device 4 (step S83). When the increase button 71 is operated, the control unit 11 makes a correction to increase the number of cells. When the decrease button 72 is operated, the control unit 11 makes a correction to decrease the number of cells.

If the end button 73 is operated, the cell count correction process ends, and the process proceeds to step S84. The control unit 11 determines whether the cell count has been corrected (step S84). If it is determined that the cell count has been corrected (step S84: YES), the control unit 11 stores the corrected cell count in association with the partial image P1 (step S85). If the process of step S85 has been completed, or if it is determined that no correction has been made (step S84: NO), the control unit 11 ends the enlarged display process (step S86) and determines whether to end the display and confirmation process of the cell counting results (step S87). The user can use the input device 4 to instruct the display and confirmation process of the cell counting results to end.

If it is determined that the display process should not be ended (step S87: NO), the control unit 11 returns the process to step S81 and continues the process related to the enlarged display and the acceptance of corrections to the cell count. If it is determined that the display process should be ended (step S87: YES), the control unit 11 ends the process related to the display of the cell count results.

The computer program 5a, cell counting method, cell counting device 1, image display processing method, and image display processing device of embodiment 1 configured in this manner can measure the number of cells contained in the microscopic image P0, including cells that overlap each other, with high accuracy.

Furthermore, according to this embodiment 1, the nuclear portion can be accurately detected from the partial image P1 of a living cell that is not fluorescently stained. In a microscope image P0 that is not fluorescently stained, for example, a microscope image P0 obtained using a phase-contrast microscope, the nuclear portion is clear and it is difficult to find the feature amount of the nuclear portion. Therefore, the cell counting device 1 according to this embodiment 1 is equipped with a trained neural network 6. The cell counting device 1 detects the nuclear portion from the partial image P1 using the trained neural network 6, and therefore can accurately detect the nuclear portion of a living cell.

Furthermore, the cell counting device 1 is configured to extract a partial image P1 of an area containing a single cell or a partial image P1 of an area containing multiple overlapping cells from the microscopic image P0, and input it to the trained neural network 6. Therefore, the cell counting device 1 can detect nuclear portions more efficiently and accurately than a configuration in which a large microscopic image P0 is input to the trained neural network 6.

Furthermore, the cell counting device 1 is configured to extract partial image P1 that includes objects other than cells, and input it to the trained neural network 6. In other words, because the trained neural network 6 is configured to distinguish between cells and objects other than cells, the process of extracting partial image P1 is simple, and the distinction between cells and the above-mentioned objects can be made with high accuracy.

Furthermore, the cell counting device 1 is configured to generate a mask image M and extract a partial image P1 by extracting an image of the cell portion. In other words, the partial image P1 does not contain any extra information other than the cell portion. Therefore, the cell counting device 1 can detect the nuclear portion of the cell with higher accuracy by inputting the partial image P1 into the trained neural network 6.

Furthermore, the cell counting device 1 is configured to input a partial image P1 of less than a predetermined size to the trained neural network 6, so that the detection accuracy of the nuclear portion can be guaranteed to be constant.

Furthermore, the cell counting device 1 can display a symbol image C1, a graphic image C2, and a color image C3 indicating the number of cells at a position on the microscope image P0 corresponding to the partial image P1. Therefore, the user can check the number of cells contained in the image area corresponding to the partial image P1 by using the symbol image C1, graphic image C2, and color image C3.

Furthermore, the cell counting device 1 can superimpose the symbol image C1, graphic image C2, and color image C3 while avoiding the center position and nucleus of the cell. The image of the nucleus of the cell is not hidden by the symbol image C1, graphic image C2, or color image C3. Therefore, there is no risk of the user being hindered in checking the cell count, and the cell count can be checked efficiently.

Furthermore, if the cell nucleus is not detected from the partial image P1, the cell counting device 1 can display a specified image C0 on the microscope image P0 in association with the partial image P1. Therefore, the user can recognize the cell images for which cell counting has not been performed.

Furthermore, the cell counting device 1 can superimpose the cell contour image B on the microscope image P0. Therefore, the user can easily recognize the shape of the cells detected by the cell counting device 1.
The cell counting device 1 can superimpose the contour image B of the object other than the cell on the microscope image P0. Therefore, the user can easily recognize the shape of the object other than the cell detected by the cell counting device 1.
The user can also intuitively know whether the calculation of the cell number has been successful or unsuccessful based on the shape of the contour image B.

Furthermore, the cell counting device 1 can display the microscopic image P0 and the superimposed image P0' side by side. The user can check the number of cells contained in the microscopic image P0 by comparing the microscopic image P0 and the superimposed image P0'.

Furthermore, the cell counting device 1 can accept modifications to the number of cells contained in the partial image P1 and correct the number of cells.

Furthermore, the microscope image P0 and the superimposed image P0' can be enlarged. The user can confirm and correct the cell count using the microscope image P0 and the superimposed image P0'.

In this embodiment 1, an example has been described in which the cell counting device 1 is separate from the phase contrast microscope and the imaging device 2, but the cell counting device 1 may be provided at any location in a phase contrast microscope system including a phase contrast microscope, the imaging device 2, and a computer for image processing. For example, the cell counting device 1 and the imaging device 2 may be configured as one unit. Also, the phase contrast microscope, the cell counting device 1, and the imaging device 2 may be configured as one unit.

In the first embodiment, the cell counting device 1, which is a computer in which the computer program 5a is installed, is disposed next to a phase-contrast microscope, but the computer program 5a may be configured to operate on a cloud server. The cloud server operates as the cell counting device 1.
The phase-contrast microscope system includes a phase-contrast microscope, an imaging device 2, and a communication device that transmits a microscope image P0 to a cloud server. The phase-contrast microscope system transmits the microscope image P0 captured by the imaging device 2 to the cloud server. The cloud server receives the microscope image P0, executes the same processing as the cell counting device 1 according to the first embodiment, and transmits data indicating the cell counting result or an image indicating the cell counting result to the phase-contrast microscope system. The phase-contrast microscope system receives the data indicating the cell counting result or the image indicating the cell counting result, and displays the cell counting result.

(Embodiment 2)
The computer program 5a, cell counting method, cell counting device 1, image display processing method, and image display processing device according to the second embodiment differ from the first embodiment in the configuration for automatically enlarging and displaying partial images P1 in which the number of cells was undetectable and partial images P1 in which the number of cells is multiple, and accepting corrections to the number of cells, and therefore the above differences will be mainly described below. The other configurations and effects are the same as those of the first embodiment, so the same reference numerals are used for corresponding parts and detailed descriptions are omitted.

FIG. 20 is a flowchart showing a display process procedure according to the second embodiment. The control unit 11 executes the same processes as steps S71 to S75 of the first embodiment. The control unit 11 judges whether the number of cells corresponding to the partial image P1 is stored (step S75). When it is judged that the number of cells is stored (step S75: YES), the control unit 11 judges whether the number of cells stored is a plurality of cells (step S201). When it is judged that the number of cells is a single number (step S201: NO), the control unit 11 superimposes a symbol image C1 indicating the number of cells by a symbol on the position of the microscope image P0 corresponding to the partial image P1 in the same manner as the process of step S76 of the first embodiment (step S206). When it is judged that the number of cells is not stored (step S75: NO) or when it is judged that the number of cells stored is a plurality of cells (step S201: YES), the control unit 11 enlarges and displays the microscope image P0 and the superimposed image P0' (step S202). The control unit 11 accepts the correction of the cell number from the input device 4 (step S203), and stores the corrected cell number in association with the partial image P1 (step S204). When the end button 73 is operated, the control unit 11 ends the correction process of the cell number and the enlarged display process (step S205), and superimposes a symbol image C1 that indicates the number of cells by a symbol at the position of the microscope image P0 corresponding to the partial image P1, similar to the process of step S76 in the first embodiment (step S206).
After completing the process of step S206, the control unit 11 executes the same processes as those of step S78 and thereafter in the first embodiment.

According to the computer program 5a, cell counting method, cell counting device 1, image display processing method, and image display processing device of embodiment 1 configured in this manner, when there are multiple cells or when the nucleus cannot be detected, the magnified microscope image P3 and magnified superimposed image P3' of the partial image P1 that requires confirmation are automatically displayed, allowing the user to efficiently confirm and correct the cell count.

(Embodiment 3)
The computer program 5a, cell counting method, cell counting device 1, image display processing method, and image display processing device according to the third embodiment differ from those of the first embodiment in that an image showing the type of cells is displayed on the microscope image P0 in addition to the number of cells, and therefore the above differences will be mainly described below. The other configurations and effects are the same as those of the first embodiment, so the same reference numerals are used for corresponding parts and detailed descriptions will be omitted.

The control unit 11 according to the third embodiment has a function of detecting the type of cell contained in the partial image P1. The control unit 11 may use the trained neural network 6 to identify the type of cell.
For example, when a partial image P1 is input to a first trained neural network, deep learning is performed so that an image of a cell related to type α is output. Also, when a partial image P1 is input to a second trained neural network, deep learning is performed so that an image of a cell related to type β is output. By inputting the partial image P1 to each of the first and second trained neural networks, information indicating the cell types α and β can be obtained.
As another method, when a partial image P1 is input to the trained neural network 6, it may be trained to output a nuclear image P2 in which pixels corresponding to cells of cell type α are of class "1" (white image), pixels corresponding to cells of cell type β are of class "2" (red image), and pixels corresponding to the background image are of class "0" (black image). By inputting the partial image to such a trained neural network 6, information indicating the cell types α and β can be obtained.

Figure 21 is an explanatory diagram showing an example of the display of cell counting results according to embodiment 3. In Figure 21, ellipses indicate cells of type α, and squares indicate cells of type β. The control unit 11 generates a superimposed image P0' by superimposing the cell types α and β contained in each partial image P1 and a symbol image C1 indicating the number of cells on the microscope image P0, and outputs the generated superimposed image P0' to the display device 3.

The computer program 5a, cell counting method, cell counting device 1, image display processing method, and image display processing device according to embodiment 1 configured in this manner allow the user to check the cell types α and β along with the number of cells by referring to the superimposed image P0'.

(Embodiment 4)
The computer program 5a, cell counting method, cell counting device 1, image display processing method, and image display processing device according to embodiment 4 differ from embodiment 1 in that the cell counting device 1 is configured to transmit original training images for the trained neural network 6 to an external server 9 and retrain the trained neural network 6, and therefore the above differences will be mainly described below. The other configurations and effects are the same as those of embodiment 1, so the same reference numerals are used for corresponding parts and detailed descriptions will be omitted.

Figure 22 is a block diagram showing a cell counting system according to embodiment 4. The cell counting device 1 of the cell counting system according to embodiment 4 has the same configuration as that of embodiment 1, and further includes a communication unit 16. The communication unit 16 is a communication device that communicates with the external server 9 in a wired or wireless manner.

The external server 9 collects original training data from one or more cell counting devices 1 for retraining the trained neural network 6, and retrains the trained neural network 6 using the collected original training data. The external server 9 distributes update data for the trained neural network 6 after retraining to the cell counting device 1.

Figure 23 is a flowchart showing the processing steps involved in re-learning the trained neural network 6. The control unit 11 of the cell counting device 1 stores the partial image P1 of the cell region where detection of each part has failed in the memory unit 12 as original data for re-learning (step S411). When a predetermined amount of original data for re-learning has been stored, the control unit 11 transmits the original data for re-learning stored in the memory unit 12 to the external server 9 via the communication unit 16 (step S412).

The external server 9 receives the original data for re-learning transmitted from the cell counting device 1 (step S413). The external server 9 then creates re-learning data from the original data for learning, i.e., the partial image P1 (step S414). The re-learning data is data for the partial image P1 that indicates whether the pixels of the partial image P1 are nuclear parts or background.

Next, the external server 9 retrains the trained neural network 6 using the retraining data (step S415). The trained neural network 6 to be retrained is the same as the one already used by the cell counting device 1.

The external server 9 transmits update data for updating the trained neural network 6 to the cell counting device 1 (step S416). The update data is information such as weighting coefficients that define the trained neural network 6.

The cell counting device 1 receives the update data distributed from the external server 9 (step S417) and updates the trained neural network 6 (step S418).

According to embodiment 4, re-learning data can be collected from the cell counting device 1, and the trained neural network 6 can be re-trained.

(Embodiment 5)
The computer program 5a, cell counting method, cell counting device 1, image display processing method, and image display processing device according to embodiment 5 differ from embodiment 1 in that they are provided with a function for supporting the creation of a confirmation report of the cell counting results, and therefore the above differences will be mainly described below. The other configurations and effects are the same as those of embodiment 1, so the same reference numerals are used for corresponding parts and detailed descriptions will be omitted.

The cell counting device 1 displays, for example, the magnified microscope image P3 and the magnified superimposed image P3' of each of the multiple partial images P1 and the confirmation result input section arranged vertically and horizontally on the display device 3. The confirmation result input section displays the number of cells counted in each partial image P. The confirmation result input section also includes a change operation section that accepts changes to the cell count, a check box for checking whether or not confirmation has been performed, and the like. The user can modify the cell count by operating the change operation section of the confirmation result input section. Also, by checking the check box, it is possible to record whether or not the cell count has been confirmed. The check box is not a required component. The cell counting device 1 also displays a confirmation operator information input section on the display device 3. The confirmation operator information input section is a text input box or the like for accepting input of the name of the confirmation operator, the confirmation date, and the like.

The cell counting device 1 stores information on the cell count, information indicating whether corrections were made, information indicating whether confirmation of the cell count was performed, information on the person who performed the confirmation, information on the date of confirmation, etc., in association with the multiple partial images P1. The cell counting device 1 outputs report information in which each piece of stored information is arranged in an arbitrary format.

According to the fifth embodiment, it is possible to output report information summarizing information related to the results of the confirmation work of the cell counting results.

REFERENCE SIGNS LIST 1 Cell counting device 1a Image acquisition section 1b Cell region extraction section 1c Nucleus portion detection section 1d Counting section 1e Display processing section 2 Imaging device 3 Display device 4 Input device 5 Recording medium 6 Trained neural network 5a Computer program 9 External server 11 Control section 12 Storage section 13 Image input section 14 Image output section 15 Input I/F
16 Communication unit 61 Input layer 62 Convolution layer 63 Deconvolution layer 64 Output layer 71 Increase button 72 Decrease button 73 End button A Culture vessel B Contour image M Mask image C0 Predetermined image C1 Symbol image C2 Graphic image C3 Color image P0 Microscope image P0' Superimposed image P1, P11, P12, P13, P14 Partial image P2 Nuclear image P3 Enlarged microscope image P3' Enlarged superimposed image

Claims (31)

On the computer,
A process of acquiring captured images obtained by capturing images of a plurality of cells;
an extraction process for extracting a partial image of a region including one cell and a partial image of a region including a plurality of overlapping cells from the captured image;
a nuclear portion detection process for detecting nuclear portions of cells included in each partial image by inputting each extracted partial image to a trained neural network that has been machine-learned so that when a partial image of a region including a single cell or a partial image of a region including multiple overlapping cells is input, a nuclear image is output in which the nuclear portion and background of each of the single cell or the multiple cells are represented in pixel units using different pixel values;
A counting process for calculating the number of detected nuclear portions as the cell number. The trained neural network is
The system includes an input layer to which a partial image of a region including a single cell or a partial image of a region including multiple overlapping cells is input , a convolution layer and a deconvolution layer to extract image features, and an output layer to output a nuclear image representing the single cell or each of the multiple cells.
2. The computer program product of claim 1. The extraction process includes:
Furthermore, extract partial images that contain objects other than cells.
3. A computer program according to claim 1 or 2 . The extraction process includes:
A mask image showing a cellular region is generated by performing edge detection processing, binarization processing, and morphology processing on the captured image, and the partial image is extracted using the generated mask image.
A computer program according to any one of claims 1 to 3 . The partial image having a size smaller than a predetermined size is input to the trained neural network.
A computer program according to any one of claims 1 to 4 . The computer includes:
A process of displaying an image indicating the number of cells contained in the partial image on the captured image in association with the partial image is executed.
A computer program according to any one of claims 1 to 5 . The image showing the number of cells is
The partial image includes at least one of a symbol image that indicates the number of cells included in the partial image by a symbol, a graphic image that indicates the number by a graphic, and a color image that has a different color according to the number.
7. A computer program according to claim 6 . The computer includes:
When the nucleus of the cell is not detected from the partial image, a process of displaying a predetermined image on the captured image in association with the partial image is executed.
A computer program according to any one of claims 1 to 7 . The computer includes:
A process of detecting a type of cell contained in the partial image;
The computer program product according to claim 1 , further comprising: a process of displaying an image indicating a type of cell on the captured image in association with the partial image. The computer includes:
A process of creating an image showing the outline of a cell included in the partial image;
The computer program product according to claim 1 , further comprising: a process of displaying the image showing a contour of a cell on the captured image . The computer includes:
creating an image showing the contours of objects other than cells contained in the partial image;
The computer program product according to claim 1 , further comprising: a process of displaying the image showing an outline of an object other than a cell on the captured image . The computer includes:
Executing a process of displaying, side by side, the captured image in which an image showing the number of cells included in the partial image is not displayed and the captured image in which an image showing the number of cells included in the partial image is displayed.
A computer program according to any one of claims 6 to 11 . The computer includes:
A process of accepting a correction to the number of cells included in the partial image calculated by the counting process;
The computer program product according to claim 1 , further comprising: a process of correcting the number of cells included in the partial image based on a result of the reception. The computer includes:
A process of accepting an area of the captured image to be enlarged and displayed;
The computer program product according to claim 1 , further comprising: a process of extracting the accepted region from the captured image and enlarging the region. The computer includes:
A process of extracting the partial image including a plurality of cells from the captured image and enlarging it is executed.
A computer program according to any one of claims 1 to 14 . The computer includes:
A process is executed to extract the partial image in which the nucleus of the cell cannot be detected from the captured image and enlarge it.
A computer program according to any one of claims 1 to 15 . The computer includes:
When the number of cells is corrected, a process of storing the partial image in which the number of cells is corrected as original data for re-learning of the trained neural network;
The computer program product according to claim 1 , further comprising: a process of transmitting the stored re-learning original data to an external server. Acquiring captured images of a plurality of cells;
A partial image of a region including one cell and a partial image of a region including a plurality of overlapping cells are extracted from the captured image.
A trained neural network that has been trained to output a nuclear image in which the nuclear portion and background of each of the single cell or the multiple cells are represented in pixel units using different pixel values when a partial image of a region including a single cell or a partial image of a region including multiple overlapping cells is input, detects the nuclear portion of the cell included in each partial image by inputting each of the extracted partial images ;
A cell counting method in which the number of detected nuclei is calculated as the cell number. an image acquisition unit that acquires captured images obtained by imaging a plurality of cells;
an extracting unit that extracts a partial image of a region including one cell and a partial image of a region including a plurality of overlapping cells from the captured image;
a nuclear part detection unit that detects nuclear parts of cells included in each partial image by inputting each extracted partial image into a trained neural network that has been machine-trained so that, when a partial image of a region including a single cell or a partial image of a region including multiple overlapping cells is input, a nuclear image is output in which the nuclear parts and background of the single cell or the multiple cells are represented in pixel units using different pixel values;
and a counting unit that calculates the number of detected nuclear portions as the number of cells. On the computer,
a process of displaying an image indicating the number of cells included in a partial image of an area including a plurality of overlapping cells in a captured image obtained by capturing an image of a plurality of cells, in association with the partial image on the captured image ;
and if there is a single cell in the captured image that is not overlapping with other cells, a process of displaying an image on the captured image indicating that the number of the cell is single, in correspondence with a partial image of the area containing the single cell. The image showing the number of cells is
The computer program according to claim 20 , comprising at least one of a symbol image indicating the number of cells, a graphic image indicating the number of cells , and an image having a different color depending on the number of cells . An image showing the number of cells is displayed at a position away from the center of gravity of the cells.
22. A computer program according to claim 20 or claim 21 . An image showing the number of cells is displayed at a position outside the nucleus of the cells.
A computer program according to any one of claims 20 to 22 . The computer includes:
When the captured image includes a partial image in which the nucleus of a cell is unclear, a process is executed to display a predetermined image on the captured image in association with the partial image.
A computer program according to any one of claims 20 to 23 . The computer includes:
A process is executed in which an image showing the type of the cell is displayed on the captured image in association with each of a plurality of cells included in the captured image .
A computer program according to any one of claims 20 to 24 . The computer includes:
A process of creating an image showing the outline of a cell included in the captured image ;
The computer program product according to any one of claims 20 to 25, further comprising: a process of displaying the image showing a contour of a cell on the captured image . The computer includes:
creating an image showing the contours of objects other than cells included in the captured image ;
27. The computer program product according to claim 20 , further comprising: a process of displaying an image showing an outline of an object other than a cell on the captured image . The computer includes:
The computer program according to any one of claims 20 to 27, further comprising a process for displaying, side by side, the captured image on which the image indicating the number of cells is not superimposed and the captured image on which the image indicating the number of cells is superimposed. The computer includes:
Execute a process of extracting and enlarging a partial area of the captured image.
A computer program according to any one of claims 20 to 28 . The computer includes:
A process of extracting a partial image including a plurality of cells from the captured image and enlarging the partial image is executed.
30. A computer program according to any one of claims 20 to 29 . The computer includes:
When the captured image contains a partial image in which the nucleus of a cell is unclear, a process is executed to extract the partial image from the captured image and enlarge it.
A computer program according to any one of claims 20 to 30 .

Images