JP6138935B2 - Cell image processing apparatus, cell image recognition apparatus, and cell image recognition method - Google Patents

Description

  The present invention relates to a cell image processing technique.

  Currently, there is a cell culture technique for culturing cells ex vivo and enhancing their functions into tissues and organs. Cell culture technology enables the production of test cells with high reproducibility of biological phenomena and high operability and economy. In industrial fields such as pharmaceuticals, the use of test cells is expected to accelerate research and development and clinical trials. In addition, cell culture technology is expected to be applied to regenerative medicine that aims to treat diseases by transplanting cells cultured outside the body. The cell culture device is described in Patent Document 1, for example. Cells and tissues for regenerative medicine are manufactured by skilled engineers in cell processing facilities that require a high degree of cleanability.

JP 2012-130297 A

  However, such a manufacturing form has a problem that the manufacturing cost is high, the degree of dependence on the skill level of engineers is high, and the risk of biological contamination is high. For example, in currently available cell culture devices, engineers regularly check the cell status to check the cell growth status, and if there is a problem, the technician adjusts the culture environment. is necessary. Furthermore, the contents to be checked vary depending on the stage of cell growth. For example, it is necessary to check whether or not the cells are growing smoothly in the early and middle stages of the culture and in the later stage, the quality of the grown cells. For this reason, control according to the growth stage is necessary.

  If monitoring of the state of the cells according to each growth stage can be substituted by image recognition, it is possible to reduce periodic manual checks, dependence on the skill level of engineers, and the risk of biological contamination.

  Therefore, the present invention proposes a technique capable of automatically monitoring the growth state of cells by performing image processing on microscopic images taken at each stage during cell culture.

  In order to solve the above problems, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above-mentioned problems. To give an example of this, “a background separation image and a cell contour image are generated from an image of cells cultured in a cell culture device, and the background The noise component included in the separated image is removed by the cell outline image, the noise component contained in the cell outline image is removed by the background separated image, and the background separated image and the cell outline image from which the noise component has been removed are individually used. A cell recognition unit for recognizing a cell region of the cell, a cell growth stage determination unit for determining a cell growth stage based on the recognition result, and an abnormality based on the number of cells and the cell position in the first stage of the cell growth stage In the later stage of the cell growth stage, an abnormality determination unit that determines the presence / absence of an abnormality based on the cell occupancy rate and an abnormality in the presence / absence determination of the abnormality are An abnormal state processing unit that issues a control or alarm, and determines whether or not the cell culture by the cell culture device is completed based on the cell occupancy rate, and / or outputs a quality evaluation value based on the cell occupancy rate A cell image processing apparatus having a completion determination unit.

  ADVANTAGE OF THE INVENTION According to this invention, the growth condition of the cell in a cell culture apparatus can be monitored automatically and quantitatively. This reduces costs (labor costs), dependence on the skill level of engineers, and the risk of biological contamination as compared with the case where cell growth is regularly checked manually. Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments.

The figure which shows the main components of the cell culture apparatus which concerns on an Example. The whole figure of the cell culture device concerning an example. The front view of the cell culture apparatus which concerns on an Example. The side view of the cell culture apparatus which concerns on an Example. The figure explaining the flow of the process performed at the time of culture | cultivation. The figure explaining the flow of cell image processing. The figure which shows the example of a microscope image (input image). The figure explaining the production | generation process of the cell separation image by the schematic diagram of a cell, and a cell outline image. 5 is a flowchart showing a specific example of background separation processing 602. The figure which shows the example of the background separation image obtained by a binarization process. The figure which shows the example of a background separation image after applying a background isolated point removal process to the image shown in FIG. The figure which shows the example of the background separation image A after applying the intracellular hole-filling process to the image shown in FIG. The figure which shows the example of the cell outline image A. The figure which changed the black pixel of the part corresponding to the black pixel of the background separation image A among input images. The figure which changed the black pixel of the part corresponding to the black pixel of the background separation image B among input images. The figure which shows the schematic diagram of the cell outline image C. FIG. The figure which shows the example of the cell outline image C. FIG. The figure explaining the image of a cell segmentation process. 12 is a flowchart illustrating another specific example of the background separation process 602.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment of the present invention is not limited to the embodiments described later, and various modifications can be made within the scope of the technical idea.

  Here, an embodiment of a cell culture device equipped with an image processing function (cell image processing function) proposed in this specification will be described with reference to the drawings.

(Device configuration)
FIG. 1 shows main components of a cell culture device 101 according to the present embodiment. A cell culture device 101 shown in FIG. 1 includes an input device 102, a display device 103, an image acquisition device 104, a communication device 105, a computing device (CPU) 106, an external storage device 107, a culture vessel 108, and an environment control device 109. Yes.

  The input device 102 is a keyboard, mouse, or the like for inputting commands and the like. The input device 102 is used to input a command executed for controlling a program executed by the arithmetic unit (CPU) 106 and a device connected to the cell culture device 101. The display device 103 is a device such as a display that displays processing contents as appropriate. The image acquisition device 104 is an image acquisition device such as a microscope or a camera. The image acquisition device 104 is used to acquire an image obtained by enlarging cells in the culture vessel 108 with a microscope. The acquired image may be stored in an external storage device or the like. The communication device 105 is used to exchange data with an external device such as a PC or a server. The communication device 105 is used for purposes such as acquisition of an execution command by an user from an external device, acquisition of data such as an image and text from an external device, and the like. The communication device 105 is also used for the purpose of transmitting processing contents and images in the cell culture device 101 to an external device.

  The computing device (CPU) 106 is a computing device that executes cell image recognition processing and the like. The external storage device 107 is a storage device such as an HDD or a memory. The external storage device 107 stores various data such as images captured by the image acquisition device 104, images being processed, and processing results. The external storage device 107 is also used for temporary storage of data or the like generated during processing executed by the arithmetic unit (CPU) 106. Note that the cell culture device 101 does not have to include the input device 102, the display device 103, the image acquisition device 104, and the communication device 105. For example, when the input device 102 is not provided, the start of processing is instructed from an external device using the communication device 105, or is automatically performed by time designation or the like. If the display device 103 is not provided, the processing result is transmitted to an external device using the communication device 105 or stored in the external storage device 107.

  Output and input of a module that executes processing may be performed via the external storage device 107. That is, when the first module outputs the processing result to the second module and the second module receives the processing result as an input, the first module actually sends the processing result to the external storage device 107. The second module may obtain the output result of the first module from the external storage device 107.

  The cell culture apparatus 101 according to the present embodiment is also equipped with a mechanism unit and a control unit (not shown) for seeding a cell seed in a culture container and growing the cell to produce a cell sheet. Further, the cell culture apparatus 101 according to the present embodiment is equipped with an image processing function (cell image processing function) described later so that the growth state of the cells can be automatically and quantitatively monitored.

  Here, the cell culture device 101 periodically and automatically checks the cell growth status, and if there is a problem with the check result, realizes a corresponding operation such as automatically adjusting the culture environment. There is a need. For example, when cell growth is in the early and middle stages, the cell culture device 101 is required to have a function of checking whether the cells are growing along a normal growth curve. In this case, a normal growth curve given in advance is used as a criterion. In addition, when cell growth is late, the cell culture device 101 is required to have a function of checking whether or not the hole (the portion in the sheet where there is no cell) is small in order to maintain the quality of the cell sheet. The

  For this reason, as an image processing function (cell image processing function) proposed in this embodiment, a function of counting (counting) the number of cells in the early and middle stages of cell amplification and recognizing a part having no cells in the later stage. prepare. However, the shape of the cell appearing in the cell image to be processed is not limited to a relatively simple shape such as an ellipse or a line, and it is also necessary to handle cells having complicated shapes and different sizes. In addition, in a cell image, it is assumed that it is difficult to correctly recognize individual cells, for example, when cells are in contact with each other or when features such as brightness and color that separate cells are poor. An image processing function (cell image processing function) described later is provided with a mechanism for accurately recognizing individual cells even for cell images that are generally difficult to recognize.

  Subsequently, the automatic culture apparatus 210 will be described with reference to FIGS. The automatic culture apparatus 210 is one example of the cell culture apparatus 101 shown in FIG. 2 is an overall view of the automatic culture apparatus 210, FIG. 3 is a front view of the cell culture apparatus, and FIG. 4 is a side view of the cell culture apparatus. 2 to 4, the same parts are denoted by the same reference numerals.

  A cell culture chamber door 211 and an intermediate chamber door 217 are attached to the front side of the casing of the automatic culture apparatus 210 so as to be openable and closable. A cell culture chamber 213 is provided behind the cell culture chamber door 211, and an intermediate chamber 216 is disposed behind the intermediate chamber door 217. These doors are provided with a stopper and a heat insulating mechanism for sealing the inside when they are closed. By closing the cell culture chamber door 211 and the intermediate chamber door 217, it is possible to maintain the airtightness and temperature inside the automatic culture apparatus 210.

  A refrigerator 214 is disposed inside the intermediate chamber 216, and a refrigerator door 212 is attached to the front side thereof so as to be opened and closed. The refrigerator door 212 is also provided with the same mechanism as the door described above. A control unit 215 is disposed below the automatic culture apparatus 210. The control unit 215 is independent of other compartments and blocks the temperature, humidity, and carbon dioxide in the cell culture chamber 213.

  The cell culture chamber 213 includes a culture vessel 220, a culture vessel base 221 that holds the culture vessel, a rotation mechanism 222 that rotationally drives the culture vessel base 221, a pump 223, a valve 224, a tank 225, a drive base 227, a microscope 228, a stage. 229, a connector board 230, a connector 231, a flow path 240, a tube 241, a syringe 243, and a syringe pump 244 are arranged. Among these, the culture vessel 220 corresponds to the culture vessel 108 (FIG. 1), and the microscope 228 corresponds to the image acquisition device 104 (FIG. 1) as an imaging device.

  A medium base 232 is disposed in the refrigerator 214. Between the cell culture chamber 213 and the refrigerator 214, a seal 233 that seals both spaces is disposed. Note that a claw portion 234 for installing the seal 233 is provided on the walls of the cell culture chamber 213 and the refrigerator 214.

  In addition, the automatic culture apparatus 210 includes a fan (for an intermediate chamber) 250, a filter (for an intermediate chamber) 251, a fan (for a control unit) 252, an intake filter 253, and an exhaust gas used for adjusting temperature, humidity, and the like. A filter 254 is provided. In FIG. 4, the air flow in the intermediate chamber 216 is indicated by an arrow 255, and the air flow in the control unit 215 is indicated by an arrow 256.

  Here, a general term for various environmental control devices that control air conditioning equipment such as fans corresponds to the environmental control device 109 (FIG. 1). Further, the microscope 228 may incorporate the arithmetic device 106 (FIG. 1) and the external storage device 107 (FIG. 1). The processing result of the arithmetic device 106 may be transmitted from the cell culture device 101 to an external computer through the communication device 105, and the processing result may be displayed on the external computer. In addition, an image captured by the microscope 228 is transmitted to an external computer (a computer including the arithmetic device 106, the external storage device 107, the input device 102, and the display device 103) through the communication device 105, and the image is processed in the external computer. May be.

(Flow of processing executed during culture)
Here, the flow of the culture process performed by the automatic culture apparatus 210 will be described with reference to FIG. The automatic culture apparatus 210 according to the present embodiment periodically captures cells in culture, recognizes the position of the cells through image processing of the captured images, counts the number of cells, and occupies the cells. Calculate. In addition, the automatic culture apparatus 210 determines the growth stage of the cells in culture based on the processing results, changes the report content according to the growth stage, changes the abnormality determination method, and outputs an alarm when abnormal. And execute environmental control.

  The following processing is realized through execution of a program in the arithmetic device 106. First, the automatic culture apparatus 210 seeds cell seeds in the culture vessel 220 (501). Next, the automatic culture device 210 captures an enlarged image of the cells in the culture vessel 220 using the microscope 228, and the arithmetic device 106 recognizes the position of the cells in the image (502). The computing device 106 also counts the number of recognized cells, calculates the area occupied by the cells, and calculates the ratio occupied by the cells (occupation rate). This calculation process is programmed so that the first execution time is preset and the process (502) is started after a certain time from the first execution time or periodically according to the growth state of the cells. The

  Next, the computing device 106 determines a cell growth stage based on the calculated number of cells, occupation rate, elapsed time, and the like (503). Below, some examples of determination criteria will be described. For example, when the cell type and the seeding density are determined in advance, the growth stage can be determined based on the elapsed time. If these pieces of information are not determined, it may be determined that the growth phase has been entered later when the cell occupancy rate exceeds a certain threshold. Also, the occupation rate and the number of cells are recorded every hour, and if the increase in the number of cells or the rate of increase in the occupation rate slows down (when the increase rate falls below a certain threshold value), it is judged that it has entered the later stage of the growth stage. May be. In other cases, it is determined as the initial period or the middle period.

  Incidentally, when it is determined that the cell is in the early or middle stage of growth, whether or not the number of cells is increasing according to the growth curve, whether or not the number of dead cells is abnormal is used as a criterion. On the other hand, when it is determined that it is in the later stage of the growth stage, the size of the unoccupied portion of the cell and the number of cells are used as the determination criteria.

  In the above-described determination, when it is determined that the growth stage is in the initial or middle period, the arithmetic unit 106 records the number of cells counted up to the current time and the recognized cell position as a determination history (504). The computing device 106 refers to the determination history of the number of cells and the cell position up to the current time, and determines whether there is an abnormality in cell growth (506).

  First, the case where the number is used as a determination criterion will be described. In this case, the arithmetic unit 106 evaluates based on a difference from a preset normal growth curve (number of cells), a difference in the shape of the growth curve, and the like, and determines that an abnormality occurs when the difference exceeds a certain threshold value. To do. When evaluating based on the difference in the shape of the growth curve, for example, the scale of the actual growth curve before the current time is changed so that it approximates the shape of the normal growth curve, and the same time based on the normal growth curve. There is a method for obtaining the difference between the number of cells and the actual number of cells. Another possible method is to obtain the least square error of the number of cells before the current time of the normal growth curve and the actual growth curve.

  Next, a case where the cell position is used as a determination criterion will be described. The computing device 106 can track individual cells based on the recorded information of individual cell positions in the past. In this case, the arithmetic unit 106 can determine that the cell with little movement is dead. This number of dead cells is also a criterion for determining whether or not there is an abnormality.

  The computing device 106 determines whether or not there is an abnormality in cell growth based on the difference from the normal growth curve as described above, the number of dead cells, and the like.

  When it is determined that the growth stage is late, the computing device 106 records the size, number, etc. of the cell non-occupied areas as a determination history (505). The computing device 106 determines whether the quality is abnormal based on the size, number, etc. of the cell non-occupied areas up to the current time (506).

  In the later stage of the growth stage, the smaller the part without cells (the part not occupied by cells), the better the quality. For example, the computing device 106 has an area of a certain size or more among the connected components of the non-occupied region, whether the non-occupancy rate is greater than a certain threshold, When referring to the past history, it is determined whether or not there is an abnormality in cell growth based on whether there is a connected component of the unoccupied region that is not easily filled even after a lapse of time.

  When it is determined that there is an abnormality, the arithmetic device 106 performs two operations. As one of the operations, the arithmetic unit 106 executes a control operation (environment control) for automatically adjusting the culture environment such as temperature and humidity (507). The contents of the control operation are programmed in advance based on experience. Further, the arithmetic unit 106 executes alarm notification processing as one of the operations (508). The alarm notification is executed when it is determined that manual control is necessary, such as when it is determined that automatic environmental control is impossible. Here, when it is determined that automatic environmental control is impossible, for example, a case where the degree of abnormality is significant or a situation cannot be improved from a history of past control results of processing 507 is assumed. The alarm is notified to the worker through a display device or a communication device.

  After the environmental control or when it is not determined to be abnormal, the arithmetic unit 106 proceeds to one of the processes 509 and 510 depending on the growth stage. When the growth stage is in the initial stage or the middle stage, the computing device 106 outputs the number of cells, the number of deaths, the growth curve of the cells so far, etc., which are useful information at this stage (509). After a predetermined time has elapsed since the execution of the process 509, the arithmetic unit 106 returns to the process 502. When the growth stage is the latter stage, the arithmetic unit 106 outputs the occupation rate, quality evaluation value, etc., which are useful information at this stage (510). Here, the quality evaluation value includes the number of connected components in the non-occupied region, the size of the maximum component among the connected components in the non-occupied region, the unoccupied rate, the history of the unoccupied rate of each connected component, and the like.

  If the growth stage is late, it is further determined whether or not the cell culture can be terminated (511). That is, completion determination is executed. For example, when the quality evaluation value clears a certain standard, the arithmetic unit 106 determines that the quality does not improve even if a longer time elapses (the rate at which the unoccupied area becomes smaller with reference to the past history) It is determined that the culture has been completed. When it is determined that the culture is completed, the arithmetic device 106 notifies the completion of the culture through a display device, a communication device, etc. (512). On the other hand, when the culture is not completed, the arithmetic unit 106 returns to the process 502 after a predetermined time of completion determination.

(Cell image recognition processing)
The flow of cell image processing according to the present embodiment will be described with reference to FIG. This cell image processing is executed by the arithmetic device 106 in the processing 502 (FIG. 5). The arithmetic device 106 acquires, as the input image 601, a microscopic image of cultured cells taken in the culture device. In the cell image processing, the position of each cell is recognized from the microscope image, and the number of cells is counted (counting).

  By the way, there are the following technical difficulties in the calculation device 106 automatically recognizing the position of each cell from the microscope image and counting (counting) the number of cells. First, the shape of the cell handled by the cell culture device is not limited to a relatively simple shape such as an ellipse or a line, and it is assumed that it is necessary to handle a cell having a complicated shape. When the shape is complicated, a method such as a pattern matching method that is assumed in advance cannot be used for cell recognition. In addition, the size of cells to be handled is not constant but varies, and there is a possibility that a plurality of cells overlap and contact each other. In addition, the microscopic image may have poor color and brightness characteristics that separate the interior of the cell from the background, and binarization based on color and brightness cannot accurately separate the cell and background. There is a case.

  Therefore, in the cell image processing of the present embodiment, the background separation processing and the cell contour are performed so that the generation accuracy of the background separation image and the cell contour extraction image is improved even in the situation where it is difficult to separate the cell portion and the background portion as described above. Adopt a method of combining each other while taking advantage of each other's advantages of the extraction process.

  For example, the background separation image may include a region that is recognized as the inside of the cell even though it is originally a region outside the cell. Therefore, in the cell image processing of the present embodiment, a method of removing the noise component of the background separation image by removing the component inside the cell that is outside the cell outline from the outline of the cell outline image is adopted.

  In addition, the cell contour image includes a background noise component and a noise component due to the internal structure of the cell. The background noise component removes the contour located in the background area of the background separation image, and the noise component inside the cell removes the non-contact connected component from the background of the background separation image. The method of removing the noise component contained in is adopted.

  In the cell image processing of this embodiment, the cell cluster in the background separated image from which the noise component has been removed is separated into individual cells using the cell contour image from which the noise component has been removed, and the position of each cell is determined. Recognize and count the number of cells.

(Input image 601)
The input image 601 is a microscopic image of cells cultured in the culture vessel 220 of the automatic culture apparatus 210. An image 701 shown in FIG. 7 represents an image obtained by converting the input image 601 into a gray scale and further converting it into a newspaper printing format. The actual input image is color, and there is unevenness in light due to uneven illumination. Also, the distinct color features that separate the cell interior and cell exterior are generally poor. Among the cell boundaries, the boundary between the background and the cell part has a high brightness due to light hitting the convex part, but it is only a part of the boundary and does not surround the whole cell. In addition, the intensity of the boundary brightness varies depending on the location. Further, it can be seen that the shape of the cells is generally complicated, the shape is different from cell to cell, and there is contact between cells. Therefore, in order to accurately count the number of cells, it is not sufficient to calculate the cell occupancy rate.

  As will become apparent later, according to the cell image processing of this embodiment, the position of each individual cell can be accurately recognized from the cell image even under such a disadvantageous condition for image processing. Moreover, if the position of each cell can be recognized, the division status and movement status of each cell can be tracked. Thereby, in the cell image processing of the present embodiment, additional information for automatically determining the cell growth state can be obtained.

(Background separation processing 602 / Background separation image 603)
The background separation process 602 is a process of separating a cell internal area and a background area in the input image 601 and generating a “background separated image A” 603. The “background separated image A” 603 is an image in which the internal area of the cell is expressed by white pixels and the background area is expressed by black pixels. In the background separation process 602, it is necessary to distinguish the internal area of the cell from the background area by some method, but there is a possibility that a background part having a short distance from the cell is also determined as the inside of the cell. Various factors such as those caused by the imaging system and those caused by the processing method of image processing can be considered as factors that cause such erroneous determination.

  A creation image of “background separation image A” will be schematically described with reference to FIG. Here, it is assumed that the cells in the input image 601 are represented in an elliptical shape as illustrated in a schematic diagram 801. The actual shape of the cell is not simple as shown in the schematic diagram 801, but for easy understanding, the cell is represented by an elliptical shape.

  Next, a specific example of the background separation process 602 will be described with reference to FIG. In the background separation process 602, the input image 601 is separated into an internal region and an external region of the cell by extracting irregularities on the image due to the internal structure of the cell. First, the arithmetic unit 106 converts the input image 601 into an HSV expression in the HSV conversion process 901. When expressing the color of an image, RGB expression (a method of expressing a color with three colors of red (R), green (G), and blue (B)) is common, but here, HSV expression (hue ( H), saturation (S), and lightness (V).

  In the next graying process 902, the arithmetic unit 106 converts the image into a gray image based on the brightness. The lightness (V) is given by a value from 0.0 to 1.0. By re-expressing this value to 256 gradation values from 0 to 255, a gray image is generated. For example, when the brightness is v, the integer part of the value calculated by 255 * v is output. Note that “*” is a multiplication operator. By the HSV conversion process 901 and the graying process 902 described above, the input image 601 is converted into a gray image of 256 gradations.

  However, when the input image 601 is a gray image, these processes are unnecessary. Also, other graying processing methods such as a graying process based on light intensity instead of lightness, and a graying process based on a value obtained by adding lightness and lightness with weights may be used.

  In the next unevenness extraction process 903, the arithmetic unit 106 extracts uneven portions due to the internal structure of the cell from the image. For this process, a method of calculating the smoothness of the image can be used. For example, a “Sobel filter” can be used. The calculation by the “Sobel filter” is performed as follows.

  When f (i, j) is a pixel value in the pixel (i, j) of the gray image, the horizontal gradient fx (i, j) and the vertical gradient fy (i, j) by the “Sobel filter” are Calculated as follows:

[Formula 1]
Fx (i, j)
= {F (i + 1, j + 1) + 2f (i + 1, j) + f (i + 1, j-1) -f (i-1, j + 1) -2f (i-1, j) -f (i-1, j-1)}
[Formula 2]
Fy (i, j)
= {F (i + 1, j + 1) + 2f (i, j + 1) + f (i-1, j + 1) -f (i + 1, j-1) -2f (i, j- 1) -f (i-1, j-1)}
Here, the pixel value f outside the image is set to 0 for the pixel (i, j) located at the boundary of the image. Thus, the smoothness g (i, j) at the pixel (i, j) can be calculated by the following equation.

[Formula 3]
g (i, j) = fx (i, j) × fx (i, j) + fy (i, j) × fy (i, j)
In the next binarization processing 904, the arithmetic unit 106 determines that a white pixel is obtained if the value of g (i, j) is equal to or greater than a predetermined threshold value, and a black pixel is otherwise obtained. As a result, a background separated image 1001 as shown in FIG. 10 is obtained. In the image at this stage, there is a portion that is clearly a background region but a white pixel due to local uneven noise in the background portion. In addition, some white pixels may protrude from the background outside the cell, resulting in a white pixel area.

  An image after binarization processing is shown in a schematic diagram 802 of FIG. In the schematic diagram 802, the portion determined to be inside the cell is represented by white, the portion determined to be the background is represented by black, and the hatched portion is determined to be inside the cell, but actually represents the portion that is the background. There are several factors that cause the white pixels in the shaded area to protrude outside the cell. For example, it is a problem of the imaging system. Depending on how the light hits, the resolution at the time of shooting, the pseudo color, etc., there are cases where irregularities occur about several pixels from the actual boundary. Further, for example, when calculating the smoothness of the image as in the previous equation, the surrounding pixel values must be taken into account, and therefore, unevenness may occur by several pixels.

  In the background isolated point removal processing 905, the arithmetic unit 106 calculates a connected component of white pixels, and removes background noise by removing those below a certain size (making them black pixels). FIG. 11 shows a background isolated image 1101 obtained by removing background noise from the background isolated image 1001 (FIG. 10). As can be seen by comparing FIG. 10 and FIG. 11, isolated points are removed from the background of the background separated image 1101. However, although the background separated image 1101 is clearly a region inside the cell, there is a portion represented by black pixels due to the influence of a locally smooth portion inside the cell. Also in the schematic diagram 802 of FIG. 8, black pixels remaining inside the cell are represented by black dots.

  In the intra-cell filling process 906, the arithmetic unit 106 calculates a black pixel connected component and removes a black pixel block having a certain size or less (conversion to a white pixel), contrary to the background isolated point removal process 905. To fill the inside of the cell. As a result, a background separated image 1201 shown in FIG. 12 is obtained. This is the “background separated image A” 603 in FIG.

  Through the series of processes described above (that is, the background separation process 602), the “background separation image A” 603 is obtained from the input image 601. However, as described above, “background separated image A” includes a region in which white pixels protrude somewhat from the background outside the cell and are white pixels even though they are the background. A schematic diagram 803 of FIG. 8 shows an image of “background separated image A”. A hatched portion in the schematic diagram 803 indicates a region which is a white pixel although it is the background.

(Contour extraction processing 604 / contour extraction image 605)
In the cell image 601, as described above, it is assumed that illumination is applied to the convex portion at the boundary portion of the cell and the brightness is high. Also in the aforementioned input image 701 (FIG. 7), light hits the convex portion at the boundary of the cell portion, and the brightness is high. In the contour extraction process 604, in order to extract the portion with high brightness, pixels whose brightness changes greatly are extracted as cell outlines.

  The arithmetic unit 106 grays the input image 601 on the basis of brightness, as in the HSV conversion process 901 (FIG. 9) and the graying process 902 (FIG. 9), for example. Apply a filter.

  FIG. 13 shows a cell contour image 1301 obtained by applying a canny filter to a gray image. This image is a “cell outline image A” 605 in FIG. The edge portion (white pixel) is the contour portion. At this stage, there are white pixels not only in the cell boundary but also in the background and inside the cell. A schematic diagram 804 in FIG. 8 is an image of the “cell outline image A” 605.

(Outside cell outline removal processing 606 / cell outline image 607)
In the cell outer contour removal processing 606, the noise component generated in the background portion of the cell among the noise components mixed in the “cell contour image A” 605 is filtered by the background region of the “background separated image A” 603 and removed. To do. That is, white pixels in the background portion mixed in “cell outline image A” 605 are removed (set to black pixels). Specifically, the arithmetic unit 106 converts the white pixel of the “cell outline image A” 605 corresponding to the black pixel of the “background separated image A” 603 to a black pixel. As a result, a “cell outline image B” 607 from which the noise of the cell outline in the background portion is removed is generated. “Cell outline image B” 607 is an image in which a portion corresponding to the black pixel of the background separated image 1201 shown in FIG. 12 in the cell outline image 1301 shown in FIG. When this image is schematically represented, a schematic diagram 805 in FIG. 8 is obtained. As can be seen from comparison with the schematic diagram 804, white pixels corresponding to the background portion are removed.

(Extracellular component removal processing 608 / background separation image 609)
In the extracellular component removal processing 608, the noise component that is mixed in the “background separated image A” 603 and is recognized as an internal region of the cell although it is originally an external portion of the cell is removed. Specifically, in the “background separation image A” 603, the region outside the outline of the “cell outline image B” 607 is regarded as the outside area of the cell, and is removed. In other words, among the white pixels included in the “background separated image A” 603, pixels originally located outside the cell are removed (made black pixels) using the outline of the “cell outline image B” 607.

  Here, the arithmetic unit 106 determines that the pixel of the “cell outline image B” 607 corresponding to the white pixel adjacent to the black pixel of the “background separated image A” 603 is not a white pixel (outline) (that is, a black pixel). The white pixel of the “background separated image A” 603 that is the determination target is changed to a black pixel. By this process, a pixel portion that is not a contour among white pixels adjacent to the background (black pixel) is changed to a black pixel. By executing this process several times, extracellular components located outside the outline of “cell outline image B” 607 are removed.

  However, as shown in the schematic diagram 805 of FIG. 8, the outline of the “cell outline image B” 607 does not surround the entire cell so as to surround the cell. There is a possibility that the inside also becomes black pixels.

  Therefore, the arithmetic device 106 is set in advance so as to repeat the above-described black pixel conversion processing as many times as the number corresponding to the average protruding pixel. In the cell segmentation process 612 described later, since a small isolated component is not regarded as a cell, even if the external component of the cell cannot be completely removed, it is sufficient that the portion is small. On the other hand, even if the inside of the cell is made somewhat black by the extracellular component removal processing 608, there is no problem in the final output result.

  In order to reduce these influences, the black pixel conversion process described above may be executed after the outline of the “cell outline image B” 607 is first bolded. That is, the process of making the black pixel in contact with the contour portion (white pixel) of the “cell contour image A” 603 into a white pixel may be repeated many times.

  As described above, the white pixel mixed in the “background separated image A” 603 is converted into a black pixel by converting the portion from the background portion to the outline of the cell into a black pixel. As a result of this processing, “background separated image B” 609 is obtained. An image of “background separated image B” 609 is shown in a schematic diagram 806 of FIG.

  Here, the difference between “background separated image A” 603 and “background separated image B” 609 is confirmed. FIG. 14 shows an image obtained by converting a portion corresponding to a black pixel (background) in the “background separated image A” 603 out of the pixels of the input image 601 into a black pixel. FIG. 15 is an image in which the portion corresponding to the black pixel (background) in the “background separated image B” 609 among the pixels of the input image is changed to a black pixel. All are color images, but they are grayed out and converted to a dot diagram in newspaper printing format. In FIG. 14, there are portions that are not black pixels outside the cell, whereas in FIG. 15, these portions are removed.

(Cell inner contour removal processing 610 / cell contour image 611)
In the cell inner contour removal processing 610, among the noise components mixed in the “cell contour image B” 607, the noise components generated in the region inside the cell are in contact with the background of the “background separated image B” 609. Remove no connected regions. Specifically, contours other than the cell boundaries (that is, contours present inside the cells) are removed from the white pixels included in the “cell contour image B” 607.

  As can be seen from the “cell outline image A” 605 in FIG. 13, there are also white pixels other than the cell boundary (inside cell region). By the way, it is considered that the outline in contact with the black pixel of “background separated image B” 609 is a boundary. Therefore, black pixels are formed except for a part of all the components connected to the outline of the “cell outline image B” 609 (connected components of white pixels) where a part or all of the components are in contact with the black pixels at a certain level.

  Note that instead of the connected component, a process may be used in which only the white pixel that is in contact with the black pixel of the “background separated image B” 609 or the white pixel close to the pixel is left and the other white pixels are converted to black pixels. good.

  In any case, the “cell outline image C” 611 is generated by executing the cell inner outline removal processing 610. FIG. 16 schematically shows a generation image of “cell outline image C” 611. In the drawing, a schematic diagram 1601 is an image of “cell outline image C” 611. FIG. 17 shows an image image of the cell contour image 1701 corresponding to the “cell contour image C” 611. Comparing FIG. 17 and FIG. 13, it can be seen that the “cell outline image C” 611 has a background portion and the outline (white pixel) inside the cell removed compared to the “cell outline image A” 605.

  Through the above processing, a “cell outline image C” 611 in which the cell region and the background region are separated using the “background separated image B” 609 is obtained. In addition, an image of a cell outline portion is obtained by “cell outline image C” 611.

(Cell segmentation process 612)
When “background separation image B” 609 and “cell outline image C” 611 are obtained by the above-described processing, the arithmetic unit 106 executes the cell segmentation processing 612. That is, a process of dividing or separating individual cell regions from the cell region (cell cluster) obtained by the “background separated image B” 609 is executed. At this time, the arithmetic unit 106 uses the outline of the “cell outline image C” 611 for dividing or separating individual cell regions. Here, the arithmetic unit 106 regards the internal area of the cell surrounded by the outline as an individual cell area.

  However, as described above, the outline of the cell extracted by the image processing may be missing at a part of the cell boundary. This can also be seen from the “cell outline image C” 611 shown in FIG.

  Therefore, the arithmetic unit 106 considers that a part of the outline is missing and executes the following processing. FIG. 17 shows an image of the cell segmentation process 612. First, the arithmetic unit 106 thickens the outline of the “cell outline image C” 1802 and generates a “cell outline image C” 1803 that has been bolded.

  Next, the arithmetic unit 106 converts the pixel portion corresponding to the white pixel in the “cell outline image C” 1803 that has been bolded out of the white pixels in the “background separated image B” 1801 into a black pixel in the outline portion. A “background separated image B” 1804 is generated.

  Thereafter, the arithmetic unit 106 extracts a connected component of the white pixel from the contour portion black pixelation “background separated image B” 1804, and selects a component having a size equal to or larger than a predetermined threshold among the connected components as one cell. Recognize as Further, the arithmetic device 106 calculates the position of each cell as, for example, the position of the center of gravity of the connected component. The arithmetic device 106 also calculates the number of cells included in the input image 601, the area occupied by the cells, the ratio of cells occupied with respect to the input image 601, and the like according to the growth stage. The area occupied by the cells can be calculated based on the area and ratio of the white pixel portion of the “background separated image B” 609.

(Output processing 613)
After the execution of the cell segmentation process 612, the arithmetic unit 106 outputs an image in which the position of the cell is marked, the number of cells included in the input image 601, the area occupied by the cell, the cell occupation rate, and the like.

(Summary of cell image recognition processing)
As described above, in the cell image recognition process according to the present embodiment, prior to the execution of the cell segmentation process 612, the noise component included in each of the background separation image and the cell outline image is removed using the other image. The background separation accuracy and the cell contour extraction accuracy can be improved. As a result, it is possible to improve the recognition accuracy of the position of the cell included in the input image and the recognition accuracy of the cell occupation region.

  In the present embodiment, the extracellular component removal processing 608 is executed after the cell outer contour removal processing 606, and then the cell inner contour removal processing 610 is executed. The reason is as follows. First, in order to execute the cell inner contour removal processing 610, the background of the “background separated image A” 602 needs to be in contact with the cell boundary. However, as described above, the portion of the “background separated image A” 602 that is recognized as being inside the cell includes a region portion (extracellular component noise) that is not originally inside the cell. As described above, if an area portion that is not originally inside the cell is included, the cell inner contour removal processing 610 becomes difficult, and thus the extracellular component removal processing 608 is executed in advance.

  The extracellular component removal process 608 is a process of removing extracellular components from the “background separated image A” 602 by relying on the contour. In this case, even if the noise of the contour is included inside the cell. It doesn't matter, but there is a problem if there is contour noise outside the cell. Therefore, prior to the processing of the extracellular component removal processing 608, the external cell contour removal processing 606 is executed to remove contour noise outside the cell.

(Effect of Example 1)
As described above, the automatic culture apparatus 210 according to the present embodiment periodically photographs the cells in the culture container, recognizes the position of the cells and the cell occupation area from the acquired image, and the cell growth stage. Automatically, cell growth abnormality determination according to the growth stage, quality evaluation, etc. are automatically executed. Thereby, the automatic culture apparatus 210 which concerns on a present Example can monitor the growth condition etc. of a cell quantitatively. As a result, it is possible to reduce costs (labor costs), dependence on the skill level of engineers, and the risk of biological contamination, compared with the case where cell growth is regularly checked manually. . Moreover, appropriate environmental control and information provision according to the cell growth stage can be automatically executed.

  In addition, the actual shape of the cells to be cultured is not limited to a relatively simple shape (for example, an ellipse, a line, etc.) and is complex, and the size of the cells varies. If the cell image recognition processing installed in is used, it is possible to handle cells having complicated shapes and cells having different sizes. If the cell image recognition process is used, an image of a cell having poor characteristics such as contact between cells, brightness for separating the inside and outside of the cell, and color can be handled as the input image 601. As a result, even when an input image that cannot be handled by a conventional method is used, the cell growth state can be monitored, and the cell occupancy and number can be calculated quantitatively with high accuracy. In addition, if moving image processing is used, cell tracking can be facilitated, and useful information for monitoring the cell growth status can be acquired.

  The present embodiment proposes an automatic culture apparatus 210 that generates a “background separated image A” 603 by a procedure different from the background separation processing 602 (FIG. 9) of the first embodiment. Other configurations and processing contents of the automatic analyzer 210 in the present embodiment are the same as those in the first embodiment.

  The background separation processing 602 used in this embodiment will be described with reference to FIG. Background separation processing 602 in the present embodiment is a method for discriminating cell regions and background regions by machine learning, and the processing phase can be separated into a learning phase and an identification phase.

  First, the learning phase will be described. In the learning phase, first, a database in which labeled input images 1901 are collected is created. At this time, the arithmetic unit 106 cuts out a part of the cell image and performs labeling based on whether or not the cell is reflected in the image. It is convenient to process the cell image into a rectangular image having a predetermined size. Prepare as many image and label pairs as possible (for example, several hundred or more).

  In the feature extraction process 1902, the arithmetic unit 106 performs a process of converting one image into a vector value. For this processing, for example, a feature extraction method used in the field of image recognition such as HOG feature or SIFT feature can be used. The simplest processing method includes a method in which a vector in which all pixel values are arranged is used as a pixel feature. In this case, if it is a gray scale, it will become a vector of a pixel number dimension, and if it is RGB, it will become a (pixel number * 3) dimension vector. Incidentally, “*” is a multiplication operator. Let n be the number of dimensions of the vector.

  In the identification learning process 1903, the arithmetic unit 106 takes a real value as a value and generates a function f (x) with a parameter having an n-dimensional vector as an argument. For example, the value of the function f (x) with respect to the vector x extracted from the image showing the cell is positive (> 0), and the arithmetic device 106 has the function f (with respect to the vector x extracted from the image where the cell is not shown. Parameters are learned as much as possible so that the value of x) is negative (<0). As the learning algorithm, a neural network, SVM (Support Vector Machine), or the like can be used. The function f (x) generated in the identification learning process 1903 is stored in the identification dictionary 1904.

  Next, the identification phase will be described. In the identification phase, feature extraction processing 1905 is executed for the input image 601. In the feature extraction processing 1905, the arithmetic unit 106 divides the input image 601 into rectangles having the same size as the labeled input image 1901, and converts each rectangle into a vector by the same method as the feature extraction processing 1902. At this time, the rectangles may overlap each other. In the identification process 1906, the value of the function f (x) is calculated for the vector x extracted in the feature extraction process 1905. If f (x)> 0, it is determined as a cell region, and if f (x) <0, it is determined as a background region. For each pixel of the input image 601, the distances from the center positions of all the rectangles are compared, and the identification result of the rectangle having the closest distance is set as the identification result of the corresponding pixel, and the cell region and the background are determined. As a result, a “background separated image A” 603 is generated in which the cell region is a white pixel and the background region is a black pixel.

  In the above-described first and second embodiments, the case where the cell culture apparatus 101 is also provided with a cell image processing function has been dealt with. However, the cell culture apparatus 101 may be equipped with only an image acquisition function, and part or all of the cell image processing function may be implemented in a personal computer connected to the outside of the cell culture apparatus 101. Further, only the cell image processing function may be an independent software or image processing apparatus.

  The present invention is not limited to the configuration of the embodiment described above, and includes various modifications. In the above-described embodiment, the case where the “background separated image B” 609 is generated using the “background separated image A” 603 and the “cell outline image B” 607 has been described. However, the method is different from the method illustrated in FIG. “Background separation image B” 609 may be generated. For example, “background separated image B” 609 may be generated using “background separated image A” 603 and “cell outline image A” 605. Further, for example, “background separated image B” 609 may be generated by using the cell contour image obtained by removing the background noise component from “cell contour image A” 605 and “background separated image A” 603.

  In the above-described embodiment, an image captured by the cell culture device 101 is used as the input image 601. However, the input image 601 is arbitrary, and for example, noise components are removed by the method described in the above-described embodiment. You may use the image.

  The above-described embodiments are detailed descriptions of some embodiments of the present invention, and it is not always necessary to provide all the configurations described. Further, a part of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. It is also possible to add other configurations to the configuration of each embodiment, replace a partial configuration of each embodiment with another configuration, or delete a partial configuration of each embodiment.

DESCRIPTION OF SYMBOLS 101 Cell culture apparatus 102 Input apparatus 103 Display apparatus 104 Image acquisition apparatus 105 Communication apparatus 106 Arithmetic apparatus (CPU)
107 External storage device 108 Culture vessel 109 Environmental control device 210 Automatic culture device 211 Cell culture chamber door 212 Refrigerator door 213 Cell culture chamber 214 Refrigerator 215 Control unit 216 Intermediate chamber 217 Intermediate chamber door 220 Culture vessel 221 Culture vessel base 222 Rotating mechanism 223 Pump 224 Valve 225 Tank 227 Drive base 228 Microscope 229 Stage 230 Connector board 231 Connector 232 Medium base 233 Seal 234 Claw 240 Channel 241 Tube 243 Syringe 244 Syringe pump 250 Fan (intermediate chamber)
251 Filter (intermediate chamber)
252 Fan (control unit)
253 Intake filter 254 Exhaust filter

Claims (13)

A background separation image and a cell outline image are generated from an image obtained by imaging cells cultured in a cell culture device, noise components included in the background separation image are removed by the cell outline image, and included in the cell outline image A cell recognition unit that removes noise components from the background separation image, recognizes individual cell regions using the background separation image and the cell contour image from which the noise components are removed, and a cell growth stage based on the recognition result. A cell growth stage determination unit for determining;
In the first stage of the cell growth stage, the presence or absence of abnormality is determined based on the number of cells and the cell position calculated in the cell recognition unit, and in the second stage of the cell growth stage, the cell occupancy calculated in the cell recognition unit An abnormality determination unit for determining the presence or absence of an abnormality based on
When an abnormality is recognized in the abnormality determination, an abnormal state processing unit that issues environmental control or an alarm,
A cell image processing comprising: a completion determination unit that determines whether or not cell culture by the cell culture device is completed based on the cell occupancy rate and / or outputs a quality evaluation value based on the cell occupancy rate apparatus. The cell image processing apparatus according to claim 1,
The cell recognition unit
Cell image processing characterized by generating a cell contour image from which the noise component has been removed by removing a contour component located in a background portion of the background separation image from the contour components included in the cell contour image apparatus. The cell image processing apparatus according to claim 1,
The cell recognition unit
Among the intracellular portions in the background separation image, the intracellular portion located outside the contour component of the cell contour image is in contact with the background region from the portion in contact with the background region or until the background region is contacted. A cell image processing apparatus, wherein a background separated image from which the noise component is removed is generated by removing a predetermined amount from a portion in the direction of the contour. The cell image processing apparatus according to claim 1,
The cell recognition unit
A cell contour image in which the noise component is removed is generated by removing a component connected to a contour that is not in contact with the background of the background separation image from the contour components included in the cell contour image. A cell image processing apparatus. The cell image processing apparatus according to claim 1,
The cell recognition unit
Of the contour components included in the cell contour image, by removing the contour component located in the background portion of the background separation image, to generate a cell contour image from which the noise component has been removed,
Among the intracellular portions in the background separation image, the intracellular portion located outside the contour component of the cell contour image from which the noise component has been removed, until the portion comes into contact with the contour from the portion that contacts the background region, or By removing a predetermined amount from the region in contact with the background region in the direction of the contour, a background separated image from which the noise component has been removed is generated,
Among the contour components included in the cell contour image from which the noise component has been removed, the noise component is further removed by removing the connected component of the contour that is not in contact with the background region of the background separated image from which the noise component has been removed. Generated cell contour image,
An individual cell region is recognized using a background separation image from which the noise component has been removed and a cell outline image from which the noise component has been further removed. The cell image processing apparatus according to claim 1,
The cell growth stage determination unit
A cell image processing apparatus, wherein when the calculated cell occupancy is equal to or less than a predetermined threshold value, it is determined as the first stage of the growth stage, and other cases are determined as the second stage of the growth stage. The cell image processing apparatus according to claim 1,
The abnormality determination unit
A cell image processing apparatus, wherein a normal number of cells acquired in advance for the same type of cells is used as a reference value, and an abnormality is determined when a deviation of a predetermined value or more occurs with respect to the reference value. A memory for storing images of cells,
A background separation image and a cell outline image are generated from the image, a noise component included in the background separation image is removed by the cell outline image, a noise component contained in the cell outline image is removed by the background separation image, A cell image recognition apparatus comprising: a background separation image from which a noise component has been removed; and a cell recognition unit that recognizes individual cell regions using a cell contour image. In the cell image recognition device according to claim 8,
The cell recognition unit
Cell image recognition characterized by generating a cell contour image from which the noise component is removed by removing a contour component located in a background portion of the background separation image from the contour components included in the cell contour image apparatus. In the cell image recognition device according to claim 8,
The cell recognition unit
Among the intracellular portions in the background separation image, the intracellular portion located outside the contour component of the cell contour image is in contact with the background region from the portion in contact with the background region or until the background region is contacted. A cell image recognition apparatus characterized by generating a background separated image from which the noise component has been removed by removing a predetermined amount from a portion in the direction of the contour. In the cell image recognition device according to claim 8,
The cell recognition unit
A cell contour image in which the noise component is removed is generated by removing a component connected to a contour that is not in contact with the background of the background separation image from the contour components included in the cell contour image. A cell image recognition apparatus. In the cell image recognition device according to claim 8,
The cell recognition unit
Of the contour components included in the cell contour image, by removing the contour component located in the background portion of the background separation image, to generate a cell contour image from which the noise component has been removed,
Among the intracellular portions in the background separation image, the intracellular portion located outside the contour component of the cell contour image from which the noise component has been removed, until the portion comes into contact with the contour from the portion that contacts the background region, or By removing a predetermined amount from the region in contact with the background region in the direction of the contour, a background separated image from which the noise component has been removed is generated,
Among the contour components included in the cell contour image from which the noise component has been removed, the noise component is further removed by removing the connected component of the contour that is not in contact with the background region of the background separated image from which the noise component has been removed. Generated cell contour image,
An individual cell region is recognized using a background separated image from which the noise component has been removed and a cell outline image from which the noise component has been further removed. In a cell image recognition method executed in a cell image recognition apparatus having a memory for storing an image obtained by imaging a cell and a cell recognition unit for recognizing an individual cell region from the image,
Processing to generate a background separation image and a cell contour image from the image;
A process of removing noise components included in the background separation image in the cell contour image;
A process of removing noise components included in the cell outline image by the background separation image;
A cell image recognition method comprising: a background separated image from which noise components have been removed, and a process of recognizing individual cell regions using a cell contour image.

Images