JP7803799B2 - Learning model generation method, information processing device, and program - Google Patents

Description

The present invention relates to a method for generating a learning model, an information processing device, and a program.

In testing for bacteria and other microorganisms, samples are stained and then visually observed under a microscope to determine their species. However, determining the bacterial species requires a skilled laboratory technician, and a method for doing this mechanically is desirable.

For example, Patent Document 1 discloses an image processing device that uses a learning model generated by machine learning to detect bacteria from images of Gram-stained specimens, and if bacteria are detected, classifies the bacteria according to their type.

Japanese Patent Application Laid-Open No. 2022-20559

However, constructing a learning model for classifying microorganisms, as in the invention of Patent Document 1, requires a huge amount of training data.

One aspect of the present invention is to provide a method for generating a learning model that can effectively generate a learning model for classifying microorganisms.

In one aspect, a method for generating a learning model involves a computer executing a process in which an image containing microorganisms is divided into a plurality of first segmented images, the first segmented images are further divided into a plurality of second segmented images, the second segmented images are combined to generate a combined image, training data is generated that associates the combined image with classification information indicating the types of microorganisms contained in the combined image, and a learning model is generated based on the training data that outputs classification information for microorganisms when the first segmented images are input.

In one aspect, it is possible to effectively generate a learning model for classifying microorganisms.

FIG. 1 is an explanatory diagram illustrating an example of the configuration of an inspection system. FIG. 2 is a block diagram illustrating an example of the configuration of a server. FIG. 2 is a block diagram illustrating an example of the configuration of a terminal. FIG. 1 is an explanatory diagram showing problems with bacterial testing using a machine learning model. FIG. 10 is an explanatory diagram relating to a process for generating a combined image. FIG. 10 is an explanatory diagram illustrating an example of an annotation screen. FIG. 10 is an explanatory diagram regarding a learning model generation process. FIG. 10 is an explanatory diagram showing the verification results of prediction accuracy. 10 is a flowchart showing the steps of a learning model generation process. 10 is a flowchart showing the procedure for classifying samples using a learning model.

The present invention will be described in detail below with reference to the drawings showing embodiments thereof.
(Embodiment)
1 is an explanatory diagram showing an example of the configuration of an inspection system. In this embodiment, a learning model 50 is constructed that, when an image of Gram-stained bacteria is input, outputs classification information indicating the type of bacteria contained in the image, and an inspection system is described that uses this model to inspect the type of bacteria contained in a specimen. The inspection system includes an information processing device 1 and a terminal 2. Each device is connected to a network N such as the Internet.

In this embodiment, the test subject is described as bacteria, but microorganisms other than bacteria (such as mold (fungi)) may also be the target. In other words, bacteria are an example of a microorganism, and other microorganisms may also be the target.

The information processing device 1 is an information processing device capable of various information processing and information transmission and reception, such as a server computer, personal computer, tablet computer, smartphone, etc. In this embodiment, the information processing device 1 is assumed to be a server computer, and for simplicity, will be referred to as server 1 below. The server 1 generates the above-mentioned learning model 50 by learning training data in which classification information for the bacteria contained in each Gram-stained image of bacteria observed under the microscope 3 is associated with the image. Specifically, as described below, the server 1 divides images containing only one type of bacteria into predetermined units and combines these to virtually generate a combined image containing multiple types of bacteria, which is used as training data.

Terminal 2 is a terminal device used by a user of this system (e.g., a medical technician), and may be, for example, a personal computer, tablet terminal, or smartphone. As described below, when annotating a group of training images, the user operates Terminal 2 to perform the annotations (see Figure 6).

2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit .
The control unit 11 has one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc., and performs various information processing, control processing, etc. by reading and executing a program P1 stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside.

The auxiliary memory unit 14 is a non-volatile storage area such as a large-capacity memory or hard disk, and stores the program P1 (program product) and other data necessary for the control unit 11 to execute processing. The auxiliary memory unit 14 also stores the learning model 50. The learning model 50 is a machine learning model that has learned training data as described above, such as CNN (ResNet, VGG, DenseNet, MobileNet, ResNeXt). The learning model 50 is expected to be used as a program module that constitutes part of artificial intelligence software.

The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may also be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed using software.

In addition, in this embodiment, the server 1 is not limited to the above configuration and may include, for example, an input unit that accepts operational input, a display unit that displays images, etc. The server 1 may also be equipped with a reading unit that reads portable storage media 1a such as CD (Compact Disk)-ROM or DVD (Digital Versatile Disc)-ROM, and may read and execute the program P1 from the portable storage medium 1a.

3 is a block diagram showing an example of the configuration of the terminal 2. The terminal 2 includes a control unit 21, a main memory unit 22, a communication unit 23, a display unit 24, an input unit 25, and an auxiliary memory unit 26.
The control unit 21 is an arithmetic processing device such as one or more CPUs, MPUs, etc., and performs various information processing by reading and executing the program P2 stored in the auxiliary memory unit 26. The main memory unit 22 is a temporary storage area such as RAM, and temporarily stores data necessary for the control unit 21 to execute arithmetic processing. The communication unit 23 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside. The display unit 24 is a display screen such as an LCD display, and displays images. The input unit 25 is an operation interface such as a keyboard or mouse, and accepts operation input from the user. The auxiliary memory unit 26 is a non-volatile storage area such as a hard disk or large-capacity memory, and stores the program P2 (program product) and other data necessary for the control unit 21 to execute processing.

The terminal 2 may also be equipped with a reading unit that reads portable storage medium 2a such as a CD-ROM, and reads and executes the program P2 from the portable storage medium 2a.

Figure 4 is an explanatory diagram showing the problems with bacterial testing using machine learning models. The following provides an overview of this embodiment.

When building a machine learning model for bacterial testing, one first consideration is to identify individual bacteria through segmentation, as shown in Figure 4A. However, in clinical testing, which requires efficient testing of a large number of specimens, there are technical limitations to the magnification of the object being observed using the microscope used. As a result, the size of bacteria in images observed with a typical microscope 3 is small and the resolution is low, making it difficult to identify individual bacterial species. Therefore, it is not possible to build a machine learning model with high predictive accuracy.

Next, as shown in Figure 4B, bacterial testing can be considered as an image classification problem, and images can be classified according to the type of bacteria contained in the image. However, when multiple types of bacteria are contained in a single image, it is extremely labor-intensive for an examiner to determine the type of bacteria and label each image, making it difficult to build a model. For example, of the 20 types of bacterial species frequently observed in clinical testing, combinations of 1 to 4 types are frequently observed. Therefore, for each of the ₂₀ C ₄ = 4,854 combinations, more than 100 images are required as training images, which is an enormous number.

In this embodiment, training data to be used as a learning target is created by virtually generating an image containing multiple types of bacteria from an image containing only one type of bacteria.

Figure 5 is an explanatory diagram of the process for generating a combined image. Figure 5 illustrates how an image containing a first bacterium (e.g., Staphylococcus aureus) and an image containing a second bacterium (e.g., E. coli) are each divided and then randomly combined to generate a combined image.

In this embodiment, images containing only one type of bacteria are used as training images. For example, as shown in Figure 5, the server 1 uses an image containing only a first type of bacteria and an image containing only a second type of bacteria as training images. The server 1 then divides the training image into a predetermined number of images. The example on the left side of Figure 5 shows the image being divided into nine. Note that dividing the image into nine images is just an example, and the number of divisions may be eight or fewer, or ten or more.

In the following explanation, the images obtained by dividing the original image into a specified number of images will be referred to as "first divided images."

The server 1 further divides the first divided image into a predetermined number of images into a predetermined number of images. The example in the center of Figure 5 illustrates the image being divided into two. Note that dividing the image into two images is just an example, and the number of divided images may be three or more. The number of divided images is preferably determined according to the number of types of bacteria that may be contained in one image (in this example, two types: Staphylococcus aureus and E. coli).

In the following description, the image obtained by further dividing the first divided image will be referred to as the "second divided image."

Through the above process, the original image is divided into 9 x 2 = 18 pieces. Server 1 combines each second divided image with a second divided image that is different from the second divided image in the pair before division, to generate a combined image of the same size as the original first divided image. Specifically, server 1 randomly combines and combines two second divided images from the 18 second divided images obtained by dividing the image containing the first bacterium and the 18 second divided images obtained by dividing the image containing the second bacterium, to generate a greater number of combined images than the original first divided images. An example of a combined image is shown on the right side of Figure 5. As shown in the example on the right side of Figure 5, three patterns of combined images are generated: one that includes both the first bacterium and the second bacterium, one that includes only the first bacterium, and one that includes only the second bacterium.

Server 1 generates training data by assigning classification information to each combined image indicating the type of bacteria contained in that combined image. In the example on the right side of Figure 5, the first combined image from the top is assigned classification information that classifies it as a first bacterium and a second bacterium. Next, the second combined image is assigned classification information that classifies it as a first bacterium. The third combined image is assigned classification information that classifies it as a second bacterium. In this way, server 1 assigns classification information according to the combination of bacteria contained in each of the second segmented images that make up the combined image. In this way, server 1 generates training data to be used in machine learning.

In the following description, training data that associates a combined image formed by combining a second segmented image containing a first bacterium with a second segmented image containing a second bacterium, and classification information that classifies the result as the first bacterium and the second bacterium, will be referred to as "first training data." Furthermore, training data that associates a combined image formed by combining second segmented images containing the first bacterium with classification information that classifies the result as the first bacterium will be referred to as "second training data." Furthermore, training data that associates a combined image formed by combining second segmented images containing the second bacterium with classification information that classifies the result as the second bacterium will be referred to as "third training data."

It is preferable that server 1 generates more combined images (first training data) containing a mixture of multiple types of bacteria than combined images (second training data and third training data) containing a single type of bacteria. This allows for learning more about situations in which multiple types of bacteria are mixed, improving prediction accuracy.

Figure 6 is an explanatory diagram showing an example of an annotation screen. Figure 6 illustrates an operation screen displayed by terminal 2 when generating training data as described above. This screen includes a first number specification field 61, a second number specification field 62, a bacterial species specification field 63, an image specification field 64, and a divided image 65. The first number specification field 61 is a specification input field for specifying the number of images to be divided into first divided images from the original image. The second number specification field 62 is a specification input field for specifying the number of images to be divided into second divided images from the first divided image. The bacterial species specification field 63 is a specification input field for specifying the type of bacteria (classification information) contained in the image. The image specification field 64 is an operation input field for specifying the image to be divided. The divided image 65 is a display field for displaying the divided image.

Terminal 2 displays this screen and accepts various setting inputs from the user. That is, terminal 2 accepts input specifying the number of images to divide the original image into first divided images in response to an operation input into the first number specification field 61. Next, terminal 2 accepts input specifying the number of images to divide the first divided image into second divided images (the number of bacterial species that may be contained in the image) in response to an operation input into the second number specification field 62. Next, terminal 2 accepts input specifying the type of bacteria contained in the image (annotation) in response to an operation input into the bacterial species specification field 63.

Furthermore, in response to an operation input into the image specification field 64, terminal 2 reads out the image for which the bacterial species was specified above. Terminal 2 divides the image up to the second divided image and displays it as divided image 65. The corresponding bacterial species name is registered (annotated) in the displayed divided areas, except for the second divided image (blank area) that does not contain any bacteria.

In this case, terminal 2 binarizes the image using Otsu's method or the like, identifies second divided images (blank areas) that do not contain bacteria, and registers (annotates) the blank area instead of the bacterial species name for the identified divided area, which is displayed in a different display mode (for example, a different display color) from other areas related to the second divided image. In the example of Figure 6, areas related to the second divided image that do not contain bacteria are shown hatched. Second divided images that do not contain bacteria (blank areas) can be added or deleted as selection targets using the mouse pointer or keyboard directional keys. Note that in the example of Figure 6, a second divided image (blank area) that does not contain bacteria is selected, but instead of selecting a blank area that does not contain bacteria, a divided area that includes the bacterial species specified in the bacterial species specification field 63 may be selected.

Device 2 excludes second segmented images that do not contain bacteria from the second segmented images used to generate the combined image. This makes it possible to avoid learning areas that do not contain bacteria.

The terminal 2 accepts the above operations for each image, divides the image, and accepts setting input (annotation) of classification information (type of bacteria) for each second divided image. The server 1 combines these second divided images to generate a combined image and assigns classification information to the combined image according to the combination of the second divided images. In this way, the server 1 generates training data.

Based on the training data, the server 1 generates a learning model 50 that outputs classification information when the first segmented image is input. Figure 7 is an explanatory diagram of the generation process of the learning model 50. The left side of Figure 7 illustrates how the combined image of the first to third training data is given to the learning model 50 and learning is performed.

The learning model 50 is a machine learning model constructed through deep learning, such as a CNN (ResNet, etc.). The learning model 50 may also be a machine learning model other than a CNN, such as a Vision Transformer. The server 1 inputs training combined images into the learning model 50, outputting classification information. The server 1 compares the output classification information with a correct answer value and adjusts parameters such as weights so that the two are close to each other. The server 1 provides each combined image related to the first to third training data to the learning model 50, optimizes the parameters, and ultimately generates the learning model 50.

Furthermore, as shown on the right side of Figure 7, the server 1 verifies the prediction accuracy of the learning model 50 generated above. To verify the prediction accuracy, an image containing a mixture of multiple types of bacteria is used. Here, we consider the case where an image containing a mixture of two types of bacteria, a first bacterium (Staphylococcus aureus) and a second bacterium (E. coli), is used.

First, the server 1 divides the image into first divided images. The server 1 inputs the first divided images into the learning model 50 to output classification information, and verifies the prediction accuracy by comparing the output classification information with the correct answer value.

The server 1 further divides the first divided image to generate a second divided image. The server 1 then randomly combines the generated second divided images to generate a combined image. The server 1 also uses this combined image to verify the prediction accuracy. The server 1 inputs the combined image into the learning model 50 to output classification information, and verifies the prediction accuracy by comparing the output classification information with the correct answer value.

The reason why not only the first segmented image but also the combined image is used to verify the prediction accuracy is to avoid the influence of the boundary between the two second segmented images that exists in the combined image used as the learning target. By using not only the first segmented image but also the combined image for verification, the prediction accuracy of the learning model 50 can be suitably verified.

When an actual test is performed, the server 1 makes predictions using the learning model 50 generated above. Specifically, the server 1 acquires an image of the sample from the terminal 2 and divides it into first divided images. The server 1 then inputs the divided first divided images into the learning model 50, thereby outputting classification information. The server 1 returns the output classification information to the terminal 2 and presents it to the user (laboratory technician).

Figure 8 is an explanatory diagram showing the results of verification of prediction accuracy. Figures 8A to 8C each illustrate the experimental results of verification of the prediction accuracy of the learning model 50 according to this embodiment.

Figure 8A shows the results of verifying the prediction accuracy of learning model 50, which was trained not on combined images but on images containing only one type of bacteria. As shown in Figure 8A, when making predictions on images containing only one type of bacteria, the hit rate (precision), sensitivity (recall), and F-value (F1 score) all showed high values. However, because images containing a mixture of multiple types of bacteria were not trained, the prediction accuracy was low when making predictions on images containing a mixture of multiple types of bacteria.

Figures 8B and 8C show the results of a verification of the prediction accuracy of learning model 50 when a combined image containing a mixture of different bacteria is used as the learning target. More specifically, Figure 8B shows the results of a verification using a virtual combined image containing a mixture of multiple types of bacteria, while Figure 8C shows the results of a verification using an actual image (first segmented image). In both Figures 8B and 8C, it can be seen that the prediction accuracy (third row of the table) for images containing a mixture of multiple types of bacteria is significantly improved compared to Figure 8A. In this way, by learning a combined image formed by combining multiple second segmented images, it is now possible to accurately detect the type of bacteria even when multiple types of bacteria are present.

Figure 9 is a flowchart showing the steps in the process of generating the learning model 50. The process of generating the learning model 50 through machine learning will be explained using Figure 9 .

The control unit 11 of the server 1 accepts setting inputs, such as the number of images to be divided into first segmented images, the number of images to be divided into second segmented images, and the type of bacteria contained in the images (classification information), via the annotation screen displayed on the terminal 2 (step S11). The control unit 11 acquires a group of training images (step S12). The control unit 11 divides each acquired image into multiple first segmented images according to the set number (step S13). The control unit 11 further divides the first segmented images into multiple second segmented images (step S14). Note that after division into second segmented images, second segmented images that do not contain bacteria are excluded from subsequent processing.

The control unit 11 randomly combines and combines the divided second divided images to generate a combined image (step S15). Specifically, the control unit 11 combines a second divided image with a second divided image that is different from the second divided image in the pair before division, generating a combined image of the same size as the first image when divided into the first number of images. In this case, the control unit 11 randomly combines the second divided images to generate a greater number of combined images than the original first divided images. Furthermore, it is preferable that the control unit 11 generate more combined images (first training data) in which second divided images of different types of bacteria are combined than more combined images (second training data and third training data) in which second divided images of the same type of bacteria are combined.

The control unit 11 generates training data that associates the generated combined image with classification information for the bacteria in the combined image (step S16). That is, the control unit 11 assigns classification information to the combined image according to the combination of bacteria contained in each of the second segment images that make up the combined image. As a result, the control unit 11 generates first training data in which the bacteria contained in the combined image differ between the first bacterium and the second bacterium, second training data in which the bacteria contained in the combined image are common to the first bacterium, and third training data in which the bacteria contained in the combined image are common to the second bacterium.

Based on the training data, the control unit 11 generates a learning model 50 that outputs classification information for the bacteria in the first segmented image when the first segmented image is input (step S17). For example, the control unit 11 generates a CNN (ResNet) as the learning model 50. The control unit 11 outputs classification information by inputting the training combined image into the learning model 50, and compares the output classification information with the correct value of the classification information associated with the training combined image. The control unit 11 optimizes parameters such as weights so that the two are similar, and generates the learning model 50.

The control unit 11 acquires a group of test images (step S18). The test image group is a mixture of images of multiple types of bacteria. The control unit 11 divides the test images into first segmented images, and further divides them into second segmented images (step S19). The control unit 11 generates a combined image by combining the divided second segmented images (step S20). The control unit 11 then inputs the first segmented images and the combined image by combining the second segmented images into the learning model 50 to output classification information, and verifies the prediction accuracy of the learning model 50 by comparing the output classification information with the correct answer (step S21). The control unit 11 then completes this series of processes.

Figure 10 is a flowchart showing the procedure for classifying samples using the learning model 50. The details of the bacterial classification process using the learning model 50 will be explained based on Figure 10. Note that the procedure for classifying samples may be either Figure 10 or the processing details for the test image group (the process of outputting classification information from steps S18 to S21).

The control unit 11 of the server 1 acquires an image of the specimen to be inspected from the terminal 2 (step S31). The control unit 11 divides the acquired image into first divided images (step S32). The control unit 11 inputs the divided images into the learning model 50, thereby outputting bacterial classification information (step S33). The control unit 11 ends the series of processes.

As described above, according to this embodiment, the training image is divided into second divided images, which are then combined, and the resulting combined image is learned to generate a learning model 50. This reduces the amount of training data that needs to be manually created, while still allowing accurate prediction of bacterial species even in specimens containing a mixture of multiple types of bacteria.

The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the meaning described above, and is intended to include all modifications within the meaning and scope of the claims.

The matters described in the embodiments can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the citation format. Furthermore, although the claims use a format in which a claim cites two or more other claims (multi-claim format), this is not limited to this. A multiple claim citing at least one other multiple claim (multi-multi-claim) can also be used.

1. Server (information processing device)
REFERENCE SIGNS LIST 11 Control unit 12 Main memory unit 13 Communication unit 14 Auxiliary memory unit P1 Program 2 Terminal 21 Control unit 22 Main memory unit 23 Communication unit 24 Display unit 25 Input unit 26 Auxiliary memory unit P2 Program

Claims (11)

Dividing the image containing the microorganism into a plurality of first divided images;
further dividing the first divided image into a plurality of second divided images;
generating a combined image by combining the second divided images;
generating training data in which the combined image is associated with classification information indicating the types of microorganisms contained in the combined image;
A learning model generation method in which a computer executes a process of generating a learning model that outputs classification information of a microorganism when the first segmented image is input, based on the training data. The image contains one type of microorganism,
The method for generating a learning model according to claim 1 , further comprising: dividing an image including a first microorganism and an image including a second microorganism into the second divided images and combining the divided images to generate the combined image. first training data in which the combined image obtained by combining the second divided image including the first microorganism and the second divided image including the second microorganism is associated with the classification information for the first microorganism and the second microorganism;
second training data in which the combined image obtained by combining the second divided images including the first microorganism is associated with the classification information identifying the first microorganism;
The method for generating a learning model described in claim 2, further comprising generating third training data that corresponds the combined image obtained by combining the second divided images containing the second microorganism with the classification information identifying the second microorganism. The method for generating a learning model according to claim 3 , further comprising generating the learning model based on the first training data, the second training data, and the third training data. The method for generating a learning model according to claim 3 , wherein the first training data is generated in greater quantities than the second training data and the third training data. The method for generating a learning model described in claim 1, wherein the combined image is generated by combining the second divided image with a second divided image that is different from the second divided image of the pair before division so that the combined image is the same size as the first divided image. The method for generating a learning model according to claim 1 , further comprising randomly combining the second divided images to generate a greater number of combined images than the first divided images. generating a first divided image obtained by dividing an image including a plurality of types of microorganisms and a combined image obtained by combining second divided images obtained by further dividing the first divided image;
The method for generating a learning model according to claim 1 , further comprising inputting the generated first divided image and combined image into the learning model to verify prediction accuracy. An information processing device including a control unit,
The control unit
Dividing the image containing the microorganism into a plurality of first divided images;
further dividing the first divided image into a plurality of second divided images;
generating a combined image by combining the second divided images;
generating training data in which the combined image is associated with classification information indicating the types of microorganisms contained in the combined image;
an information processing device that generates a learning model that outputs classification information of microorganisms when the first segmented image is input based on the training data. Dividing the image containing the microorganism into a plurality of first divided images;
further dividing the first divided image into a plurality of second divided images;
generating a combined image by combining the second divided images;
generating training data in which the combined image is associated with classification information indicating the types of microorganisms contained in the combined image;
A program that causes a computer to execute a process of generating a learning model that outputs classification information of microorganisms when the first segmented image is input, based on the training data. Acquire an image containing microorganisms,
Dividing the acquired image into first divided images;
A program that causes a computer to execute a process of inputting training data that associates the divided first divided image, a combined image obtained by combining second divided images obtained by dividing the first divided image, and classification information that indicates the type of microorganism contained in the combined image into a trained learning model, and outputting the classification information.