JP7187557B2 - MEDICAL IMAGE LEARNING APPARATUS, METHOD AND PROGRAM - Google Patents

Description

The present invention relates to a medical image learning apparatus, method, and program, and more particularly to a technique of performing learning for accurate image recognition with a relatively small amount of data.

In the medical field, examinations using endoscope apparatuses are performed. In recent years, it has been known to detect lesions contained in an image by image analysis, to classify lesions by type, or the like, and to support examinations by performing recognition and reporting.

Machine learning of images including deep learning is widely used in image analysis for recognition (for example, Non-Patent Document 1).

Deep learning enables image recognition such as classification and detection by learning a large amount of images according to the problem.

By the way, medical images include special light images captured by a light source called special light in addition to normal light images captured by a light source called normal light.

Since visual inspection primarily uses normal light images, a relatively large number of normal light images can be collected.

On the other hand, since the special light image is selectively used according to the purpose of observation, the amount of data is not so large compared to the normal light image. Therefore, it is difficult to sufficiently perform the learning using the special light image, and it is difficult to improve the recognition accuracy of the special light image by the learned recognizer.

On the other hand, in Patent Document 1, at least one characteristic of the shape of the subject captured in the target image group of the learning object, the tissue structure of the subject captured in the target image group, and the imaging system of the device that captured the target image group is similar. Pre-learning is performed based on a group of similar images, and main learning is performed based on the result of pre-learning and a group of target images. An image processing apparatus has been proposed that includes a unit. As a result, highly accurate learning can be performed even if the amount of data in the target image group used for main learning is small.

WO2017/221412

A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012

The image group used for pre-learning described in Patent Document 1 is a similar image group similar in at least one characteristic of the target image group to be learned, the shape of the subject, the tissue structure of the subject, and the imaging system of the imaging device. . That is, in Patent Document 1, a group of target images to be learned and a group of similar images are a group of special light images captured with special observation light and a group of normal light images captured with normal observation light. It does not describe that (the images are groups of images captured with different observation lights).

The present invention has been made in view of such circumstances, and provides a medical image learning device, method, and program capable of generating a model for highly accurate image recognition of medical images captured with special light. intended to

In order to achieve the above object, a medical image learning apparatus according to one aspect of the present invention performs learning using at least a first image group composed of a plurality of first medical images captured with normal light, thereby learning images for image recognition. and a second image group consisting of a plurality of second medical images captured with special light based on the first learning unit that generates the first model of the second medical image. and a second learning unit that generates a second model that performs image recognition for the.

According to one aspect of the present invention, the first learning unit performs learning (pre-learning) using a first image group composed of first medical images captured with normal light, which can be collected in relatively large numbers. , can well generate the first model for image recognition. Then, based on the first model, the second learning unit performs learning (also referred to as “fine tuning”) using a second image group consisting of a plurality of second medical images. It is possible to generate a second model that performs appropriate image recognition on the second medical image even from two image groups.

In the medical image learning device according to another aspect of the present invention, it is preferable that the first learning unit also use the second medical image to generate the first model. As a result, more medical images can be used to generate the first model.

A medical image learning device according to still another aspect of the present invention includes an image processing unit that converts a first medical image having a plurality of color channels into a first medical image having a single color channel; is preferably learned using a first set of images with one transformed color channel. By converting the first medical image into a first medical image having one color channel, the feature amount of the first medical image can be brought closer to the feature amount of the second medical image, and the first model can be generated more easily. can be done properly.

In the medical image learning device according to still another aspect of the present invention, the image processing unit converts a first medical image having a plurality of color channels into a first medical image having only luminance signals, thereby obtaining a single color channel Preferably, the first medical image has As a result, the first learning unit can extract basic feature amounts such as edges and identification rules that are not affected by color information consisting of a plurality of color channels, and can generate an appropriate first model. can.

A medical image learning apparatus according to still another aspect of the present invention converts a first medical image and a second medical image having a plurality of color channels into a first medical image and a second medical image each having a single color channel. Preferably, the first training unit trains using the first and second sets of images having one transformed color channel.

When the first group of images and the second group of images are used for training in the first training unit, the first group of images and the second group of images, which consist of multiple color channels, are converted into groups of images each having one color channel. By using the obtained one, it is possible to generate the first model more properly.

In the medical image learning device according to still another aspect of the present invention, the image processing unit converts the first medical image and the second medical image having a plurality of color channels into the first medical image and the second medical image having only luminance signals. resulting in a first medical image and a second medical image having one color channel. As a result, the first learning unit can extract basic feature amounts such as edges that are not affected by color information made up of a plurality of color channels, and can generate an appropriate first model.

A medical image learning device according to still another aspect of the present invention comprises an extraction unit for extracting a first medical image having one color channel from a first medical image having a plurality of color channels, the first learning unit comprising , preferably using a first set of images with one color channel extracted. By extracting a first medical image having one color channel from a first image group consisting of a plurality of color channels and learning using the extracted first image group having one color channel, It is possible to extract basic feature amounts such as edges that are not affected by color information, and to generate an appropriate first model.

In a medical image learning device according to still another aspect of the present invention, a first medical image and a second medical image each having a single color channel are extracted from a first medical image and a second medical image having a plurality of color channels. and the first learning unit learns using the first group of images and the second group of images having one extracted color channel.

When using the first group of images and the second group of images for training in the first learning unit, the first medical image having one color channel from each of the first group of images and the second group of images each having a plurality of color channels. and a second medical image, and the extracted image group of one color channel is used for learning, so that the first model can be generated more appropriately.

In the medical image learning device according to still another aspect of the present invention, the plurality of color channels are preferably three channels of three primary colors, or three channels of a luminance signal and two color difference signals.

In the medical image learning device according to still another aspect of the present invention, a third model for performing image recognition on the first medical image is generated by learning using the first image group based on the first model. 3 learning units. The first model can also be used to generate a third model for performing image recognition on the first medical image.

In the medical image learning device according to still another aspect of the present invention, the first medical image and the second medical image are preferably images captured by an endoscope device.

In the medical image learning device according to still another aspect of the present invention, it is preferable that the first model and the second model are configured by convolutional neural networks.

A medical image learning method according to still another aspect of the present invention provides a first image group consisting of a plurality of first medical images captured with normal light and a second image group consisting of a plurality of second medical images captured with special light. preparing a group of images; generating a first model for image recognition by a first learning unit by learning using at least the first group of images; generating a second model for performing image recognition on the second medical image by training using the second set of images in the step of .

In a medical image learning method according to still another aspect of the present invention, the image processing unit converts a first medical image having a plurality of color channels into a first medical image having a single color channel, The step of generating one model preferably trains using a first set of images having one transformed color channel.

In the medical image learning method according to still another aspect of the present invention, the image processing unit converts the first medical image and the second medical image having a plurality of color channels into the first medical image and the second medical image having one color channel. Preferably, the step of generating the first model includes transforming to medical images and training using a first set of images and a second set of images having one color channel that has been transformed.

In a medical image learning method according to still another aspect of the present invention, the extracting unit extracts a first medical image having one color channel from a first medical image having a plurality of color channels, The step of generating the model is preferably trained using a first set of images having one color channel extracted.

In the medical image learning method according to still another aspect of the present invention, the extracting unit extracts the first medical image and the second medical image having one color channel from the first medical image and the second medical image having a plurality of color channels. Preferably, the step of generating the first model comprises extracting each of the images, and training using the first and second sets of images with one extracted color channel.

A medical image learning program according to still another aspect of the present invention provides a first image group consisting of a plurality of first medical images captured with normal light and a second image group consisting of a plurality of second medical images captured with special light. A function of acquiring each image group, a function of generating a first model for image recognition by learning using at least the first image group, and learning using a second image group based on the first model. causes the computer to realize a function of generating a second model for performing image recognition on the second medical image.

According to the present invention, a good first model is generated by performing learning using at least the first image group composed of the first medical images captured with normal light, which can be collected in relatively large numbers, and this first model Based on one model, learning is performed using the second image group consisting of a plurality of second medical images captured with special light, so even from the second image group with a relatively small amount of data, can be used to generate a second model that performs appropriate image recognition.

FIG. 1 is a block diagram showing an example of the hardware configuration of a medical image learning device according to the present invention. FIG. 2 is a block diagram showing the first embodiment of the medical image learning device 10-1 according to the present invention. FIG. 3 is a functional block diagram showing an embodiment of the first learning section 30. As shown in FIG. FIG. 4 is a block diagram showing a second embodiment of the medical image learning device 10-2 according to the present invention. FIG. 5 is a block diagram showing a third embodiment of the medical image learning device 10-3 according to the present invention. FIG. 6 is a block diagram showing a fourth embodiment of the medical image learning device 10-4 according to the present invention. FIG. 7 is a flowchart illustrating an embodiment of a medical image learning method according to the present invention. FIG. 8 is a diagram showing step S14-1 showing a first modification of step S14 shown in FIG. FIG. 9 is a diagram showing step S14-2 showing a second modification of step S14 shown in FIG.

Preferred embodiments of the medical image learning apparatus and method according to the present invention will be described below with reference to the accompanying drawings.

[Hardware configuration of medical image learning device]
FIG. 1 is a block diagram showing an example of the hardware configuration of a medical image learning device according to the present invention.

A personal computer or a workstation can be used as the medical image learning device 10 shown in FIG. It comprises a second database 16 , an operation section 18 , a CPU (Central Processing Unit) 20 , a RAM (Random Access Memory) 22 , a ROM (Read Only Memory) 24 and a display section 26 .

The communication unit 12 is a part that performs wired or wireless communication processing with an external device and exchanges information with the external device.

The first database 14 includes a first image group (normal light image group) composed of a plurality of first medical images (normal light images) captured with normal light, and correct data indicating correct image recognition results of each normal light image. The second database 16 stores a second image group (special A second data set for image recognition (for learning) consisting of correct data indicating the correct image recognition result of each special light image is stored.

Here, the normal light image (first medical image) and the special light image (second medical image) are color images captured by the endoscope apparatus under different light sources.

Ordinary light is light (white light) in which light in all wavelength bands of visible light is substantially evenly mixed, and ordinary light images are used for ordinary observation. Therefore, a relatively large number of normal light images can be collected.

On the other hand, the special light is light in one specific wavelength band or light in various wavelength bands in accordance with the purpose of observation, which is a combination of light in a plurality of specific wavelength bands, and has a narrower band than the white wavelength band. and is used for narrow band imaging (NBI (Narrow band imaging), FICE (Flexible spectral imaging color enhancement)).

A first example of a specific wavelength band is, for example, the blue band or the green band in the visible range. The wavelength band of this first example includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light of the first example has a peak wavelength within the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. .

A second example of a specific wavelength band is, for example, the visible red band. The wavelength band of this second example includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the second example has a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. .

A third example of the specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the third example has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients. have The wavelength band of this third example includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example is the above 400±10 nm, 440±10 nm, 470 It has a peak wavelength in the wavelength band of ±10 nm or from 600 nm to 750 nm.

A fourth example of the specific wavelength band is the wavelength band (390 nm to 470 nm) of excitation light used for observation of fluorescence emitted by a fluorescent substance in the living body (fluorescence observation) and for exciting this fluorescent substance.

A fifth example of the specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and the light of the fifth example has a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

Since lesions are difficult to see in special light images captured under special light having such a specific wavelength band, they are only used for observation purposes such as observing surface structures, and the number of data is not large.

In this example, the first data set of the normal light image group stored in the first database 14 is prepared more than the second data set of the special light image group stored in the second database 16. and

Further, in the first database 14 and the second database 16, the correct data stored in association with each normal light image and each special light image are, for example, the types of lesions shown in the normal light image and the special light image, Data indicating the position of a lesion, case-specific identification information, and the like are conceivable. Classification of lesions includes two classifications of neoplastic and non-neoplastic, NICE classification, and the like. Data indicating the position of a lesion may be information on a rectangle surrounding the lesion, mask data that hides the lesion, or the like.

In this example, the first database 14 and the second database 16 are provided in the medical image learning device 10, but they may be provided outside. In this case, a data set for learning can be acquired from an external database via the communication unit 12 .

The operation unit 18 uses a keyboard and a mouse that are wired or wirelessly connected to the computer, and receives various operation inputs for machine learning.

The CPU 20 reads various programs (including a medical image learning program according to the present invention) stored in the ROM 24 or a hard disk device (not shown) or the like, and executes various processes. The RAM 22 is used as a work area for the CPU 20 and is used as a storage unit for temporarily storing read programs and various data.

Various monitors such as a liquid crystal monitor that can be connected to a computer are used as the display unit 26 , and the display unit 26 is used as part of the user interface together with the operation unit 18 .

In the medical image learning apparatus 10 configured as described above, the CPU 20 reads out a medical image learning program stored in the ROM 24, hard disk device, or the like in response to an instruction input from the operation unit 18, and executes the medical image learning program, thereby performing the operation as described later. function as a medical image learning device.

[First embodiment of medical image learning device]
FIG. 2 is a block diagram showing the first embodiment of the medical image learning device 10-1 according to the present invention, and is a functional block diagram showing main functions of the medical image learning device 10 shown in FIG.

The medical image learning device 10-1 shown in FIG. 2 includes a first learning unit 30 and a second learning unit 40.

The first learning unit 30 learns for image recognition by learning using a data set of normal light images stored in the first database 14 and a data set of special light images stored in the second database 16. Generate a model (first model). In this example, a convolution neural network (CNN), which is one of learning models, is constructed.

FIG. 3 is a functional block diagram showing an embodiment of the first learning section 30. As shown in FIG.

The first learning unit 30 shown in FIG. 3 is mainly composed of a CNN 32, an error calculation unit 34, and a parameter updating unit 36.

The CNN 32 is, for example, a part corresponding to a recognizer that recognizes the types of lesions appearing in medical images, has a multiple layer structure, and holds multiple weight parameters. The CNN 32 can change from an unlearned model to a learned model by updating the weight parameter from the initial value to the optimum value.

This CNN 32 comprises an input layer 32A, a hidden layer 32B having multiple sets of convolutional and pooling layers, and a fully connected layer, and an output layer 32C, each layer having a plurality of "nodes" as "edges". It has a structure connected by

The normal light image 14A to be learned is input to the input layer 32A.

The intermediate layer 32B has multiple sets of convolutional layers and pooling layers, and a fully connected layer, and is a part for extracting features from an image input from the input layer. The convolutional layer filters (convolves with filters) the nodes that are nearby in the previous layer to get a "feature map". The pooling layer reduces the feature map output from the convolution layer to a new feature map. The “convolution layer” plays a role of extracting features such as edge extraction from an image, and the “pooling layer” plays a role of providing robustness so that the extracted features are not affected by translation or the like. Note that the intermediate layer 32B is not limited to the case where the convolutional layer and the pooling layer are set as one set, but also includes the case where the convolutional layers are continuous and the normalization layer.

The output layer 32C is a portion that outputs recognition results for classifying the types of lesions appearing in medical images based on the features extracted by the intermediate layer 32B. In the trained CNN 32, for example, medical images are classified into three categories of "neoplastic", "non-neoplastic", and "other", and the recognition results are classified into "neoplastic", "non-neoplastic" and "other ” (the sum of the three scores is 100%).

Arbitrary initial values are set for the filter coefficients and offset values applied to each convolutional layer of the CNN 32 before learning, and the weight of the connection with the next layer in the fully connected layer.

The error calculator 34 acquires the recognition result output from the output layer 32C of the CNN 32 and the correct data for the normal light image 14A, and calculates the error between them. Examples of error calculation methods include softmax cross entropy and sigmoid.

The parameter updating unit 36 adjusts the weight parameters of the CNN 30 by error backpropagation based on the error calculated by the error calculating unit 34 .

This parameter adjustment processing is repeated, and learning is repeated until the difference between the output of the CNN 32 and the correct data becomes small.

The first learning unit 30 uses all data sets of the normal light image group stored in the first database 14, and performs learning to optimize each parameter of the CNN 32, a learned model (first model) to generate

Returning to FIG. 2 , the second learning unit 40 uses the trained model trained by the first learning unit 30 (using the parameters of the trained CNN 32 as initial values), the special model stored in the second database 16 A learning model (second model) for performing image recognition on the special light image is generated by re-learning using only the data set of the light image.

Since the second learning section 40 is configured in the same manner as the first learning section 30 shown in FIG. 3, detailed description thereof will be omitted. Note that the second learning section 40 may be configured by the same learning section as the first learning section 30 .

According to the medical image learning device 10-1 of the first embodiment, learning (pre-learning) is performed using a data set of normal light images that can be collected in relatively large numbers by the first learning unit 30. can successfully generate a learning model of Then, based on the learning model generated by the first learning unit 30, the second learning unit 40 performs learning (also referred to as “fine tuning”) using only the special light image group. It is possible to generate a learning model that performs appropriate image recognition for special light images even from a small group of special light images.

The first learning unit 30 of the medical image learning device 10-1 of the first embodiment learns using the normal light image data set and the special light image data set, but only the normal light image data set may be learned using

[Second Embodiment of Medical Image Learning Apparatus]
FIG. 4 is a block diagram showing a second embodiment of the medical image learning device 10-2 according to the present invention. In the medical image learning device 10-2 shown in FIG. 4, the parts common to the medical image learning device 10-1 of the first embodiment shown in FIG. omitted.

The medical image learning device 10-2 of the second embodiment shown in FIG. 4 differs from the medical image learning device 10-1 of the first embodiment shown in FIG. 2 mainly in that an image processing unit 50 is added. do.

The normal light image stored in the first database 14 and the special light image stored in the second database 16 are each color images consisting of three color channels, red (R), green (G) and blue (B). be an image.

The image processing unit 50 processes a normal light image having three RGB color channels stored in the first database 14 and a special light image having three RGB color channels stored in the second database 16, It converts into a normal light image and a special light image each having one color channel, and outputs the normal light image and the special light image having one color channel after conversion to the first learning unit 30 as a data set for learning. .

The image processing unit 50 performs monochrome processing on the RGB color image to convert it into a normal light image and a special light image having one color channel.

Here, as an example of monochrome processing, for example, a process of generating a luminance signal (Y signal) from RGB color signals (R signal, G signal, B signal) according to the following equation can be considered.

[Number 1]
Y=0.3R+0.59G+0.11B
The first learning unit 30 acquires the normal light image and the special light image (monochrome image) having one color channel converted by the image processing unit 50 as a learning data set, and uses the acquired data set for image recognition. Generate a learning model (first model) for

The second learning unit 40 uses the parameters of the learned CNN learned by the first learning unit 30 as initial values, and re-learns using only the RGB special light image data set stored in the second database 16. By doing so, a learning model for performing image recognition on the special light image is generated.

Here, since the data set used for learning in the first learning unit 30 is a 1-channel image (monochrome image), the filter used at least in the first convolution layer of the CNN is also 1-channel. On the other hand, the data set used for learning in the second learning unit 40 is a special light image of multiple color channels (RGB), so at least the filters used in the first convolution layer of the CNN are also multiple channels (3 channels) become. Therefore, the same parameters are used as initial values for the three-channel filters used in the first convolutional layer of the CNN of the second learning unit 40 .

According to the medical image learning device 10-2 of the second embodiment, pre-learning can be performed by bringing the feature amount of the normal light image closer to the feature amount of the special light image. Re-learning with only the dataset can be performed more appropriately.

Note that the image processing unit 50 of the medical image learning device 10-2 of the second embodiment performs monochrome processing on the normal light image and the special light image, respectively. 30 may be trained using only monochrome processed normal light image data sets.

Further, the image processing unit 50 is not limited to generating a "black and white image" made up of luminance signals, and may generate a single color (one channel) image. An image is preferably generated.

[Third Embodiment of Medical Image Learning Apparatus]
FIG. 5 is a block diagram showing a third embodiment of the medical image learning device 10-3 according to the present invention. In the medical image learning device 10-3 shown in FIG. 5, the parts common to the medical image learning device 10-1 of the first embodiment shown in FIG. omitted.

The medical image learning device 10-3 of the third embodiment shown in FIG. 5 differs from the medical image learning device 10-1 of the first embodiment shown in FIG. 2 mainly in that an extraction unit 60 is added. .

The normal light image stored in the first database 14 and the special light image stored in the second database 16 are assumed to be color images each consisting of three color channels of RGB.

The extraction unit 60 extracts from the normal light image having three color channels of RGB stored in the first database 14 and the special light image having three color channels of RGB stored in the second database 16, respectively. A normal light image and a special light image with one color channel are extracted.

Here, if the special light image stored in the second database 16 is an endoscopic image captured with special light having a peak wavelength in the blue or blue-violet wavelength band, the extraction unit 60 extracts one channel of B It is preferable to extract only normal light images and special light images. Further, when the special light image stored in the second database 16 is an endoscopic image captured with special light having a peak wavelength in the red or near-infrared wavelength band, the extraction unit 60 selects 1 of R It is preferable to extract the channel-only normal and special light images. That is, the extraction unit 60 preferably extracts an image of one color channel that is close to the color of the special light image from the normal light image and the special light image having three color channels of RGB.

The first learning unit 30 acquires the normal light image and the special light image having one color channel extracted by the extraction unit 60 as a data set for learning, and uses the acquired data set as a learning model for image recognition ( 1st model).

The second learning unit 40 uses the parameters of the learned CNN learned by the first learning unit 30 as initial values, and re-learns using only the RGB special light image data set stored in the second database 16. By doing so, a learning model for performing image recognition on the special light image is generated.

According to the medical image learning device 10-3 of the third embodiment, pre-learning can be performed by bringing the feature amount of the normal light image closer to the feature amount of the special light image. Re-learning with only the dataset can be performed more appropriately.

The extraction unit 60 of the medical image learning device 10-3 of the third embodiment extracts normal light images and special light images having one color channel from RGB normal light images and special light images. Only the normal light image having one color channel is extracted from the RGB normal light image, and the first learning unit 30 may learn using only the extracted data set of the normal light image having one color channel. good.

[Fourth Embodiment of Medical Image Learning Apparatus]
FIG. 6 is a block diagram showing a fourth embodiment of the medical image learning device 10-4 according to the present invention. In the medical image learning device 10-4 shown in FIG. 6, the parts common to the medical image learning device 10-2 of the second embodiment shown in FIG. omitted.

The medical image learning device 10-4 of the fourth embodiment shown in FIG. 6 is different from the medical image learning device 10-2 of the second embodiment shown in FIG. 4 mainly in that a third learning unit 70 is added. differ.

The third learning unit 70 uses the learned CNN parameters learned by the first learning unit 30 as initial values, and re-learns using only the RGB normal light image data set stored in the first database 14. Thus, a learning model (third model) for performing image recognition on a normal light image is generated.

According to the medical image learning device 10-4 of the fourth embodiment, the parameters of the learned CNN learned by the first learning unit 30 are used as initial values. The learning time can be shortened compared to the case of learning using only the data set of optical images.

The third learning unit 70 may use the learned CNN parameters learned by the first learning unit 30 of the medical image learning device 10-3 of the third embodiment shown in FIG. 5 as initial values. good.

The normal light image and the special light image stored in the first database 14 and the second database 16 of the present embodiment are RGB "three primary colors of light", respectively. A color image of "three primary colors" of cyan (C), magenta (M), and yellow (Y), which are complementary colors, may also be used.

The normal light image and the special light image stored in the first database 14 and the second database 16 are composed of a luminance signal (Y) generated from RGB color signals and two color difference signals (Cr, Cb). A 3-channel color image may also be used.

Needless to say, RGB, CMY, and YCrCb color images can be mutually converted.

[Medical image learning method]
FIG. 7 is a flow chart showing an embodiment of the medical image learning method according to the present invention, and shows processing procedures of each part of the medical image learning apparatus 10-1 of the first embodiment shown in FIG.

In the first database 14 and the second database 16, a data set of normal light image groups and a data set of special light image groups for learning are stored in advance (step of preparing data sets).

The first learning unit 30 acquires the data set of the normal light image group and the data set of the special light image group from the first database 14 and the second database 16 (steps S10 and S12). The first learning unit 30 may acquire the data set in units of about 10 to 100 mini-batches, or may acquire the data sets one by one.

The first learning unit 30 generates a learning model (first model) for image recognition by learning using the acquired data sets of the normal light image and the special light image (step S14). In this example, a trained CNN, which is one of the learning models, is constructed.

Subsequently, the second learning unit 40, based on the learned model learned by the first learning unit 30 (using the parameters of the learned CNN as initial values), the special light image stored in the second database 16 By learning again using only the data set, a learning model (second model) for performing image recognition on the special light image is generated (step S16).

According to the medical image learning method of this embodiment, since learning (pre-learning) is performed using a data set of normal light images that can be collected in relatively large numbers by the first learning unit 30, a learning model for image recognition is prepared. can be produced well. Then, based on the learning model generated by the first learning unit 30, the second learning unit 40 re-learns using only the special light image group. It is possible to generate a learning model that performs appropriate image recognition for special light images.

In step S14, the first learning unit 30 generates a learning model for image recognition by learning using the data sets of the normal light image and the special light image, but uses only the data set of the normal light image. You can learn

<First modification>
FIG. 8 is a diagram showing step S14-1 showing a first modification of step S14 shown in FIG.

In FIG. 8, the image processing unit 50 (FIG. 4) processes a normal light image having three color channels of RGB stored in the first database 14 and a three color channel of RGB stored in the second database 16. The special light image having channels is converted into a normal light image and a special light image (monochrome image) each having one color channel (step S20).

The first learning unit 30 acquires the monochrome image converted by the image processing unit 50 as a learning data set, and uses the acquired data set to acquire a CNN learning model (first learning model), which is one of the learning models for image recognition. 1 model) is generated (step S22). The parameters of the learned CNN learned in step S22 are used as the initial values of the CNN of the second learning unit 40 in step S16 shown in FIG.

<Second modification>
FIG. 9 is a diagram showing step S14-2 showing a second modification of step S14 shown in FIG.

9, the extractor 60 (FIG. 5) extracts a normal light image with three RGB color channels stored in the first database 14 and a three RGB color channel image stored in the second database 16. A normal light image and a special light image each having one color channel are extracted from the special light image having , (step S30). The extraction unit 60 preferably extracts an image of one color channel that is close to the color of the special light image from the normal light image and the special light image having three color channels of RGB.

The first learning unit 30 acquires the normal light image and the special light image having one color channel extracted by the extraction unit 60 as a learning data set, and uses the acquired data set to develop a learning model for image recognition. A learning model (first model) of one CNN is generated (step S32). The parameters of the learned CNN learned in step S32 are used as the initial values of the CNN of the second learning unit 40 in step S16 shown in FIG.

[others]
The CNN 32 shown in FIG. 3 is a learning model for image recognition of the types of lesions shown in medical images, but it may be a learning model that performs segmentation to recognize the position (lesion area) of lesions shown in medical images. . The CNN in this case preferably applies a fully convolutional network (FCN), which is a type of CNN, and can grasp the position of a lesion in a medical image at the pixel level.

The present invention can also be applied to machine learning models other than CNN, such as DBN (Deep Belief Network) and SVM (Support Vector Machine).

Further, hardware structures for executing various controls of the medical image learning apparatus 10 of this embodiment are various processors as shown below. For various processors, the circuit configuration can be changed after manufacturing such as CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), which is a general-purpose processor that executes software (program) and functions as various control units. Programmable Logic Device (PLD), which is a processor, ASIC (Application Specific Integrated Circuit), etc. be

One processing unit may be composed of one of these various processors, or composed of two or more processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA). may Also, a plurality of control units may be configured by one processor. As an example of configuring a plurality of control units with one processor, first, one processor is configured with a combination of one or more CPUs and software, as typified by a computer such as a client or a server. There is a form in which a processor functions as multiple controllers. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple control units with a single IC (Integrated Circuit) chip. be. In this way, various control units are configured using one or more of the above various processors as a hardware structure.

Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit combining circuit elements such as semiconductor elements.

The present invention also includes a medical image learning program that is installed in a computer to function as the medical image learning device according to the present invention, and a recording medium in which the medical image learning program is recorded.

Furthermore, the present invention is not limited to the embodiments described above, and it goes without saying that various modifications are possible without departing from the spirit of the present invention.

10, 10-1, 10-2, 10-3, 10-4 Medical image learning device 12 Communication unit 14 First database 14A Normal light image 16 Second database 18 Operation unit 20 CPU
22 RAMs
24 ROMs
26 display unit 30 first learning unit 32A input layer 32B intermediate layer 32C output layer 34 error calculation unit 36 parameter updating unit 40 second learning unit 50 image processing unit 60 extraction unit 70 third learning unit S10, S12, S14, S14- 1, S14-2, S16, S20, S22, S30, S32 Step

Claims (18)

a first learning unit that generates a first model for image recognition by learning using at least a first image group composed of a plurality of first medical images captured with normal light;
Based on the first model, learning is performed using a second image group consisting of a plurality of second medical images captured with special light to generate a second model for image recognition of the second medical image. a second learning unit;
an image processor that transforms the first medical image having multiple color channels into the first medical image having a single color channel;
The image processing unit converts the first medical image having the plurality of color channels into a monochrome image that approximates the hue of the second medical image, thereby obtaining the first medical image having the one color channel. year,
The first learning unit learns using the first image group having the transformed one color channel.
Medical image learning device. a first learning unit that generates a first model for image recognition by learning using at least a first image group composed of a plurality of first medical images captured with normal light;
Based on the first model, learning is performed using a second image group consisting of a plurality of second medical images captured with special light to generate a second model for image recognition of the second medical image. a second learning unit;
an image processor that transforms the first medical image having multiple color channels into the first medical image having a single color channel;
The image processing unit converts the first medical image having the plurality of color channels into the first medical image having only a luminance signal to obtain the first medical image having the one color channel;
The first learning unit learns using the first image group having the transformed one color channel.
Medical image learning device. a first learning unit that generates a first model for image recognition by learning using at least a first image group composed of a plurality of first medical images captured with normal light;
Based on the first model, learning is performed using a second image group consisting of a plurality of second medical images captured with special light to generate a second model for image recognition of the second medical image. a second learning unit;
an image processing unit that converts the first medical image and the second medical image having multiple color channels into the first medical image and the second medical image each having one color channel;
The image processing unit converts the first medical image and the second medical image having the plurality of color channels into a monochrome image that approximates the hue of the second medical image, thereby converting the one color channel The first medical image and the second medical image having
The first learning unit learns using the first group of images and the second group of images having one converted color channel.
Medical image learning device. a first learning unit that generates a first model for image recognition by learning using at least a first image group composed of a plurality of first medical images captured with normal light;
Based on the first model, learning is performed using a second image group consisting of a plurality of second medical images captured with special light to generate a second model for image recognition of the second medical image. a second learning unit;
an image processing unit that converts the first medical image and the second medical image having multiple color channels into the first medical image and the second medical image each having one color channel;
The image processing unit converts the first medical image and the second medical image having the plurality of color channels into the first medical image and the second medical image having only luminance signals, thereby performing the first medical image and the second medical image. the first medical image and the second medical image having two color channels;
The first learning unit learns using the first group of images and the second group of images having one converted color channel.
Medical image learning device. a first learning unit that generates a first model for image recognition by learning using at least a first image group composed of a plurality of first medical images captured with normal light;
Based on the first model, learning is performed using a second image group consisting of a plurality of second medical images captured with special light to generate a second model for image recognition of the second medical image. a second learning unit;
an extraction unit for extracting the first medical image having one color channel from the first medical image having multiple color channels;
The extracting unit extracts an image having a color close to that of the second medical image from the first medical image having the plurality of color channels to obtain the first medical image having the one color channel,
The first learning unit learns using the first image group having the extracted one color channel.
Medical image learning device. a first learning unit that generates a first model for image recognition by learning using at least a first image group composed of a plurality of first medical images captured with normal light;
Based on the first model, learning is performed using a second image group consisting of a plurality of second medical images captured with special light to generate a second model for image recognition of the second medical image. a second learning unit;
an extraction unit for extracting the first medical image and the second medical image having one color channel from the first medical image and the second medical image having multiple color channels, respectively;
The extraction unit extracts an image having a color close to that of the second medical image from the first medical image and the second medical image having the plurality of color channels, thereby obtaining the image having the one color channel. As a first medical image and the second medical image,
The first learning unit learns using the first image group and the second image group having the extracted one color channel.
Medical image learning device. The medical image learning apparatus according to any one of claims 1 to 6 , wherein the plurality of color channels are three channels of three primary colors, or three channels of a luminance signal and two color difference signals. a third learning unit that generates a third model for image recognition of the first medical image by learning using the first image group based on the first model;
The medical image learning device according to any one of claims 1 to 7 , comprising: The medical image learning device according to any one of claims 1 to 8 , wherein the first medical image and the second medical image are images captured by an endoscope apparatus. 10. The medical image learning device according to any one of claims 1 to 9 , wherein the first model and the second model are configured by convolutional neural networks. preparing a first image group consisting of a plurality of first medical images captured with normal light and a second image group consisting of a plurality of second medical images captured with special light;
a first learning unit generating a first model for image recognition by learning using at least the first image group;
a second learning unit learning using the second image group based on the first model to generate a second model for performing image recognition on the second medical image;
an image processor transforming the first medical image having multiple color channels into the first medical image having a single color channel;
The converting step converts the first medical image having the plurality of color channels into a monochrome image that approximates the hue of the second medical image, thereby obtaining the first medical image having the one color channel. year,
generating the first model trains using the first set of images having the transformed one color channel;
Medical image learning device. preparing a first image group consisting of a plurality of first medical images captured with normal light and a second image group consisting of a plurality of second medical images captured with special light;
a first learning unit generating a first model for image recognition by learning using at least the first image group;
a second learning unit learning using the second image group based on the first model to generate a second model for performing image recognition on the second medical image;
an image processor transforming the first medical image having multiple color channels into the first medical image having a single color channel;
the converting step includes converting the first medical image having the plurality of color channels into the first medical image having only a luminance signal, thereby forming the first medical image having the one color channel;
generating the first model trains using the first set of images having the transformed one color channel;
Medical image learning method. preparing a first image group consisting of a plurality of first medical images captured with normal light and a second image group consisting of a plurality of second medical images captured with special light;
a first learning unit generating a first model for image recognition by learning using at least the first image group;
a second learning unit learning using the second image group based on the first model to generate a second model for performing image recognition on the second medical image;
an image processor transforming the first medical image and the second medical image having multiple color channels into the first medical image and the second medical image having a single color channel;
The converting step converts the first medical image and the second medical image having the plurality of color channels into a monochrome image that approximates the hue of the second medical image, thereby converting the one color channel. The first medical image and the second medical image having
generating the first model trains using the first set of images and the second set of images having the transformed one color channel;
Medical image learning method. preparing a first image group consisting of a plurality of first medical images captured with normal light and a second image group consisting of a plurality of second medical images captured with special light;
a first learning unit generating a first model for image recognition by learning using at least the first image group;
a second learning unit learning using the second image group based on the first model to generate a second model for performing image recognition on the second medical image;
an image processor transforming the first medical image and the second medical image having multiple color channels into the first medical image and the second medical image having a single color channel;
The converting step converts the first medical image and the second medical image having the plurality of color channels into the first medical image and the second medical image having only luminance signals, The first medical image and the second medical image having color channels;
generating the first model trains using the first set of images and the second set of images having the transformed one color channel;
Medical image learning method. preparing a first image group consisting of a plurality of first medical images captured with normal light and a second image group consisting of a plurality of second medical images captured with special light;
a first learning unit generating a first model for image recognition by learning using at least the first image group;
a second learning unit learning using the second image group based on the first model to generate a second model for performing image recognition on the second medical image;
an extractor extracting the first medical image having one color channel from the first medical image having multiple color channels;
The extracting step extracts an image having a color close to that of the second medical image from the first medical image having the plurality of color channels, so as to obtain the first medical image having the one color channel. ,
generating the first model trains using the first set of images having the extracted one color channel;
Medical image learning method. preparing a first image group consisting of a plurality of first medical images captured with normal light and a second image group consisting of a plurality of second medical images captured with special light;
a first learning unit generating a first model for image recognition by learning using at least the first image group;
a second learning unit learning using the second image group based on the first model to generate a second model for performing image recognition on the second medical image;
an extractor extracting the first medical image and the second medical image having one color channel from the first medical image and the second medical image having multiple color channels, respectively;
The extracting step includes extracting an image close to the color tone of the second medical image from the first medical image and the second medical image having the plurality of color channels, and having the one color channel. The first medical image and the second medical image,
generating the first model trains using the first set of images and the second set of images having the extracted one color channel;
Medical image learning method. A medical image learning program that causes a computer to implement the medical image learning method according to any one of claims 12 to 16 . A non-transitory computer-readable recording medium, in which the medical image learning program according to claim 17 is recorded.

Images