JP7106741B2 - Learning method, learning device, generative model and program - Google Patents

Description

The present invention relates to a learning method, a learning device, a generation model, and a program, and more particularly to technology for generating high-resolution images.

Generative Adversarial Networks (GANs) are known that alternately learn a "Generator" that creates data and a "Discriminator" that identifies data. Non-Patent Document 1 describes research on GANs. According to GAN, it is possible to learn a generative model that generates highly accurate data along the characteristics of learning data.

In addition, research is being conducted on techniques to which GANs are applied. Non-Patent Literature 2 describes a method of learning a pair of an input image and an output image using a GAN. According to the technique described in Non-Patent Document 2, it is possible to learn a generative model with less burden of parameter adjustment.

GAN image generation has a problem that learning is difficult when there is diversity in the input data. For example, in a medical image, if the input image has a wide variety of parts and slice conditions, the features of the generated image are averaged. Even in dealing with various input data in Non-Patent Document 2, the features of learning data are averaged.

On the other hand, Non-Patent Document 3 describes a method of using data category information for GAN learning. According to the method described in Non-Patent Document 3, the generative model can be adjusted based on the category information.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio “Generative Adversarial Nets”, arXiv:1406.2661 Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros “Image-to-Image Translation with Conditional Adversarial Networks”,CVPR2016 Mehdi Mirza, Simon Osindero “Conditional Generative Adversarial Nets”, arXiv:1411.1784

However, in the GAN configuration described in Non-Patent Document 3, it is necessary to input category information to the Generator, so there is a problem that the network architecture needs to be significantly modified and the Generator interface becomes complicated. .

In addition, GAN is strongly affected by some learning data, and there is a problem that it may fall into a state called Mode collapse, in which images to be generated become limited and cannot cope with diversity.

The present invention has been devised in view of such circumstances, and is capable of performing diverse learning and generating images containing high-resolution information without adjusting parameters or significantly modifying the network architecture. An object is to provide a method, a learning device, a generative model and a program.

One aspect of the learning method for achieving the above object is generation of estimating a second image containing high-resolution information with relatively high resolution from a first image containing low-resolution information with relatively low resolution. A method of training a model, comprising: a generator that is a generative model that uses a first image to generate a virtual second image; and a discriminator that distinguishes between the training second image and the virtual second image. a generator input step of inputting only the first image to the generator of the target generation network; a first discriminator input step of inputting the second learning image and part information of the second learning image to the discriminator; a second discriminator input step of inputting a second virtual image and part information of the second virtual image to a discriminator.

According to this aspect, only the first image is input to the generator of the hostile generation network, part information is input to the discriminator, and learning is performed for each condition by adding the part information to the input. , it can learn with diversity and generate images containing high-resolution information without parameter tuning and significant modification of the network architecture.

The first image containing low-resolution information with relatively low resolution is, for example, low-resolution. The second image containing high-resolution information with relatively high resolution is, for example, a high-resolution image. Also, the second image may be a high-frequency component image containing high-resolution information with relatively high resolution.

The part information preferably includes the head, chest, abdomen, and legs of the human body. As a result, an image containing high-resolution information can be generated from an image containing low-resolution information of various parts of the human body.

The first image is a three-dimensional tomographic image, and the resolution is preferably the resolution in the slice thickness direction. Thereby, a three-dimensional tomographic image including high-resolution information in the slice thickness direction can be generated.

Preferably, the first discriminator input step and the second discriminator input step input slice information of the first image to the discriminator. As a result, a high-resolution image can be generated even when slice information of an input image is diverse.

The slice information is preferably a slice interval. As a result, a high-resolution image can be generated even when the input image has a variety of slice intervals.

The slice information is preferably slice thickness. As a result, a high-resolution image can be generated even when the slice thickness of the input image is diverse.

Preferably, the first discriminator input step and the second discriminator input step input the first image to the discriminator. Thereby, the discriminator can appropriately identify the second learning image and the virtual second image.

Preferably, the first image generating step generates the first image from the second learning image, and the generator input step inputs only the generated first image to the generator. Thereby, an image including low-resolution information for inputting to the generator can be properly acquired.

One aspect of a program to be executed by a computer to achieve the above object is a program for causing a computer to execute the learning method described above.

One aspect of the generative model for achieving the above object is a high-resolution relatively high-resolution first image containing relatively low-resolution low-resolution information learned by the learning method described above. A generative model for estimating a second image containing image information.

One aspect of the learning device for achieving the above object is generation of estimating a second image containing high-resolution information with relatively high resolution from a first image containing low-resolution information with relatively low resolution. Apparatus for learning a model, comprising: a generator, a generative model that uses a first image to generate a virtual second image; and a discriminator that distinguishes between the training second image and the virtual second image. a generator input unit that inputs only the first image to the generator; and a first discriminator input unit that inputs the second learning image and part information of the second learning image to the discriminator; and a second discriminator input unit for inputting a second virtual image and part information of the second virtual image to the discriminator.

According to this aspect, only the first image is input to the generator of the hostile generation network, part information is input to the discriminator, and learning is performed for each condition by adding the part information to the input. , it can learn with diversity and generate images containing high-resolution information without parameter tuning and significant modification of the network architecture.

According to the present invention, it is possible to perform diverse learning and generate images containing high-resolution information without adjusting parameters and modifying network architecture significantly.

FIG. 1 is a block diagram showing an example of the hardware configuration of a medical image learning device according to the present invention. FIG. 2 is a diagram for explaining the slice interval and slice thickness of a CT image. FIG. 3 is a diagram for explaining the slice interval and slice thickness of a CT image. FIG. 4 is a diagram for explaining the slice interval and slice thickness of a CT image. FIG. 5 is a diagram for explaining the slice interval and slice thickness of a CT image. FIG. 6 is a functional block diagram showing learning processing of the medical image learning device 10 according to the first embodiment. FIG. 7 is a diagram for explaining identification by the identification unit 44 during learning. FIG. 8 is a flow chart showing an example of the medical image learning method of the medical image learning device 10. As shown in FIG. FIG. 9 is a diagram for explaining image generation by learning a whole-body image without providing the meta-information D1. FIG. 10 is a diagram for explaining image generation by learning whole-body images without providing the meta information D1. FIG. 11 is a diagram showing an image of a leg. FIG. 12 is a diagram for explaining the effect of the embodiment; FIG. 13 is a diagram for explaining the effect of the embodiment; FIG. 14 is a diagram for explaining the effect of the embodiment; FIG. 15 is a functional block diagram showing learning processing of the medical image learning device 10 according to the second embodiment. FIG. 16 is a functional block diagram showing learning processing of the medical image learning device 10 according to the third embodiment. FIG. 17 is a functional block diagram showing learning processing of the medical image learning device 10 according to the fourth embodiment.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

<Hardware configuration of medical image learning device>
FIG. 1 is a block diagram showing an example of the hardware configuration of a medical image learning device according to the present invention. The medical image learning device 10 is configured by a personal computer or workstation. The medical image learning device 10 mainly includes a communication unit 12, an image database 14, an operation unit 16, a CPU (Central Processing Unit) 18, a RAM (Random Access Memory) 20, a ROM (Read Only Memory) 22, and a display unit 24. be done.

The communication unit 12 is an interface that performs wired or wireless communication processing with an external device and exchanges information with the external device.

The image database 14 is a large-capacity storage device that stores CT reconstructed images (CT images) captured by a medical X-ray CT (Computed Tomography) apparatus. The image database 14 may be provided outside the medical image learning device 10 . In this case, the medical image learning device 10 acquires CT images from the image database 14 via the communication unit 12 .

The operation unit 16 is an input interface that receives various operation inputs to the medical image learning device 10 . The operation unit 16 uses a keyboard, mouse, or the like (not shown) that is wired or wirelessly connected to the computer.

The CPU 18 reads various programs stored in the ROM 22 or a hard disk device (not shown) and executes various processes. RAM 20 is used as a work area for CPU 18 . Also, the RAM 20 is used as a storage unit that temporarily stores read programs and various data. The medical image learning device 10 may include a GPU (Graphics Processing Unit).

The display unit 24 is an output interface that displays information necessary for the medical image learning device 10 . A liquid crystal monitor (not shown) or the like connectable to a computer is used as the display unit 24 .

In the medical image learning apparatus 10, the CPU 18 reads out a medical image learning program stored in the ROM 22, a hard disk device, or the like, and executes the medical image learning program according to an instruction input from the operation unit 16. FIG. As a result, a medical image learning method, which will be described later, is performed to estimate a second image containing high-resolution information with a relatively high resolution from a first image containing low-resolution information with a relatively low resolution in the CT image. A generative model is trained.

A medical image learning program that causes a computer to execute the medical image learning method may be stored in a computer-readable non-temporary recording medium and provided.

<Part information and slice information of CT image>
The CT images stored in the image database 14 are medical images obtained by imaging a human body (subject), and are three-dimensional tomographic images including a plurality of slice images (axial cross-sectional images). Here, each slice image is an image parallel to the mutually orthogonal X and Y directions. The Z direction perpendicular to the X and Y directions is the body axis direction of the subject and is also called the slice thickness direction.

The CT images stored in the image database 14 are images of parts of the human body. The CT image may be an image captured for each part, or may be an image cut out for each part from an image obtained by capturing the whole body. In each CT image stored in the image database 14, part information and slice information are stored in association with each other.

The part information is information indicating the part shown in the CT image. The part information includes the head, chest, abdomen, and legs. The part information may include the waist.

Also, the slice information includes at least one of the slice interval and slice thickness of the CT image. The slice interval is the difference in Z-direction position between adjacent slice images, and is also called slice spacing. The slice thickness is the width (thickness) of one slice image in the Z direction, and is also called slice thickness.

2 to 5 are diagrams for explaining the slice interval and slice thickness of CT images. A CT image G1 shown in FIG. 2 includes slice images G11, G12, and G13. The CT image G1 has a slice interval P of 4 mm and a slice thickness T of 4 mm. That is, the difference in Z-direction position between slice image G11 and slice image G12 and the difference in Z-direction position between slice image G12 and slice image G13 are each 4 mm. Also, the thickness in the Z direction of the slice images G11, G12, and G13 is 4 mm. Since the CT image G1 has the same slice interval P and slice thickness T, the slice image G11 and the slice image G12, and the slice image G12 and the slice image G13 are in contact with each other in the Z direction.

A CT image G2 shown in FIG. 3 includes slice images G21, G22, and G23. The CT image G2 has a slice interval P of 4 mm and a slice thickness T of 6 mm. Since the slice thickness T of the CT image G2 is larger than the slice interval P, the slice images G21 and G22, and the slice images G22 and G23 overlap in the Z direction. do.

A CT image G3 shown in FIG. 4 includes slice images G31, G32, and G33. The CT image G3 has a slice interval P of 8 mm and a slice thickness T of 4 mm. Since the slice interval P of the CT image G3 is larger than the slice thickness T , the slice images G31 and G32 , and the slice images G32 and G33 are separated in the Z direction. .

A CT image G4 shown in FIG. 5 includes slice images G41, G42, and G43. The CT image G4 has a slice interval P of 1 mm and a slice thickness T of 1 mm. CT image G4 has more information in the Z direction than CT images G1, G2, and G3. That is, the CT image G4 has relatively higher resolution in the Z direction than the CT images G1, G2, and G3.

The slice intervals and slice thicknesses of CT images are set under various conditions according to the preferences of facilities, doctors, etc., using the CT apparatus. A CT image preferably has a high resolution for diagnosis, but there is a problem that the dose of exposure to the subject increases. In addition, since the CT image has a large amount of data, it may be saved at a low resolution in order to reduce the capacity.

The medical image learning apparatus 10 uses low-resolution CT images with slice intervals and slice thicknesses under various conditions as shown in FIGS. A generative model for estimating CT images is trained.

<First embodiment>
[Functional block of medical image learning device]
FIG. 6 is a functional block diagram showing learning processing of the medical image learning device 10 according to the first embodiment. The medical image learning device 10 includes an input section 30 and a learning section 40 . The functions of the input unit 30 and the learning unit 40 are realized by each unit of the medical image learning device 10 shown in FIG.

The input unit 30 generates data (learning data) necessary for the learning unit 40 to learn, and inputs the generated learning data to the learning unit 40 . The input unit 30 receives an original high-resolution image GR (Real Thin image) captured by a CT apparatus (not shown) from the image database 14 (see FIG. 1), and part information and slice information associated with the original high-resolution image GR. get.

The input unit 30 has a low resolution conversion unit 32 . The low-resolution conversion unit 32 artificially generates various low-resolution images GL1 (thick images) based on the input data of the original high-resolution image GR. For example, the low-resolution conversion unit 32 performs attitude conversion on a slice image isotropic to 1 mm, cuts out regions at random, and converts a low-resolution image GL1 with a slice interval and a slice thickness of 4 mm or 8 mm, etc., into a virtual image. to generate

Alternatively, the low-resolution conversion unit 32 may generate the low-resolution image GL1 by down-sampling processing that simply thins out and reduces the input original high-resolution image GR.

The input unit 30 inputs the original high resolution image GR and the low resolution image GL1 to the learning unit 40 . Furthermore, the input unit 30 inputs meta information D1 (Condition) to the learning unit 40 .

The meta information D1 includes part information and slice information. The part information is information common to the original high-resolution image GR and the low-resolution image GL1. As the slice information, the slice information of the low-resolution image GL1 is used. The slice information of the low-resolution image GL1 can be obtained from the slice information associated with the original high-resolution image GR and the contents of the low-resolution conversion processing performed by the low-resolution conversion unit 32.

The learning unit 40 is based on a structure obtained by extending the architecture described in Non-Patent Document 3 from two-dimensional data to three-dimensional data. The learning unit 40 builds Generative Adversarial Networks (GAN). A GAN has a "Generator" that creates data and a "Discriminator" that identifies data. The learning unit 40 includes a generating unit 42 corresponding to "Generator" and a discriminating unit 44 corresponding to "Discriminator". The learning unit 40 learns the generation unit 42 by repeating hostile learning using the identification unit 44 based on the data input from the input unit 30 . The generation unit 42 corresponds to a generation model for estimating a second image containing high-resolution information with relatively high resolution from a first image containing low-resolution information with relatively low resolution.

Only the low-resolution image GL1 (an example of the first image) is input to the generation unit 42 from the input unit 30 (an example of the generator input unit). The generator 42 generates a virtual high-resolution image GV having the same resolution as the original high-resolution image GR from the input low-resolution image GL1.

The identifying unit 44 receives a pair of an original high-resolution image GR (an example of a second image for learning) and a low-resolution image GL1 input from the input unit 30, or a pair from the generating unit 42 (an example of a discriminator input unit). A pair of an input virtual high-resolution image GV (an example of a second virtual image) and a low-resolution image GL1 is input.

Furthermore, a new channel is reserved for the identification unit 44 . Meta information D1 is input to the new channel of the identification unit 44 . Using the low-resolution image GL1 and the meta information D1, the identification unit 44 identifies whether the input high-resolution image is the original high-resolution image GR or the virtual high-resolution image GV. The identification result of the identification unit 44 is input to the generation unit 42 .

FIG. 7 is a diagram for explaining identification by the identification unit 44 during learning. In the case of F7A shown in FIG. 7, the low-resolution image GL1, the meta information D1, and the original high-resolution image GR are input to the identification unit 44. FIG. Therefore, if the identifying unit 44 identifies the input high-resolution image as the original high-resolution image GR, it is correct, and if it identifies it as the virtual high-resolution image GV, it is incorrect.

On the other hand, in the case of F7B shown in FIG. 7, the low-resolution image GL1, the meta information D1, and the virtual high-resolution image GV generated by the generation unit 42 are input to the identification unit 44. FIG. Therefore, if the identifying unit 44 identifies the input high-resolution image as the original high-resolution image GR, it is incorrect, and if it identifies it as the virtual high-resolution image GV, it is correct.

The identification unit 44 correctly identifies whether the input high-resolution image is a real CT image captured by a CT device (not shown) or a virtual CT image generated by the generation unit 42. is learned as On the other hand, the generation unit 42 generates a virtual CT image that resembles a real CT image taken by a CT device (not shown), and the identification unit 44 learns to make the identification incorrect.

As the learning progresses, the generation unit 42 and the identification unit 44 mutually improve their accuracy, and the generation unit 42 can generate a virtual high-resolution image GV that is closer to the real CT image and is not identified by the identification unit 44 . The generator 42 acquired by this learning is the generative model. The generation unit 42 and the identification unit 44 are trained using, for example, a convolutional neural network.

[Medical image learning method]
FIG. 8 is a flow chart showing an example of the medical image learning method of the medical image learning device 10. As shown in FIG. As shown in FIG. 8, the medical image learning method includes an original image acquisition step (step S1), a low resolution image generation step (step S2), a low resolution image input step (step S3), a virtual high resolution image generation step (step S4), a meta information input step (step S5), a high resolution image input step (step S6), and a learning step (step S7).

At step S<b>1 , the input unit 30 acquires the original high-resolution image GR from the image database 14 . Here, an original high-resolution image GR with a slice interval of 1 mm and a slice thickness of 1 mm is acquired.

In step S2 (an example of the first image generating step), the low-resolution converter 32 generates a low-resolution image GL1 from the input original high-resolution image GR. Here, the low-resolution converter 32 generates a low-resolution image GL1 with a slice interval of 5 mm and a slice thickness of 1 mm. That is, the original high-resolution image GR and the low-resolution image GL1 have different resolutions in the Z direction, but have the same resolution in the X and Y directions.

In step S<b>3 (an example of a generator input step), the input unit 30 inputs the low-resolution image GL<b>1 generated in step S<b>2 to the generation unit 42 of the learning unit 40 . Only the low-resolution image GL1 is input to the generation unit 42 .

In step S4, the generator 42 generates a virtual high-resolution image GV from the input low-resolution image GL1. Here, the generator 42 generates a virtual high-resolution image GV with a slice interval of 1 mm and a slice thickness of 1 mm.

In step S5 (an example of a first discriminator input step, an example of a second discriminator input step), the input unit 30 (an example of a first discriminator input unit, an example of a second discriminator input unit) , the meta information D1 is input to the identification unit 44 of the learning unit 40 . The meta information D1 is the part information of the high-resolution image and the slice information of the low-resolution image GL1 that are input to the identification unit 44 in step S6. The part information of the original high-resolution image GR and the virtual high-resolution image GV is common to the part information of the low-resolution image GL1. Therefore, the input unit 30 may input the part information of the low-resolution image GL1 to the identifying part 44 as the part information of the meta information D1.

In step S<b>6 , the input unit 30 inputs the high-resolution image to the identification unit 44 of the learning unit 40 . The high resolution image is the original high resolution image GR or the virtual high resolution image GV. That is, the input unit 30 (an example of a first discriminator input unit) inputs the original high-resolution image GR to the identification unit 44 (an example of a first discriminator input step), or the input unit 30 (an example of a second discriminator input step). An example of a discriminator input unit) inputs the virtual high-resolution image GV to the identification unit 44 (an example of a second discriminator input step). Furthermore, the input unit 30 inputs the low-resolution image GL1 to the identification unit 44 (an example of the first discriminator input unit, an example of the second discriminator input unit).

In step S7, the identification unit 44 identifies whether the input high-resolution image is the original high-resolution image GR or the virtual high-resolution image GV using the input low-resolution image GL1 and meta information D1. do. The identification result of the identification unit 44 is input to the generation unit 42 . Thereby, the generation unit 42 and the identification unit 44 are learned.

By repeating the learning method described above, the identification unit 44 learns so as to be able to more preferably identify whether the input high-resolution image is the original high-resolution image GR or the virtual high-resolution image GV. On the other hand, the generator 42 learns to generate a virtual high-resolution image GV close to the original high-resolution image GR. Thereby, the generator 42 can estimate the virtual high-resolution image GV with relatively high resolution from the low-resolution image GL1 with relatively low resolution.

9 and 10 are diagrams for explaining image generation in the generation unit 42 that has learned the whole body image without providing the identification unit 44 with the meta information D1. FIG. 9 shows a sagittal cross-sectional image of a CT image of the head. F9A shown in FIG. 9 is the low-resolution image input to the generation unit 42. FIG. F9B shown in FIG. 9 is the virtual high-resolution image output from the generator 42. FIG. F9C shown in FIG. 9 is the ground truth of F9B and the original high-resolution image used as the original data of F9A. FIG. 10 shows a sagittal cross-sectional image of a CT image of the waist. F10A, F10B , and F10C shown in FIG. 10 are an input low-resolution image, an output virtual high-resolution image, and correct data, respectively.

Ideally, the generating unit 42 outputs a high-resolution image close to the correct data when a low-resolution image is input, regardless of the body part from the head to the legs. However, as shown in FIGS. 9 and 10, when learning is performed without providing the meta-information D1 to the identifying unit 44, the virtual high-resolution image generated by the generating unit 42 is affected by learning of other parts, and the features are different. Averaging resulted in a blurred image. The reason for this is thought to be the diversity of input data.

FIG. 11 shows a sagittal section image of a CT image of the leg. F11A shown in FIG. 11 is a low-resolution image of the leg. F11B shown in FIG. 11 is a high-resolution image of the leg. As shown in FIG. 11, the leg image has a lot of background and vertical information, and the change between the low resolution image and the high resolution image is relatively small. For this reason, the generation unit 42 learns to “almost do nothing” for generation of high-resolution images of the legs. However, there is a problem that this learning adversely affects the learning for generating high-resolution images of the head, chest, and abdomen.

In addition, the identification unit 44 has a problem that the accuracy is not improved due to confusion due to the diversity of the parts of the image.

As one method for solving such a problem, it is conceivable to learn and create a classification (super-resolution image output model) for each part. However, with this method, there are new problems such as the need to create models for each part, the need to determine the part of the data at the time of inference, and the need to process data that spans parts. .

12 to 14 are diagrams for explaining the effect of this embodiment. FIG. 12 shows a sagittal cross-sectional image of a CT image of the head. F12A shown in FIG. 12 is a low-resolution image input to the generator 42, which is a learned generative model. F12B shown in FIG. 12 is the virtual high-resolution image output from the generation unit 42 . F12C shown in FIG. 12 is ground truth data of F12B, and is an original high-resolution image used as original data of F12A. FIG. 13 shows a sagittal cross-sectional image of a CT image of the abdomen. F13A, F13B, and F13C shown in FIG. 13 are an input low-resolution image, an output virtual high-resolution image, and correct data, respectively. FIG. 14 shows a sagittal cross-sectional image of a CT image of the waist. F14A, F14B, and F14C shown in FIG. 14 are images of the waist, which are an input low-resolution image, an output virtual high-resolution image, and correct data, respectively.

As shown in FIGS. 12 to 14, according to the generating unit 42 learned according to the present embodiment, by inputting the meta information D1 into the identifying unit 44 and learning, high-precision images can be obtained regardless of the site, slice interval, or the like. image generation is possible. This generator 42 can be applied to a VTS (Virtual Thin Slice) device.

According to this embodiment, there is no need to prepare a generative model for each part. In this embodiment, since only low-resolution images are input to the generation unit 42, significant modification of the network architecture is unnecessary. Further, the identification unit 44 can learn using part information as a feature amount. As a result, the trained generative model does not need information about which part of the image the input low-resolution image is, and can generate a high-resolution image regardless of which part of the image is input. can. Therefore, assets of past CT images can be effectively used. Note that this generation model can generate a high-resolution image even when an image spanning a plurality of parts is input.

Here, the input unit 30 inputs the part information and the slice information to the identifying part 44 as the meta information D1, but the meta information D1 may include at least the part information. Further, the meta-information D1 may include information on the degree of blurring of the low-resolution image. Blur strength information may be a classification such as “strong blur” or “weak blur”.

Further, in this embodiment, a pair of the original high-resolution image GR and the low-resolution image GL1 or a pair of the virtual high-resolution image GV and the low-resolution image GL1 input from the generation unit 42 are input to the identifying unit 44. However, it is not essential to input the low-resolution image GL1. In other words, the original high resolution image GR or the virtual high resolution image GV may be input to the identification unit 44 .

Even if the low-resolution image GL1 is not input to the identification unit 44, if the virtual high-resolution image GV is input to the identification unit 44, the slice information of the low-resolution image GL1 is used as the slice information of the meta information D1. do. This is to consider under what conditions the virtual high-resolution image GV input to the identification unit 44 was generated based on the low-resolution image GL1.

When the original high-resolution image GR is input to the identification unit 44, the slice information is given a random value for learning because the slice information has no meaning. As a result, the identifying unit 44 determines whether a high-resolution image generated from a low-resolution image with a slice interval of 4 mm and possibly a fake is true, Discrimination can be learned according to conditions such as "determine the authenticity of a high-resolution image that has the possibility of

<Second embodiment>
FIG. 15 is a functional block diagram showing learning processing of the medical image learning device 10 according to the second embodiment. The same reference numerals are given to the same parts as those in the functional block diagram shown in FIG. 6, and detailed description thereof will be omitted.

The medical image learning device 10 includes an interpolation section 34 in the input section 30 . The interpolation unit 34 generates a low-resolution image GL2 from the low-resolution image GL1 generated by the low-resolution conversion unit 32. The interpolation unit 34 performs spline interpolation processing on the low-resolution image GL1 in the Z direction. Further, the interpolation unit 34 performs Gaussian filter processing on the low-resolution image GL1 that has been subjected to the spline interpolation processing. An image generated by spline interpolation processing and Gaussian filtering processing is the low-resolution image GL2.

For example, the low-resolution image GL1 has a slice interval of 5 mm and a slice thickness of 1 mm, and the low-resolution image GL2 generated by the interpolation unit 34 has a slice interval of 1 mm and a slice thickness of 1 mm. The slice interval and slice thickness of the low-resolution image GL2 are the same as those of the original high-resolution image GR, but the low-resolution image GL2 is blurred in the Z direction compared to the original high-resolution image GR. Become.

The low-resolution image GL2 generated by the interpolating section 34 is input to the generating section 42 of the learning section 40 . That is, in this embodiment, the generator 42 generates the virtual high-resolution image GV from the low-resolution image GL2. Note that the slice interval and slice thickness of the virtual high-resolution image GV are the same as those of the low-resolution image GL2, but the virtual high-resolution image GV is sharper in the Z direction than the low-resolution image GL2. becomes an image.

Also, the low-resolution image GL2 generated by the interpolation unit 34 is input to the identification unit 44 in a pair with the high-resolution image. Using the low-resolution image GL2 and the meta information D1, the identification unit 44 identifies whether the input high-resolution image is the original high-resolution image GR or the virtual high-resolution image GV. Note that the low-resolution image GL2 does not have to be input to the identification unit 44 .

According to the present embodiment, the generating unit 42 can learn to generate a high-resolution image from the low-resolution image GL2 obtained by low-resolution conversion from the original high-resolution image GR and spline interpolation processing. Therefore, by combining learning for generating a high-resolution image from the low-resolution image GL1, it is possible to obtain a generative model adapted to various input low-resolution images.

<Third Embodiment>
FIG. 16 is a functional block diagram showing learning processing of the medical image learning device 10 according to the third embodiment. The same reference numerals are assigned to the same parts as those in the functional block diagram shown in FIG. 6, and detailed description thereof will be omitted. The medical image learning apparatus 10 according to the present embodiment generates and identifies high-frequency component images containing high-resolution information with relatively high resolution instead of high-resolution images.

The medical image learning device 10 includes a high-frequency component extraction section 36 in the input section 30 . The high-frequency component extraction unit 36 extracts high-frequency components from the original high-resolution image GR and generates an original high-frequency component image CR (an example of a high-frequency component image for learning). High-frequency components are extracted using a high-pass filter. Similar to the original high-resolution image GR, the original high-frequency component image CR has a slice interval of 1 mm and a slice thickness of 1 mm.

The original high frequency component image CR generated by the high frequency component extraction unit 36 is input to the identification unit 44 of the learning unit 40 .

Also, the generation unit 42 of the learning unit 40 generates a virtual high-frequency component image CV having the same resolution as the original high-frequency component image CR from the low-resolution image GL1 input from the low-resolution conversion unit 32 of the input unit 30 . Here, the generator 42 generates a virtual high-frequency component image CV with a slice interval of 1 mm and a slice thickness of 1 mm.

In the identification unit 44, a pair of the original high-frequency component image CR (an example of the second image for learning) and the low-resolution image GL1 input from the input unit 30, or from the generation unit 42 (an example of the discriminator input unit) A pair of an input virtual high-resolution image GV (an example of a second virtual image) and a low-resolution image GL1 is input.

The identification unit 44 identifies whether the input high-frequency component image is the original high-frequency component image CR or the virtual high-frequency component image CV using the low-resolution image GL1 and the meta information D1 input from the input unit 30. do.

According to this embodiment, the generator 42 can learn to generate a high-frequency component image from the low-resolution image GL1. The generating unit 42 can obtain a high-resolution image by adding the high-frequency component image generated by the generating unit 42 and the low-resolution image GL1 input to the generating unit 42 .

<Fourth Embodiment>
FIG. 17 is a functional block diagram showing learning processing of the medical image learning device 10 according to the fourth embodiment. 15 and 16 are denoted by the same reference numerals, and detailed description thereof will be omitted.

A low-resolution conversion unit 32 of the input unit 30 generates a low-resolution image GL1 from the original high-resolution image GR. The interpolation unit 34 generates a low resolution image GL2 from the low resolution image GL1.

The generation unit 42 of the learning unit 40 generates a virtual high-frequency component image CV having the same resolution as the original high-resolution image GR from the low-resolution image GL2 input from the interpolation unit 34 of the input unit 30 .

The medical image learning apparatus 10 includes an addition section 46 in the learning section 40 . The adder 46 adds the virtual high-frequency component image CV generated by the generator 42 and the low-resolution image GL2 generated by the interpolator 34 . Thereby, the adder 46 generates a virtual high-resolution image GV having the same resolution as the original high-resolution image GR.

The identifying unit 44 receives an original high-resolution image GR and low-resolution image GL2 pair input from the input unit 30 or a virtual high-resolution image GV and low-resolution image GL2 pair input from the adding unit 46 .

Using the low-resolution image GL1 and the meta information D1, the identification unit 44 identifies whether the input high-resolution image is the original high-resolution image GR or the virtual high-resolution image GV. The identification result of the identification unit 44 is input to the generation unit 42 . Thereby, the generation unit 42 and the identification unit 44 are learned.

According to this embodiment, the generator 42 can learn to generate a high-frequency component image from the low-resolution image GL1. Further, the identification unit 44 can learn to identify high-resolution images.

〔others〕
Although the medical image learning apparatus 10 is configured by one personal computer or workstation here, it may be configured by a plurality of personal computers or the like. For example, the input unit 30 and the learning unit 40 may be configured by different personal computers or the like. With this configuration, the generation of learning data and the learning of the generative model can be performed without being physically or temporally bound to each other.

Although the method of learning a super-resolution generative model for CT images has been described here, the method of learning a generative model according to this embodiment can be applied to three-dimensional tomographic images having part information. For example, MR images acquired by MRI (Magnetic Resonance Imaging) devices, PET images acquired by PET (Positron Emission Tomography) devices, OCT images acquired by OCT (Optical Coherence Tomography) devices, and three-dimensional ultrasound imaging devices. It may be a three-dimensional ultrasound image or the like acquired by.

In addition, the generative model learning method according to the present embodiment can be applied to a two-dimensional image having part information. For example, it may be an X-ray image. Moreover, it is not limited to medical images, and can be applied to normal camera images. When applied to a two-dimensional image, sharpness (reconstruction function) information may be used in place of the slice interval of the slice information. Pixel spacing information, that is, pixel size information may be used instead of the slice thickness of the slice information.

In the embodiments described so far, the hardware structure of the processing unit that executes various processes of the medical image learning apparatus 10 is various processors as shown below. Various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing such as FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc. Dedicated to execute specific processing It includes a dedicated electrical circuit, which is a processor with designed circuitry, and the like.

One processing unit may be composed of one of these various processors, or two or more processors of the same or different type (for example, a plurality of FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with a single processor, first, as represented by computers such as servers and clients, a single processor is configured with a combination of one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including a plurality of processing units with a single IC (Integrated Circuit) chip. be. In this way, various processing units are configured using one or more various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit combining circuit elements such as semiconductor elements.

The technical scope of the present invention is not limited to the scope described in the above embodiments. Configurations and the like in each embodiment can be appropriately combined between each embodiment without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10... Medical image learning apparatus 12... Communication part 14... Image database 16... Operation part 18... CPU
20 RAM
22 ROM
24 Display unit 30 Input unit 32 Low resolution conversion unit 34 Interpolation unit 36 High frequency component extraction unit 40 Learning unit 42 Generation unit 44 Identification unit 46 Addition unit CR Original high frequency component image CV Virtual high frequency Component image D1 Meta information G1, G2, G3, G4 CT images G11, G12, G13 Slice images G21, G22, G23 Slice images G31, G32, G33 Slice images G41, G42, G43 Slice image GL1, GL2: Low-resolution image GR: Original high-resolution image GV: Virtual high-resolution image S1 to S7 Each step of the medical image learning method

Claims (12)

A generative model learning method for estimating a second image containing high-resolution information with relatively high resolution from a first image containing low-resolution information with relatively low resolution, comprising:
to the generator of a generative adversarial network comprising: a generator that is a generative model that generates a virtual second image using a first image; and a discriminator that distinguishes between a training second image and the virtual second image: a generator input step of inputting only the first image;
a first discriminator input step of inputting the second learning image and part information of the second learning image to the discriminator;
a second discriminator input step of inputting the virtual second image and part information of the virtual second image into the discriminator;
A learning method with 2. The learning method according to claim 1, wherein the part information includes the head, chest, abdomen, and legs of the human body. The first image is a three-dimensional tomographic image,
3. The learning method according to claim 1, wherein said resolution is a resolution in a slice thickness direction. 4. The learning method according to claim 3, wherein said first discriminator input step and said second discriminator input step input slice information of said first image to said discriminator. 5. The learning method according to claim 4, wherein the slice information is a slice interval. 6. The learning method according to claim 4, wherein the slice information is slice thickness. 7. The learning method according to claim 1, wherein said first discriminator input step and said second discriminator input step input said first image into said discriminator. A first image generating step of generating the first image from the second learning image,
7. The learning method according to any one of claims 1 to 6, wherein the generator input step inputs only the generated first image to the generator. A program for causing a computer to execute the learning method according to any one of claims 1 to 8. A non-transitory computer-readable recording medium that causes a computer to execute the program according to claim 9 when instructions stored in the recording medium are read by the computer. A second image containing high-resolution information having a relatively high resolution from a first image containing low-resolution information having a relatively low resolution learned by the learning method according to any one of claims 1 to 8 A generative model for a computer to realize the function of estimating an image. A generative model learning device for estimating a second image containing high-resolution information having a relatively high resolution from a first image containing low-resolution information having a relatively low resolution,
a generative adversarial network comprising: a generator that is a generative model that generates a virtual second image using a first image; and a discriminator that distinguishes between a training second image and the virtual second image;
a generator input unit that inputs only the first image to the generator;
a first discriminator input unit for inputting the second learning image and part information of the second learning image to the discriminator;
a second discriminator input unit for inputting the second virtual image and part information of the second virtual image to the discriminator;
A learning device with

Images