JP6915685B2 - Learning device, super-resolution device, learning method, super-resolution method and program - Google Patents

Description

The present invention relates to a learning device, a super-resolution device, a learning method, a super-resolution method, a learning program, and a super-resolution program for performing a process of converting a low-resolution image into a high-resolution image.

In recent years, with the widespread use of surveillance cameras and Web cameras, the amount of image data and video data captured by these cameras has increased. Humans often recognize suspicious persons and suspicious objects by using these data. However, when the resolution of the captured data is low, there is a limit to human recognition of suspicious persons and suspicious objects.

For example, if an existing camera is inexpensive, it is assumed that the specifications of the camera are also low. While it is difficult to acquire high-resolution data with such an existing camera, there is also a demand for using this camera to recognize people and objects in the distance with high accuracy. Therefore, a super-resolution technique for generating high-resolution data from low-resolution data may be used.

Patent Document 1 describes an image processing apparatus that generates a restored image from a deteriorated image as a case-based learning type super-resolution method. The image processing apparatus described in Patent Document 1 deteriorates a high-resolution image which is a learning image to generate a deteriorated image, and creates a dictionary based on a pair of the high-resolution image and the corresponding deteriorated image. As a result, the optimum patch of the input image to be image-processed is searched for, and the synthesis processing is performed using the high-resolution learning image paired with the obtained patch.

In addition, SRCNN (Super-Resolution Convolutional Neural Network) and VDSR (Very Deep Super Resolution) are examples of deep learning-based learning-type super-resolution methods with improved accuracy compared to case-based learning-type super-resolution methods. Are known.

Patent Document 2 describes a detector that performs detection at a plurality of stages having different resolutions. The detector described in Patent Document 2 performs a plurality of steps of processing in which the attributes of the object included in the input data are the plurality of attribute values for each of the plurality of modality types, the resolutions of which are different from each other. Detect with.

International Publication No. 2014/087652 Japanese Unexamined Patent Publication No. 2012-022535

On the other hand, when the magnification for super-resolution from a low-resolution image is high, there is a problem that it is difficult to obtain a high-precision image by super-resolution. For example, the above-mentioned SRCNN and VDSR assume super-resolution at a low magnification. Therefore, when super-resolution is directly converted from a low-resolution image to a target resolution, the higher the magnification for super-resolution from the low-resolution image, the larger the number of parameters, and a highly accurate image can be obtained. Becomes difficult.

FIG. 15 is an explanatory diagram showing an example of super-resolution. As illustrated in FIG. 15, it is assumed that a 2 × 2 low resolution image L1 exists. Here, the squares on which the numbers are not described correspond to the parts to be super-resolutiond and complemented. For example, when the image L1 is super-resolutiond into a 4 × 4 image L2, it can be said that it is relatively easy to obtain a high-precision image. However, when the image L1 is super-resolutiond into an 8 × 8 image L3, the number of parameters is larger than when the image L2 is super-resolutiond, so that it is difficult to obtain a highly accurate image.

Further, both of the contents described in Patent Document 1 and Patent Document 2 select an image having an optimum resolution, and it is difficult to improve the accuracy of the image to be image-processed. Further, the image processing apparatus described in Patent Document 1 searches the created dictionary for the optimum restoration patch corresponding to the patch of the input image to be image processed, and generates a super-resolution image by compositing processing. do. However, the deteriorated image registered in the dictionary created here includes a deteriorated image in consideration of various situations. Therefore, when super-resolution with a high magnification as described above is performed, the number of parameters also increases, and it is difficult to obtain a highly accurate image.

Therefore, the present invention provides a learning device, a super-resolution device, a learning method, a super-resolution method, a learning program, and a super-resolution program capable of generating a high-precision high-resolution image from a low-resolution image. The purpose is to do.

The learning device according to the present invention is a combination of a low resolution unit that generates a low resolution image that is an image having a lower resolution than the learning image based on the learning image, and two adjacent resolutions. A learning unit that learns parameters for super-resolution from a first-resolution image to a second-resolution image using each of the first-resolution and second-resolution images, and stores parameters for each combination of learned resolutions. A storage unit and a super-resolution unit that generates an image having a higher resolution than the target image by using the parameters stored in the storage unit are provided, and the low-resolution unit is based on the learning image. A low-resolution image with a plurality of resolutions is generated stepwise, and the super-resolution unit generates a low-resolution image generated by the low-resolution unit or a low-resolution image generated by the super-resolution unit. The super-resolution is generated to generate a higher-resolution image, and the learning unit uses the image of the adjacent resolution among the learning image and the low-resolution image as the first-resolution image and the second-resolution image. It is characterized in that parameters are learned by using a low-resolution image generated by the super-resolution unit as a first-resolution image.

The super-resolution device according to the present invention is a super-resolution device that generates a high-resolution image that is an image having a higher resolution than the target image from the input target image, and is stepwise from the target image. A super-resolution unit that produces a high-resolution image, parameters that super-resolution from an intermediate resolution, which is the resolution between the resolution of the target image and the resolution of the high-resolution image, to the resolution of the high-resolution image, and , A storage unit that stores parameters that super-resolution from the resolution of the target image to an intermediate resolution, and the storage unit uses images of the first resolution and the second resolution, which are a combination of two adjacent resolutions. It is a parameter for super-resolution from the first-resolution image to the second-resolution image learned in the above process, and has a lower resolution than the training image generated by the super-resolution unit by super-resolution. A low-resolution image, which is an image, is used as the first-resolution image to store the learned parameters, and the super-resolution unit uses the above parameters to generate at least one intermediate-resolution image from the target image. , It is characterized in that a high resolution image is generated from the generated intermediate resolution image.

In the learning method according to the present invention, a low-resolution image, which is an image having a lower resolution than the learning image, is gradually generated for a plurality of resolutions based on the learning image, and a combination of two adjacent resolutions is used. Using each of the first and second resolution images, the parameters for super-resolution from the first resolution image to the second resolution image are learned, and the parameters are stored for each combination of the learned resolutions. An image generated using the parameters stored in the storage unit or a low-resolution image generated stepwise is super-resolved to generate a higher-resolution image during learning. Of the training image and the low resolution image, the image of the adjacent resolution is used as the first resolution image and the second resolution image, and the super-resolution low resolution image is used as the first resolution image. It is characterized by learning the parameters.

The super-resolution method according to the present invention is a super-resolution method for generating a high-resolution image, which is an image having a higher resolution than the target image, from the input target image, and is a super-resolution method stepwise from the target image. From the parameters that generate a high resolution image and super-resolution from the intermediate resolution, which is the resolution between the resolution of the target image and the resolution of the high resolution image, to the resolution of the high resolution image, and the resolution of the target image. Refer to the storage unit that stores the parameters to be super-resolutiond to the intermediate resolution, generate at least one intermediate resolution image from the target image using the parameters stored in the storage unit, and generate the generated intermediate resolution image. A high resolution image was generated from the image, and the parameters stored in the storage unit were learned using the images of the first resolution and the second resolution, which are a combination of two adjacent resolutions. A low-resolution image, which is a parameter for super-resolution from an image to a second-resolution image and has a lower resolution than the training image generated by the super-resolution, is used as the first-resolution image. It is characterized in that it is a parameter learned by using it.

The learning program according to the present invention is a low-resolution process that generates a low-resolution image, which is an image having a lower resolution than the learning image, on a computer based on the learning image, and is a combination of two adjacent resolutions. in a using each of the image of the first resolution and second resolution, the learning process for learning the parameters of the super-resolution of the first resolution of the image to a second image resolution, parameters for each combination of learned resolution The storage process for storing the image in the storage unit and the super-resolution processing for generating an image having a higher resolution than the target image using the parameters stored in the storage unit are executed, and the low-resolution processing is performed. Based on the training image, low-resolution images with multiple resolutions are generated step by step, and the low-resolution image generated by the low-resolution processing or the super-resolution processing is performed by the super-resolution processing. The low-resolution image generated in is super-resolution to generate a higher-resolution image, and in the training process, the image with the closest resolution among the training image and the low-resolution image is set to the first resolution. It is characterized in that it is used as an image and a second resolution image, and a low resolution image generated by the super-resolution processing is used as a first resolution image to train parameters.

The super-resolution program according to the present invention is a super-resolution program applied to a computer that generates a high-resolution image that is an image having a higher resolution than the target image from the input target image, and is a computer. Is subjected to a super-resolution process that gradually generates a high-resolution image from the target image, and the super-resolution process is an intermediate resolution between the resolution of the target image and the resolution of the high-resolution image. Refer to the storage unit that stores the parameters that super-resolution from the resolution to the resolution of the high-resolution image and the parameters that super-resolution from the resolution of the target image to the intermediate resolution, and refer to the storage unit for the parameters stored in the storage unit. It is used to generate at least one intermediate resolution image from the target image, generate a high resolution image from the generated intermediate resolution image , and the parameters stored in the storage unit are a combination of two adjacent resolutions. It is a parameter for super-resolution from the first-resolution image to the second-resolution image, which is learned using each of the first-resolution and second-resolution images, and is super-solved by the super-resolution processing. It is characterized in that it is a parameter learned by using a low resolution image, which is an image having a resolution lower than that of the learning image, generated by imaging as a first resolution image.

According to the present invention, a high-precision high-resolution image can be generated from a low-resolution image.

It is a block diagram which shows the structural example of the 1st Embodiment of the super-resolution system by this invention. It is explanatory drawing which shows the example of the learning part of 1st Embodiment. It is explanatory drawing which shows the structural example of the super-resolution part. It is explanatory drawing which shows the operation example which performs a learning process. It is a flowchart which shows the operation example which performs super-resolution processing. It is a flowchart which shows the apparatus which performs the general high-resolution processing. It is explanatory drawing which shows the operation example which performs super-resolution. It is explanatory drawing which shows the example of the learning part of the 2nd Embodiment. It is explanatory drawing which shows the example of the learning part of 3rd Embodiment. It is explanatory drawing which shows the example of the learning part of 4th Embodiment. It is a block diagram which shows the structural example of the 5th Embodiment of the super-resolution system by this invention. It is explanatory drawing which shows the example of the learning part of 6th Embodiment. It is a block diagram which shows the outline of the learning apparatus by this invention. It is a block diagram which shows the outline of the super-resolution apparatus by this invention. It is explanatory drawing which shows the example of super-resolution.

Embodiment 1.
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of a first embodiment of the super-resolution system according to the present invention. The super-resolution system 100 of the present embodiment includes a learning image input unit 10, a learning unit 20, a parameter holding unit 30, a super-resolution target image input unit 40, and a super-resolution unit 50. It includes an image output unit 60. The unidirectional arrow shown in FIG. 1 simply indicates the direction of information flow, and does not exclude bidirectionality.

The learning image input unit 10 inputs an image used for learning by the learning unit 20 described later. The learning image input unit 10 may input an image received via the communication network into the learning unit 20, or read an image from a storage unit (not shown) in which the learning image is stored and read the image. May be input to the learning unit 20.

The learning unit 20 learns the parameters to be super-resolution, and stores the learned parameters in the parameter holding unit 30, which will be described later. The parameter holding unit 30 may be included in the learning unit 20.

FIG. 2 is an explanatory diagram showing an example of the learning unit 20 of the first embodiment. The learning unit 20 of the present embodiment includes the parameter holding unit 30. Further, as in FIG. 1, the unidirectional arrow shown in FIG. 2 simply indicates the direction of information flow, and does not exclude bidirectionality. The same applies to the following drawings. The learning unit 20 includes a plurality of low-resolution learning means 21a, 21b, 21c, a plurality of super-resolution learning means 22a, 22b, 22c, and a parameter holding unit 30. Further, the parameter holding unit 30 has a plurality of learning parameter holding units 30a, 30b, 30c.

In the following description, when the processing common to the low resolution means 21a, 21b, 21c is described, each low resolution means 21a, 21b, 21c is simply referred to as a low resolution means. Further, when the processing common to the super-resolution learning means 22a, 22b, 22c is explained, each super-resolution learning means 22a, 22b, 22c is simply referred to as a super-resolution learning means.

Further, in the present embodiment, the case where the low resolution means is realized by a plurality of low resolution means 21a, 21b, 21c is illustrated, but each of the low resolution means 21a, 21b, 21c performs. The processing may be performed collectively by one low resolution means. Similarly, in the present embodiment, the case where the super-resolution learning means is realized by a plurality of super-resolution learning means 22a, 22b, 22c is illustrated, but the super-resolution learning means 22a, 22b, One super-resolution learning means may collectively perform the processing performed by 22c.

Further, in the present embodiment, the case where the parameter holding unit 30 is realized by a plurality of learning parameter holding units 30a, 30b, 30c is illustrated, but one storage means is the learning parameter holding units 30a, 30b, 30c. The information to be memorized may be memorized respectively. The parameter holding unit 30 is realized by, for example, a magnetic disk device.

The low-resolution means generates a low-resolution image, which is an image having a resolution lower than that of the learning image, based on the learning image input by the learning image input unit 10. In the following description, a learning image that is a source for generating a low-resolution image may be referred to as a high-resolution image.

The low-resolution means gradually generates low-resolution images having a plurality of resolutions based on the learning image. Specifically, in the present embodiment, the low-resolution means 21a, 21b, and 21c generate low-resolution images having different resolutions based on the learning images, respectively. In this embodiment, it is assumed that the magnification of the low-resolution image to be generated is predetermined.

Here, since the low-resolution image is generated stepwise, the resolution of the low-resolution image generated by the low-resolution means 21b is lower than that of the low-resolution image generated by the low-resolution means 21a. It is assumed that the resolution of the low-resolution image generated by the low-resolution means 21c is lower than that of the low-resolution image generated by the low-resolution means 21b. In the following description, the magnification at which the low resolution means 21a lowers the image is described as (1 / N ₁ ), and the magnification at which the low resolution means 21b lowers the image is (1 / N). ₂ ), and the magnification at which the low-resolution means 21c reduces the resolution of the image is described _{as (1 / N 3).} Further, in the following description, the resolution may be represented by the above magnification. That is, it means that the resolution of the image at the magnification (1 / N _{3) is the lowest.}

The method by which the low resolution means produces a low resolution image is arbitrary. As the low-resolution means, for example, a low-resolution image may be generated by using a Bicubic method, a Nearest neighbor method, a Bilinear method, or the like. However, the method by which the low-resolution means produces a low-resolution image is not limited to these methods.

The super-resolution learning means learns the parameters to be super-resolution using two images having different resolutions. Specifically, the super-resolution learning means uses an image of a combination of two adjacent resolutions (hereinafter referred to as a first resolution and a second resolution), and uses an image of the first resolution to an image of the second resolution. Learn the parameters to be super-resolution. It is assumed that the first resolution has a lower resolution than the second resolution. The super-resolution learning means uses images having adjacent resolutions as the first-resolution image and the second-resolution image among the learning image and the low-resolution image.

In the present embodiment, the super-resolution learning means 22c has an image having a resolution (magnification: 1 / N ₃ ) generated by the low-resolution means 21c and a resolution (magnification: 1) generated by the low-resolution means 21b. / _{N 2)} by using the image of the resolution (magnification: 1 / _{N 3} image from the resolution (magnification): 1 / _{N 2)} images to learn the parameters of the super-resolution of. Then, the super-resolution learning means 22c stores the learned parameters in the learning parameter holding unit 30c.

Similarly, the super-resolution learning means 22b has an image having a resolution (magnification: 1 / N ₂ ) generated by the low-resolution means 21b and a resolution (magnification: 1 / N) generated by the low-resolution means 21a. by using the image of _one), the resolution (magnification: 1 / N 2 images from the resolution _{(magnification):} 1 / N ₁₎ the image to learn the parameters of the super-resolution of. Then, the super-resolution learning means 22b stores the learned parameters in the learning parameter holding unit 30b.

Then, the super-resolution learning means 22a uses the image of the resolution (magnification: 1 / N ₁ ) generated by the low-resolution means 21a and the image for learning, and has a resolution (magnification: 1 / N ₁ ). Learn the parameters for super-resolution from the image of the above to the image of the original resolution. Then, the super-resolution learning means 22a stores the learned parameters in the learning parameter holding unit 30a.

In this way, the parameter holding unit 30 stores the parameters for each combination of learned resolutions.

The super-resolution target image input unit 40 inputs an image used for super-resolution by the super-resolution unit 50, which will be described later. The super-resolution target image input unit 40 may input an image received via a communication network into the super-resolution unit 50, or an image is input from a storage unit (not shown) in which the target image is stored. The read image may be input to the super-resolution unit 50.

FIG. 3 is an explanatory diagram showing a configuration example of the super-resolution unit 50. The super-resolution unit 50 includes super-resolution means 50a, 50b, 50c. In the following description, when the processing common to the super-resolution means 50a, 50b, 50c is described, each super-resolution means 50a, 50b, 50c is simply referred to as a super-resolution unit 50.

Further, in the present embodiment, the case where the super-resolution unit 50 is realized by a plurality of super-resolution means 50a, 50b, 50c is illustrated, but each super-resolution means 50a, 50b, 50c The processing to be performed may be collectively performed by one super-resolution unit 50.

The super-resolution unit 50 uses the parameters stored in the parameter holding unit 30 to gradually generate a high-resolution image from the target image input by the super-resolution target image input unit 40. The super-resolution unit 50 of the present embodiment does not directly generate the final high-resolution image from the input target image, but has a resolution between the target image and the high-resolution image (hereinafter referred to as an intermediate resolution). After the image of (described) is generated, the final high-resolution image is generated. That is, the super-resolution unit 50 generates at least one intermediate-resolution image from the target image, and generates a high-resolution image from the generated intermediate-resolution image.

In the example shown in FIG. 3, two types of intermediate resolution images are generated. However, the types of intermediate resolution images generated are not limited to two types. The generated intermediate resolution is determined based on the parameters held by the parameter holding unit 30.

That is, the parameter holding unit 30 super-resolutions the target image from the resolution ((1 / N ₃ ) illustrated in FIG. 2) to the intermediate resolution ((1 / N ₂ ) illustrated in FIG. 2), and , The parameter for super-resolution from the intermediate resolution (exemplified in FIG. 2 (1 / N ₁ )) to the resolution of the high-resolution image is stored. In addition, the parameter holding unit 30 also includes parameters for the first intermediate resolution (exemplified in FIG. 2 (1 / N ₂ )) to the second intermediate resolution (illustrated in FIG. 2 (1 / N ₁ )). Hold.

In this way, the parameter holding unit 30 stores the parameters for super-resolution to a plurality of intermediate resolutions having different resolutions, so that the super-resolution unit 50 uses these parameters to increase the resolution. Generate intermediate resolution images in stages.

The method in which the super-resolution unit 50 performs super-resolution is arbitrary. The super-resolution unit 50 may perform super-resolution using, for example, SRCNN or VSDR. However, the method in which the super-resolution unit 50 performs super-resolution is not limited to these methods.

The image output unit 60 outputs the generated high-resolution image.

As described above, learning of the parameters used when performing super-resolution is realized by the learning image input unit 10, the learning unit 20, and the parameter holding unit 30. Therefore, a device including a learning image input unit 10, a learning unit 20, and a parameter holding unit 30 can be called a learning device.

The learning image input unit 10 and the learning unit 20 are realized by a CPU of a computer that operates according to a program (learning program). For example, the program may be stored in a storage unit (not shown) included in the learning device, and the CPU may read the program and operate as the learning image input unit 10 and the learning unit 20 according to the program.

Further, the process of generating a high-resolution image from the target image (that is, super-resolution) is performed by the parameter holding unit 30, the super-resolution target image input unit 40, the super-resolution unit 50, and the image output unit 60. It will be realized. Therefore, a device including a parameter holding unit 30, a super-resolution target image input unit 40, a super-resolution unit 50, and an image output unit 60 can be called a super-resolution device.

The super-resolution target image input unit 40, the super-resolution unit 50, and the image output unit 60 are realized by a computer CPU that operates according to a program (super-resolution program). For example, the program is stored in a storage unit (not shown) included in the super-resolution device, the CPU reads the program, and the super-resolution target image input unit 40 and the super-resolution unit 50 follow the program. And may operate as the image output unit 60.

Next, it is explanatory drawing which shows the operation example of the super-resolution system of this embodiment. FIG. 4 is a flowchart showing an operation example in which the learning process is performed using the super-resolution system of the present embodiment. The learning image input unit 10 inputs the learning image to the learning unit 20 (step S11). The learning unit 20 gradually generates low-resolution images having a plurality of resolutions based on the learning image (step S12). The learning unit 20 learns the parameters used for super-resolution by using the images having adjacent resolutions as the first-resolution image and the second-resolution image among the learning image and the low-resolution image (step S13). ). Then, the learning unit 20 stores the parameters in the parameter holding unit 30 for each combination of the learned resolutions (step S14).

FIG. 5 is a flowchart showing an operation example of performing super-resolution processing using the super-resolution system of the present embodiment. The super-resolution target image input unit 40 inputs an image to be super-resolution into the super-resolution unit 50 (step S21). The super-resolution unit 50 gradually generates a high-resolution image from the input target image (step S22). Specifically, the super-resolution unit 50 generates at least one intermediate-resolution image from the target image using the parameters stored in the parameter holding unit 30, and high-resolution from the generated intermediate-resolution image. Generate an image image. Then, the image output unit 60 outputs the generated high-resolution image (step S23).

As described above, in the present embodiment, the low-resolution learning means gradually generates low-resolution images having a plurality of resolutions based on the learning image, and the super-resolution learning means has the first resolution. And, using each of the images of the second resolution, the parameters for super-resolution from the image of the first resolution to the image of the second resolution are learned, and each combination of the learned resolutions is stored in the parameter holding unit 30. .. Further, the super-resolution learning means uses images having adjacent resolutions as the first-resolution image and the second-resolution image among the learning image and the low-resolution image. Therefore, a high-precision high-resolution image can be generated from the low-resolution image.

FIG. 6 is an explanatory diagram showing an apparatus that performs general high-resolution processing. As in the learning unit 120 illustrated in FIG. 6, a parameter for super-resolution to a high-resolution image is learned using a low-resolution image whose resolution has been reduced to the target resolution for super-resolution. It is also possible. However, as described above, the higher the super-resolution magnification, the larger the number of parameters, and it is difficult to generate a high-precision, high-resolution image.

Specifically, the low-resolution means 121 lowers the resolution of the learning image input from the learning image input unit 110, and the super-resolution learning means 122 is obtained from the low-resolution image and the learning image. Learns the parameters for super-resolution and stores them in the learning parameter holding unit 130. As a result of increasing this parameter, the super-resolution means 150a of the super-resolution unit 150 super-resolutions the image input from the super-resolution target image input unit 140, and the image output unit 160 outputs the image. However, it is difficult to increase the resolution of the image with high accuracy.

However, in the present embodiment, parameters for stepwise super-resolution are learned, and super-resolution is performed using the parameters. FIG. 7 is an explanatory diagram showing an operation example in which the super-resolution device of the present embodiment performs super-resolution. In the example shown in FIG. 7, the 2 × 2 low-resolution image L1 is super-resolutiond into a 4 × 4 image L2, and the 4 × 4 image L2 is super-resolutiond into an 8 × 8 image L3. .. Therefore, the number of parameters used in each step illustrated in FIG. 7 can be reduced as compared with the method illustrated in FIG. Therefore, it is possible to generate a high-precision and high-resolution image.

Embodiment 2.
Next, a second embodiment of the super-resolution system of the present invention will be described. In the first embodiment, a method in which the super-resolution learning means of the learning unit 20 learns parameters using the low-resolution image generated by the low-resolution means has been described. In the present embodiment, a method of super-resolution a low-resolution image using the learned parameters and learning the parameters using the super-resolution low-resolution image will be described.

The configuration of the super-resolution system of the present embodiment is the same as that of the first embodiment. However, the content of the learning unit 20 is different from that of the first embodiment.

FIG. 8 is an explanatory diagram showing an example of the learning unit 20 of the second embodiment. The learning unit 20 includes a plurality of low-resolution means 21a, 21b, 21c, a plurality of super-resolution learning means 23a, 23b, 23c, a parameter holding unit 30, and super-resolution means 71a, 71b, 71c. And include. The contents of the low resolution means 21a, 21b, 21c are the same as those in the first embodiment. Further, the parameter holding unit 30 stores the learned parameters as in the first embodiment. However, the stored content is different from that of the first embodiment.

In the following description, when the processing common to the super-resolution means 71a, 71b, 71c is described, each super-resolution means 71a, 71b, 71c is simply referred to as a super-resolution means. Further, when the processing common to the super-resolution learning means 23a, 23b, 23c is explained, each super-resolution learning means 23a, 23b, 23c is simply referred to as a super-resolution learning means.

The super-resolution means uses the parameters stored in the parameter holding unit 30 to generate an image having a higher resolution than the target image. Specifically, the super-resolution means super-resolutions a low-resolution image generated by the low-resolution means or a low-resolution image generated by the super-resolution means to have a higher resolution. Generate an image.

In the case of the example shown in FIG. 8, the super-resolution means 71c has a higher resolution than the low-resolution image generated by the low-resolution means 21c by using the parameters stored in the learning parameter holding unit 30c. Generate an image. Further, the super-resolution means 71b uses the parameters stored in the learning parameter holding unit 30b to generate a higher-resolution image from the low-resolution image generated by the super-resolution means 71c. Even if the super-resolution means 71a uses the parameters stored in the learning parameter holding unit 30a to generate a higher-resolution image from the low-resolution image generated by the super-resolution means 71b. good.

When a low-resolution image generated by the super-resolution means exists, the super-resolution learning means uses the low-resolution image as a first-resolution image and sets parameters used for super-resolution. learn.

In the case of the example shown in FIG. 8, the lowest resolution low-resolution image used by the super-resolution learning means 23c for learning is not the low-resolution image generated by the super-resolution learning means. Therefore, in the super-resolution learning means 23c, as in the first embodiment, the image of the resolution (magnification: 1 / N ₃ ) generated by the low-resolution means 21c and the low-resolution means 21b are generated. Learn the parameters for super-resolution from an image with a resolution (magnification: 1 / N ₃ ) to an image with a resolution (magnification: 1 / N ₂ ) using the image with the same resolution (magnification: 1 / N _2). .. Then, the super-resolution learning means 23c stores the learned parameters in the learning parameter holding unit 30c.

The super-resolution means 71c uses the parameters stored in the learning parameter holding unit 30c to super-resolution an image having _{a resolution (magnification: 1 / N 3) generated by the low-resolution means 21c to achieve a resolution.} Generates an image with a resolution higher than (magnification: 1 / N ₃ ) (magnification: 1 / N _2').

Next, the super-resolution learning means 23b includes an image having a resolution (magnification: 1 / N ₂ ') generated by the super-resolution means 71c and a resolution (magnification: 1 /) generated by the low-resolution means 21a. by using the image of N _1), a resolution (1 / _{N 2} ') image from the resolution (magnification: ₁ / _N 1) the image to learn the parameters of the super-resolution of. Then, the super-resolution learning means 23b stores the learned parameters in the learning parameter holding unit 30b.

The super-resolution means 71b uses the parameters stored in the learning parameter holding unit 30b to super-resolution an image having a resolution (magnification: 1 / N ₂ ') generated by the super-resolution means 71b. Generates an image with a resolution higher than the resolution (1 / N ₂ ') (magnification: 1 / N _1').

Next, the super-resolution learning means 23a uses an image having a resolution (magnification: 1 / N ₁ ') generated by the super-resolution means 71b and a high-resolution image for learning, and has a resolution (magnification). : Learn the parameters for super-resolution from 1 / N ₁ ') images to high-resolution images. Then, the super-resolution learning means 23a stores the learned parameters in the learning parameter holding unit 30a.

As described above, in the present embodiment, the learning unit 20 (more specifically, the super-resolution learning means) uses the low-resolution image generated by the super-resolution means as the first-resolution image. , Learn the parameters. That is, in the present embodiment, learning is performed step by step, and the super-resolution low-resolution image at each step is used as learning data in the next step. Therefore, in addition to the effect of the first embodiment, a higher-definition high-resolution image can be generated. This is because in learning-type super-resolution, learning is performed using a pair of a low-resolution image and a high-resolution image, and when a low-resolution image to be super-resolution is given, the resolution of the image is obtained. This is because the closer the resolution is to the low-resolution image used during learning, the higher the definition of the finally generated high-resolution image.

Embodiment 3.
Next, a third embodiment of the super-resolution system of the present invention will be described. In the first embodiment, the case where the low-resolution means of the learning unit 20 generates a low-resolution image based on the high-resolution image for learning has been described. In this embodiment, a method of generating a low-resolution image for learning from an already generated low-resolution image will be described.

The configuration of the super-resolution system of the present embodiment is the same as that of the first embodiment. However, the content of the learning unit 20 is different from that of the first embodiment.

FIG. 9 is an explanatory diagram showing an example of the learning unit 20 of the third embodiment. The learning unit 20 includes a plurality of low-resolution learning means 24a, 24b, 24c, a parameter holding unit 30, and super-resolution learning means 22a, 22b, 22c. The contents of the super-resolution learning means 22a, 22b, 22c are the same as those in the first embodiment. Further, the parameter holding unit 30 stores the learned parameters as in the first embodiment.

In the following description, when the processing common to the low resolution means 24a, 24b, 24c is described, each low resolution means 24a, 24b, 24c is simply referred to as a low resolution means. The low-resolution means generates a low-resolution image, which is an image having a resolution lower than that of the learning image, based on the learning image input by the learning image input unit 10. However, the low-resolution means of the present embodiment generates a low-resolution image from the learning image, and sequentially generates a low-resolution image having a lower resolution from the generated low-resolution image. In the present embodiment, it is assumed that the magnification of the low-resolution image to be generated is predetermined.

In the case of the example shown in FIG. 9, the low-resolution means 24a generates a low-resolution image having a _{resolution (magnification: 1 / N 1) from the learning image.} Next, the low-resolution means 24b generates a low-resolution image with a resolution (magnification: 1 / N ₂ ) from the generated low-resolution image with _{a resolution (magnification: 1 / N 1).} Further, the low resolution means 24c generates a low resolution image with a resolution (magnification: 1 / N ₃ ) from the generated low resolution image with _{a resolution (magnification: 1 / N 2).}

As described above, in the present embodiment, the learning unit 20 (more specifically, the low-resolution means) generates a low-resolution image from the learning image, and further from the generated low-resolution image. Generates low resolution images with low resolution. Therefore, in addition to the effect of the first embodiment, a higher-definition high-resolution image can be generated.

That is, in the first embodiment, the low resolution means directly generates all the low resolution images from the learning image. On the other hand, in the present embodiment, the low-resolution means generates a low-resolution image from the learning image, and sequentially generates a low-resolution image using the low-resolution image. Therefore, it is possible to further suppress the loss of information when the image is reduced.

Embodiment 4.
Next, a fourth embodiment of the super-resolution system of the present invention will be described. In the fourth embodiment, the learning unit includes the low-resolution means of the third embodiment, and the super-resolution means and the super-resolution learning means of the second embodiment.

FIG. 10 is an explanatory diagram showing an example of the learning unit 20 of the fourth embodiment. As described above, the learning unit 20 includes a plurality of low-resolution means 24a, 24b, 24c, a plurality of super-resolution learning means 23a, 23b, 23c, a parameter holding unit 30, and a super-resolution means. Includes 71a, 71b, 71c. The contents of the low resolution means 24a, 24b, 24c are the same as those in the third embodiment. Further, the contents of the super-resolution learning means 23a, 23b, 23c and the super-resolution means 71a, 71b, 71c are the same as those in the second embodiment.

Even with such a configuration, as shown in the effects of the first embodiment, the second embodiment, and the third embodiment, a highly accurate high-resolution image can be generated from the low-resolution image.

Embodiment 5.
Next, a fifth embodiment of the super-resolution system of the present invention will be described. In the first to fourth embodiments, the case where the magnification for lowering the resolution is predetermined has been described. In this embodiment, a method of dynamically determining the magnification for lowering the resolution will be described.

FIG. 11 is a block diagram showing a configuration example of a fifth embodiment of the super-resolution system according to the present invention. The super-resolution system 500 of the embodiment includes a learning image input unit 10, a learning unit 20, a parameter holding unit 30, a super-resolution target image input unit 40, a super-resolution unit 50, and an image. It includes an output unit 60 and a magnification determination unit 70.

The contents of the learning image input unit 10, the parameter holding unit 30, the super-resolution target image input unit 40, the super-resolution unit 50, and the image output unit 60 are the same as those in the first to fourth embodiments. Is. Further, the configuration of the learning unit 20 is the same as the configuration of any one of the first to fourth embodiments. However, in the present embodiment, the magnification at which each low-resolution means in the learning unit 20 produces a low-resolution image is determined by the magnification determination unit 70.

The magnification determination unit 70 determines the magnification at which the low-resolution means generates a low-resolution image for each low-resolution means. Specifically, the magnification determination unit 70 specifies a magnification at which the image deterioration of images having close resolutions is lower than a predetermined threshold value. The closer the resolution between the images used for learning is, the better the accuracy of the image generated by super-resolution (that is, the image deterioration is low), but the more parameters to be learned. Therefore, the magnification determination unit 70 may specify the magnification at which the image deterioration due to the low resolution processing performed by the low resolution means is lower than the predetermined threshold value Th, and the lowest resolution can be achieved.

PSNR (Peak Signal-to-Noise Ratio) can be mentioned as an example of an index for evaluating image deterioration. It can be said that the higher the PSNR, the lower the image deterioration.

For example, it is assumed that the magnification determination unit 70 determines the magnification at which the low-resolution means of the learning unit 20 illustrated in FIG. 2 produces a low-resolution image. First, the magnification determination unit 70 generates a low-resolution image with a magnification determined in the initial state from the learning image. The magnification determined in the initial state is arbitrary. When the image deterioration of the low-resolution image generated from the learning image is larger than the predetermined threshold value Th, the magnification determination unit 70 changes the magnification so that the image deterioration does not exceed the threshold value Th, and re-creates the low-resolution image. Generate. The magnification determination unit 70 repeats this process until the image deterioration becomes lower than the predetermined threshold value Th. When the image deterioration of the low-resolution image generated from the learning image is smaller than the predetermined threshold value Th, the magnification determination unit 70 changes the magnification so that the image deterioration approaches the threshold value Th, and the low-resolution image again. May be generated. In this way, the magnification determination unit 70 determines the magnification (1 / N ₁ ) when the low resolution means 21a generates a low resolution image.

Next, the magnification determination unit 70 generates a low-resolution image having a predetermined magnification from the low-resolution image generated by the determined magnification, and similarly, the image deterioration becomes lower than the predetermined threshold value Th. Identify the magnification. In this way, the magnification determining unit 70 determines the magnification (1 / N ₂ ) when the low resolution means 21b generates a low resolution image and when the low resolution means 21c generates a low resolution image. Magnification (1 / N ₃ ) is determined in sequence.

When the above-mentioned psnr is used as an index of image deterioration, the magnification determining unit 70 sets the magnifications 1 / N ₁ , 1 / N ₂ , 1 / N ₃ satisfying the relationship shown in the following equation 1 to the magnification of low resolution. May be specified.

psnr (1,1 / N ₁ )> Th
psnr (1 / N ₁ , 1 / N ₂ )> Th
psnr (1 / N ₂ , 1 / N ₃ )> Th (Equation 1)

As described above, in the present embodiment, the magnification determination unit 70 sets the magnification when the low resolution unit generates a low resolution image lower than the threshold value for image deterioration of images having close resolutions. Decide to be. Therefore, in addition to the effects of the above-described embodiment, it is possible to learn so as to suppress image deterioration.

Embodiment 6.
Next, a sixth embodiment of the super-resolution system of the present invention will be described. In the fifth embodiment, a method of determining the magnification of the low resolution image from the beginning based on the value of psnr has been described. In the present embodiment, a method of determining the magnification will be described using the low-resolution image generated by the super-resolution means in the second embodiment or the fourth embodiment.

The configuration of the super-resolution system of the present embodiment is the same as that of the second embodiment or the fourth embodiment. However, the content of the learning unit 20 is different from that of the second embodiment or the fourth embodiment.

FIG. 12 is an explanatory diagram showing an example of the learning unit 20 of the sixth embodiment. The learning unit 20 includes a plurality of low-resolution means 21a, 21b, 21c, a plurality of super-resolution learning means 23a, 23b, 23c, a parameter holding unit 30, and super-resolution means 71a, 71b, 71c. And the image deterioration determining means 72a, 72b, 72c. The contents of the low resolution means 21a, 21b, 21c are the same as those in the second embodiment. The low resolution means may be realized by the low resolution means 24a, 24b, 24c in the fourth embodiment.

In the following description, when the processing common to the image deterioration determining means 72a, 72b, 72c is described, each image deterioration determining means 72a, 72b, 72c is simply referred to as an image deterioration determining means. The contents of the low-resolution means, the super-resolution means, the parameter holding unit 30, and the super-resolution means in the present embodiment are the same as those in the second embodiment or the fourth embodiment. However, it differs from the second embodiment or the fourth embodiment in that the types of resolutions to be processed increase depending on the determination result of the image deterioration determining means.

The image deterioration determining means determines the degree of image deterioration by comparing the low-resolution image generated by the low-resolution means with the low-resolution image generated by the super-resolution means. Specifically, the image deterioration determining means super-resolutions the second-resolution image generated by the low-resolution means and the parameters stored in the parameter holding unit 30 from the first-resolution image. The image deterioration is judged by comparing with the image. If the image degradation is higher than a predetermined threshold Th, the image degradation determination means specifies a magnification that produces a low resolution image between the first and second resolutions.

Once the magnification is specified, the low resolution means produces a low resolution image at the specified magnification. Then, the super-resolution learning means learns the parameters for super-resolution from the first-resolution image into the generated low-resolution image. Further, the super-resolution learning means learns parameters for super-resolution from the generated low-resolution image to a second-resolution image.

Hereinafter, the operation of the learning unit 20 of the present embodiment will be specifically described with reference to FIG. First, as in the first embodiment, the low-resolution means generates a plurality of low-resolution images (magnification: 1 / N ₁ , 1 / N ₂ , 1 / N ₃ ), and super-resolution learning. The means learns the parameters at the time of super-resolution and stores them in the parameter holding unit 30.

Next, the super-resolution means 71c, using the stored in the learning parameter holding unit 30c parameters, low resolution image generated by the low-resolution Zoka means 21c (i.e., magnification: 1 / N ₃ of the image from), to produce a higher resolution image (hereinafter, referred to as 1 / _{N 2} 'images.). Image deterioration determining unit 72c is low resolution image low resolution Zoka means 21b is generated (hereinafter, 1 / N referred to as _second image.) And 1 / N 2 _'is compared with the image, determining an image degradation .. The image deterioration determining means 72c may determine the image deterioration by the value of psnr exemplified in the fifth embodiment, for example. In this case, when the psnr (1 / N ₂ , 1 / N ₂ ') is smaller than a certain threshold Th, the image deterioration determining means 72c produces a low-resolution image (hereinafter referred to as 1 / a). To identify.

The low-resolution means generates a low-resolution image (hereinafter referred to as 1 / a image) obtained by reducing the resolution _{of a 1 / N 2 image at a specified magnification.} The super-resolution learning means uses an image generated by the low-resolution means 21c (that is, an image having a magnification of 1 / N ₃ ) and a 1 / a image as training data, and has a magnification of 1 / N ₃ . Learn the parameters for super-resolution from an image to a 1 / a image. Then, the super-resolution learning means stores the learned parameters in the parameter holding unit 30.

Next, super-resolution means, magnification: 1 / a N ₃ of the image in 1 / a image using the parameter of super-resolution, low-resolution image (hereinafter a super-resolution target image, 1 / a'image) is generated. The image deterioration determining means compares the 1 / a image with the 1 / a'image to determine the image deterioration. The image deterioration determining means further specifies the magnification for producing a low resolution image, for example, when psnr (1 / a, 1 / a') is smaller than a certain threshold Th. When the magnification is further specified, the above-mentioned process is repeated.

As a result, the learning unit 20 newly generates and learns a low-resolution image having a magnification of 1 / a between _{the image having a magnification of 1 / N 3 and} the image having a magnification of 1 / N _2. That is, the finally generated low-resolution images for learning are images having magnifications of 1 / N ₁ , 1 / N ₂ , 1 / a, and 1 / N ₃ , respectively. _{Finally, the magnifications 1 / N 1} , 1 / N ₂ , 1 / a, 1 / N ₃ that satisfy the relationship shown in the following equation 2 are specified as the low resolution magnifications. In Equation 2, 1'is the magnification of the image finally super-resolution as a high-resolution image. The operation of the image deterioration determination units 72b and 72a is the same as the operation of the image deterioration determination unit 72c.

psnr (1 / a', 1 / a)> Th
psnr (1 / N ₂ ', 1 / N ₂ )> Th
psnr (1 / N ₁ ', 1 / N ₁ )> Th
psnr (1', 1)> Th (Equation 2)

As described above, in the present embodiment, the image deterioration determining means compares the second resolution image generated by the low resolution unit with the super-resolution image from the first resolution image. The image degradation is determined, and if the image degradation is higher than a predetermined threshold, the magnification for producing a low resolution image between the first resolution and the second resolution is specified. Therefore, as with the effect of the fifth embodiment, it is possible to learn so as to suppress image deterioration.

Next, the outline of the present invention will be described. FIG. 13 is a block diagram showing an outline of the learning device according to the present invention. The learning device 80 according to the present invention includes a low-resolution unit 81 (for example, a low-resolution means) that generates a low-resolution image that is an image having a resolution lower than that of the learning image based on the learning image. Learning unit 82 (learning parameters for super-resolution from a first-resolution image to a second-resolution image using images of the first resolution and the second resolution, which are a combination of two adjacent resolutions. For example, it includes a super-resolution learning means) and a storage unit 83 (for example, a parameter holding unit 30) that stores parameters for each combination of learned resolutions.

The low-resolution unit 81 gradually generates low-resolution images having a plurality of resolutions based on the learning image, and the learning unit 82 has the adjacent resolutions of the learning image and the low-resolution image. Images are used as first resolution images and second resolution images.

With such a configuration, a high-precision high-resolution image can be generated from a low-resolution image. That is, by using the parameters learned by the learning device 80 of the present invention, a high-precision high-resolution image can be generated from the low-resolution image.

Further, the learning device 80 may include a super-resolution unit (for example, a super-resolution means) that generates an image having a resolution higher than that of the target image by using the parameters stored in the storage unit 83. .. Then, the super-resolution unit super-resolutions the low-resolution image generated by the low-resolution unit 81 or the low-resolution image generated by the super-resolution unit to produce an image having a higher resolution. The generation unit 82 may use the low-resolution image generated by the super-resolution unit as the first-resolution image to learn the parameters.

Further, the learning device 80 super-resolutions the second-resolution image generated by the low-resolution unit 81 and the super-resolution image from the first-resolution image using the parameters stored in the storage unit 83. An image deterioration determination unit (for example, an image deterioration determination means) for determining image deterioration may be provided for comparison. Then, when the image deterioration is higher than a predetermined threshold value, the image deterioration determination unit may specify a magnification for generating a low resolution image between the first resolution and the second resolution. At this time, the low-resolution unit 81 generates a low-resolution image at a specified magnification, and the learning unit 82 creates a first-resolution image with parameters for super-resolution to the generated low-resolution image. It may be learned by using a low-resolution image in which parameters for super-resolution to a second-resolution image are generated.

Further, the low-resolution unit 81 may generate a low-resolution image from the learning image, and may generate a low-resolution image having a lower resolution from the generated low-resolution image.

Further, the learning device 80 may include a magnification determination unit (for example, a magnification determination unit 70) for determining the magnification when the low resolution unit 81 generates a low resolution image. Then, the magnification determination unit specifies a magnification at which the image deterioration of images having close resolutions becomes lower than a predetermined threshold value, and the low resolution unit 81 generates a low resolution image at the specified magnification. May be good.

FIG. 14 is a block diagram showing an outline of the super-resolution device according to the present invention. The learning device 90 according to the present invention is a super-resolution device that generates a high-resolution image that is an image having a higher resolution than the target image from the input target image, and has a stepwise higher resolution from the target image. From the intermediate resolution, which is the resolution between the resolution of the target image and the resolution of the high-resolution image, to the super-resolution unit 91 (for example, the super-resolution unit 50) that generates the image of the high-resolution image. It includes a storage unit 92 (for example, a parameter holding unit 30) that stores a parameter for super-resolution to resolution and a parameter for super-resolution from the resolution of the target image to an intermediate resolution.

The super-resolution unit 91 uses parameters to generate at least one intermediate-resolution image from the target image, and generates a high-resolution image from the generated intermediate-resolution image.

Even with such a configuration, a high-precision high-resolution image can be generated from a low-resolution image.

Further, the storage unit 92 may store parameters for super-resolution to a plurality of intermediate resolutions having different resolutions. Then, the super-resolution unit 91 may generate an image of intermediate resolution step by step so as to increase the resolution by using the parameters.

Some or all of the above embodiments may also be described, but not limited to:

(Appendix 1) A low-resolution image that generates a low-resolution image that is an image with a lower resolution than the learning image based on the learning image, and a first resolution that is a combination of two adjacent resolutions. A learning unit that learns parameters for super-resolution from the first resolution image to the second resolution image using each image of the second resolution, and stores the parameters for each combination of the learned resolutions. A storage unit is provided, and the low-resolution unit gradually generates low-resolution images having a plurality of resolutions based on the learning image, and the learning unit uses the learning image and the low-resolution image. A learning device characterized in that, among image images, images having adjacent resolutions are used as a first-resolution image and a second-resolution image.

(Appendix 2) A super-resolution unit that generates an image having a higher resolution than the target image by using the parameters stored in the storage unit is provided, and the super-resolution unit is generated by the low-resolution unit. The low-resolution image or the low-resolution image generated by the super-resolution unit is super-resolutiond to generate a higher-resolution image, and the learning unit is the low-resolution image generated by the super-resolution unit. The learning device according to Appendix 1 for learning parameters using a resolution image as a first-resolution image.

(Appendix 3) An image by comparing the second resolution image generated by the low resolution unit with the super-resolution image from the first resolution image using the parameters stored in the storage unit. An image deterioration determination unit for determining deterioration is provided, and the image deterioration determination unit determines a magnification for generating a low-resolution image between the first resolution and the second resolution when the image deterioration is higher than a predetermined threshold value. The identification and low resolution unit generates a low resolution image at the specified magnification, and the learning unit uses the first resolution image for the parameters to super-resolution the generated low resolution image. The learning apparatus according to Appendix 2 for learning and learning using a low-resolution image in which parameters for super-resolution to a second-resolution image are generated.

(Appendix 4) The low-resolution unit generates a low-resolution image from the learning image, and generates a low-resolution image having a lower resolution from the generated low-resolution image. The learning device according to any one of the above.

(Appendix 5) The low-resolution unit includes a magnification-determining unit that determines the magnification when generating a low-resolution image, and the magnification-determining unit has a magnification determination unit that causes image deterioration of images having close resolutions to be greater than a predetermined threshold. The learning device according to any one of Supplementary note 1 to Supplementary note 4, wherein the reduction factor is specified, and the low resolution unit generates a low resolution image at the specified magnification.

(Appendix 6) A super-resolution device that generates a high-resolution image, which is an image having a higher resolution than the target image, from the input target image, and gradually increases the resolution of the target image. The super-resolution unit to be generated, the parameter for super-resolution from the intermediate resolution which is the resolution between the resolution of the target image and the resolution of the high-resolution image to the resolution of the high-resolution image, and the above-mentioned The super-resolution unit includes a storage unit that stores parameters for super-resolution from the resolution of the target image to the intermediate resolution, and the super-resolution unit uses the parameters to at least one image of the intermediate resolution from the target image. A super-resolution device for generating the high-resolution image from the generated intermediate-resolution image.

(Appendix 7) The storage unit stores parameters for super-resolution to a plurality of intermediate resolutions having different resolutions, and the super-resolution unit uses the parameters to gradually increase the intermediate resolution so that the resolution becomes higher. The super-resolution device according to Appendix 6, which generates an image of.

(Appendix 8) Based on the learning image, a low-resolution image, which is an image having a lower resolution than the learning image, is gradually generated for a plurality of resolutions, and is a combination of two adjacent resolutions. Using each of the resolution and the second resolution images, the parameters for super-resolution from the first resolution image to the second resolution image are learned, and the parameters are stored for each combination of the learned resolutions. A learning method characterized in that, at the time of the learning, an image having a resolution close to the learning image and the low-resolution image is used as a first-resolution image and a second-resolution image.

(Appendix 9) Using the parameters stored in the storage unit, super-resolution processing for generating an image having a resolution higher than that of the target image is performed, and the low-resolution image generated by the super-resolution processing is performed. Alternatively, the low-resolution image generated by the super-resolution processing is super-resolutiond to generate a higher-resolution image, and the low-resolution image generated as a higher-resolution image is set to the first resolution. The learning method according to Appendix 8 for learning parameters by using them as images.

(Appendix 10) A super-resolution method for generating a high-resolution image, which is an image having a higher resolution than the target image, from the input target image, and a stepwise higher resolution image is obtained from the target image. The parameters that are generated and super-resolution from the intermediate resolution, which is the resolution between the resolution of the target image and the resolution of the high-resolution image, to the resolution of the high-resolution image, and the resolution of the target image. A storage unit that stores parameters for super-resolution to an intermediate resolution is referred to, and at least one image of the intermediate resolution is generated from the target image using the parameters stored in the storage unit. A super-resolution method characterized by generating the high-resolution image from an image having an intermediate resolution.

(Appendix 11) Refer to a storage unit that stores parameters for super-resolution into a plurality of intermediate resolutions having different resolutions, and use the parameters stored in the storage unit to gradually increase the intermediate resolution so that the resolution becomes higher. 10. The super-resolution method according to Appendix 10 for generating an image of.

(Appendix 12) A low-resolution process for generating a low-resolution image, which is an image having a lower resolution than the learning image, on a computer based on the learning image, which is a combination of two adjacent resolutions. The learning process for learning the parameters for super-resolution from the first-resolution image to the second-resolution image using the respective resolution and second-resolution images, and the above-mentioned for each combination of the learned resolutions. A storage process for storing parameters in a storage unit is executed, the low-resolution processing is used to generate low-resolution images having a plurality of resolutions stepwise based on the learning image, and the learning process is used to generate a low-resolution image having a plurality of resolutions. A learning program for using images having adjacent resolutions as a first-resolution image and a second-resolution image among the training image and the low-resolution image.

(Appendix 13) A computer is made to execute a super-resolution process for generating an image having a resolution higher than that of the target image by using the parameters stored in the storage unit, and the super-resolution process lowers the resolution. The low-resolution image generated by the unit or the low-resolution image generated by the super-resolution unit is super-resolutiond to generate a higher-resolution image, and the super-resolution processing is performed in the learning process. The learning program according to Appendix 12, wherein the parameters are trained by using the low-resolution image generated in the above as the image of the first resolution.

(Appendix 14) A super-resolution program applied to a computer that generates a high-resolution image that is an image having a higher resolution than the target image from the input target image, and the target image is applied to the computer. An intermediate resolution, which is a resolution between the resolution of the target image and the resolution of the high-resolution image, is executed in the super-resolution process to gradually generate a high-resolution image from the image. Refers to a storage unit that stores parameters for super-resolution to the resolution of the high-resolution image and parameters for super-resolution from the resolution of the target image to the intermediate resolution, and is stored in the storage unit. A super-resolution program for generating at least one intermediate-resolution image from the target image using the parameters and generating the high-resolution image from the generated intermediate-resolution image.

(Appendix 15) A computer is referred to a storage unit that stores parameters for super-resolution to a plurality of intermediate resolutions having different resolutions in the super-resolution processing, and the parameters stored in the storage unit are used. The super-resolution program according to Appendix 13, which generates an image having an intermediate resolution stepwise so as to increase the resolution.

10 Image input unit for learning 20 Learning unit 21a, 21b, 21c, 24a, 24b, 24c Low-resolution means 22a, 22b, 22c, 23a, 23b, 23c Super-resolution learning means 30 Parameter holding units 30a, 30b, 30c Learning parameter holding unit 40 Super-resolution target image input unit 50 Super-resolution unit 50a, 50b, 50c Super-resolution means 60 Image output unit 70 Magnification determination unit 71a, 71b, 71c Super-resolution means 72a, 72b, 72c Image deterioration determination means

Claims (9)

A low-resolution unit that generates a low-resolution image that is an image with a lower resolution than the learning image based on the learning image.
A learning unit that learns parameters for super-resolution from the first-resolution image to the second-resolution image using each of the first-resolution and second-resolution images, which are a combination of two adjacent resolutions. ,
A storage unit that stores the parameters for each combination of learned resolutions ,
It is provided with a super-resolution unit that generates an image having a resolution higher than that of the target image by using the parameters stored in the storage unit.
The low-resolution unit gradually generates low-resolution images having a plurality of resolutions based on the learning image.
The super-resolution unit super-resolutions a low-resolution image generated by the low-resolution unit or a low-resolution image generated by the super-resolution unit to produce an image having a higher resolution. Generate and
The learning unit uses images having adjacent resolutions as the first-resolution image and the second-resolution image among the learning image and the low-resolution image, and the low-resolution image generated by the super-resolution unit. A learning device characterized in that an image is used as an image of the first resolution to learn parameters. The image deterioration is determined by comparing the second resolution image generated by the low resolution unit with the super-resolution image from the first resolution image using the parameters stored in the storage unit. Equipped with an image deterioration judgment unit
The image deterioration determination unit specifies a magnification for generating a low resolution image between the first resolution and the second resolution when the image deterioration is higher than a predetermined threshold value.
The low resolution section generates a low resolution image at a specified magnification and
The learning unit learns the parameters for super-resolution to the generated low-resolution image using the first-resolution image, and generates the parameters for super-resolution to the second-resolution image. learning device according to claim 1 trained using. The learning device according to claim 1 or 2, wherein the low-resolution unit generates a low-resolution image from a learning image, and generates a low-resolution image having a lower resolution from the generated low-resolution image. .. The low resolution unit is provided with a magnification determination unit that determines the magnification when generating a low resolution image.
The magnification determining unit specifies a magnification at which the image deterioration of images having close resolutions becomes lower than a predetermined threshold value.
The learning device according to any one of claims 1 to 3 , wherein the low resolution unit generates a low resolution image at a specified magnification. A super-resolution device that generates a high-resolution image that is an image with a higher resolution than the target image from the input target image.
A super-resolution unit that gradually generates a high-resolution image from the target image,
The parameter for super-resolution from the intermediate resolution, which is the resolution between the resolution of the target image and the resolution of the high-resolution image, to the resolution of the high-resolution image, and the resolution of the target image to the intermediate resolution. Equipped with a storage unit that stores super-resolution parameters
The storage unit super-resolutions the image of the first resolution into the image of the second resolution, which is learned using the images of the first resolution and the second resolution, which are a combination of two adjacent resolutions. Parameters that are learned by using a low-resolution image, which is an image with a lower resolution than the learning image, generated by the super-resolution unit by super-resolution as the image of the first resolution. Remember,
The super-resolution unit is characterized in that at least one image having the intermediate resolution is generated from the target image using the parameters, and the high-resolution image is generated from the generated intermediate resolution image. Super-resolution device. Based on the learning image, a low-resolution image, which is an image having a lower resolution than the learning image, is gradually generated for a plurality of resolutions.
Using the images of the first resolution and the second resolution, which are a combination of two adjacent resolutions, the parameters for super-resolution from the image of the first resolution to the image of the second resolution are learned.
The parameters are stored in the storage unit for each learned resolution combination, and the parameters are stored in the storage unit.
The image generated by using the parameter stored in the storage unit or the low-resolution image generated stepwise is super-resolution to generate a higher-resolution image.
At the time of the training, among the training image and the low resolution image, an image having a resolution close to each other is used as a first resolution image and a second resolution image, and the super-resolution low resolution image is used. A learning method characterized in that parameters are learned by using the image as the first resolution image. It is a super-resolution method that generates a high-resolution image that is an image with a higher resolution than the target image from the input target image.
A high-resolution image is gradually generated from the target image,
The parameter for super-resolution from the intermediate resolution, which is the resolution between the resolution of the target image and the resolution of the high-resolution image, to the resolution of the high-resolution image, and the resolution of the target image to the intermediate resolution. With reference to a storage unit that stores the parameters to be super-resolution, at least one image having the intermediate resolution is generated from the target image using the parameters stored in the storage unit.
The high resolution image is generated from the generated intermediate resolution image, and the image is generated.
The parameters stored in the storage unit are from the image of the first resolution to the image of the second resolution, which are learned using the images of the first resolution and the second resolution, which are a combination of two adjacent resolutions. A parameter learned by using a low-resolution image, which is a parameter generated by the super-resolution and having a lower resolution than the learning image, as the image of the first resolution. super-resolution method, characterized in that it. On the computer
A low-resolution process that generates a low-resolution image that is an image with a lower resolution than the learning image based on the learning image.
A learning process that learns parameters for super-resolution from an image of the first resolution to an image of the second resolution using images of the first resolution and the second resolution, which are a combination of two adjacent resolutions .
A storage process that stores the parameters in the storage unit for each learned resolution combination , and
Using the parameters stored in the storage unit, a super-resolution process for generating an image having a resolution higher than that of the target image is executed.
In the low resolution process, low resolution images having a plurality of resolutions are gradually generated based on the learning image.
In the super-resolution processing, the low-resolution image generated by the low-resolution processing or the low-resolution image generated by the super-resolution processing is super-resolutiond to have a higher resolution. Generate high images,
In the learning process, of the training image and the low resolution image, images having adjacent resolutions are used as the first resolution image and the second resolution image, and the low resolution generated by the super-resolution processing is performed. A learning program for learning parameters by using an image as an image of the first resolution. A super-resolution program applied to a computer that generates a high-resolution image that is an image with a higher resolution than the target image from the input target image.
On the computer
A super-resolution process for gradually generating a high-resolution image from the target image is executed.
In the super-resolution processing, a parameter for super-resolution from an intermediate resolution, which is a resolution between the resolution of the target image and the resolution of the high-resolution image, to the resolution of the high-resolution image, and the target. A storage unit that stores parameters for super-resolution from the image resolution to the intermediate resolution is referred to, and at least one image of the intermediate resolution is generated from the target image using the parameters stored in the storage unit. And generate the high resolution image from the generated intermediate resolution image .
The parameters stored in the storage unit are from the image of the first resolution to the image of the second resolution, which are learned using the images of the first resolution and the second resolution, which are a combination of two adjacent resolutions. A low-resolution image, which is a parameter for super-resolution and is an image having a lower resolution than the learning image, which is generated by super-resolution in the super-resolution processing, is used as the first resolution image. It is a parameter learned using
A super-resolution program characterized by this.

Images