JP7637282B2 - Medical image processing device, medical image processing method and program - Google Patents

Description

The present invention relates to a medical image processing device, a medical image processing method, and a program.

In the medical field, images are acquired by various imaging devices to identify the disease of a subject or observe the extent of the disease, and image diagnosis is performed by medical professionals. For example, in the field of radiology, types of imaging devices include X-ray devices, X-ray computed tomography (CT) devices, magnetic resonance imaging (MRI) devices, positron emission tomography (PET) devices, and single photon emission computed tomography (SPECT) devices. In the field of ophthalmology, for example, there are fundus cameras, scanning laser ophthalmoscopes (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

To perform image diagnosis accurately and complete it in a short time, it is important that the images acquired by the imaging device have high image quality, such as low noise, high resolution and spatial resolution, and appropriate gradation. Images that highlight the area or lesion to be observed can also be useful.

However, with many imaging devices, some trade-off is required to obtain images suitable for diagnostic imaging, such as high image quality. For example, one option is to purchase a high-performance imaging device to obtain high-image-quality images, but this often requires a larger investment than a low-performance device.

In addition, for example, in CT, the subject's radiation dose may need to be increased to obtain an image with less noise. In addition, for example, in MRI, a contrast agent that carries a risk of side effects may be used to obtain an image that highlights the area to be observed. In addition, for example, in OCT, if the area to be photographed is large or high spatial resolution is required, the imaging time may be longer. In addition, for example, some imaging devices require multiple images to be taken in order to obtain an image with high image quality, which increases the imaging time.

Patent Document 1 discloses a technology that uses an artificial intelligence engine to convert previously acquired images into images with higher resolution in order to accommodate rapid advances in medical technology and the need for simple imaging in emergencies. With this technology, for example, it is possible to convert an image acquired by simple imaging with little cost into an image with higher resolution.

JP 2018-5841 A

However, even high-resolution images may not be suitable for image diagnosis. For example, even if the image has high resolution, if there is a lot of noise or low contrast, the object to be observed may not be properly identified.

In response to this, one of the objectives of the present invention is to provide a medical image processing device, a medical image processing method, and a program that can generate images that are more suitable for image diagnosis than conventional methods.

A medical image processing device according to one embodiment of the present invention includes an acquisition unit that acquires a first image, which is an image of a specific part of a subject, and a display control unit that controls a display unit to display either a second image generated by inputting the first image into an image quality improvement engine including a machine learning engine as an input image of the image quality improvement engine, or the first image, and when the shooting conditions of the first image are an image quality improvement angle that can be improved by the image quality improvement engine, the display unit is controlled to display the second image, and when the shooting conditions of the first image are not an image quality improvement angle that can be improved by the image quality improvement engine, the display unit is controlled to display the first image .

In addition, a medical image processing method according to another embodiment of the present invention includes acquiring a first image, which is an image of a specific part of a subject, and controlling a display unit to display either a second image generated by inputting the first image into an image quality improvement engine including a machine learning engine as an input image of the image quality improvement engine, or the first image, wherein if the shooting conditions of the first image are an image quality improvement angle that can be improved by the image quality improvement engine, the display unit is controlled to display the second image, and if the shooting conditions of the first image are not an image quality improvement angle that can be improved by the image quality improvement engine, the display unit is controlled to display the first image .

According to one aspect of the present invention, it is possible to generate images that are more suitable for image diagnosis than conventional methods.

1 shows an example of the configuration of a neural network related to image quality improvement processing. 1 shows an example of the configuration of a neural network related to a shooting location estimation process. 1 shows an example of the configuration of a neural network related to image authenticity evaluation processing. 1 illustrates an example of a schematic configuration of an image processing apparatus according to a first embodiment. FIG. 4 is a flowchart showing an example of a flow of image processing according to the first embodiment. FIG. 11 is a flowchart showing another example of the flow of image processing according to the first embodiment. FIG. 11 is a flowchart showing an example of the flow of image processing according to the second embodiment. FIG. 13 is a diagram for explaining image processing according to a fourth embodiment. FIG. 13 is a flowchart showing an example of the flow of image quality improvement processing according to the fourth embodiment. FIG. 13 is a diagram for explaining image processing according to the fifth embodiment. FIG. 13 is a flowchart showing an example of the flow of image quality improvement processing according to the fifth embodiment. FIG. 13 is a diagram for explaining image processing according to the sixth embodiment. FIG. 23 is a flowchart showing an example of the flow of image quality improvement processing according to the sixth embodiment. FIG. 13 is a diagram for explaining image processing according to the sixth embodiment. 23 shows an example of a schematic configuration of an image processing device according to a seventh embodiment. FIG. 23 is a flowchart showing an example of the flow of image processing according to the seventh embodiment. 23 shows an example of a user interface according to the seventh embodiment. 23 shows an example of a schematic configuration of an image processing device according to a ninth embodiment. FIG. 23 is a flowchart showing an example of the flow of image processing according to the ninth embodiment. 23 shows an example of a schematic configuration of an image processing device according to a twelfth embodiment. FIG. 23 is a flowchart showing an example of the flow of image quality improvement processing according to the thirteenth embodiment. FIG. 23 is a flowchart showing another example of the flow of image quality improvement processing according to the thirteenth embodiment. 23 shows an example of a schematic configuration of an image processing device according to a seventeenth embodiment. FIG. 23 is a flow diagram showing an example of the flow of image processing according to the seventeenth embodiment.

Below, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. In addition, the same reference numerals are used between the drawings to indicate elements that are identical or functionally similar.

<Terminology>
First, the terms used in this specification will be explained.

In the network described in this specification, each device may be connected by a wired or wireless line. Here, the lines connecting each device in the network include, for example, a dedicated line, a local area network (hereinafter referred to as LAN) line, a wireless LAN line, an Internet line, Wi-Fi (registered trademark), and Bluetooth (registered trademark), etc.

The medical image processing device may be composed of two or more devices capable of communicating with each other, or may be composed of a single device. Furthermore, each component of the medical image processing device may be composed of a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Furthermore, each component may be composed of a circuit that performs a specific function, such as an ASIC. Furthermore, it may be composed of a combination of any other hardware and any software.

In addition, the medical images processed by the medical image processing device or medical image processing method according to the following embodiments include images acquired using any modality (imaging device, imaging method). The medical images to be processed can include medical images acquired by any imaging device, etc., and images created by the medical image processing device or medical image processing method according to the following embodiments.

Furthermore, the medical image to be processed is an image of a specific part of the subject, and the image of the specific part includes at least a part of the specific part of the subject. The medical image may also include other parts of the subject. The medical image may be a still image or a moving image, and may be a black and white image or a color image. The medical image may also be an image that represents the structure (shape) of the specific part, or an image that represents its function. Images that represent function include, for example, images that represent hemodynamics (blood flow rate, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasonic Doppler images. The specific part of the subject may be determined according to the subject to be photographed, and may include any part such as the human eye (subject's eye), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, the head, chest, legs, and arms.

The medical image may be a tomographic image of the subject, or a frontal image. Frontal images include, for example, a frontal image of the fundus, a frontal image of the anterior segment, a fundus image photographed by fluorescence, and an En-Face image generated using data acquired by OCT from at least a partial range in the depth direction of the subject.

An imaging device is a device for capturing images used in diagnosis. Imaging devices include, for example, devices that obtain an image of a specific part of a subject by irradiating the subject with light, radiation such as X-rays, electromagnetic waves, or ultrasound, and devices that obtain an image of a specific part by detecting radiation emitted from a subject. More specifically, imaging devices according to the following embodiments include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscope.

The OCT device may include a time domain OCT (TD-OCT) device or a Fourier domain OCT (FD-OCT) device. The Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a swept wavelength OCT (SS-OCT) device. The SLO device or OCT device may include a wavefront compensation SLO (AO-SLO) device or a wavefront compensation OCT (AO-OCT) device using a wavefront compensation optical system.

The image management system is a device and system that receives and stores images captured by an imaging device and images that have been processed. The image management system can also transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. Examples of the image management system include a picture archiving and communication system (PACS). In particular, the image management system according to the embodiment described below includes a database that can store various information such as subject information and shooting time associated with received images. The image management system is also connected to a network, and can send and receive images, convert images, and send and receive various information associated with stored images in response to a request from other devices.

The imaging conditions are various pieces of information at the time of capturing an image obtained by an imaging device. The imaging conditions include, for example, information about the imaging device, information about the facility where the imaging was performed, information about the examination related to the imaging, information about the photographer, and information about the subject. The imaging conditions also include, for example, the imaging date and time, the name of the imaging part, the imaging area, the imaging angle of view, the imaging method, the image resolution and gradation, the image size, the applied image filter, information about the image data format, and information about the radiation dose. The imaging area can include a surrounding area that is shifted from a specific imaging part, an area that includes multiple imaging parts, etc.

The shooting conditions can be stored in the data structure constituting the image, or as shooting condition data separate from the image, or stored in a database or image management system related to the shooting device. Therefore, the shooting conditions can be acquired by a procedure corresponding to the storage means of the shooting conditions of the shooting device. Specifically, the shooting conditions are acquired, for example, by analyzing the data structure of the image output by the shooting device, acquiring shooting condition data corresponding to the image, or accessing an interface for acquiring the shooting conditions from a database related to the shooting device.

Depending on the imaging device, there may be some imaging conditions that cannot be acquired because they have not been saved. For example, this may be the case when the imaging device does not have the function to acquire or save specific imaging conditions, or when such a function is disabled. In addition, there may be cases where the imaging conditions are not saved because they are not related to the imaging device or imaging. Furthermore, there may be cases where the imaging conditions are hidden, encrypted, or cannot be acquired without the right. However, there are cases where even imaging conditions that have not been saved can be acquired. For example, the name of the imaging part or the imaging area can be identified by performing image analysis.

A machine learning model is a model that has been trained (learned) in advance for an arbitrary machine learning algorithm using appropriate teacher data (learning data). The teacher data consists of one or more pairs of input data and output data. The format and combination of the input data and output data of the pairs that make up the teacher data may be suitable for the desired configuration, such as one being an image and the other being a number, one being composed of a group of images and the other being a string, or both being images.

Specific examples of such training data include training data (hereinafter referred to as "first training data") that is composed of a set of pairs of an image acquired by OCT and an imaging site label corresponding to the image. The imaging site label is a unique numerical value or character string that represents the site. Other examples of training data include training data (hereinafter referred to as "second training data") that is composed of a set of pairs of a noisy low-quality image acquired by normal OCT imaging and a high-quality image that has been imaged multiple times by OCT and processed to improve image quality.

When input data is input to the machine learning model, output data according to the design of the machine learning model is output. The machine learning model outputs output data that is likely to correspond to the input data, for example, according to the tendency trained using the teacher data. The machine learning model can also output the possibility of each type of output data corresponding to the input data as a numerical value, for example, according to the tendency trained using the teacher data. Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first teacher data, the machine learning model outputs the imaging site label of the imaging site captured in the image, or outputs the probability for each imaging site label. Also, for example, when a noisy low-quality image acquired by normal OCT imaging is input to a machine learning model trained with the second teacher data, the machine learning model outputs a high-quality image equivalent to an image captured multiple times by OCT and processed for high image quality. Note that the machine learning model can be configured not to use the output data it outputs as teacher data from the viewpoint of maintaining quality.

The machine learning algorithm also includes deep learning techniques such as convolutional neural networks (CNN). In deep learning techniques, different parameter settings for layers and nodes constituting a neural network may result in different degrees of reproducibility of trends trained using training data in output data. For example, in a deep learning machine learning model using first training data, setting more appropriate parameters may increase the probability of outputting a correct imaging site label. Also, in a deep learning machine learning model using second training data, setting more appropriate parameters may result in higher quality images being output.

Specifically, parameters in a CNN can include, for example, the filter kernel size, number of filters, stride value, and dilation value set for the convolutional layer, as well as the number of nodes output by the fully connected layer. The parameter group and the number of training epochs can be set to values that are favorable for the usage form of the machine learning model based on the training data. For example, a parameter group and the number of epochs can be set based on the training data to output a correct imaging site label with a higher probability or to output a higher quality image.

One method for determining such a parameter set and the number of epochs is exemplified below. First, 70% of the pair sets constituting the teacher data are set for training, and the remaining 30% are set for evaluation, randomly. Next, the machine learning model is trained using the pair sets for training, and at the end of each epoch of training, a training evaluation value is calculated using the pair sets for evaluation. The training evaluation value is, for example, the average value of a group of values obtained by evaluating the output when the input data constituting each pair is input to the machine learning model being trained, and the output data corresponding to the input data, using a loss function. Finally, the parameter set and the number of epochs when the training evaluation value is the smallest are determined as the parameter set and the number of epochs of the machine learning model. In this way, by dividing the pair sets constituting the teacher data into those for training and those for evaluation and determining the number of epochs, it is possible to prevent the machine learning model from overlearning the pair sets for training.

The high-quality image engine is a module that outputs a high-quality image by improving the quality of an input low-quality image. In this specification, high-quality image refers to converting an input image into an image with a quality more suitable for image diagnosis, and a high-quality image refers to an image converted into an image with a quality more suitable for image diagnosis. In addition, a low-quality image is, for example, a two-dimensional image or a three-dimensional image acquired by X-ray photography, CT, MRI, OCT, PET, or SPECT, or a three-dimensional moving image of a CT taken continuously without being set to obtain a particularly high quality image. Specifically, low-quality images include, for example, images acquired by low-dose photography using an X-ray imaging device or CT, photography using an MRI without using a contrast agent, images acquired by short-time photography using OCT, and OCTA images acquired with a small number of photography sessions.

The content of image quality suitable for image diagnosis depends on what is being diagnosed with various types of image diagnosis. Therefore, it is not possible to make a general statement, but for example, image quality suitable for image diagnosis includes image quality with little noise, high contrast, colors and gradations that make it easy to observe the subject, large image size, and high resolution. It can also include image quality in which non-existent objects and gradations that are drawn during the image generation process are removed from the image.

In addition, when high-quality images with low noise and high contrast are used for image analysis such as vascular analysis processing of images such as OCTA, or area segmentation processing of images such as CT or OCT, the analysis can often be performed more accurately than when low-quality images are used. Therefore, high-quality images output by the image quality engine can be useful not only for image diagnosis, but also for image analysis.

The image processing method constituting the image quality improvement method in the following embodiment performs processing using various machine learning algorithms such as deep learning. Note that in addition to processing using machine learning algorithms, the image processing method may also perform any existing processing such as various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing.

In particular, as an example of a CNN configuration for improving the image quality of a two-dimensional image, there is a configuration shown in FIG. 1. The configuration of the CNN includes a group of multiple convolution processing blocks 100. The convolution processing block 100 includes a convolution layer 101, a batch normalization layer 102, and an activation layer 103 using a normalized linear function (Rectifier Linear Unit). The configuration of the CNN also includes a merger layer 104 and a final convolution layer 105. The merger layer 104 merges the output value group of the convolution processing block 100 with the pixel value group constituting the image by connecting or adding them. The final convolution layer 105 outputs the pixel value group constituting the high-image-quality image Im120 merged in the merger layer 104. In this configuration, the pixel values constituting the input image Im110 are output through the convolution processing blocks 100, and the pixel values constituting the input image Im110 are synthesized in the synthesis layer 104. The synthesized pixel values are then shaped into a high-quality image Im120 in the final convolution layer 105.

For example, a certain level of image quality improvement can be achieved by setting the number of convolution processing blocks 100 to 16, setting the parameters of the convolution layer 101 group to a filter kernel size of 3 pixels wide and 3 pixels high, and setting the number of filters to 64. However, in practice, as described in the above description of the machine learning model, a better parameter group can be set using training data according to the usage form of the machine learning model. If three-dimensional or four-dimensional images need to be processed, the filter kernel size may be expanded to three or four dimensions.

Note that when using some image processing methods, such as image processing using CNN, attention must be paid to image size. Specifically, it should be noted that in order to address issues such as the issue of the peripheral areas of a high-quality image not being of sufficient quality, different image sizes may be required for the input low-quality image and the output high-quality image.

For the sake of clarity, this will not be specified in the embodiments described below, but when a high-image-quality engine is adopted that requires different image sizes for the image input to the high-image-quality engine and the image output, the image size is adjusted appropriately. Specifically, the image size is adjusted for input images, such as images used as teacher data for training a machine learning model or images input to the high-image-quality engine, by padding or combining the surrounding shooting area of the input image. Note that the area to be padded is filled with a fixed pixel value, filled with nearby pixel values, or mirror padded according to the characteristics of the high-image-quality technique so that the image quality can be effectively improved.

The image quality improvement method may be implemented using only one image processing method, or may be implemented using a combination of two or more image processing methods. In addition, multiple image quality improvement methods may be implemented in parallel to generate multiple high-quality image groups, and the highest quality image may be selected as the final high-quality image. The highest quality image may be selected automatically using an image quality assessment index, or may be selected according to the examiner's (user's) instructions by displaying multiple high-quality image groups on a user interface provided on an arbitrary display unit, etc.

In some cases, an input image that has not been enhanced in image quality may be more suitable for image diagnosis, so the input image may be included in the final image selection. Parameters may also be input to the image enhancement engine along with the low-image quality image. For example, parameters specifying the degree of image enhancement or parameters specifying the image filter size used in the image processing method may be input to the image enhancement engine along with the input image.

The shooting location estimation engine is a module that estimates the shooting part and shooting area of the input image. The shooting location estimation engine can output the location of the shooting part and shooting area depicted in the input image, or the probability that it is the shooting part or shooting area for each shooting part label or shooting area label of the required level of detail.

Depending on the imaging device, the imaging part and imaging area may not be saved as imaging conditions, or may not be acquired by the imaging device and saved. Even if the imaging part and imaging area are saved, the imaging part and imaging area may not be saved at the required level of detail. For example, the imaging part may only be saved as "posterior segment," but it is not clear whether it is the "macular area," "optic disc," "macular area and optic disc," or "other." In another example, the imaging part may only be saved as "breast," but it is not clear whether it is the "right breast," "left breast," or "both." Therefore, by using an imaging location estimation engine, it is possible to estimate the imaging part and imaging area of the input image in these cases.

The image and data processing method constituting the estimation method of the imaging location estimation engine performs processing using various machine learning algorithms such as deep learning. Note that the image and data processing method may perform any existing estimation processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge-based processing, etc. in addition to or instead of processing using a machine learning algorithm. Note that the teacher data used to train the machine learning model constructed using the machine learning algorithm can be images labeled with the imaging site and imaging area. In this case, the images of the teacher data are used as input data, and the labels of the imaging site and imaging area are used as output data.

In particular, as an example of a CNN configuration that estimates the shooting location of a two-dimensional image, there is a configuration shown in FIG. 2. The CNN configuration includes a group of multiple convolution processing blocks 200, each of which is composed of a convolution layer 201, a batch normalization layer 202, and an activation layer 203 using a normalized linear function. The CNN configuration also includes a final convolution layer 204, a fully connected layer 205, and an output layer 206. The fully connected layer 205 fully connects the output value groups of the convolution processing blocks 200. The output layer 206 also uses a Softmax function to output the probability of each expected shooting part label for the input image Im210 as an estimation result (Result) 207. In this configuration, for example, if the input image Im210 is an image of the "macular region", the highest probability is output for the shooting part label corresponding to the "macular region.

For example, the imaging site can be estimated with a certain degree of accuracy by setting the number of convolution processing blocks 200 to 16, the parameters of the convolution layer 201 group to a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters to 64. However, in practice, as described in the above description of the machine learning model, a better parameter group can be set using training data according to the usage form of the machine learning model. If it is necessary to process three-dimensional or four-dimensional images, the filter kernel size may be expanded to three or four dimensions. The estimation method may be implemented using only one image and data processing method, or may be implemented using a combination of two or more image and data processing methods.

The image quality assessment engine is a module that outputs an image quality assessment index for an input image. The image quality assessment processing method for calculating the image quality assessment index performs processing using various machine learning algorithms such as deep learning. Note that the image quality assessment processing method may perform any existing evaluation processing such as an image noise measurement algorithm and a matching process using a database of image quality assessment indices corresponding to similar images and base images. Note that these evaluation processes may be performed in addition to or instead of processing using a machine learning algorithm.

For example, the image quality assessment index can be obtained from a machine learning model constructed using a machine learning algorithm. In this case, the pair of input data constituting the teacher data for training the machine learning model is a group of images consisting of a group of low-quality images and a group of high-quality images captured in advance under various shooting conditions. In addition, the pair of output data constituting the teacher data for training the machine learning model is, for example, a group of image quality assessment indices set for each of the image groups of the input data by an examiner performing image diagnosis.

The authenticity evaluation engine in the description of this invention is a module that evaluates the rendering of an input image and evaluates with a certain degree of accuracy whether the image was captured and acquired by the target imaging device. The authenticity evaluation processing method performs processing using various machine learning algorithms such as deep learning. Note that the authenticity evaluation processing method may perform any existing evaluation processing such as knowledge-based processing in addition to or instead of processing using machine learning algorithms.

For example, the authenticity evaluation process can be performed by a machine learning model constructed using a machine learning algorithm. First, the training data of the machine learning model will be described. The training data includes a pair group of high-quality images captured in advance under various shooting conditions and a label (hereinafter, genuine label) indicating that the images were captured and acquired by the target imaging device. The training data also includes a pair group of high-quality images generated by inputting low-quality images to an image quality improvement engine (first level image quality improvement engine) and a label (hereinafter, counterfeit label) indicating that the images were not captured and acquired by the target imaging device. A machine learning model trained using such training data outputs a counterfeit label when a high-quality image generated by the first level image quality improvement engine is input.

In particular, FIG. 3 shows an example of a CNN configuration for performing authenticity evaluation processing of two-dimensional images. The CNN configuration includes a group of multiple convolution processing blocks 300, each of which is composed of a convolution layer 301, a batch normalization layer 302, and an activation layer 303 using a normalized linear function. The CNN configuration also includes a final convolution layer 304, a fully connected layer 305, and an output layer 306. The fully connected layer 305 fully connects the output value groups of the convolution processing blocks 300. The output layer 306 also uses a sigmoid function to output a value of 1 (true) representing an authentic label or a value of 0 (false) representing a counterfeit label as the result (Result) 307 of the authenticity evaluation processing for the input image Im 310.

Note that by setting the number of convolution processing blocks 300 to 16, the parameters of the convolution layer 301 group to a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters to 64, it is possible to obtain accurate authenticity evaluation results with a certain degree of precision. However, as described in the above explanation of the machine learning model, in practice, a better parameter group can be set using training data according to the usage form of the machine learning model. Note that when three-dimensional or four-dimensional images need to be processed, the filter kernel size may be expanded to three or four dimensions.

The authenticity evaluation engine may output an authenticity label when a high-quality image generated by a high-quality engine (second-level image quality engine) that provides higher image quality than the first-level image quality engine is input. In other words, the authenticity evaluation engine cannot reliably evaluate whether an input image is an image that was captured and acquired by a camera device, but it can evaluate whether the image has the resemblance of an image that was captured and acquired by a camera device. By utilizing this characteristic, a high-quality image generated by a high-quality engine can be input to the authenticity evaluation engine, making it possible to evaluate whether the high-quality image generated by the high-quality engine has been sufficiently enhanced.

The efficiency and accuracy of both engines may be improved by training the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine in cooperation with each other. In this case, first, the machine learning model of the image quality improvement engine is trained so that when the authenticity evaluation engine evaluates a high-quality image generated by the image quality improvement engine, an authenticity label is output. In parallel, the machine learning model of the authenticity evaluation engine is trained so that when the authenticity evaluation engine evaluates an image generated by the image quality improvement engine, an authenticity label is output. In parallel, the machine learning model of the authenticity evaluation engine is trained so that when the authenticity evaluation engine evaluates an image acquired by a photographing device, an authenticity label is output. This improves the efficiency and accuracy of the image quality improvement engine and the authenticity evaluation engine.

First Embodiment
The medical image processing apparatus according to the first embodiment will be described below with reference to Figures 4 and 5. Figure 4 shows an example of a schematic configuration of the image processing apparatus according to this embodiment.

The image processing device 400 is connected to the image capture device 10 and the display unit 20 via a circuit or a network. The image capture device 10 and the display unit 20 may also be directly connected. Note that although these devices are separate devices in this embodiment, some or all of these devices may be configured as an integrated device. Furthermore, these devices may be connected to any other device via a circuit or a network, or may be configured as an integrated device with any other device.

The image processing device 400 is provided with an acquisition unit 401, an imaging condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405 (display control unit). The image processing device 400 may be composed of multiple devices in which some of these components are provided. The acquisition unit 401 can acquire various data and images from the imaging device 10 or other devices, and can acquire input from the examiner via an input device (not shown). The input device may be a mouse, a keyboard, a touch panel, or any other arbitrary input device. The display unit 20 may also be configured as a touch panel display.

The imaging condition acquisition unit 402 acquires the imaging conditions of the medical image (input image) acquired by the acquisition unit 401. Specifically, the imaging condition group stored in the data structure constituting the medical image is acquired according to the data format of the medical image. Note that if the imaging conditions are not stored in the medical image, it is possible to acquire an imaging information group including the imaging condition group from the imaging device 10 or the image management system via the acquisition unit 401.

The image quality improvement possibility determination unit 403 determines whether the medical image can be handled by the image quality improvement unit 404 using the group of shooting conditions acquired by the shooting condition acquisition unit 402. The image quality improvement unit 404 improves the image quality of the medical image that can be handled, and generates a high-quality image suitable for image diagnosis. The output unit 405 displays the high-quality image generated by the image quality improvement unit 404, the input image, various information, etc. on the display unit 20. The output unit 405 may also store the generated high-quality image, etc. in a storage device connected to the image processing device 400.

Next, the image quality improvement unit 404 will be described in detail. The image quality improvement unit 404 is equipped with an image quality improvement engine. The image quality improvement method equipped with the image quality improvement engine according to this embodiment performs processing using a machine learning algorithm.

In this embodiment, training of a machine learning model related to a machine learning algorithm uses teacher data consisting of a pair group of input data, which is a low-quality image having specific shooting conditions assumed to be processed, and output data, which is a high-quality image corresponding to the input data. The specific shooting conditions specifically include a predetermined shooting part, shooting method, shooting angle of view, image size, etc.

In this embodiment, the input data of the teacher data is a low-quality image acquired using the same model and settings as the image capture device 10. The output data of the teacher data is a high-quality image acquired using settings and image processing provided in the same model as the image capture device 10. Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by performing superimposition processing such as averaging on a group of images (original images) acquired by capturing images multiple times.

The overlaid image created by the overlay process emphasizes pixels that are commonly depicted in the original images, resulting in a high-quality image suitable for image diagnosis. In this case, the high-quality image generated is a high-contrast image with a clear distinction between low-brightness and high-brightness areas, as a result of emphasizing the commonly depicted pixels. In addition, for example, in the overlaid image, random noise that occurs with each capture can be reduced, and areas that were not well depicted in the original images at a certain point in time can be interpolated using other original images.

Note that multiple pair groups can be generated by using each of the original image groups as input data and the corresponding overlaid images as output data. For example, when obtaining one overlaid image from a group of 15 original images, a pair of the first original image in the original image group and an overlaid image, and a pair of the second original image in the original image group and an overlaid image can be generated. In this way, when obtaining one overlaid image from a group of 15 original images, 15 pair groups can be generated by using one image in the original image group and an overlaid image.

In addition, when the input data for a machine learning model needs to be composed of multiple images, the required number of original image groups can be selected from the original image group and used as input data. For example, when obtaining one overlaid image from a group of 15 original images, if two images are required as input data for the machine learning model, 105 (15C2=105) pair groups can be generated.

Note that, among the pairs that make up the training data, pairs that do not contribute to high image quality can be removed from the training data. For example, if the high-image-quality image that is the output data that makes up a pair of training data has image quality that is unsuitable for image diagnosis, the image output by the image quality improvement engine that has learned using the training data may also have image quality that is unsuitable for image diagnosis. Therefore, by removing pairs whose output data has image quality that is unsuitable for image diagnosis from the training data, the possibility that the image quality improvement engine will generate an image with image quality that is unsuitable for image diagnosis can be reduced.

In addition, if the average luminance or luminance distribution of a pair of images differs significantly, the image quality improvement engine that has trained using the training data may output an image that is not suitable for image diagnosis, with a luminance distribution that differs significantly from that of the low-quality image. For this reason, it is possible to remove from the training data pairs of input data and output data that differ significantly in average luminance or luminance distribution.

Furthermore, if the structure or position of the subject depicted in the paired images differs significantly, the image quality improvement engine that has learned using the training data may output an image that is not suitable for image diagnosis, in which the subject is depicted in a structure or position that differs significantly from that of the low-quality image. For this reason, it is possible to remove from the training data pairs of input data and output data in which the structure or position of the depicted subject differs significantly. Furthermore, from the perspective of maintaining quality, the image quality improvement engine can be configured not to use the high-quality images it outputs as training data.

By using an image quality improvement engine that has performed machine learning in this way, when a medical image acquired in a single shot is input, the image quality improvement unit 404 can output a high-image quality image in which contrast has been increased and noise has been reduced by overlay processing. Therefore, the image quality improvement unit 404 can generate a high-image quality image suitable for image diagnosis based on a low-image quality image that is the input image.

Next, a series of image processing steps according to this embodiment will be described with reference to the flow diagram in FIG. 5. FIG. 5 is a flow diagram of a series of image processing steps according to this embodiment. First, when a series of image processing steps according to this embodiment is started, the process proceeds to step S510.

In step S510, the acquisition unit 401 acquires an image captured by the imaging device 10 as an input image from the imaging device 10 connected via a circuit or network. The acquisition unit 401 may acquire the input image in response to a request from the imaging device 10. Such a request may be issued, for example, when the imaging device 10 generates an image, before or after the image generated by the imaging device 10 is saved in a recording device provided in the imaging device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

The acquisition unit 401 may acquire data for generating an image from the imaging device 10, and acquire an image generated by the image processing device 400 based on the data as an input image. In this case, any existing image generation method may be adopted as the image generation method for the image processing device 400 to generate various images.

In step S520, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image. Specifically, the shooting condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. Note that, as described above, if the shooting conditions are not stored in the input image, the shooting condition acquisition unit 402 can acquire a group of shooting information including the group of shooting conditions from the shooting device 10 or an image management system (not shown).

In step S530, the image quality improvement possibility determination unit 403 uses the acquired group of shooting conditions to determine whether the input image can be improved in image quality by the image quality improvement engine provided in the image quality improvement unit 404. Specifically, the image quality improvement possibility determination unit 403 determines whether the shooting part, shooting method, shooting angle of view, and image size of the input image match the conditions that can be handled by the image quality improvement engine.

When the image quality improvement feasibility determination unit 403 has determined that all of the shooting conditions are manageable, the process proceeds to step S540. On the other hand, when the image quality improvement feasibility determination unit 403 has determined that the image quality improvement engine cannot handle the input image based on these shooting conditions, the process proceeds to step S550.

Depending on the settings and implementation of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, imaging angle of view, and image size. For example, such processing may be performed when the image quality improvement engine is assumed to be comprehensively compatible with any imaging part of the subject and is implemented to be able to handle even unknown imaging parts included in the input data. Furthermore, the image quality improvement feasibility determination unit 403 may determine whether or not at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image matches a condition that can be handled by the image quality improvement engine, depending on the desired configuration.

In step S540, the image quality improvement unit 404 uses the image quality improvement engine to improve the image quality of the input image and generate a high-image quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to the image quality improvement engine, which generates a high-image quality image with improved image quality. The image quality improvement engine generates a high-image quality image that is similar to an overlay process using the input image, based on a machine learning model that has performed machine learning using training data. Therefore, the image quality improvement engine can generate a high-image quality image with reduced noise and enhanced contrast compared to the input image.

Depending on the settings and implementation of the image processing device 400, the image quality improvement unit 404 may input parameters to the image quality improvement engine together with the input image according to the group of shooting conditions to adjust the degree of image quality improvement, etc. Also, the image quality improvement unit 404 may input parameters according to the examiner's input to the image quality improvement engine together with the input image to adjust the degree of image quality improvement, etc.

In step S550, if a high-quality image was generated in step S540, the output unit 405 outputs the high-quality image and displays it on the display unit 20. On the other hand, if it is determined in step S530 that high-quality processing is not possible, the input image is output and displayed on the display unit 20. Note that instead of displaying the output image on the display unit 20, the output unit 405 may cause the image capture device 10 or another device to display or store the output image. Depending on the settings and implementation form of the image processing device 400, the output unit 405 may process the output image so that it can be used by the image capture device 10 or another device, or convert the data format so that it can be transmitted to an image management system, etc.

As described above, the image processing device 400 according to this embodiment includes an acquisition unit 401 and an image quality improvement unit 404. The acquisition unit 401 acquires an input image (first image) that is an image of a specific part of a subject. The image quality improvement unit 404 uses an image quality improvement engine including a machine learning engine to generate a high image quality image (second image) from the input image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image. The image quality improvement engine includes a machine learning engine that uses the image obtained by the overlay process as learning data.

With this configuration, the image processing device 400 according to this embodiment can output a high-quality image in which noise has been reduced and contrast has been enhanced from the input image. As a result, the image processing device 400 can obtain images suitable for image diagnosis, such as clearer images and images in which the area or lesion to be observed is emphasized, at a lower cost than in the past, without increasing the invasiveness or effort of the photographer or subject.

The image processing device 400 further includes a high image quality determination unit 403 that determines whether a high image quality image can be generated for an input image using a high image quality engine. The high image quality determination unit 403 makes this determination based on at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image.

With this configuration, the image processing device 400 according to this embodiment can exclude input images that cannot be processed by the image quality improvement unit 404 from the image quality improvement process, thereby reducing the processing load on the image processing device 400 and the occurrence of errors.

In this embodiment, the output unit 405 (display control unit) is configured to display the generated high-quality image on the display unit 20, but the operation of the output unit 405 is not limited to this. For example, the output unit 405 can also output the high-quality image to the imaging device 10 or other devices connected to the image processing device 400. Therefore, the high-quality image can be displayed on the user interface of these devices, saved in any recording device, used for any image analysis, or sent to an image management system.

In this embodiment, the image quality improvement possibility determination unit 403 determines whether the input image can be improved by the image quality improvement engine, and if the input image can be improved, the image quality improvement unit 404 improves the image quality. On the other hand, if the image capturing device 10 captures images only under shooting conditions that allow for high image quality improvement, the image obtained from the image capturing device 10 may be unconditionally improved in image quality. In this case, as shown in FIG. 6, the processes of steps S520 and S530 can be omitted, and step S540 can be performed after step S510.

In this embodiment, the output unit 405 is configured to display a high-quality image on the display unit 20. However, the output unit 405 may also display a high-quality image on the display unit 20 in response to an instruction from the examiner. For example, the output unit 405 may display a high-quality image on the display unit 20 in response to the examiner pressing an arbitrary button on the user interface of the display unit 20. In this case, the output unit 405 may display the high-quality image in switchover with the input image, or may display the high-quality image alongside the input image.

Furthermore, when the output unit 405 displays a high-quality image on the display unit 20, the output unit 405 may display, together with the high-quality image, a display indicating that the displayed image is a high-quality image generated by processing using a machine learning algorithm. In this case, the display allows the user to easily identify that the displayed high-quality image is not the image itself obtained by shooting, thereby reducing misdiagnosis and improving diagnostic efficiency. Note that the display indicating that the high-quality image is generated by processing using a machine learning algorithm may be in any form as long as it allows the user to distinguish between the input image and the high-quality image generated by the processing.

The output unit 405 may also cause the display unit 20 to display, in addition to a display indicating that the image is a high-quality image generated by processing using a machine learning algorithm, a display indicating what kind of training data the machine learning algorithm used to learn. Such a display may include an explanation of the types of input data and output data of the training data, or any display regarding the training data, such as the imaging site included in the input data and output data.

In the image quality improvement engine according to this embodiment, a superimposed image is used as output data for the teacher data, but the teacher data is not limited to this. As output data for the teacher data, a high-quality image obtained by performing at least one of a superimposition process, a group of processes described below, or a shooting method described below, which are means for obtaining a high-quality image, may be used.

For example, high-quality images obtained by performing maximum a posteriori probability estimation processing (MAP estimation processing) on a group of original images may be used as the output data of the training data. In the MAP estimation processing, a likelihood function is calculated from the probability density of each pixel value in multiple low-quality images, and the true signal value (pixel value) is estimated using the calculated likelihood function.

The high-quality image obtained by the MAP estimation process is a high-contrast image based on pixel values close to the true signal value. In addition, since the estimated signal value is calculated based on probability density, randomly occurring noise is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained by the MAP estimation process as training data, the image quality improvement engine can generate a high-quality image suitable for image diagnosis, with reduced noise and high contrast, from the input image. Note that the method for generating pairs of input data and output data for the training data may be the same as when an overlaid image is used as training data.

In addition, a high-quality image obtained by applying a smoothing filter process to the original image may be used as the output data of the training data. In this case, the image quality improvement engine can generate a high-quality image in which random noise has been reduced from the input image. Furthermore, an image obtained by applying a tone conversion process to the original image may be used as the output data of the training data. In this case, the image quality improvement engine can generate a high-quality image in which contrast has been enhanced from the input image. Note that the method of generating pairs of input data and output data of the training data may be the same as when an overlaid image is used as the training data.

The input data of the teacher data may be an image obtained from a photographing device having the same image quality tendency as the photographing device 10. The output data of the teacher data may be a high-quality image obtained by a high-cost process such as successive approximation, or may be a high-quality image obtained by photographing a subject corresponding to the input data with a photographing device with higher performance than the photographing device 10. Furthermore, the output data may be a high-quality image obtained by performing a rule-based noise reduction process. Here, the noise reduction process may include, for example, a process of replacing a single high-luminance pixel that appears in a low-luminance area and is clearly noise with the average value of nearby low-luminance pixels. For this reason, the image quality improvement engine may use as learning data an image taken with a photographing device with higher performance than the photographing device used to take the input image, or an image taken in a photographing process with more steps than the photographing process of the input image.

Although not described in the present embodiment, a high-quality image generated from multiple images and used as output data for training data can be generated from multiple images that have already been aligned. The alignment process may involve, for example, selecting one of the multiple images as a template, determining the similarity with other images while changing the position and angle of the template, determining the amount of misalignment with the template, and correcting each image based on the amount of misalignment. Any other existing alignment process may also be performed.

When aligning a three-dimensional image, the three-dimensional image may be decomposed into multiple two-dimensional images, and the two-dimensional images may be aligned and integrated together to align the three-dimensional image. The two-dimensional image may be decomposed into one-dimensional images, and the two-dimensional images may be aligned together to align the two-dimensional image. The alignment may be performed on the data used to generate the images, rather than on the images themselves.

In addition, in this embodiment, when the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement unit 404, the process proceeds to step S540, and the image quality improvement process by the image quality improvement unit 404 is started. In response to this, the output unit 405 may display the determination result by the image quality improvement possibility determination unit 403 on the display unit 20, and the image quality improvement unit 404 may start the image quality improvement process in response to an instruction from the examiner. At this time, the output unit 405 may display the input image and the imaging conditions such as the imaging part acquired for the input image, together with the determination result, on the display unit 20. In this case, the image quality improvement process is performed after the examiner determines whether the determination result is correct or not, so that the image quality improvement process based on an erroneous determination can be reduced.

In addition, instead of making a judgment by the image quality improvement possibility judgment unit 403, the output unit 405 may display the input image and the imaging conditions such as the imaging part acquired for the input image on the display unit 20, and the image quality improvement unit 404 may start the image quality improvement process in response to an instruction from the examiner.

Second Embodiment
Next, an image processing apparatus according to a second embodiment will be described with reference to Figures 4 and 7. In the first embodiment, the image quality improvement unit 404 includes one image quality improvement engine. In contrast, in this embodiment, the image quality improvement unit includes multiple image quality improvement engines that perform machine learning using different teacher data, and generates multiple high-image-quality images for an input image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with two or more image quality improvement engines that have undergone machine learning using different training data. Here, a method for creating a training data group according to this embodiment will be described. Specifically, first, a group of pairs of original images as input data and superimposed images as output data, in which various imaging sites are imaged, is prepared. Next, the pair groups are grouped for each imaging site to create a training data group. For example, a training data group is created in such a way that a first training data group is made up of a pair group acquired by imaging a first imaging site, and a second training data group is made up of a pair group acquired by imaging a second imaging site.

Then, separate image quality improvement engines are made to perform machine learning using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, and a second image quality improvement engine corresponding to a machine learning model trained with the second training data.

Since each of these image quality improvement engines uses different teacher data to train the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine provides a high degree of image quality improvement for the input image acquired by shooting the first shooting part, and a low degree of image quality improvement for the image acquired by shooting the second shooting part. Similarly, the second image quality improvement engine provides a high degree of image quality improvement for the input image acquired by shooting the second shooting part, and a low degree of image quality improvement for the image acquired by shooting the first shooting part.

Since each piece of training data is composed of pairs grouped by the imaging location, the image quality tendencies of the images constituting the pair groups are similar. Therefore, the image quality improvement engine can improve image quality more effectively than the image quality improvement engine according to the first embodiment, as long as the imaging location corresponds. Note that the imaging conditions for grouping pairs of training data are not limited to the imaging location, and may be the imaging angle of view, image resolution, or a combination of two or more of these.

Below, a series of image processing according to this embodiment will be described with reference to FIG. 7. FIG. 7 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S710 and S720 is similar to steps S510 and S520 according to the first embodiment, and therefore a description thereof will be omitted. Note that if the image quality of the input image is to be unconditionally improved, after the processing of step S720, the processing of step S730 may be omitted and the processing may proceed to step S740.

When the shooting conditions of the input image are acquired in step S720, the process proceeds to step S730. In step S730, the image quality improvement feasibility determination unit 403 uses the group of shooting conditions acquired in step S720 to determine whether any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

If the image quality improvement capability determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S760. On the other hand, if the image quality improvement capability determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S740. Note that, depending on the settings and implementation form of the image processing device 400, step S740 may be performed, as in the first embodiment, even if the image quality improvement engines have determined that some shooting conditions cannot be handled.

In step S740, the image quality improvement unit 404 selects an image quality improvement engine that performs image quality improvement processing from the image quality improvement engine group based on the shooting conditions of the input image acquired in step S720 and the teacher data information of the image quality improvement engine group. Specifically, for example, for an image shooting part in the shooting condition group acquired in step S720, an image quality improvement engine that has teacher data information related to the same image shooting part or surrounding image shooting parts and has a high degree of image quality improvement is selected. In the above example, when the image shooting part is the first image shooting part, the image quality improvement unit 404 selects the first image quality improvement engine.

In step S750, the image quality improvement unit 404 uses the image quality improvement engine selected in step S740 to generate a high-quality image by improving the image quality of the input image. After that, in step S760, if a high-quality image was generated in step S750, the output unit 405 outputs the high-quality image and displays it on the display unit 20. On the other hand, if it is determined in step S730 that image quality improvement processing is not possible, the input image is output and displayed on the display unit 20. Note that when displaying the high-quality image on the display unit 20, the output unit 405 may display that the high-quality image was generated using the image quality improvement engine selected by the image quality improvement unit 404.

As described above, the image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines that have been trained using different learning data. Here, each of the plurality of image quality improvement engines has been trained using different learning data for at least one of the imaging region, imaging angle of view, and image resolution. The image quality improvement unit 404 generates a high-quality image using an image quality improvement engine that corresponds to at least one of the imaging region, imaging angle of view, and image resolution of the input image.

With this configuration, the image processing device 400 according to this embodiment can generate high-quality images more effectively.

In this embodiment, the image quality improvement unit 404 selects an image quality improvement engine to be used for image quality improvement processing based on the shooting conditions of the input image, but the image quality improvement engine selection process is not limited to this. For example, the output unit 405 may display the shooting conditions of the acquired input image and the image quality improvement engine group on the user interface of the display unit 20, and the image quality improvement unit 404 may select an image quality improvement engine to be used for image quality improvement processing in response to an instruction from the examiner. The output unit 405 may display information on the teacher data used to learn each image quality improvement engine along with the image quality improvement engine group on the display unit 20. The display format of the information on the teacher data used to learn the image quality improvement engine may be arbitrary, and for example, the image quality improvement engine group may be displayed using a name related to the teacher data used for learning.

The output unit 405 may also display the image quality improvement engine selected by the image quality improvement unit 404 on the user interface of the display unit 20 and accept instructions from the examiner. In this case, the image quality improvement unit 404 may determine, in response to instructions from the examiner, whether or not to ultimately select the image quality improvement engine as the image quality improvement engine to be used for image quality improvement processing.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Third Embodiment
Next, an image processing device according to a third embodiment will be described with reference to Figs. 4 and 7. In the first and second embodiments, the imaging condition acquisition unit 402 acquires an imaging condition group from the data structure of an input image, etc. In contrast, in this embodiment, the imaging condition acquisition unit estimates an imaging part or imaging area of an input image based on the input image using an imaging location estimation engine.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the second embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the second embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first and second embodiments, the configuration shown in FIG. 4 is indicated using the same reference numerals and description will be omitted.

The imaging condition acquisition unit 402 according to this embodiment is equipped with an imaging location estimation engine that estimates the imaging part or imaging area depicted in the input image acquired by the acquisition unit 401. The imaging location estimation method provided by the imaging location estimation engine according to this embodiment performs estimation processing using a machine learning algorithm.

In this embodiment, training of a machine learning model related to a method for estimating an imaging location using a machine learning algorithm uses teacher data consisting of pairs of input data, which is an image, and output data, which is imaging site labels or imaging area labels corresponding to the input data. Here, the input data refers to an image having specific imaging conditions that is assumed to be the processing target (input image). The input data is preferably an image acquired from an imaging device having the same image quality tendency as the imaging device 10, and is preferably the same model with the same settings as the imaging device 10. The types of imaging site labels and imaging area labels, which are output data, may be imaging sites and imaging areas at least partially included in the input data. For example, in the case of OCT, the types of imaging site labels, which are output data, may be "macular region", "optic disc", "macular region and optic disc", and "others", etc.

The shooting location estimation engine according to this embodiment can output the location of the shooting part or shooting area depicted in the input image by learning using such teacher data. In addition, the shooting location estimation engine can also output the probability of the shooting part or shooting area for each shooting part label or shooting area label of the required level of detail. By using the shooting location estimation engine, the shooting condition acquisition unit 402 can estimate the shooting part or shooting area of the input image based on the input image and acquire it as the shooting condition for the input image. Note that when the shooting location estimation engine outputs the probability of the shooting part or shooting area for each shooting part label or shooting area label, the shooting condition acquisition unit 402 acquires the shooting part or shooting area with the highest probability as the shooting condition of the input image.

Next, as in the second embodiment, a series of image processing steps according to this embodiment will be described with reference to the flow diagram in FIG. 7. Note that the processing steps S710 and steps S730 to S760 according to this embodiment are similar to those steps in the second embodiment, and therefore will not be described. Note that if the image quality of the input image is to be unconditionally improved, after the processing step S720, the processing step S730 may be omitted and the processing may proceed to step S740.

When the input image is acquired in step S710, the process proceeds to step S720. In step S720, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image acquired in step S710.

Specifically, the imaging condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. Furthermore, if the imaging condition group does not include information on the imaging part or imaging area, the imaging condition acquisition unit 402 inputs the input image to an imaging location estimation engine and estimates which imaging part was imaged to acquire the input image. Specifically, the imaging condition acquisition unit 402 inputs the input image to an imaging location estimation engine, evaluates the probability output for each imaging part label group, and sets and acquires the imaging part with the highest probability as the imaging condition for the input image.

If no imaging conditions other than the imaging part and imaging area are stored in the input image, the imaging condition acquisition unit 402 can acquire a group of imaging information including a group of imaging conditions from the imaging device 10 or an image management system (not shown).

The subsequent processing is similar to the series of image processing in the second embodiment, so a detailed explanation will be omitted.

As described above, the imaging condition acquisition unit 402 according to this embodiment functions as an estimation unit that estimates at least one of the imaging part and imaging area of the input image. The imaging condition acquisition unit 402 includes an imaging location estimation engine that uses images labeled with the imaging part and imaging area as learning data, and estimates the imaging part and imaging area of the input image by inputting the input image to the imaging location estimation engine.

As a result, the image processing device 400 according to this embodiment can obtain the shooting conditions for the shooting part and shooting area of the input image based on the input image.

In this embodiment, the shooting condition acquisition unit 402 estimates the shooting part and shooting area of the input image using the shooting location estimation engine when the shooting condition group does not include information on the shooting part and shooting area. However, the situation in which the shooting part and shooting area are estimated using the shooting location estimation engine is not limited to this. The shooting condition acquisition unit 402 may estimate the shooting part and shooting area using the shooting location estimation engine even when information on the shooting part and shooting area included in the data structure of the input image is insufficient as information at the required level of detail.

In addition, the imaging condition acquisition unit 402 may estimate the imaging part and imaging area of the input image using the imaging location estimation engine, regardless of whether the data structure of the input image includes information on the imaging part and imaging area. In this case, the output unit 405 may display the estimation result output from the imaging location estimation engine and information on the imaging part and imaging area included in the data structure of the input image on the display unit 20, and the imaging condition acquisition unit 402 may determine these imaging conditions according to the examiner's instructions.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Fourth Embodiment
Next, an image processing device according to a fourth embodiment will be described with reference to Figures 4, 5, 8 and 9. In this embodiment, an image quality improvement unit enlarges or reduces an input image so that the input image has an image size that the image quality improvement engine can handle. The image quality improvement unit also reduces or enlarges an output image from the image quality improvement engine so that the image size of the output image becomes the image size of the input image, thereby generating a high-image quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to the image quality improvement engine according to the first embodiment. However, in this embodiment, as the teacher data used for learning the image quality improvement engine, a group of pairs of input data and output data, which are composed of a group of images in which the input data images and the output data images are enlarged or reduced to a certain image size, are used.

Now, referring to FIG. 8, the teacher data of the image quality improvement engine according to this embodiment will be described. As shown in FIG. 8, for example, consider the case where there is a low-quality image Im810 and a high-quality image Im820 that are smaller than a certain image size set for the teacher data. In this case, the low-quality image Im810 and the high-quality image Im820 are each enlarged so that they become the certain image size set for the teacher data. Then, the enlarged low-quality image Im811 and the enlarged high-quality image Im821 are paired, and the pair is used as one piece of teacher data.

As in the first embodiment, the input data for the training data uses an image with specific shooting conditions that are assumed to be the processing target (input image), and the specific shooting conditions are a predetermined shooting part, shooting method, and shooting angle of view. In other words, unlike the first embodiment, the specific shooting conditions in this embodiment do not include image size.

The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image and generate a high-image quality image. At this time, the image quality improvement unit 404 generates a deformed image by enlarging or reducing the input image so that it has a certain image size set for the teacher data, and inputs the deformed image to the image quality improvement engine. The image quality improvement unit 404 also reduces or enlarges the output image from the image quality improvement engine so that it has the image size of the input image, generating a high-image quality image. Therefore, the image quality improvement unit 404 according to this embodiment can use the image quality improvement engine to improve the image quality of even an input image of an image size that could not be handled in the first embodiment, and generate a high-image quality image.

Next, a series of image processing steps according to this embodiment will be described with reference to Figs. 5 and 9. Fig. 9 is a flow diagram of the image quality improvement processing according to this embodiment. Note that the processing steps S510, S520, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to shooting conditions other than image size, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S520, as in the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

If the image quality improvement feasibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in FIG. 9 is started. In the image quality improvement process according to this embodiment, first, in step S910, the image quality improvement unit 404 enlarges or reduces the input image to a certain image size set for the training data, and generates a deformed image.

Next, in step S920, the image quality improvement unit 404 inputs the generated deformed image to an image quality improvement engine to obtain a high-image-quality deformed image.

Then, in step S930, the image quality improvement unit 404 reduces or enlarges the high-quality transformed image to the image size of the input image to generate a high-quality image. Once the image quality improvement unit 404 generates a high-quality image in step S930, the image quality improvement process according to this embodiment ends, and the process proceeds to step S550. The process of step S550 is similar to step S550 in the first embodiment, and therefore a description thereof will be omitted.

As described above, the image quality improvement unit 404 according to this embodiment adjusts the image size of the input image to an image size that the image quality improvement engine can handle, and inputs the image to the image quality improvement engine. The image quality improvement unit 404 also generates a high-quality image by adjusting the output image from the image quality improvement engine to the original image size of the input image. As a result, the image processing device 400 according to this embodiment can use the image quality improvement engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images suitable for image diagnosis.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Fifth embodiment
Next, an image processing device according to a fifth embodiment will be described with reference to Figures 4, 5, 10 and 11. In this embodiment, an image quality improvement unit generates a high-quality image by image quality improvement processing using an image quality improvement engine with a certain resolution as a reference.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, in this embodiment, the training data used for training the image quality improvement engine is different from the training data in the first embodiment. Specifically, the image group constituting the pair group of input data and output data of the training data is enlarged or reduced to an image size such that the resolution of the image group becomes a certain resolution, and then padding is performed to obtain a sufficiently large and constant image size. Here, the resolution of the image group refers to, for example, the spatial resolution of the imaging device or the resolution for the imaging area.

Now, referring to FIG. 10, the teacher data of the image quality improvement engine according to this embodiment will be described. As shown in FIG. 10, for example, consider the case where there is a low-quality image Im1010 and a high-quality image Im1020, both of which have a resolution lower than the fixed resolution set for the teacher data. In this case, the low-quality image Im1010 and the high-quality image Im1020 are each enlarged so that they have the fixed resolution set for the teacher data. Furthermore, the enlarged low-quality image Im1010 and the high-quality image Im1020 are each padded so that they have the fixed image size set for the teacher data. Then, the low-quality image Im1011 and the high-quality image Im1021 that have been enlarged and padded are paired, and the pair is used as one piece of teacher data.

The fixed image size set for the training data is the maximum possible image size when the image expected to be processed (input image) is enlarged or reduced to a fixed resolution. If the fixed image size is not large enough, when the image input to the image quality improvement engine is enlarged, the image size may become too large for the machine learning model to handle.

In addition, the area to be padded is filled with a fixed pixel value, filled with nearby pixel values, or mirror padded according to the characteristics of the machine learning model so as to effectively improve image quality. As in the first embodiment, the input data uses images with specific shooting conditions that are expected to be processed, and the specific shooting conditions are a predetermined shooting part, shooting method, and shooting angle of view. In other words, unlike the first embodiment, the specific shooting conditions in this embodiment do not include image size.

The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image to generate a high-image quality image. At this time, the image quality improvement unit 404 generates a deformed image by enlarging or reducing the input image so that it has a certain resolution set for the teacher data. The image quality improvement unit 404 also pads the deformed image so that it has a certain image size set for the teacher data to generate a padded image, and inputs the padded image to the image quality improvement engine.

The image quality improvement unit 404 also trims the high-quality padded image output from the image quality improvement engine by the area where padding was performed, to generate a high-quality deformed image. The image quality improvement unit 404 then reduces or enlarges the generated high-quality deformed image to the image size of the input image, to generate a high-quality image.

For this reason, the image quality improvement unit 404 according to this embodiment can generate a high-quality image by improving the image quality using the image quality improvement engine, even for input images with image sizes that could not be handled in the first embodiment.

Next, a series of image processing steps according to this embodiment will be described with reference to Figs. 5 and 11. Fig. 11 is a flow diagram of the image quality improvement processing according to this embodiment. Note that the processing steps S510, S520, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to shooting conditions other than image size, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S520, as in the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

If the image quality improvement feasibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in FIG. 11 is started. In the image quality improvement process according to this embodiment, first, in step S1110, the image quality improvement unit 404 enlarges or reduces the input image to a certain resolution set for the training data, and generates a deformed image.

Next, in step S1120, the image quality improvement unit 404 performs padding on the generated transformed image so that the image size is the same as the image size set for the training data, to generate a padded image. At this time, the image quality improvement unit 404 fills the area to be padded with a fixed pixel value, fills with a neighboring pixel value, or performs mirror padding in accordance with the characteristics of the machine learning model so as to effectively improve the image quality of the area to be padded.

In step S1130, the image quality improvement unit 404 inputs the padding image to the image quality improvement engine to obtain a high-image-quality padding image.

Next, in step S1140, the image quality improvement unit 404 trims the high-quality padded image by the area that was padded in step S1120 to generate a high-quality transformed image.

Then, in step S1150, the image quality improvement unit 404 reduces or enlarges the high-quality transformed image to the image size of the input image to generate a high-quality image. Once the image quality improvement unit 404 generates a high-quality image in step S1130, the image quality improvement process according to this embodiment ends, and the process proceeds to step S550. The process of step S550 is similar to step S550 in the first embodiment, and therefore a description thereof will be omitted.

As described above, the image quality improvement unit 404 according to this embodiment adjusts the image size of the input image so that the resolution of the input image becomes a predetermined resolution. Furthermore, the image quality improvement unit 404 generates a padded image for the input image whose image size has been adjusted, so that the adjusted image size becomes an image size that can be handled by the image quality improvement engine, and inputs the padded image to the image quality improvement engine. The image quality improvement unit 404 then trims the output image from the image quality improvement engine by the amount of the padded area. The image quality improvement unit 404 then generates a high-quality image by adjusting the image size of the trimmed image to the original image size of the input image.

As a result, the image quality improvement unit 404 of this embodiment can use the image quality improvement engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images. Furthermore, by using an image quality improvement engine that has learned using training data based on resolution, it may be possible to improve the image quality of input images more efficiently than the image quality improvement engine of the fourth embodiment, which simply processes images of the same image size.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Sixth embodiment
Next, an image processing apparatus according to a sixth embodiment will be described with reference to Figures 4, 5, 12 and 13. In this embodiment, an image quality improving unit improves the image quality of an input image for each region of a certain image size, thereby generating a high-quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, in this embodiment, the training data used for training the image quality improvement engine is different from the training data in the first embodiment. Specifically, the training data is made up of pairs of input data, which is a low-quality image, and output data, which is a high-quality image, and the pairs are made up of rectangular area images of a certain image size in which the positional relationship between the low-quality image and the high-quality image corresponds.

Now, referring to FIG. 12, the teacher data of the image quality improvement engine according to this embodiment will be described. As shown in FIG. 12, consider a case where one of the pairs constituting the teacher data includes, for example, an original image Im1210, which is a low-image quality image, and an overlaid image Im1220, which is a high-image quality image. In this case, in the first embodiment, the input data of the teacher data is Im1210, and the output data is Im1220.

In contrast, in this embodiment, the rectangular area image R1211 of the original image Im1210 is used as input data, and the rectangular area image R1221, which is the same shooting area as the rectangular area image R1211 in the superimposed image Im1220, is used as output data. Then, the rectangular area image R1211, which is the input data, and the rectangular area image R1221, which is the output data, form a pair of teacher data (hereinafter, the first rectangular area image pair). Here, the rectangular area image R1211 and the rectangular area image R1221 are images of a certain image size. Note that the original image Im1210 and the superimposed image Im1220 may be aligned by any method. Also, the corresponding positional relationship between the rectangular area image R1211 and the rectangular area image R1221 may be specified by any method such as template matching. Note that, depending on the design of the image quality improvement engine, the image size and the number of dimensions of the input data and the output data may differ. For example, if the processing target is an OCT image, and the input data is part of a B-scan image (two-dimensional image), the output data may be part of an A-scan image (one-dimensional image).

The fixed image size for the rectangular area images R1211 and R1221 can be determined, for example, from the common divisor of the pixel counts of the corresponding dimensions for the image size group of the image assumed to be processed (input image). In this case, it is possible to prevent the positional relationship of the rectangular area images output by the image quality improvement engine from overlapping. Specifically, for example, consider a case where the image assumed to be processed is a two-dimensional image, and the first image size of the image size group is 500 pixels wide and 500 pixels high, and the second image size is 100 pixels wide and 100 pixels high. Here, the fixed image size for the rectangular area images R1211 and R1221 is selected from the common divisor of each side. In this case, for example, the fixed image size is selected from 100 pixels wide and 100 pixels high, 50 pixels wide and 50 pixels high, 25 pixels wide and 25 pixels high, etc.

When the image to be processed is three-dimensional, the number of pixels is determined for width, height, and depth. Note that multiple rectangular areas can be set for one pair of a low-quality image corresponding to the input data and a high-quality image corresponding to the output data. For this reason, for example, rectangular area image R1212 of original image Im1210 is set as input data, and rectangular area image R1222, which is the same shooting area as rectangular area image R1212 in superimposed image Im1220, is set as output data. Then, a pair of teacher data is formed by rectangular area image R1212, which is the input data, and rectangular area image R1222, which is the output data. This makes it possible to create a rectangular area image pair other than the first rectangular area image pair.

By creating a large number of pairs of rectangular area images while changing the images of rectangular areas to images with different coordinates, the group of pairs that make up the training data can be enriched, and efficient image quality improvement can be expected by an image quality improvement engine that has trained using the training pairs. However, pairs that do not contribute to improving the image quality of the machine learning model can be prevented from being added to the training data. For example, if a rectangular area image created from a high-quality image that is output data that makes up a pair has image quality that is not suitable for diagnosis, the image output by an image quality improvement engine that has trained using such training data may also have image quality that is not suitable for image diagnosis. Therefore, pairs that include such high-quality images can be removed from the training data.

In addition, for example, if the average luminance or luminance distribution of a pair of rectangular area images created from a low-quality image and a high-quality image differs significantly, such pairs can be removed from the training data. If learning is performed using such training data, the image quality improvement engine may output an image that is unsuitable for image diagnosis, with a luminance distribution that differs significantly from the input image.

Furthermore, consider a case where the structure and position of the object depicted in a pair of rectangular area images, one created from a low-image quality image and the other created from a high-image quality image, differ significantly. In this case, an image quality improvement engine that has learned using such training data may output an image that is unsuitable for image diagnosis, in which the object is depicted in a structure and position that differs significantly from that of the input image. For this reason, such pairs can be removed from the training data.

As with the first embodiment, images with specific shooting conditions that are expected to be processed are used as input data for the training data, but the specific shooting conditions are a predetermined shooting area, shooting method, and shooting angle of view. In other words, unlike the first embodiment, the specific shooting conditions in this embodiment do not include image size.

The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image and generate a high-quality image. In this case, the image quality improvement unit 404 divides the input image into a group of continuous rectangular area images with no gaps and a fixed image size set for the teacher data. The image quality improvement unit 404 improves the image quality of each of the divided rectangular area images using the image quality improvement engine, generating a group of high-quality rectangular area images. The image quality improvement unit 404 then arranges and combines the generated high-quality rectangular area images according to the positional relationship of the input image, generating a high-quality image.

In this way, the image quality improvement unit 404 of this embodiment improves the image quality of the input image in units of rectangular regions, and by combining the improved image quality images, it is possible to improve the image quality of images of image sizes that could not be handled in the first embodiment, thereby generating a high-image quality image.

Next, a series of image processing steps according to this embodiment will be described with reference to Figs. 5, 13, and 14. Fig. 13 is a flow diagram of the image quality improvement processing according to this embodiment. Note that the processing steps S510, S520, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to shooting conditions other than image size, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S520, as in the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

If the image quality improvement feasibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in FIG. 13 is started. In the image quality improvement process according to this embodiment, first, in step S1310, as shown in FIG. 14, the input image is divided into a group of rectangular area images of a certain image size set for the teacher data, which are continuous without gaps. Here, FIG. 14 shows an example in which the input image Im1410 is divided into a group of rectangular area images R1411 to R1426 of a certain image size. Note that, as described above, depending on the design of the image quality improvement engine, the image sizes and number of dimensions of the input image and output image of the image quality improvement engine may differ. In this case, the division positions of the input image can be adjusted by overlapping or separating so that there are no defects in the combined high-image quality image generated in step S1320.

Next, in step S1320, the image quality improvement unit 404 improves the image quality of each of the rectangular area images R1411 to R1426 using an image quality improvement engine, generating a group of high-image-quality rectangular area images.

Then, in step S1330, the image quality improving unit 404 places each of the generated high-image-quality rectangular area images in the same positional relationship as each of the rectangular area images R1411 to R1426 divided from the input image, and combines them to generate a high-image-quality image.

As described above, the image quality improvement unit 404 according to this embodiment divides the input image into a plurality of rectangular area images (third images) R1411 to R1426 of a predetermined image size. The image quality improvement unit 404 then inputs the divided rectangular area images R1411 to R1426 to an image quality improvement engine to generate a plurality of fourth images, and integrates the plurality of fourth images to generate a high-quality image. Note that if the rectangular area groups overlap in positional relationship during integration, the pixel values of the rectangular area groups can be integrated or overwritten.

As a result, the image quality improvement unit 404 of this embodiment can use the image quality improvement engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images. Furthermore, if the teacher data is created from multiple images obtained by dividing low-quality images and high-quality images into a predetermined image size, a large amount of teacher data can be created from a small number of images. Therefore, in this case, the number of low-quality images and high-quality images used to create the teacher data can be reduced.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Seventh embodiment
Next, an image processing apparatus according to a seventh embodiment will be described with reference to Figures 15 to 17. In this embodiment, an image quality evaluation unit selects the image with the highest image quality from among multiple high-image-quality images output from multiple image quality improvement engines in response to an instruction from an examiner.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

Figure 15 shows a schematic configuration of an image processing device 1500 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image quality evaluation unit 1506 is provided in the image processing device 1500 according to this embodiment. Note that the image processing device 1500 may be composed of multiple devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the configuration shown in Figure 4 is indicated using the same reference numerals and description will be omitted.

Furthermore, the image processing device 1500 may be connected to the imaging device 10, the display unit 20, and other devices (not shown) via any circuit or network, similar to the image processing device 400 according to the first embodiment. Furthermore, these devices may be connected to any other device via a circuit or network, or may be configured integrally with any other device. Note that, although these devices are separate devices in this embodiment, some or all of these devices may be configured integrally.

The image quality improvement unit 404 according to this embodiment is equipped with two or more image quality improvement engines that have undergone machine learning using different teacher data. Here, a method for creating a teacher data group according to this embodiment will be described. Specifically, first, a group of pairs of input data, which is a low-quality image, and output data, which is a high-quality image, captured under various shooting conditions is prepared. Next, a teacher data group is created by grouping the pair groups according to any combination of shooting conditions. For example, a teacher data group is created in such a way that a first teacher data group is composed of a pair group acquired by a first combination of shooting conditions, and a second teacher data group is composed of a pair group acquired by a second combination of shooting conditions.

Then, separate image quality improvement engines are made to perform machine learning using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, and a second image quality improvement engine corresponding to a machine learning model trained with the second training data.

Since each of these image quality improvement engines uses different teacher data to train the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine provides a high degree of image quality improvement for input images captured and acquired under the first combination of shooting conditions, and a low degree of image quality improvement for images captured and acquired under the second combination of shooting conditions. Similarly, the second image quality improvement engine provides a high degree of image quality improvement for input images captured and acquired under the second shooting conditions, and a low degree of image quality improvement for images captured and acquired under the first shooting conditions.

Since each piece of training data is composed of pairs grouped according to a combination of shooting conditions, the image quality tendencies of the images constituting the pair groups are similar. Therefore, the image quality improvement engine can improve image quality more effectively than the image quality improvement engine according to the first embodiment, so long as the combination of shooting conditions corresponds. Note that the combination of shooting conditions for grouping pairs of training data may be any combination, and may be, for example, a combination of two or more of the shooting part, shooting angle of view, and image resolution. Furthermore, training data may be grouped based on a single shooting condition, as in the second embodiment.

The image quality assessment unit 1506 selects the high-quality image with the highest image quality from the multiple high-quality images generated by the image quality improvement unit 404 using multiple image quality improvement engines, in accordance with the examiner's instructions.

The output unit 405 can display the high-quality image selected by the image quality evaluation unit 1506 on the display unit 20 or output it to another device. The output unit 405 can display multiple high-quality images generated by the image quality improvement unit 404 on the display unit 20, and the image quality evaluation unit 1506 can select the high-quality image with the highest image quality in response to instructions from the examiner who checks the display unit 20.

This allows the image processing device 1500 to output the highest quality image in accordance with the examiner's instructions from among multiple high quality images generated using multiple image quality improvement engines.

Below, a series of image processing according to this embodiment will be described with reference to Figs. 16 and 17. Fig. 16 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S1610 and S1620 according to this embodiment is similar to the processing of steps S510 and S520 in the first embodiment, and therefore a description thereof will be omitted. Note that, when improving the image quality of an input image unconditionally with respect to the shooting conditions, after the processing of step S1620, the processing of step S1630 may be omitted and the processing may proceed to step S1640.

In step S1620, as in the first embodiment, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, and the process proceeds to step S1630. In step S1630, as in the second embodiment, the image quality improvement feasibility determination unit 403 uses the acquired group of shooting conditions to determine whether or not any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

If the image quality improvement capability determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S1660. On the other hand, if the image quality improvement capability determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S1640. Note that, depending on the settings and implementation form of the image processing device 1500, step S1640 may be performed, as in the first embodiment, even if the image quality improvement engines have determined that some shooting conditions cannot be handled.

In step S1640, the image quality improvement unit 404 inputs the input image acquired in step S1610 to each of the image quality improvement engines, and generates a group of high-image-quality images.

In step S1650, the image quality assessment unit 1506 selects the image with the highest image quality from the group of high-image-quality images generated in step S1640. Specifically, first, the output unit 405 displays the group of high-image-quality images generated in step S1640 on the user interface of the display unit 20.

Here, FIG. 17 shows an example of the interface. The interface displays input image Im1710, and high-quality images Im1720, Im1730, Im1740, and Im1750 output by each of the image quality improvement engines. The examiner operates any input device (not shown) to specify the image with the highest image quality, that is, the image most suitable for image diagnosis, from the image group (high-quality images Im1720 to Im1750). Note that since an input image that has not been improved in image quality by the image quality improvement engine may be more suitable for image diagnosis, the input image may be added to the image group that is the subject of the examiner's specification.

The image quality assessment unit 1506 then selects the high-quality image indicated by the examiner as the image with the highest quality.

In step S1660, the output unit 405 displays the image selected in step S1650 on the display unit 20 or outputs it to another device. However, if it is determined in step S1630 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. Note that if an input image is specified by the examiner or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

As described above, the image quality improvement unit 404 according to this embodiment generates multiple high-quality images from an input image using multiple image quality improvement engines, and the output unit 405 of the image processing device 1500 outputs at least one of the multiple high-quality images in response to the examiner's instructions. In particular, in this embodiment, the output unit 405 outputs the image with the highest image quality in response to the examiner's instructions. This allows the image processing device 1500 to output the high-quality image with the highest image quality in response to the examiner's instructions from among the multiple high-quality images generated using the multiple image quality improvement engines.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1500, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Eighth embodiment
Next, an image processing device according to an eighth embodiment will be described with reference to Figures 15 and 16. In this embodiment, an image quality evaluation unit uses an image quality evaluation engine to select an image with the highest image quality from among a plurality of high-image-quality images output from a plurality of image quality improvement engines.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 1500 according to the seventh embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the seventh embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the seventh embodiment, the configuration shown in FIG. 15 is indicated using the same reference numerals and description will be omitted.

The image quality evaluation unit 1506 according to this embodiment is equipped with an image quality evaluation engine that evaluates the image quality of an input image. The image quality evaluation engine outputs an image quality evaluation index for the input image. The image quality evaluation processing method for calculating the image quality evaluation index in the image quality evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm. The input data of the pairs constituting the teacher data for training the machine learning model are image groups consisting of a group of low-image-quality images and a group of high-image-quality images captured in advance under various shooting conditions. In addition, the output data of the pairs constituting the teacher data for training the machine learning model are, for example, a group of image quality evaluation indices set for each of the image groups of the input data by an examiner performing image diagnosis.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 16. Note that the processing steps S1610, S1620, S1630, and S1660 according to this embodiment are similar to those steps in the seventh embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S1620, the processing step S1630 may be omitted and the processing may proceed to step S1640.

In step S1630, as in the seventh embodiment, if the image quality improvement capability determination unit 403 determines that one of the image quality improvement engines can handle the input image, the process proceeds to step S1640. Note that, as in the first embodiment, depending on the settings and implementation form of the image processing device 1500, step S1640 may be performed even if the image quality improvement engine determines that some shooting conditions cannot be handled by the image quality improvement engine.

In step S1640, the image quality improvement unit 404 inputs the input image acquired in step S1610 to each of the image quality improvement engines, and generates a group of high-image-quality images.

In step S1650, the image quality evaluation unit 1506 selects the image with the highest image quality from the group of high-quality images generated in step S1640. Specifically, the image quality evaluation unit 1506 first inputs the group of high-quality images generated in step S1640 to the image quality evaluation engine. The image quality evaluation engine calculates an image quality evaluation index for each input high-quality image based on learning. The image quality evaluation unit 1506 selects the high-quality image with the highest image quality evaluation index calculated from the calculated image quality evaluation indexes. Note that since an input image that has not been improved in quality by the image improvement engine may be more suitable for image diagnosis, the image quality evaluation unit 1506 may also input the input image to the image quality evaluation engine and add the image quality evaluation index for the input image to the selection. Step S1660 is similar to step S1660 in the seventh embodiment, and therefore a description thereof will be omitted.

As described above, the image processing device 1500 according to this embodiment further includes an image quality evaluation unit 1506 that evaluates the image quality of the high-quality image. The image quality improvement unit 404 generates multiple high-quality images from an input image using multiple image quality improvement engines, and the output unit 405 of the image processing device 1500 outputs at least one of the multiple high-quality images according to the evaluation result by the image quality evaluation unit 1506. In particular, the image quality evaluation unit 1506 according to this embodiment includes an image quality evaluation engine that uses an evaluation value by a predetermined evaluation method as learning data. The image quality evaluation unit 1506 selects the high-quality image that has the highest evaluation result using the image quality evaluation engine by the image quality evaluation unit 1506 from the multiple high-quality images. The output unit 405 outputs the high-quality image with the highest evaluation value selected by the image quality evaluation unit 1506.

As a result, the image processing device 1500 according to this embodiment can easily output the high-quality image most suitable for image diagnosis from among multiple high-quality images based on the output of the image quality evaluation engine.

In this embodiment, the image quality evaluation unit 1506 selects a high-quality image with the highest image quality evaluation index among the image quality evaluation indices output by the image quality evaluation engine, and the output unit 405 displays the selected high-quality image on the display unit 20. However, the configuration of the image quality evaluation unit 1506 is not limited to this. For example, the image quality evaluation unit 1506 may select high-quality images with the top several image quality evaluation indices among the image quality evaluation indices output by the image quality evaluation engine, and the output unit 405 may display the selected high-quality image on the display unit 20. In addition, the output unit 405 may display the image quality evaluation index output by the image quality evaluation engine together with the corresponding high-quality image on the display unit 20, and the image quality evaluation unit 1506 may select the image with the highest image quality in accordance with the examiner's instructions.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1500, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Ninth embodiment
Next, an image processing device according to a ninth embodiment will be described with reference to Figures 18 and 19. In this embodiment, an authenticity evaluation unit uses an authenticity evaluation engine to evaluate whether or not the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

Figure 18 shows a schematic configuration of an image processing device 1800 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image processing device 1800 according to this embodiment is provided with an authenticity evaluation unit 1807. Note that the image processing device 1800 may be composed of multiple devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the configuration shown in Figure 4 is indicated using the same reference numerals and description will be omitted.

Furthermore, the image processing device 1800 may be connected to the imaging device 10, the display unit 20, and other devices (not shown) via any circuit or network, similar to the image processing device 400 according to the first embodiment. Furthermore, these devices may be connected to any other device via a circuit or network, or may be configured integrally with any other device. Note that, although these devices are separate devices in this embodiment, some or all of these devices may be configured integrally.

The authenticity evaluation unit 1807 is equipped with an authenticity evaluation engine. The authenticity evaluation unit 1807 uses the authenticity evaluation engine to evaluate whether the high-quality image generated by the image quality improvement engine has been sufficiently improved in quality. The authenticity evaluation processing method in the authenticity evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm.

The training data for training the machine learning model includes pairs of high-quality images taken in advance under various shooting conditions and labels indicating that the images were taken and acquired with the target imaging device (hereinafter, genuine labels). The training data also includes pairs of high-quality images generated by inputting low-quality images into an image-enhancing engine with poor image-enhancing accuracy and labels indicating that the images were not taken and acquired with the target imaging device (hereinafter, counterfeit labels).

An authenticity evaluation engine that has been trained using such teacher data cannot reliably evaluate whether an input image is an image that was captured and acquired by a camera device, but it can evaluate whether the image resembles an image that was captured and acquired by a camera device. Using this characteristic, the authenticity evaluation unit 1807 can input a high-quality image generated by the image quality improvement unit 404 to the authenticity evaluation engine, and evaluate whether the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality.

When the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved, the output unit 405 displays the high-quality image on the display unit 20. On the other hand, when the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved, the output unit 405 displays the input image on the display unit 20. Note that, when displaying the input image, the output unit 405 can display on the display unit 20 that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved or that the image being displayed is the input image.

Below, a series of image processing according to this embodiment will be described with reference to FIG. 19. FIG. 19 is a flow diagram of a series of image processing according to this embodiment. Note that the processing from step S1910 to step S1940 according to this embodiment is similar to the processing from step S510 to step S540 in the first embodiment, and therefore a description thereof will be omitted. Note that, if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S1920, the processing of step S1930 may be omitted and the processing may proceed to step S1940.

In step S1940, when the image quality improvement unit 404 generates a group of high-quality images, the process proceeds to step S1950. In step S1950, the authenticity evaluation unit 1807 inputs the high-quality images generated in step S1940 to an authenticity evaluation engine, and performs an authenticity evaluation based on the output of the authenticity evaluation engine. Specifically, if the authenticity evaluation engine outputs a genuine label (genuine), the authenticity evaluation unit 1807 evaluates that the generated high-quality images have been sufficiently improved in image quality. On the other hand, if the authenticity evaluation engine outputs a counterfeit label (fake), the authenticity evaluation unit 1807 evaluates that the generated high-quality images have not been sufficiently improved in image quality.

In step S1960, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality, the output unit 405 causes the display unit 20 to display the high-quality image. On the other hand, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved in quality, the output unit 405 causes the display unit 20 to display the input image.

As described above, the image processing device 1800 according to this embodiment further includes an authenticity evaluation unit 1807 that evaluates the image quality of a high-quality image, and the authenticity evaluation unit 1807 includes an authenticity evaluation engine that evaluates the authenticity of an image. The authenticity evaluation engine includes a machine learning engine that uses as learning data images generated by an image quality improvement engine that has lower (worse) image quality improvement processing accuracy than the image quality improvement engine of the image quality improvement unit 404. The output unit 405 of the image processing device 1800 outputs a high-quality image when the output from the authenticity evaluation engine of the authenticity evaluation unit is true.

As a result, with the image processing device 1800 according to this embodiment, the examiner can efficiently check high-quality images that have been sufficiently improved.

In addition, the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine may be trained in coordination to improve the efficiency and accuracy of both engines.

In this embodiment, the image quality improvement unit 404 generates one high-quality image, and the authenticity evaluation unit 1807 evaluates the generated high-quality image, but the evaluation by the authenticity evaluation unit 1807 is not limited to this. For example, as in the second embodiment, when the image quality improvement unit 404 generates multiple high-quality images using multiple image quality improvement engines, the authenticity evaluation unit 1807 may be configured to evaluate at least one of the multiple high-quality images generated. In this case, for example, the authenticity evaluation unit 1807 may evaluate all of the multiple high-quality images generated, or may evaluate only the image of the multiple high-quality images specified by the examiner.

Furthermore, the output unit 405 may cause the display unit 20 to display the result of the judgment made by the authenticity evaluation unit 1807 as to whether or not the high-quality image has been sufficiently improved in quality, and output the high-quality image according to the examiner's instructions.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1800, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Tenth embodiment
Next, an image processing device according to a tenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improvement unit divides a three-dimensional input image into a plurality of two-dimensional images, inputs the images to an image quality improvement engine, and combines output images from the image quality improvement engine to generate a three-dimensional high-quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires a three-dimensional image composed of a group of structurally continuous two-dimensional images. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a group of OCT B-scan images (tomographic images). Also, for example, it is a three-dimensional CT volume image composed of a group of axial tomographic images.

The image quality improvement unit 404 is equipped with an image quality improvement engine, as in the first embodiment. Note that the pair group of input data and output data, which is the teacher data for the image quality improvement engine, is composed of a group of two-dimensional images. The image quality improvement unit 404 divides the acquired three-dimensional image into multiple two-dimensional images, and inputs each two-dimensional image to the image quality improvement engine. This allows the image quality improvement unit 404 to generate multiple two-dimensional high-quality images.

The output unit 405 combines the multiple two-dimensional high-quality images generated by the image quality improvement unit 404 for each two-dimensional image of the three-dimensional image, and outputs a three-dimensional high-quality image.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S510 to S530 and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. However, in step S510, the acquisition unit 401 acquires a three-dimensional image. Note that, if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. The image quality improvement possibility determination unit 403 may make the determination based on the shooting conditions of the three-dimensional image, or may make the determination based on the shooting conditions for the multiple two-dimensional images that make up the three-dimensional image. In step S540, the image quality improvement unit 404 divides the acquired three-dimensional image into multiple two-dimensional images. The image quality improvement unit 404 inputs each of the multiple divided two-dimensional images to the image quality improvement engine to generate multiple two-dimensional high-quality images. The image quality improvement unit 404 combines the multiple generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

In step S550, the output unit 405 displays the generated high-quality three-dimensional image on the display unit 20. Note that the display format of the high-quality three-dimensional image may be arbitrary.

As described above, the image quality improvement unit 404 according to this embodiment divides a three-dimensional input image into multiple two-dimensional images and inputs them to the image quality improvement engine. The image quality improvement unit 404 combines the multiple two-dimensional high-quality images output from the image quality improvement engine to generate a three-dimensional high-quality image.

As a result, the image quality improvement unit 404 according to this embodiment can improve the image quality of a three-dimensional image using an image quality improvement engine that has been trained using teacher data of two-dimensional images.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Eleventh embodiment
Next, an image processing device according to an eleventh embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improvement unit divides a three-dimensional input image into a plurality of two-dimensional images, improves the image quality of the plurality of two-dimensional images in parallel by a plurality of image quality improvement engines, and generates a high-quality three-dimensional image by combining output images from the image quality improvement engines.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the tenth embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the tenth embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first and tenth embodiments, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is provided with a plurality of image quality improvement engines similar to those in the tenth embodiment. The plurality of image quality improvement engines provided in the image quality improvement unit 404 may be implemented in two or more device groups via a circuit or network so as to enable distributed processing, or may be implemented in a single device.

As in the tenth embodiment, the image quality improvement unit 404 divides the acquired three-dimensional image into multiple two-dimensional images. The image quality improvement unit 404 uses multiple image quality improvement engines to improve the image quality of the multiple two-dimensional images in a shared manner (in parallel), generating multiple two-dimensional high-quality images. The image quality improvement unit 404 combines the multiple two-dimensional high-quality images output from the multiple image quality improvement engines based on the three-dimensional image to be processed, generating a three-dimensional high-quality image.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S510 to S530 and S550 according to this embodiment are similar to those steps in the tenth embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may make this determination based on the shooting conditions of the three-dimensional image, or may make this determination based on the shooting conditions for the multiple two-dimensional images that make up the three-dimensional image.

In step S540, the image quality improvement unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality improvement unit 404 inputs each of the divided two-dimensional images into a plurality of image quality improvement engines, and performs image quality improvement processing in parallel to generate a plurality of two-dimensional high-quality images. The image quality improvement unit 404 combines the generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

In step S550, the output unit 405 displays the generated high-quality three-dimensional image on the display unit 20. Note that the display format of the high-quality three-dimensional image may be arbitrary.

As described above, the image quality improvement unit 404 according to this embodiment includes multiple image quality improvement engines. The image quality improvement unit 404 divides a three-dimensional input image into multiple two-dimensional images, and generates multiple two-dimensional high-quality images using the multiple image quality improvement engines in parallel. The image quality improvement unit 404 generates a three-dimensional high-quality image by integrating the multiple two-dimensional high-quality images.

As a result, the image quality improvement unit 404 according to this embodiment can improve the image quality of a three-dimensional image using an image quality improvement engine that has been trained using teacher data of two-dimensional images. Furthermore, compared to the tenth embodiment, it is possible to improve the image quality of a three-dimensional image more efficiently.

The training data for the multiple image quality improvement engines may be different training data depending on the processing target to be processed by each image quality improvement engine. For example, the first image quality improvement engine may learn from training data for the first imaging area, and the second image quality improvement engine may learn from training data for the second imaging area. In this case, each image quality improvement engine can improve the image quality of the two-dimensional image with greater accuracy.

In addition, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. In addition, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. In other words, a high-quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing may be used.

Twelfth embodiment
Next, an image processing apparatus according to a twelfth embodiment will be described with reference to Figs. 5 and 20. In this embodiment, an acquisition unit 401 acquires an input image from an image management system 2000, not from an image capture device.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as that of the image processing device 400 according to the first embodiment, the same reference symbols are used for the configuration shown in FIG. 4 and the description will be omitted.

Figure 20 shows a schematic configuration of an image processing device 400 according to this embodiment. The image processing device 400 according to this embodiment is connected to an image management system 2000 and a display unit 20 via any circuit or network. The image management system 2000 is a device and system that receives and stores images captured by any imaging device or processed images. The image management system 2000 can also transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. The image management system can include, for example, a picture archiving and communication system (PACS).

The acquisition unit 401 according to this embodiment can acquire an input image from the image management system 2000 connected to the image processing device 400. In addition, the output unit 405 can output a high-quality image generated by the image quality improvement unit 404 to the image management system 2000.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S520 to S540 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S510, the acquisition unit 401 acquires an image stored in the image management system 2000 as an input image from the image management system 2000 connected via a circuit or network. The acquisition unit 401 may acquire the input image in response to a request from the image management system 2000. Such a request may be issued, for example, when the image management system 2000 saves an image, before transmitting the saved image to another device, or when displaying the saved image on the display unit 20. The request may also be issued, for example, when a user operates the image management system 2000 to request high image quality processing, or when a high image quality image is used for the image analysis function provided by the image management system 2000.

The processing of steps S520 to S540 is the same as that in the first embodiment. When the image quality improvement unit 404 generates a high-quality image in step S540, the processing proceeds to step S550. In step S550, if a high-quality image was generated in step S540, the output unit 405 outputs the high-quality image to the image management system 2000 as an output image. If a high-quality image was not generated in step S540, the input image is output to the image management system 2000 as an output image. Depending on the settings and implementation of the image processing device 400, the output unit 405 may process the output image so that it can be used by the image management system 2000, or convert the data format of the output image.

As described above, the acquisition unit 401 according to this embodiment acquires an input image from the image management system 2000. Therefore, the image processing device 400 according to this embodiment can output a high-quality image suitable for image diagnosis based on the image stored in the image management system 2000, without increasing the invasiveness or labor of the photographer or the subject. The output high-quality image can be stored in the image management system 2000, or displayed on a user interface provided by the image management system 2000. The output high-quality image can be used for the image analysis function provided by the image management system 2000, or can be transmitted via the image management system 2000 to other devices connected to the image management system 2000.

The image processing device 400, image management system 2000, and display unit 20 may be connected to other devices (not shown) via a circuit or network. In this embodiment, these devices are separate devices, but some or all of these devices may be configured as an integrated unit.

Furthermore, as in the first embodiment, the output unit 405 may output the generated high-quality image to the image management system 2000 or another device connected to the image processing device 400.

Thirteenth embodiment
Next, an image processing apparatus according to a thirteenth embodiment will be described with reference to Figures 4, 5, 21A and 21B. In this embodiment, an image quality improving unit receives a plurality of images as input images and generates one high-quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 in this embodiment acquires multiple images from the imaging device 10 or another device as input data to be processed.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. The teacher data may also be similar to that of the first embodiment. The image quality improvement unit 404 inputs each of the multiple images acquired by the acquisition unit 401 to the image quality improvement engine, and performs a process of overlaying the multiple high-quality images output to generate a final high-quality image. Note that the image quality improvement unit 404 may align the multiple high-quality images by any method before performing a process of overlaying the multiple high-quality images.

The output unit 405 displays the final high-quality image generated by the image quality improvement unit 404 on the display unit 20. The output unit 405 may display a plurality of input images on the display unit 20 together with the final high-quality image. The output unit 405 may display the generated plurality of high-quality images on the display unit 20 together with the final high-quality image and the input images.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5 and FIG. 21A. FIG. 21A is a flow diagram of the image quality improvement processing according to this embodiment. Note that the processing steps S510 to S530 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described.

However, in step S510, the acquisition unit 401 acquires multiple images, and in steps S520 and S530, the shooting conditions are acquired for each of the multiple images, and it is determined whether or not the image quality can be improved by the image quality improvement engine. If the image quality of the input image is to be improved unconditionally in terms of the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540. Furthermore, if it is determined that some of the multiple images cannot be handled by the image quality improvement engine, the image may be excluded from subsequent processing.

If the image quality improvement possibility determination unit 403 determines in step S530 that the multiple input images can be handled by the image quality improvement engine, the process proceeds to step S540. When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in FIG. 21A is started. In the image quality improvement process according to this embodiment, first, in step S2110, the image quality improvement unit 404 inputs each of the multiple input images to the image quality improvement engine and generates a group of high-image-quality images.

Next, in step S2120, the image quality improvement unit 404 performs an overlay process on the generated high-quality images to generate a single final high-quality image. The overlay process may be performed by averaging such as additive averaging or any other existing process. When overlaying, the image quality improvement unit 404 may align the multiple high-quality images by any method before overlaying them. Once the image quality improvement unit 404 has generated the final high-quality image, the process proceeds to step S550.

In step S550, the output unit 405 causes the display unit 20 to display the generated final high-quality image.

As described above, the image quality improvement unit 404 according to this embodiment generates one final high-quality image from multiple input images. Since the image quality improvement by the image quality improvement engine is based on the input image, for example, if a lesion or the like is not displayed properly in a certain input image, the pixel value will be low in the high-quality image obtained by improving the image quality of the input image. On the other hand, there are cases where the lesion or the like is displayed properly in another input image capturing the same location, and the pixel value is high in the high-quality image obtained by improving the image quality of the other input image. Therefore, by superimposing these high-quality images, it becomes possible to properly display the areas with low or high pixel values, and a high-quality image with high contrast can be generated. In addition, by making the number of input images smaller than the number required for conventional superimposition, the cost of longer imaging time, etc., as in the conventional case, can be reduced.

This effect is particularly noticeable when using input images that use motion contrast data such as OCTA images. Here, motion contrast data refers to data used in OCTA and the like, which is obtained by repeatedly capturing images of the same location of the subject and detecting changes in the subject over time between captures. In OCTA, an OCTA image is generated by generating a frontal image using data from the calculated motion contrast data in a desired range in the depth direction of the subject.

Motion contrast data is the detection of temporal changes in the subject during time intervals when the same location on the subject is repeatedly photographed. For example, at one time interval, only slight movement of the subject may be detected. In contrast, when photographs are taken at a different time interval, the movement of the subject may be detected as greater. Therefore, by overlaying high-quality images of the motion contrast images from each case, it is possible to interpolate motion contrast that did not occur or was only slightly detected at a particular time. Therefore, this type of processing makes it possible to generate a motion contrast image in which the contrast of more of the subject's movements is enhanced, allowing the examiner to grasp the condition of the subject more accurately.

Therefore, when an image that depicts an area that changes over time, such as an OCTA image, is used as the input image, a specific area of the subject can be imaged in more detail by overlaying high-quality images taken at different times.

In this embodiment, a high quality image is generated from each of a plurality of input images, and the high quality images are then superimposed to generate a single final high quality image; however, the method of generating a single high quality image from a plurality of input images is not limited to this. For example, in another example of the image quality improvement process of this embodiment shown in FIG. 21B, when the image quality improvement process is started in step S540, in step S2130, the image quality improvement unit 404 superimposes the input images to generate a single superimposed input image.

Then, in step S2140, the image quality improvement unit 404 inputs one superimposed input image to an image quality improvement engine to generate one high-image quality image. Even with this type of image quality improvement process, as with the image quality improvement process described above, it becomes possible to appropriately display areas with low or high pixel values in multiple input images, and to generate a high-image quality image with high contrast. This process can also have a remarkable effect when a motion contrast image such as the OCTA image described above is used as the input image.

When performing this high image quality processing, the image quality engine uses superimposed images of the same number of input images as the multiple input images to be processed as input data for the teacher data of the image quality engine. This allows the image quality engine to perform appropriate image quality processing.

In addition, in the image quality improvement process according to this embodiment and the other image quality improvement processes described above, the process of combining the high-image-quality image group or the input image group is not limited to overlay. For example, a single image may be generated by applying a MAP estimation process to these image groups. Also, a single image may be generated by synthesizing the high-image-quality image group or the input image group.

When generating a single image by synthesizing a group of high-quality images or a group of input images, for example, an image with a wide range of gradations in the high-luminance region and an image with a wide range of gradations in the low-luminance region may be used as the input image. In this case, for example, an image with a wide range of gradations in the high-luminance region and an image with a wide range of gradations in the low-luminance region are synthesized to generate a high-quality image. This makes it possible to generate an image that can express a wider range of brightness (dynamic range). In this case, the input data of the training data for the high-quality engine can be an image with a wide range of gradations in the high-luminance region or a low-quality image with a wide range of gradations in the low-luminance region, which are to be processed. Also, the output data of the training data for the high-quality engine can be a high-quality image corresponding to the input data.

Also, an image having a wide range of gradations in high-luminance areas and an image having a wide range of gradations in low-luminance areas may be synthesized, and the synthesized image may be enhanced in image quality by the image quality enhancement engine. In this case, an image capable of expressing a wider range of brightness may be generated. In this case, the input data of the training data for the image quality enhancement engine may be an image obtained by synthesizing a low-quality image having a wide range of gradations in high-luminance areas and a low-quality image having a wide range of gradations in low-luminance areas, which are to be processed. Furthermore, the output data of the training data for the image quality enhancement engine may be a high-quality image corresponding to the input data.

In these cases, the image quality improvement engine can be used to improve the image quality of images that can express a wider range of brightness, making it possible to process fewer images than before and providing images suitable for image analysis at a lower cost.

In addition, any method may be used to capture an image having a wide range of gradations in the high-luminance region and an image having a wide range of gradations in the low-luminance region, such as shortening or lengthening the exposure time of the imaging device. Furthermore, the method of dividing the range of gradations is not limited to low-luminance and high-luminance regions, and may be any method.

In addition, in the image quality improvement process according to this embodiment, multiple input images may be processed in parallel using multiple image quality improvement engines. Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing. In other words, a high-quality image obtained by performing at least one of the following processing groups or imaging methods may be used: overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

<Fourteenth embodiment>
Next, an image processing device according to a fourteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving section uses a medium quality image generated from a plurality of low quality images as an input image and generates a high quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires a medium quality image obtained by superimposing multiple low quality images as input data to be processed from the image capture device 10 or another device. When superimposing the low quality images, any positioning process may be performed.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, the image quality improvement engine of this embodiment is designed to input a medium image quality image, which is of medium image quality, and output a high image quality image. A medium image quality image is a composite image generated by superimposing a group of low image quality images. A high image quality image is an image of higher quality than a medium image quality image. Also, for the group of pairs that constitute the teacher data used to train the image quality improvement engine, the input data that constitutes each pair is a medium image quality image generated in the same manner as the medium image quality image, and the output data is a high image quality image.

The output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the display unit 20. The output unit 405 may display the input image on the display unit 20 together with the high-quality image, and in this case, the output unit 405 may display on the display unit 20 that the input image is an image generated from a plurality of low-quality images.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S520 to S550 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described.

In step S510, the acquisition unit 401 acquires a medium quality image as an input image from the image capture device 10 or another device. The acquisition unit 401 may acquire a medium quality image generated by the image capture device 10 as an input image in response to a request from the image capture device 10. Such a request may be issued, for example, when the image capture device 10 generates an image, before or after the image generated by the image capture device 10 is saved in a recording device provided in the image capture device 10, when the saved image is displayed on the display unit 20, when a high quality image is used for image analysis processing, etc.

The subsequent processing is similar to that in the first embodiment, so a detailed explanation is omitted.

As described above, the acquisition unit 401 according to this embodiment acquires, as an input image, a medium quality image, which is an image generated using multiple images of a specific part of the subject. In this case, since the input image is a clearer image, the high quality engine can generate a high quality image with higher accuracy. Note that the number of low quality images used to generate the medium quality image may be less than the number of images used to generate a conventional overlaid image.

Note that the medium quality image is not limited to an image obtained by superimposing multiple low quality images, but may be, for example, an image obtained by applying a MAP estimation process to multiple low quality images, or an image obtained by synthesizing multiple low quality images. When synthesizing multiple low quality images, images with different gradations may be synthesized.

In addition, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. In addition, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. In other words, a high-quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing may be used.

<Fifteenth embodiment>
Next, an image processing device according to a fifteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving unit performs image quality improvement according to the first embodiment and the like as well as image size increase (size increase) of an input image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires an image of low image size (low-size image) as an input image. Note that a low-size image is an image that comprises fewer pixels than an image of high image size (high-size image) output by a high-image-quality engine described below. Specifically, for example, when the image size of a high-size image is 1024 pixels wide, 1024 pixels high, and 1024 pixels deep, the image size of a low-size image is 512 pixels high, 512 pixels deep, and so on. In this regard, in this specification, "enhancing the image size" refers to a process of increasing the number of pixels per image and enlarging the image size.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine, as in the first embodiment. However, the image quality improvement engine of this embodiment is configured to reduce noise and enhance contrast for an input image, and to increase the image size of the input image. Therefore, the image quality improvement engine of this embodiment is configured to input a low-size image and output a high-size image.

In relation to this, for the group of pairs that constitute the training data for the image quality improvement engine, the input data that constitutes each pair is a low-size image, and the output data is a high-size image. Note that the high-size images used for the output data can be obtained from a device with higher performance than the imaging device that obtained the low-size images, or can be obtained by changing the settings of the imaging device. Furthermore, if a group of high-size images already exists, the group of low-size images to be used as input data may be obtained by reducing the group of high-size images to the image size of the images expected to be obtained from the imaging device 10. Furthermore, for the high-size images, low-size images that are superimposed are used, as in the first embodiment, etc.

Note that the image size enlargement of the input image by the image quality improvement unit 404 according to this embodiment is different from simple image enlargement, since it is obtained as teacher data from a device with higher performance than the image capture device 10 or by changing the settings of the image capture device 10. Specifically, the image size enlargement process of the input image by the image quality improvement unit 404 according to this embodiment can reduce degradation in resolution compared to when the image is simply enlarged.

With this configuration, the image quality improvement unit 404 according to this embodiment can reduce noise and enhance contrast from the input image, and generate a high-quality image with a large image size.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S520, S530, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S510, the acquisition unit 401 acquires a low-size image from the imaging device 10 or another device as input data to be processed. The acquisition unit 401 may acquire a low-size image generated by the imaging device 10 as an input image in response to a request from the imaging device 10. Such a request may be issued, for example, when the imaging device 10 generates an image, before or after the image generated by the imaging device 10 is saved in a recording device provided in the imaging device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

The processing of steps S520 and S530 is the same as that in the first embodiment, and therefore a description thereof will be omitted. In step S540, the image quality improvement unit 404 inputs the input image to an image quality improvement engine, and generates an image that has been subjected to noise reduction and contrast enhancement as a high-quality image and has a large image size. The processing thereafter is the same as that in the first embodiment, and therefore a description thereof will be omitted.

As described above, the image quality improvement unit 404 according to this embodiment generates a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the image size has been increased. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis, without increasing the invasiveness or effort of the photographer or subject.

In this embodiment, a high-quality image is generated by performing the high-quality image processing and the high-image size processing according to the first embodiment, etc., using one high-quality image engine, but the configuration for performing these processes is not limited to this. For example, the image quality improvement unit may include an image quality improvement engine that performs the high-quality image processing according to the first embodiment, etc., and another image quality improvement engine that performs the high-image size processing.

In this case, the image quality improvement engine that performs the image quality improvement process according to the first embodiment, etc. can use a machine learning model that has been trained in the same way as the image quality improvement engine according to the first embodiment, etc. Furthermore, as input data for the teacher data of the image quality improvement engine that performs the high image size process, the high image quality image generated by the image quality improvement engine according to the first embodiment, etc. is used. Furthermore, as output data for the teacher data of the image quality improvement engine, the high image quality image generated by the image quality improvement engine according to the first embodiment, etc. for an image acquired by a high-performance imaging device is used. In this way, the image quality improvement engine that performs the high image size process can generate a final high image quality image by increasing the image size of the high image quality image that has been subjected to the image quality improvement process according to the first embodiment, etc.

The high image quality engine may perform high image size processing before the high image quality processing engine according to the first embodiment, etc. In this case, the training data for the high image quality engine performing high image size processing is configured as a group of pairs of input data, which is a low size image acquired by the imaging device, and output data, which is a high size image. The training data for the high image quality engine performing high image quality processing according to the first embodiment, etc. is configured as a group of pairs of input data, which is a high size image, and output data, which is an image in which the high size image is superimposed.

Even with this configuration, the image processing device 400 can generate a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the image size has been enlarged.

In this embodiment, the high image quality processing according to the first embodiment and the like has been described in terms of a configuration in which the superimposed image is used as output data of the teacher data, but as with the first embodiment, the output data is not limited to this. In other words, a high image quality image obtained by performing at least one of a group of processes or imaging methods, such as superimposition processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400.

<Sixteenth embodiment>
Next, an image processing apparatus according to a sixteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving section improves spatial resolution in addition to the image quality improvement according to the first embodiment.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires a low spatial resolution image as an input image. Note that a low spatial resolution image is an image with a lower spatial resolution than the high spatial resolution image output by the image quality improvement unit 404.

The image quality improvement unit 404 is equipped with an image quality improvement engine, as in the first embodiment. However, the image quality improvement engine of this embodiment is configured to reduce noise and enhance contrast of the input image, as well as to increase the spatial resolution of the input image. Therefore, the image quality improvement engine of this embodiment is configured to input a low spatial resolution image and output a high spatial resolution image.

In relation to this, for the group of pairs that make up the training data for the image quality improvement engine, the input data that makes up each pair is a low spatial resolution image, and the output data is a high spatial resolution image. Note that the high spatial resolution image can be obtained from a device with higher performance than the imaging device that acquired the low spatial resolution image, or can be obtained by changing the settings of the imaging device. Also, for the high spatial resolution image, an image that is overlaid with a low spatial resolution image is used, as in the first embodiment, etc.

With this configuration, the image quality improvement unit 404 according to this embodiment can reduce noise and enhance contrast from the input image, and generate a high-quality image with high spatial resolution.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S520, S530, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S510, the acquisition unit 401 acquires a low spatial resolution image from the imaging device 10 or another device as input data to be processed. The acquisition unit 401 may acquire a low spatial resolution image generated by the imaging device 10 as an input image in response to a request from the imaging device 10. Such a request may be issued, for example, when the imaging device 10 generates an image, before or after the image generated by the imaging device 10 is saved in a recording device provided in the imaging device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

The processing of steps S520 and S530 is the same as that in the first embodiment, and therefore a description thereof will be omitted. In step S540, the image quality improvement unit 404 inputs the input image to an image quality improvement engine, and generates an image with high spatial resolution and noise reduction as a high-quality image, in which contrast has been enhanced. The subsequent processing is the same as that in the first embodiment, and therefore a description thereof will be omitted.

As described above, the image quality improvement unit 404 according to this embodiment generates a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the spatial resolution has been improved. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis without increasing the invasiveness or effort of the photographer or subject.

In this embodiment, a high-quality image is generated by performing high-quality image processing and high-resolution processing according to the first embodiment, etc., using a single high-quality image engine, but the configuration for performing these processes is not limited to this. For example, the image quality improvement unit may include an image quality improvement engine that performs image quality processing according to the first embodiment, etc., and another image quality improvement engine that performs high spatial resolution processing.

In this case, the image quality improvement engine that performs the image quality improvement process according to the first embodiment, etc. can use a machine learning model that has been trained in the same way as the image quality improvement engine according to the first embodiment, etc. Furthermore, as input data of the teacher data of the image quality improvement engine that performs the high spatial resolution process, the high image quality image generated by the image quality improvement engine according to the first embodiment, etc. is used. Furthermore, as output data of the teacher data of the image quality improvement engine, the high image quality image generated by the image quality improvement engine according to the first embodiment, etc. for an image acquired by a high-performance imaging device is used. In this way, the image quality improvement engine that performs the high spatial resolution process can generate a final high image quality image with high spatial resolution for the high image quality image that has been subjected to the image quality improvement process according to the first embodiment, etc.

The high spatial resolution processing by the image quality improvement engine may be performed before the high image quality processing by the image quality improvement processing engine according to the first embodiment, etc. In this case, the training data for the image quality improvement engine performing the high spatial resolution processing is configured as a group of pairs of input data, which is a low spatial resolution image acquired by the imaging device, and output data, which is a high spatial resolution image. The training data for the image quality improvement engine performing the image quality improvement processing according to the first embodiment, etc. is configured as a group of pairs of input data, which is a high spatial resolution image, and output data, which is an image in which the high spatial resolution image is superimposed.

Even with this configuration, the image processing device 400 can generate a high-quality image in which at least one of noise reduction and contrast enhancement is performed compared to the input image, and the spatial resolution is improved.

In this embodiment, the high image quality processing according to the first embodiment and the like has been described in terms of a configuration in which the superimposed image is used as output data of the teacher data, but as with the first embodiment, the output data is not limited to this. In other words, a high image quality image obtained by performing at least one of a group of processes or imaging methods, such as superimposition processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

The image quality improvement unit 404 may use an image quality improvement engine to perform the image quality improvement process according to the fifteenth embodiment in addition to the high spatial resolution process. In this case, at least one of noise reduction and contrast enhancement is performed compared to the input image, and an image with a larger image size and higher spatial resolution compared to the input image can be generated as a high-quality image. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis without increasing the invasiveness or effort of the photographer or subject.

Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400.

Seventeenth embodiment
Next, an image processing apparatus according to a seventeenth embodiment will be described with reference to Figures 22 and 23. In this embodiment, an analysis unit performs image analysis on a high-quality image generated by an image quality improvement unit.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

Figure 22 shows a schematic configuration of an image processing device 2200 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image processing device 2200 according to this embodiment is provided with an analysis unit 2208. Note that the image processing device 2200 may be composed of multiple devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the configuration shown in Figure 4 is indicated using the same reference numerals and description will be omitted.

The analysis unit 2208 applies a predetermined image analysis process to the high-quality image generated by the image quality improvement unit 404. In the field of ophthalmology, the image analysis process includes, for example, any existing image analysis process such as retinal layer segmentation, layer thickness measurement, optic nerve 3D shape analysis, lamina cribrosa analysis, vascular density measurement of OCTA images, and corneal shape analysis for images acquired by OCT. In addition, the image analysis process is not limited to analysis processes in the field of ophthalmology, and also includes, for example, any existing analysis process in the field of radiology, such as diffusion tensor analysis and VBL (Voxel-based Morphometry) analysis.

The output unit 405 can display the high-quality image generated by the image quality improvement unit 404 on the display unit 20, and can also display the analysis results of the image analysis process performed by the analysis unit 2208. The output unit 405 can display only the image analysis results performed by the analysis unit 2208 on the display unit 20, or can output the image analysis results to the imaging device 10, an image management system, or other devices. The display form of the analysis results can be arbitrary depending on the image analysis process performed by the analysis unit 2208, and can be displayed, for example, as an image, a numerical value, or a character.

A series of image processing steps according to this embodiment will be described below with reference to FIG. 23. FIG. 23 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S2310 to S2340 according to this embodiment are similar to the processing steps S510 to S540 in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S2320, the processing step S2330 may be omitted and the processing may proceed to step S2340.

In step S2340, when the image quality improving unit 404 generates a group of high-quality images, the process proceeds to step S2350. In step S2350, the analysis unit 2208 performs image analysis on the high-quality images generated in step S2340, and outputs the analysis results. The image analysis process content and processing method may be any existing image analysis process and processing method. The analysis unit 2208 may perform multiple image analysis processes. In relation to this, the analysis unit 2208 may perform some or all of the multiple image analysis processes in response to instructions from the examiner via any input device.

In step S2360, the output unit 405 displays the high-quality image generated by the image quality improvement unit 404 and the analysis results by the analysis unit 2208 on the display unit 20. The output unit 405 may output the high-quality image and the analysis results to separate display units or devices. The output unit 405 may also display only the analysis results on the display unit 20. Furthermore, when the analysis unit 2208 outputs multiple analysis results, the output unit 405 may output some or all of the multiple analysis results to the display unit 20 or another device.

As described above, the image processing device 2200 according to this embodiment further includes an analysis unit 2208 that performs image analysis on the high-quality image, and the output unit 405 causes the display unit 20 to display the analysis results by the analysis unit 2208. In this way, the image processing device 2200 according to this embodiment uses high-quality images for image analysis, thereby improving the accuracy of the analysis.

In addition, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 2200, as in the first embodiment. In addition, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. In other words, a high-quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing may be used.

In the first to seventeenth embodiments, the display of the high-quality image on the display unit 20 by the output unit 405 is basically performed automatically in response to the generation of the high-quality image by the image quality improvement unit 404 and the output of the analysis result by the analysis unit 2208. However, the display of the high-quality image may be performed in response to an instruction from the examiner. For example, the output unit 405 may display on the display unit 20 an image selected in response to an instruction from the examiner from among the high-quality image generated by the image quality improvement unit 404 and the input image. In addition, the output unit 405 may switch the display on the display unit 20 from the captured image (input image) to the high-quality image in response to an instruction from the examiner.

Furthermore, the image quality improvement unit 404 may start image quality improvement processing by the image quality improvement engine (input of an image to the image quality improvement engine) in response to an instruction from the examiner, and the output unit 405 may display the high-quality image generated by the image quality improvement unit 404 on the display unit 20. In contrast, when an input image is captured by the imaging device 10, the image quality improvement engine may automatically generate a high-quality image based on the input image, and the output unit 405 may display the high-quality image on the display unit 20 in response to an instruction from the examiner. Note that these processes can also be performed on the output of the analysis results. Furthermore, the image processing device may be configured to start processing by the imaging location estimation engine, image quality evaluation engine, authenticity evaluation engine, and evaluation unit in response to an instruction from the examiner.

Note that, in relation to the first to seventeenth embodiments, the display mode in which the output unit 405 displays the high-quality image on the display unit 20 may be arbitrary. For example, the output unit 405 may display the input image and the high-quality image side by side, or may switch between them. The output unit 405 may also display the input image and the high-quality image in order according to the part of the body that was photographed, the photographing date and time, the facility where the photographing was performed, and the like. Similarly, the output unit 405 may display the image analysis results using the high-quality image in order according to any photographing conditions of the high-quality image and the input image corresponding to the high-quality image. Furthermore, the output unit 405 may display the image analysis results using the high-quality image in order for each analysis item.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above-mentioned embodiments. The present invention also includes inventions that have been modified without going against the spirit of the present invention, and inventions that are equivalent to the present invention. In addition, the above-mentioned embodiments can be combined as appropriate without going against the spirit of the present invention.

100: Image processing device (medical image processing device), 104: Image quality improvement unit, 105: Output unit

Claims (16)

an acquisition unit that acquires a first image which is an image of a predetermined portion of a subject's eye;
a display control unit that controls a display unit to display either a second image generated by inputting the first image into an image quality improvement engine including a machine learning engine as an input image of the image quality improvement engine, or the first image;
Equipped with
When a photographing condition of the first image is a photographing angle of view that can be improved by the image quality improvement engine, the display unit is controlled to display the second image;
When the imaging condition of the first image is not an imaging angle of view that can be improved by the image quality improvement engine, the display unit is controlled to display the first image . 2. The medical image processing device according to claim 1, wherein when the shooting conditions of the first image are at a shooting angle of view that can be improved in image quality by the image quality improvement engine, the display unit is controlled to change from displaying the first image to displaying the second image in response to an instruction from an operator. 3. The medical image processing device according to claim 1, wherein when the shooting conditions of the first image are at a shooting angle of view that can be improved in image quality by the image quality improvement engine, the display unit is controlled to change from displaying the first image to displaying the second image in response to an instruction to press a button on a user interface displayed on the display unit. the first image is an image included in a moving image of a predetermined portion of the subject's eye,
The medical image processing apparatus according to claim 1 , wherein the second images are generated sequentially. The image quality improvement engine includes a machine learning engine obtained by using OCTA images as learning data;
the first image is an OCTA image;
The medical image processing apparatus according to claim 1 , wherein the OCTA image is a motion contrast image showing a temporal change in the predetermined area obtained from the results of repeatedly photographing the same area. The medical image processing apparatus according to claim 5 , wherein the image quality improvement engine is a machine learning engine obtained by using, as learning data, OCTA images obtained by superimposition processing. The medical image processing apparatus according to claim 5 , wherein the display unit is controlled to display an analysis result obtained by performing an image analysis related to blood vessel density measurement on the second image in response to an instruction from an operator. The medical image processing apparatus according to claim 1 , wherein the second image has a higher image quality than the first image. The medical image processing apparatus according to claim 1 , wherein the second image includes at least one of lower noise and higher contrast than the first image. 10. A medical image processing device according to claim 1, wherein input data of the learning data of the image quality improvement engine is an image based on one or more images of the test eye, and output data of the learning data is an image based on a number of images of the test eye that is greater than the one or more images of the test eye. 11. The medical image processing device according to claim 1, wherein the second image is generated by using an image quality improvement engine selected from among a plurality of image quality improvement engines that have been trained using learning data with different shooting conditions according to the shooting conditions corresponding to the first image. 12. The medical image processing device according to claim 1, wherein when the shooting conditions of the first image are an OCTA shooting method and a shooting angle of view that can be improved in image quality by the image quality improvement engine, the display unit is controlled to display the second image. The image quality improvement engine further includes a machine learning engine obtained by using OCT images as learning data,
13. The medical image processing device according to claim 1, wherein when the shooting conditions of the first image are an OCT shooting method that can be improved in image quality by the image quality improvement engine, the display unit is controlled to display the second image. An imaging unit that images the subject's eye;
A medical image processing apparatus according to any one of claims 1 to 13,
An optical coherence tomography apparatus comprising: acquiring a first image which is an image of a predetermined portion of a subject's eye;
Controlling a display unit to display either a second image generated by inputting the first image into an image quality improvement engine including a machine learning engine as an input image of the image quality improvement engine , or the first image;
Including,
When a photographing condition of the first image is a photographing angle of view that can be improved by the image quality improvement engine, the display unit is controlled to display the second image;
A medical image processing method , wherein, when the shooting conditions of the first image are not an angle of view at which the image quality can be improved by the image quality improvement engine, the display unit is controlled to display the first image . A program that, when executed by a processor, causes the processor to perform each step of the medical image processing method according to claim 15 .