JP7604604B2 - Medical image processing device, medical image processing method and program - Google Patents

Description

The present invention relates to a medical image processing device, a medical image processing method, and a program.

In the medical field, images are acquired by various imaging devices to identify the disease of a subject and observe the extent of the disease, and image diagnosis is performed by medical professionals. For example, in the field of radiology, types of imaging devices include X-ray devices, X-ray computed tomography (CT) devices, magnetic resonance imaging (MRI) devices, positron emission tomography (PET) devices, and single photon emission computed tomography (SPECT) devices. In the field of ophthalmology, for example, there are fundus cameras, scanning laser ophthalmoscopes (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

To perform image diagnosis accurately and complete it in a short time, it is important that the images acquired by the imaging device have high image quality, such as low noise, high resolution and spatial resolution, and appropriate gradation. Images that highlight the area or lesion to be observed can also be useful.

However, with many imaging devices, some trade-off is required to obtain images suitable for diagnostic imaging, such as high image quality. For example, one option is to purchase a high-performance imaging device to obtain high-image quality images, but this often requires a larger investment than a low-performance device.

In addition, for example, in CT, the subject's radiation dose may need to be increased to obtain an image with less noise. In addition, for example, in MRI, a contrast agent that carries a risk of side effects may be used to obtain an image that highlights the area to be observed. In addition, for example, in OCT, if the area to be imaged is large or high spatial resolution is required, the imaging time may be longer. In addition, for example, some imaging devices require multiple images to be taken in order to obtain an image with high image quality, which increases the imaging time.

Patent Document 1 discloses a technology that uses an artificial intelligence engine to convert previously acquired images into images with higher resolution in order to accommodate rapid advances in medical technology and the need for simple imaging in emergencies. With this technology, for example, it is possible to convert an image acquired by simple imaging with little cost into an image with higher resolution.

JP 2018-5841 A

However, even high-resolution images may not be suitable for image diagnosis. For example, even if the image has high resolution, if there is a lot of noise or low contrast, it may not be possible to properly grasp the object to be observed.

In response to this, one of the objectives of the present invention is to provide a medical image processing device, a medical image processing method, and a program that can generate images that are more suitable for image diagnosis than conventional methods.

A medical image processing apparatus according to an embodiment of the present invention comprises:
a designation means for designating a part of a depth range of a predetermined part of a subject in three-dimensional medical image data of the predetermined part in response to an instruction from an examiner ;
an acquisition means for acquiring a first image, which is a medical image of the predetermined portion corresponding to the specified partial depth range, by using the three-dimensional medical image data ;
The image quality improvement unit uses an image quality improvement engine including a machine learning engine obtained using learning data including a plurality of medical images corresponding to a plurality of depth ranges of a specific part of a subject to generate a second image having higher image quality than the first image from the first image , and generates a composite image by combining pixel values of corresponding pixels in the first image and the second image at a ratio that can be changed in response to instructions from an examiner .

Furthermore, a medical image processing method according to another embodiment of the present invention comprises:
Specifying a depth range of a predetermined portion of the subject's three-dimensional medical image data in response to an instruction from an examiner ;
acquiring a first image, which is a medical image of the predetermined portion corresponding to the specified partial depth range, using the three-dimensional medical image data ;
The method includes generating a second image having higher image quality than the first image from the first image using an image quality improvement engine including a machine learning engine obtained using learning data including a plurality of medical images corresponding to a plurality of depth ranges of a specific part of a subject , and generating a composite image by combining pixel values of corresponding pixels in the first image and the second image at a ratio that can be changed in response to instructions from an examiner .

According to one aspect of the present invention, it is possible to generate images that are more suitable for image diagnosis than conventional methods.

1 shows an example of the configuration of a neural network related to image quality improvement processing. 1 shows an example of the configuration of a neural network related to a shooting location estimation process. 1 shows an example of the configuration of a neural network related to image authenticity evaluation processing. 1 illustrates an example of a schematic configuration of an image processing apparatus according to a first embodiment. FIG. 4 is a flowchart showing an example of a flow of image processing according to the first embodiment. FIG. 11 is a flowchart showing another example of the flow of image processing according to the first embodiment. FIG. 11 is a flowchart showing an example of a flow of image processing according to the second embodiment. FIG. 13 is a diagram for explaining image processing according to a fourth embodiment. FIG. 13 is a flowchart showing an example of the flow of image quality improvement processing according to the fourth embodiment. FIG. 13 is a diagram for explaining image processing according to the fifth embodiment. FIG. 13 is a flowchart showing an example of the flow of image quality improvement processing according to the fifth embodiment. FIG. 13 is a diagram for explaining image processing according to the sixth embodiment. FIG. 23 is a flowchart showing an example of the flow of image quality improvement processing according to the sixth embodiment. FIG. 13 is a diagram for explaining image processing according to the sixth embodiment. 23 shows an example of a schematic configuration of an image processing device according to a seventh embodiment. FIG. 23 is a flowchart showing an example of the flow of image processing according to the seventh embodiment. 23 shows an example of a user interface according to the seventh embodiment. 23 shows an example of a schematic configuration of an image processing device according to a ninth embodiment. FIG. 23 is a flowchart showing an example of the flow of image processing according to the ninth embodiment. 23 shows an example of a schematic configuration of an image processing device according to a twelfth embodiment. FIG. 23 is a flowchart showing an example of the flow of image quality improvement processing according to the thirteenth embodiment. FIG. 23 is a flowchart showing another example of the flow of image quality improvement processing according to the thirteenth embodiment. 23 shows an example of a schematic configuration of an image processing device according to a seventeenth embodiment. FIG. 23 is a flow diagram showing an example of the flow of image processing according to the seventeenth embodiment. 1 shows an example of the configuration of a neural network related to image quality improvement processing. 23 shows an example of a schematic configuration of an image processing device according to a nineteenth embodiment. FIG. 23 is a flow diagram showing an example of the flow of image processing according to the nineteenth embodiment. A flowchart showing an example of the flow of image processing related to the twenty-first embodiment. 1 shows an example of a teacher image related to image quality improvement processing. 1 shows an example of an input image for image quality improvement processing. 23 shows an example of a schematic configuration of an image processing device according to a twenty-second embodiment. FIG. 22 is a flow diagram showing an example of the flow of image processing according to the twenty-second embodiment. FIG. 22 is a diagram for explaining a wide-angle image according to the twenty-second embodiment. 23A to 23C are diagrams for explaining image quality improvement processing according to the twenty-third embodiment. 23 shows an example of a user interface according to the twenty-fourth embodiment. 25 shows an example of a schematic configuration of an image processing device according to a twenty-fifth embodiment. 13 shows an example of the configuration of a neural network used as a machine learning engine in accordance with variant example 6. 13 shows an example of the configuration of a neural network used as a machine learning engine in accordance with variant example 6. 23 shows an example of a user interface according to the twenty-fourth embodiment.

Below, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. In addition, the same reference numerals are used in the drawings to indicate elements that are identical or functionally similar.

<Terminology>
First, the terms used in this specification will be explained.

In the network described in this specification, each device may be connected by a wired or wireless line. Here, the lines connecting each device in the network include, for example, a dedicated line, a local area network (hereinafter referred to as LAN) line, a wireless LAN line, an Internet line, Wi-Fi (registered trademark), and Bluetooth (registered trademark), etc.

The medical image processing device may be composed of two or more devices capable of communicating with each other, or may be composed of a single device. Furthermore, each component of the medical image processing device may be composed of a software module executed by a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit). Furthermore, each component may be composed of a circuit that performs a specific function, such as an ASIC. Furthermore, it may be composed of a combination of any other hardware and any software.

In addition, the medical images processed by the medical image processing device or medical image processing method according to the following embodiments include images acquired using any modality (imaging device, imaging method). The medical images to be processed can include medical images acquired by any imaging device, etc., and images created by the medical image processing device or medical image processing method according to the following embodiments.

Furthermore, the medical image to be processed is an image of a specific part of the subject (test subject), and the image of the specific part includes at least a part of the specific part of the subject. The medical image may also include other parts of the subject. The medical image may be a still image or a moving image, and may be a black and white image or a color image. The medical image may also be an image that represents the structure (shape) of the specific part, or an image that represents its function. Images that represent function include images that represent hemodynamics (blood flow rate, blood flow velocity, etc.), such as OCTA images, Doppler OCT images, fMRI images, and ultrasonic Doppler images. The specific part of the subject may be determined according to the subject to be photographed, and may include any part, such as the human eye (test subject eye), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, the head, chest, legs, and arms.

The medical image may be a tomographic image of the subject, or a frontal image. The frontal image includes, for example, a frontal image of the fundus, a frontal image of the anterior segment, a fundus image photographed by fluorescence, and an En-Face image generated using data acquired by OCT (three-dimensional OCT data) from at least a partial range in the depth direction of the subject. The En-Face image may be an OCTA En-Face image (motion contrast frontal image) generated using three-dimensional OCTA data (three-dimensional motion contrast data) from at least a partial range in the depth direction of the subject. Three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

An imaging device is a device for capturing images used in diagnosis. Imaging devices include, for example, devices that obtain an image of a specific part of a subject by irradiating the specific part with light, radiation such as X-rays, electromagnetic waves, or ultrasound, and devices that obtain an image of a specific part by detecting radiation emitted from a subject. More specifically, imaging devices according to the following embodiments include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscope.

The OCT device may include a time domain OCT (TD-OCT) device or a Fourier domain OCT (FD-OCT) device. The Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a swept wavelength OCT (SS-OCT) device. The SLO device or OCT device may include a wavefront compensation SLO (AO-SLO) device or a wavefront compensation OCT (AO-OCT) device using a wavefront compensation optical system. The SLO device or OCT device may include a polarized SLO (PS-SLO) device or a polarized OCT (PS-OCT) device for visualizing information related to polarization phase difference and depolarization.

The image management system is a device and system that receives and stores images captured by an imaging device and images that have been processed. The image management system can also transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. Examples of the image management system include a picture archiving and communication system (PACS). In particular, the image management system according to the embodiment described below includes a database that can store various information such as subject information and shooting time associated with received images. The image management system is also connected to a network, and can send and receive images, convert images, and send and receive various information associated with stored images in response to a request from other devices.

The imaging conditions are various pieces of information at the time of capturing an image obtained by an imaging device. The imaging conditions include, for example, information about the imaging device, information about the facility where the imaging was performed, information about the examination related to the imaging, information about the photographer, and information about the subject. The imaging conditions also include, for example, the imaging date and time, the name of the imaging part, the imaging area, the imaging angle of view, the imaging method, the image resolution and gradation, the image size, the applied image filter, information about the image data format, and information about the radiation dose. The imaging area can include a surrounding area that is shifted from a specific imaging part, an area that includes multiple imaging parts, etc.

The shooting conditions can be stored in the data structure constituting the image, or as shooting condition data separate from the image, or stored in a database or image management system related to the shooting device. Therefore, the shooting conditions can be acquired by a procedure corresponding to the storage means of the shooting conditions of the shooting device. Specifically, the shooting conditions are acquired, for example, by analyzing the data structure of the image output by the shooting device, acquiring shooting condition data corresponding to the image, or accessing an interface for acquiring the shooting conditions from a database related to the shooting device.

Depending on the imaging device, there may be some imaging conditions that cannot be acquired because they have not been saved. For example, this may be the case when the imaging device does not have the function to acquire or save specific imaging conditions, or when such a function is disabled. In addition, there may be cases where the imaging conditions are not saved because they are not related to the imaging device or imaging. Furthermore, there may be cases where the imaging conditions are hidden, encrypted, or cannot be acquired without the right. However, there are cases where even imaging conditions that have not been saved can be acquired. For example, the name of the imaging part or the imaging area can be identified by performing image analysis.

A machine learning model is a model that has been trained (learned) in advance using appropriate teacher data (learning data) for an arbitrary machine learning algorithm. The teacher data consists of one or more pairs of input data and output data (correct answer data). The format and combination of the input data and output data in the pairs that make up the teacher data may be suitable for the desired configuration, such as one being an image and the other being a number, one being composed of a group of multiple images and the other being a string of characters, or both being images.

Specific examples of such training data include training data (hereinafter referred to as "first training data") that is composed of a set of pairs of an image acquired by OCT and an imaging site label corresponding to the image. The imaging site label is a unique numerical value or character string that represents the site. Other examples of training data include training data (hereinafter referred to as "second training data") that is composed of a set of pairs of a noisy low-quality image acquired by normal OCT imaging and a high-quality image that has been imaged multiple times by OCT and processed to improve image quality.

When input data is input to the machine learning model, output data according to the design of the machine learning model is output. The machine learning model outputs output data that is likely to correspond to the input data, for example, according to the tendency trained using the teacher data. In addition, the machine learning model can output the possibility of corresponding to the input data as a numerical value for each type of output data, for example, according to the tendency trained using the teacher data. Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first teacher data, the machine learning model outputs the imaging site label of the imaging site captured in the image, or outputs the probability for each imaging site label. In addition, for example, when a noisy low-quality image acquired by normal OCT imaging is input to a machine learning model trained with the second teacher data, the machine learning model outputs a high-quality image equivalent to an image captured multiple times by OCT and processed for high image quality. In addition, the machine learning model can be configured not to use the output data it outputs as teacher data from the viewpoint of maintaining quality.

The machine learning algorithm also includes deep learning techniques such as convolutional neural networks (CNN). In deep learning techniques, different parameter settings for layers and nodes constituting a neural network may result in different degrees of reproducibility of trends trained using training data in output data. For example, in a deep learning machine learning model using first training data, setting more appropriate parameters may increase the probability of outputting a correct imaging site label. Also, in a deep learning machine learning model using second training data, setting more appropriate parameters may result in higher quality images being output.

Specifically, parameters in a CNN can include, for example, the filter kernel size, number of filters, stride value, and dilation value set for the convolutional layer, as well as the number of nodes output by the fully connected layer. The parameter group and the number of training epochs can be set to values that are favorable for the usage form of the machine learning model based on the training data. For example, a parameter group and the number of epochs can be set based on the training data to output a correct imaging site label with a higher probability or to output a higher quality image.

One method for determining such a parameter set and the number of epochs is exemplified below. First, 70% of the pair sets constituting the teacher data are set for training, and the remaining 30% are set for evaluation, randomly. Next, the machine learning model is trained using the pair sets for training, and at the end of each epoch of training, a training evaluation value is calculated using the pair sets for evaluation. The training evaluation value is, for example, the average value of a group of values obtained by evaluating the output when the input data constituting each pair is input to the machine learning model being trained, and the output data corresponding to the input data, using a loss function. Finally, the parameter set and the number of epochs when the training evaluation value is the smallest are determined as the parameter set and the number of epochs of the machine learning model. In this way, by dividing the pair sets constituting the teacher data into those for training and those for evaluation and determining the number of epochs, it is possible to prevent the machine learning model from overlearning the pair sets for training.

The image quality improvement engine (trained model for image quality improvement) is a module that improves the image quality of an input low-image quality image and outputs a high-image quality image. In this specification, image quality improvement refers to converting an input image into an image with image quality more suitable for image diagnosis, and a high-image quality image refers to an image converted into an image with image quality more suitable for image diagnosis. In addition, a low-image quality image is, for example, a two-dimensional image or a three-dimensional image acquired by X-ray photography, CT, MRI, OCT, PET, or SPECT, or a three-dimensional moving image of a CT taken continuously without being set to obtain a particularly high image quality. Specifically, low-image quality images include, for example, images acquired by low-dose photography using an X-ray photography device or CT, photography using an MRI without using a contrast agent, images acquired by short-time photography using OCT, and OCTA images acquired with a small number of photography sessions.

The content of image quality suitable for diagnostic imaging depends on what is being diagnosed with various types of diagnostic imaging. Therefore, it is difficult to generalize, but for example, image quality suitable for diagnostic imaging includes image quality with little noise, high contrast, colors and gradations that make it easy to observe the subject, large image size, and high resolution. It can also include image quality in which non-existent objects and gradations that are drawn during the image generation process are removed from the image.

In addition, when high-quality images with low noise and high contrast are used for image analysis such as vascular analysis processing of images such as OCTA, or area segmentation processing of images such as CT or OCT, the analysis can often be performed more accurately than when low-quality images are used. Therefore, high-quality images output by the image quality engine can be useful not only for image diagnosis, but also for image analysis.

The image processing method constituting the image quality improvement method in the following embodiment performs processing using various machine learning algorithms such as deep learning. Note that in addition to processing using machine learning algorithms, the image processing method may also perform any existing processing such as various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing.

In particular, as an example of a CNN configuration for improving the image quality of a two-dimensional image, there is a configuration shown in FIG. 1. The CNN configuration includes a group of multiple convolution processing blocks 100. The convolution processing block 100 includes a convolution layer 101, a batch normalization layer 102, and an activation layer 103 using a normalized linear function (Rectifier Linear Unit). The CNN configuration also includes a merger layer 104 and a final convolution layer 105. The merger layer 104 merges the output value group of the convolution processing block 100 with the pixel value group that constitutes the image by connecting or adding them. The final convolution layer 105 outputs the pixel value group that constitutes the high-image-quality image Im120 that is merged in the merger layer 104. In this configuration, the pixel values constituting the input image Im110 are output through the convolution processing blocks 100, and the pixel values constituting the input image Im110 are synthesized in the synthesis layer 104. The synthesized pixel values are then shaped into a high-quality image Im120 in the final convolution layer 105.

For example, a certain level of image quality improvement can be achieved by setting the number of convolution processing blocks 100 to 16, setting the parameters of the convolution layer 101 group to a filter kernel size of 3 pixels wide and 3 pixels high, and setting the number of filters to 64. However, in practice, as described in the above description of the machine learning model, a better parameter group can be set using training data according to the usage form of the machine learning model. Note that when three-dimensional or four-dimensional images need to be processed, the filter kernel size may be expanded to three or four dimensions.

Note that when using some image processing methods, such as image processing using CNN, attention must be paid to image size. Specifically, it should be noted that in order to address issues such as the issue of the peripheral areas of a high-quality image not being of sufficient quality, different image sizes may be required for the input low-quality image and the output high-quality image.

For the sake of clarity, this will not be specified in the embodiments described below, but when a high-image-quality engine is adopted that requires different image sizes for the image input to the high-image-quality engine and the image output, the image size is adjusted appropriately. Specifically, the image size is adjusted for input images, such as images used as teacher data for training a machine learning model or images input to the high-image-quality engine, by padding or combining the surrounding shooting area of the input image. Note that the area to be padded is filled with a fixed pixel value, filled with nearby pixel values, or mirror padded according to the characteristics of the high-image-quality technique so that the image quality can be effectively improved.

The image quality improvement method may be implemented using only one image processing method, or may be implemented using a combination of two or more image processing methods. In addition, multiple image quality improvement methods may be implemented in parallel to generate multiple high-quality image groups, and the highest quality image may be selected as the final high-quality image. The highest quality image may be selected automatically using an image quality assessment index, or may be selected according to the examiner's (user's) instructions by displaying multiple high-quality image groups on a user interface provided on an arbitrary display unit, etc.

In some cases, an input image that has not been enhanced in image quality may be more suitable for image diagnosis, so the input image may be included in the final image selection. Parameters may also be input to the image enhancement engine along with the low-image quality image. For example, parameters specifying the degree of image enhancement or parameters specifying the image filter size used in the image processing method may be input to the image enhancement engine along with the input image.

The shooting location estimation engine is a module that estimates the shooting part and shooting area of the input image. The shooting location estimation engine can output where the shooting part and shooting area depicted in the input image are, or the probability that it is the shooting part or shooting area for each shooting part label or shooting area label of the required level of detail.

Depending on the imaging device, the imaging part and imaging area may not be saved as imaging conditions, or may not be acquired by the imaging device and saved. Even if the imaging part and imaging area are saved, the imaging part and imaging area may not be saved at the required level of detail. For example, the imaging part may only be saved as "posterior segment," but it is not clear whether it is the "macular area," "optic disc," "macular area and optic disc," or "other." In another example, the imaging part may only be saved as "breast," but it is not clear whether it is the "right breast," "left breast," or "both." Therefore, by using the imaging location estimation engine, it is possible to estimate the imaging part and imaging area of the input image in these cases.

The image and data processing method constituting the estimation method of the imaging location estimation engine performs processing using various machine learning algorithms such as deep learning. Note that the image and data processing method may perform any existing estimation processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge-based processing, etc. in addition to or instead of processing using a machine learning algorithm. Note that the teacher data used to train the machine learning model constructed using the machine learning algorithm may be images labeled with the imaging site and imaging area. In this case, the images of the teacher data are used as input data, and the labels of the imaging site and imaging area are used as output data.

In particular, as an example of a CNN configuration that estimates the shooting location of a two-dimensional image, there is a configuration shown in FIG. 2. The CNN configuration includes a group of multiple convolution processing blocks 200, each of which is composed of a convolution layer 201, a batch normalization layer 202, and an activation layer 203 using a normalized linear function. The CNN configuration also includes a final convolution layer 204, a fully connected layer 205, and an output layer 206. The fully connected layer 205 fully connects the output value groups of the convolution processing blocks 200. The output layer 206 also uses a Softmax function to output the probability of each expected shooting part label for the input image Im210 as an estimation result (Result) 207. In this configuration, for example, if the input image Im210 is an image of the "macular region", the highest probability is output for the shooting part label corresponding to the "macular region.

For example, the imaging site can be estimated with a certain degree of accuracy by setting the number of convolution processing blocks 200 to 16, the parameters of the convolution layer group 201 to a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters to 64. However, in practice, as described in the above description of the machine learning model, a better parameter group can be set using training data according to the usage form of the machine learning model. If it is necessary to process three-dimensional or four-dimensional images, the filter kernel size may be expanded to three or four dimensions. The estimation method may be implemented using only one image and data processing method, or may be implemented using a combination of two or more image and data processing methods.

The image quality assessment engine is a module that outputs an image quality assessment index for an input image. The image quality assessment processing method for calculating the image quality assessment index performs processing using various machine learning algorithms such as deep learning. Note that the image quality assessment processing method may perform any existing evaluation processing such as an image noise measurement algorithm and a matching process using a database of image quality assessment indices corresponding to similar images and base images. Note that these evaluation processes may be performed in addition to or instead of processing using a machine learning algorithm.

For example, the image quality assessment index can be obtained from a machine learning model constructed using a machine learning algorithm. In this case, the pair of input data constituting the teacher data for training the machine learning model is a group of images consisting of a group of low-quality images and a group of high-quality images captured in advance under various shooting conditions. In addition, the pair of output data constituting the teacher data for training the machine learning model is, for example, a group of image quality assessment indices set for each of the image groups of the input data by an examiner performing image diagnosis.

The authenticity evaluation engine in the description of this invention is a module that evaluates the rendering of an input image and evaluates with a certain degree of accuracy whether the image was captured and acquired by the target imaging device. The authenticity evaluation processing method performs processing using various machine learning algorithms such as deep learning. Note that the authenticity evaluation processing method may perform any existing evaluation processing such as knowledge-based processing in addition to or instead of processing using machine learning algorithms.

For example, the authenticity evaluation process can be performed by a machine learning model constructed using a machine learning algorithm. First, the training data of the machine learning model will be described. The training data includes a pair group of high-quality images captured in advance under various shooting conditions and a label (hereinafter, genuine label) indicating that the images were captured and acquired by the target imaging device. The training data also includes a pair group of high-quality images generated by inputting low-quality images to an image quality improvement engine (first level image quality improvement engine) and a label (hereinafter, counterfeit label) indicating that the images were not captured and acquired by the target imaging device. A machine learning model trained using such training data outputs a counterfeit label when a high-quality image generated by the first level image quality improvement engine is input.

In particular, FIG. 3 shows an example of a CNN configuration for performing authenticity evaluation processing of two-dimensional images. The CNN configuration includes a group of multiple convolution processing blocks 300, each of which is composed of a convolution layer 301, a batch normalization layer 302, and an activation layer 303 using a normalized linear function. The CNN configuration also includes a final convolution layer 304, a fully connected layer 305, and an output layer 306. The fully connected layer 305 fully connects the output value groups of the convolution processing blocks 300. The output layer 306 also uses a sigmoid function to output a value of 1 (true) representing an authentic label or a value of 0 (false) representing a counterfeit label as the result (Result) 307 of the authenticity evaluation processing for the input image Im 310.

Note that by setting the number of convolution processing blocks 300 to 16, the parameters of the convolution layer 301 group to a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters to 64, it is possible to obtain accurate authenticity evaluation results with a certain degree of precision. However, as described in the above explanation of the machine learning model, in practice, a better parameter group can be set using training data according to the usage form of the machine learning model. Note that when three-dimensional or four-dimensional images need to be processed, the filter kernel size may be expanded to three or four dimensions.

The authenticity evaluation engine may output an authenticity label when a high-quality image generated by a high-quality engine (second-level image quality engine) that provides higher image quality than the first-level image quality engine is input. In other words, the authenticity evaluation engine cannot reliably evaluate whether an input image is an image that was captured and acquired by a camera device, but it can evaluate whether the image has the resemblance of an image that was captured and acquired by a camera device. By utilizing this characteristic, a high-quality image generated by a high-quality engine can be input to the authenticity evaluation engine, making it possible to evaluate whether the high-quality image generated by the high-quality engine has been sufficiently enhanced.

The efficiency and accuracy of both engines may be improved by training the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine in cooperation with each other. In this case, first, the machine learning model of the image quality improvement engine is trained so that when the authenticity evaluation engine evaluates a high-quality image generated by the image quality improvement engine, an authenticity label is output. In parallel, the machine learning model of the authenticity evaluation engine is trained so that when the authenticity evaluation engine evaluates an image generated by the image quality improvement engine, an authenticity label is output. In parallel, the machine learning model of the authenticity evaluation engine is trained so that when the authenticity evaluation engine evaluates an image acquired by a photographing device, an authenticity label is output. This improves the efficiency and accuracy of the image quality improvement engine and the authenticity evaluation engine.

First Embodiment
The medical image processing apparatus according to the first embodiment will be described below with reference to Figures 4 and 5. Figure 4 shows an example of a schematic configuration of the image processing apparatus according to this embodiment.

The image processing device 400 is connected to the image capture device 10 and the display unit 20 via a circuit or a network. The image capture device 10 and the display unit 20 may also be directly connected. Note that although these devices are separate devices in this embodiment, some or all of these devices may be configured as an integrated device. Furthermore, these devices may be connected to any other device via a circuit or a network, or may be configured as an integrated device with any other device.

The image processing device 400 is provided with an acquisition unit 401, an imaging condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405 (display control unit). The image processing device 400 may be composed of multiple devices in which some of these components are provided. The acquisition unit 401 can acquire various data and images from the imaging device 10 or other devices, and can acquire input from the examiner via an input device (not shown). The input device may be a mouse, a keyboard, a touch panel, or any other input device. The display unit 20 may also be configured as a touch panel display.

The imaging condition acquisition unit 402 acquires the imaging conditions of the medical image (input image) acquired by the acquisition unit 401. Specifically, the imaging condition group stored in the data structure constituting the medical image is acquired according to the data format of the medical image. Note that if the imaging conditions are not stored in the medical image, it is possible to acquire an imaging information group including the imaging condition group from the imaging device 10 or the image management system via the acquisition unit 401.

The image quality improvement possibility determination unit 403 determines whether the medical image can be handled by the image quality improvement unit 404 using the group of shooting conditions acquired by the shooting condition acquisition unit 402. The image quality improvement unit 404 improves the image quality of the medical image that can be handled, and generates a high-quality image suitable for image diagnosis. The output unit 405 displays the high-quality image generated by the image quality improvement unit 404, the input image, various information, etc. on the display unit 20. The output unit 405 may also store the generated high-quality image, etc. in a storage device (storage unit) connected to the image processing device 400.

Next, the image quality improvement unit 404 will be described in detail. The image quality improvement unit 404 is equipped with an image quality improvement engine. The image quality improvement method equipped with the image quality improvement engine according to this embodiment performs processing using a machine learning algorithm.

In this embodiment, training of a machine learning model related to a machine learning algorithm uses teacher data consisting of a pair group of input data, which is a low-quality image having specific shooting conditions assumed to be processed, and output data, which is a high-quality image corresponding to the input data. Note that the specific shooting conditions specifically include a predetermined shooting part, shooting method, shooting angle of view, image size, etc.

In this embodiment, the input data of the teacher data is a low-quality image acquired by the same model and the same settings as the imaging device 10. The output data of the teacher data is a high-quality image acquired by the settings and image processing of the same model as the imaging device 10. Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by performing superimposition processing such as averaging on a group of images (original images) acquired by multiple shooting. Here, the high-quality image and the low-quality image are explained using the motion contrast data of OCTA as an example. Here, the motion contrast data is data used in OCTA etc., which is obtained by repeatedly shooting the same location of the shooting target and detecting the temporal change of the shooting target between the shootings. At this time, the En-Face image (motion contrast front image) of OCTA can be generated by generating a front image using data of a desired range in the depth direction of the shooting target from the calculated motion contrast data (an example of three-dimensional medical image data). In the following, repeatedly shooting OCT data at the same location is called NOR (Number Of Repeat).

In this embodiment, two different methods for generating high-quality images and low-quality images through overlay processing are described with reference to FIG. 28.

The first method will be described with reference to FIG. 28(a) as an example of a high-quality image, in which motion contrast data is generated from OCT data obtained by repeatedly capturing images of the same location on the subject. In FIG. 28(a), Im2810 shows three-dimensional motion contrast data, and Im2811 shows two-dimensional motion contrast data that constitutes the three-dimensional motion contrast data. Im2811-1 to Im2811-3 show OCT tomographic images (B-scans) used to generate Im2811. Here, NOR refers to the number of OCT tomographic images in Im2811-1 to Im2811-3 in FIG. 28(a), and NOR is 3 in the example shown. Im2811-1 to Im2811-3 are captured at a predetermined time interval (Δt). The same location refers to one line in the front direction (X-Y) of the eye to be examined, and corresponds to the location of Im2811 in FIG. 28(a). The front direction is an example of a direction intersecting with the depth direction. Since the motion contrast data is data that detects a change over time, it is necessary to perform at least two NORs in order to generate this data. For example, when NOR is 2, one motion contrast data is generated. When NOR is 3, two data are generated when the motion contrast data is generated only by OCT of adjacent time intervals (first and second times, second and third times). When the motion contrast data is generated using OCT data of distant time intervals (first and third times), a total of three data are generated. That is, when NOR is increased to three times, four times, etc., the number of motion contrast data at the same location also increases. By aligning multiple motion contrast data obtained by repeatedly photographing the same location and performing superposition processing such as averaging, it is possible to generate high-quality motion contrast data. Therefore, NOR is set to at least three times, and preferably to five times or more. On the other hand, an example of a corresponding low-quality image is motion contrast data before performing superimposition processing such as averaging. In this case, it is preferable to use the low-quality image as a reference image when performing superimposition processing such as averaging. When performing superimposition processing, if the position and shape of the target image are deformed with respect to the reference image to perform positioning, there is almost no spatial positional deviation between the reference image and the image after superimposition processing. Therefore, it is easy to pair a low-quality image with a high-quality image. Note that the target image that has been subjected to image deformation processing for positioning, rather than the reference image, may be used as the low-quality image. By using each of the original image groups (reference image and target image) as input data and the corresponding superimposed image as output data, multiple pair groups can be generated. For example, when obtaining one superimposed image from 15 original image groups, a pair of the first original image and the superimposed image in the original image group and a pair of the second original image and the superimposed image in the original image group can be generated. In this way, when obtaining one overlaid image from a group of 15 original images, it is possible to generate a group of 15 pairs of one image from the group of original images and the overlaid image. It is also possible to generate high-quality three-dimensional data by repeatedly photographing the same location in the main scanning (X) direction and then scanning while shifting the images in the sub-scanning (Y) direction.

The second method is a process for generating a high-quality image by superimposing motion contrast data obtained by photographing the same area of the subject multiple times, and is described with reference to FIG. 28(b). The same area refers to an area of 3×3 mm or 10×10 mm in the front direction (X-Y) of the subject's eye, and means obtaining three-dimensional motion contrast data including the depth direction of the tomographic image. When photographing the same area multiple times and performing superimposing, it is desirable to perform NOR two or three times in order to shorten each photographing. In addition, in order to generate high-quality three-dimensional motion contrast data, at least two or more three-dimensional data of the same area are obtained. FIG. 28(b) shows an example of multiple three-dimensional motion contrast data. Im2820 to Im2840 are three-dimensional motion contrast data similar to those described in FIG. 28(a). Using these two or more three-dimensional motion contrast data, alignment processing is performed in the front direction (X-Y) and the depth direction (Z), and after removing data that becomes artifacts in each data, averaging processing is performed. This makes it possible to generate a single piece of high-quality three-dimensional motion contrast data from which artifacts have been removed. A high-quality image is obtained by generating an arbitrary plane from the three-dimensional motion contrast data. On the other hand, it is desirable for the corresponding low-quality image to be an arbitrary plane generated from reference data when performing superposition processing such as averaging. As explained in the first method, there is almost no spatial misalignment between the reference image and the image after averaging, so a low-quality image and a high-quality image can easily be paired. Note that the low-quality image may be an arbitrary plane generated from target data that has been subjected to image deformation processing for alignment, rather than the reference data.

The first method places less of a burden on the subject, as the shooting itself is completed in one go. However, the more NORs are used, the longer each shooting session takes. Also, if artifacts such as clouding of the eye or eyelashes appear during shooting, a good image may not necessarily be obtained. The second method requires multiple shots, which places a slightly greater burden on the subject. However, the time required for each shot is shorter, and even if an artifact appears in one shot, as long as the artifact is not visible in another shot, a clean image with fewer artifacts can ultimately be obtained. Taking these characteristics into consideration, an arbitrary method is selected according to the subject's situation when collecting data.

In this embodiment, the motion contrast data has been described as an example, but the present invention is not limited to this. Since OCT data is captured to generate motion contrast data, the same method can be used with OCT data as described above. Furthermore, although the description of the tracking process has been omitted in this embodiment, in order to capture images of the same location or area of the subject's eye, it is desirable to capture images while tracking the subject's eye.

In this embodiment, since a pair of three-dimensional high-quality data and low-quality data is created, any pair of two-dimensional images can be generated from the pair. This will be explained with reference to FIG. 29. For example, when the target image is an OCTA En-Face image, the OCTA En-Face image is generated in the desired depth range from the three-dimensional data. The desired depth range refers to the range in the Z direction in FIG. 28. An example of the OCTA En-Face image generated here is shown in FIG. 29(a). As the OCTA En-Face image, learning is performed using OCTA En-Face images generated in different depth ranges, such as the superficial layer (Im2910), deep layer (Im2920), outer layer (Im2930), and choroidal vascular network (Im2940). The types of OCTA En-Face images are not limited to these, and the types may be increased by generating OCTA En-Face images with different depth ranges set by changing the reference layer and offset value. When learning, learning may be performed separately for each OCTA En-Face image of different depths, or learning may be performed by combining multiple images of different depth ranges (for example, dividing them into superficial and deep layers), or OCTA En-Face images of all depth ranges may be learned together. In the case of a luminance En-Face image generated from OCT data, learning is performed using multiple En-Face images generated from any depth range, similar to the OCTA En-Face. For example, consider a case where the image quality improvement engine includes a machine learning engine obtained using learning data including multiple motion contrast front images corresponding to different depth ranges of the subject's eye. At this time, the acquisition unit can acquire, as the first image, a motion contrast front image corresponding to a part of a long depth range including different depth ranges. That is, a motion contrast front image corresponding to a depth range different from the multiple depth ranges corresponding to the multiple motion contrast front images included in the learning data can be used as an input image for image enhancement. Of course, a motion contrast front image having the same depth range as that during learning may be used as an input image for image enhancement. In addition, the part of the depth range may be set by the examiner pressing an arbitrary button on the user interface, or may be set automatically. Note that the above content is not limited to the motion contrast front image, and can also be applied to, for example, a luminance En-Face image.

When the image to be processed is a tomographic image, learning is performed using an OCT tomographic image, which is a B-scan, or a tomographic image of motion contrast data. This will be described with reference to FIG. 29(b). In FIG. 29(b), Im2951 to Im2953 are OCT tomographic images. The images are different in FIG. 29(b) because they show tomographic images at different positions in the sub-scanning (Y) direction. For tomographic images, learning may be performed together without considering the difference in position in the sub-scanning direction. However, in the case of images taken at different locations (for example, the center of the macula, the center of the optic disc), learning may be performed separately for each location, or learning may be performed together without considering the location. Note that the image features of OCT tomographic images and tomographic images of motion contrast data are significantly different, so it is better to learn them separately.

The overlaid image created by the overlay process emphasizes pixels that are commonly depicted in the original images, resulting in a high-quality image suitable for image diagnosis. In this case, the high-quality image generated is a high-contrast image with a clear distinction between low-brightness and high-brightness areas, as a result of emphasizing the commonly depicted pixels. In addition, for example, in the overlaid image, random noise that occurs with each capture can be reduced, and areas that were not well depicted in the original images at a certain point in time can be interpolated using other original images.

In addition, when the input data for a machine learning model needs to be composed of multiple images, the required number of original image groups can be selected from the original image group and used as input data. For example, when obtaining one overlaid image from a group of 15 original images, if two images are required as input data for the machine learning model, 105 (15C2=105) pair groups can be generated.

Note that, among the pairs that make up the training data, pairs that do not contribute to high image quality can be removed from the training data. For example, if the high-image-quality image that is the output data that makes up a pair of training data has image quality that is unsuitable for image diagnosis, the image output by the image quality improvement engine that has learned using the training data may also have image quality that is unsuitable for image diagnosis. Therefore, by removing pairs whose output data has image quality that is unsuitable for image diagnosis from the training data, the possibility that the image quality improvement engine will generate an image with image quality that is unsuitable for image diagnosis can be reduced.

In addition, if the average luminance or luminance distribution of a pair of images differs significantly, the image quality improvement engine that has learned using the training data may output an image that is not suitable for image diagnosis, with a luminance distribution that differs significantly from the low-quality image. For this reason, it is possible to remove pairs of input data and output data with significantly different average luminance or luminance distribution from the training data.

Furthermore, if the structure or position of the subject depicted in the paired images differs significantly, the image quality improvement engine that has learned using the training data may output an image that is not suitable for image diagnosis, in which the subject is depicted in a structure or position that differs significantly from that of the low-quality image. For this reason, it is possible to remove from the training data pairs of input data and output data in which the structure or position of the depicted subject differs significantly. Furthermore, from the perspective of maintaining quality, the image quality improvement engine can be configured not to use the high-quality images it outputs as training data.

By using an image quality improvement engine that has performed machine learning in this way, when a medical image acquired in a single shot is input, the image quality improvement unit 404 can output a high-image quality image in which contrast has been increased and noise has been reduced by overlay processing. Therefore, the image quality improvement unit 404 can generate a high-image quality image suitable for image diagnosis based on a low-image quality image that is the input image.

Next, a series of image processing steps according to this embodiment will be described with reference to the flow diagram in FIG. 5. FIG. 5 is a flow diagram of a series of image processing steps according to this embodiment. First, when a series of image processing steps according to this embodiment is started, the process proceeds to step S510.

In step S510, the acquisition unit 401 acquires an image captured by the imaging device 10 as an input image from the imaging device 10 connected via a circuit or network. The acquisition unit 401 may acquire the input image in response to a request from the imaging device 10. Such a request may be issued, for example, when the imaging device 10 generates an image, before or after the image generated by the imaging device 10 is saved in a storage device provided in the imaging device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

The acquisition unit 401 may acquire data for generating an image from the imaging device 10, and acquire an image generated by the image processing device 400 based on the data as an input image. In this case, any existing image generation method may be adopted as the image generation method for the image processing device 400 to generate various images.

In step S520, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image. Specifically, the shooting condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. Note that, as described above, if the shooting conditions are not stored in the input image, the shooting condition acquisition unit 402 can acquire a group of shooting information including the group of shooting conditions from the shooting device 10 or an image management system (not shown).

In step S530, the image quality improvement possibility determination unit 403 uses the acquired group of shooting conditions to determine whether the input image can be improved in image quality by the image quality improvement engine provided in the image quality improvement unit 404. Specifically, the image quality improvement possibility determination unit 403 determines whether the shooting part, shooting method, shooting angle of view, and image size of the input image match the conditions that can be handled by the image quality improvement engine.

When the image quality improvement feasibility determination unit 403 has determined all of the shooting conditions and determined that they can be handled, the process proceeds to step S540. On the other hand, when the image quality improvement feasibility determination unit 403 has determined that the image quality improvement engine cannot handle the input image based on these shooting conditions, the process proceeds to step S550.

Depending on the settings and implementation of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, imaging angle of view, and image size. For example, such processing may be performed when the image quality improvement engine is assumed to be comprehensively compatible with any imaging part of the subject and is implemented to be able to handle even unknown imaging parts included in the input data. Furthermore, the image quality improvement feasibility determination unit 403 may determine whether or not at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image matches a condition that can be handled by the image quality improvement engine, depending on the desired configuration.

In step S540, the image quality improvement unit 404 uses the image quality improvement engine to improve the image quality of the input image and generate a high-image quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to the image quality improvement engine, which generates a high-image quality image with improved image quality. The image quality improvement engine generates a high-image quality image that is similar to an overlay process using the input image, based on a machine learning model that has performed machine learning using training data. Therefore, the image quality improvement engine can generate a high-image quality image with reduced noise and enhanced contrast compared to the input image.

Depending on the settings and implementation of the image processing device 400, the image quality improvement unit 404 may input parameters to the image quality improvement engine together with the input image according to the group of shooting conditions to adjust the degree of image quality improvement, etc. Also, the image quality improvement unit 404 may input parameters according to the examiner's input to the image quality improvement engine together with the input image to adjust the degree of image quality improvement, etc.

In step S550, if a high-quality image was generated in step S540, the output unit 405 outputs the high-quality image and displays it on the display unit 20. On the other hand, if it is determined in step S530 that high-quality processing is not possible, the input image is output and displayed on the display unit 20. Note that instead of displaying the output image on the display unit 20, the output unit 405 may cause the image capture device 10 or another device to display or store the output image. Depending on the settings and implementation form of the image processing device 400, the output unit 405 may also process the output image so that it can be used by the image capture device 10 or another device, or convert the data format so that it can be transmitted to an image management system, etc.

As described above, the image processing device 400 according to this embodiment includes an acquisition unit 401 and an image quality improvement unit 404. The acquisition unit 401 acquires an input image (first image) that is an image of a specific part of a subject. The image quality improvement unit 404 uses an image quality improvement engine including a machine learning engine to generate a high image quality image (second image) from the input image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image. The image quality improvement engine includes a machine learning engine that uses the image obtained by the overlay process as learning data.

With this configuration, the image processing device 400 according to this embodiment can output a high-quality image in which noise has been reduced and contrast has been enhanced from the input image. As a result, the image processing device 400 can obtain images suitable for image diagnosis, such as clearer images and images in which the area or lesion to be observed is emphasized, at a lower cost than in the past, without increasing the invasiveness or effort of the photographer or subject.

The image processing device 400 further includes a high image quality determination unit 403 that determines whether a high image quality image can be generated for an input image using a high image quality engine. The high image quality determination unit 403 makes this determination based on at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image.

With this configuration, the image processing device 400 according to this embodiment can exclude input images that cannot be processed by the image quality improvement unit 404 from the image quality improvement process, thereby reducing the processing load on the image processing device 400 and the occurrence of errors.

In this embodiment, the output unit 405 (display control unit) is configured to display the generated high-quality image on the display unit 20, but the operation of the output unit 405 is not limited to this. For example, the output unit 405 can also output the high-quality image to the imaging device 10 or other devices connected to the image processing device 400. Therefore, the high-quality image can be displayed on the user interface of these devices, saved in any storage device, used for any image analysis, or sent to an image management system.

In this embodiment, the image quality improvement possibility determination unit 403 determines whether the input image can be improved by the image quality improvement engine, and if the input image can be improved, the image quality improvement unit 404 improves the image quality. On the other hand, if the image capturing device 10 captures images only under shooting conditions that allow for high image quality improvement, the image obtained from the image capturing device 10 may be unconditionally improved in image quality. In this case, as shown in FIG. 6, the processes of steps S520 and S530 can be omitted, and step S540 can be performed after step S510.

In this embodiment, the output unit 405 is configured to display a high-quality image on the display unit 20. However, the output unit 405 may also display a high-quality image on the display unit 20 in response to an instruction from the examiner. For example, the output unit 405 may display a high-quality image on the display unit 20 in response to the examiner pressing an arbitrary button on the user interface of the display unit 20. In this case, the output unit 405 may display the high-quality image in switchover with the input image, or may display the high-quality image alongside the input image.

Furthermore, when the output unit 405 displays a high-quality image on the display unit 20, the output unit 405 may display, together with the high-quality image, a display indicating that the displayed image is a high-quality image generated by processing using a machine learning algorithm. In this case, the display allows the user to easily identify that the displayed high-quality image is not the image itself obtained by shooting, thereby reducing misdiagnosis and improving diagnostic efficiency. Note that the display indicating that the high-quality image is generated by processing using a machine learning algorithm may be in any form as long as it allows the user to distinguish between the input image and the high-quality image generated by the processing.

The output unit 405 may also cause the display unit 20 to display, in addition to a display indicating that the image is a high-quality image generated by processing using a machine learning algorithm, a display indicating what kind of training data the machine learning algorithm used to learn. Such a display may include an explanation of the types of input data and output data of the training data, or any display regarding the training data, such as the imaging site included in the input data and output data.

In the image quality improvement engine according to this embodiment, a superimposed image is used as output data for the teacher data, but the teacher data is not limited to this. As output data for the teacher data, a high-quality image obtained by performing at least one of a superimposition process, a group of processes described below, or an imaging method described below, which are means for obtaining a high-quality image, may be used.

For example, high-quality images obtained by performing maximum a posteriori probability estimation processing (MAP estimation processing) on a group of original images may be used as output data for the training data. In the MAP estimation processing, a likelihood function is calculated from the probability density of each pixel value in multiple low-quality images, and the true signal value (pixel value) is estimated using the calculated likelihood function.

The high-quality image obtained by the MAP estimation process is a high-contrast image based on pixel values close to the true signal value. In addition, since the estimated signal value is calculated based on probability density, randomly occurring noise is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained by the MAP estimation process as training data, the image quality improvement engine can generate a high-quality image suitable for image diagnosis, with reduced noise and high contrast, from the input image. Note that the method for generating pairs of input data and output data for the training data may be the same as when an overlaid image is used as training data.

In addition, a high-quality image obtained by applying a smoothing filter process to the original image may be used as the output data of the training data. In this case, the image quality improvement engine can generate a high-quality image in which random noise has been reduced from the input image. Furthermore, an image obtained by applying a tone conversion process to the original image may be used as the output data of the training data. In this case, the image quality improvement engine can generate a high-quality image in which contrast has been enhanced from the input image. Note that the method of generating pairs of input data and output data of the training data may be the same as when an overlaid image is used as the training data.

The input data of the teacher data may be an image obtained from a photographing device having the same image quality tendency as the photographing device 10. The output data of the teacher data may be a high-quality image obtained by a high-cost process such as successive approximation, or may be a high-quality image obtained by photographing a subject corresponding to the input data with a photographing device with higher performance than the photographing device 10. Furthermore, the output data may be a high-quality image obtained by performing a rule-based noise reduction process. Here, the noise reduction process may include, for example, a process of replacing only one high-luminance pixel that is clearly noise and appears in a low-luminance area with the average value of nearby low-luminance pixels. For this reason, the image quality improvement engine may use as learning data an image taken with a photographing device with higher performance than the photographing device used to photograph the input image, or an image obtained in a photographing process with more labor than the photographing process of the input image. For example, when a motion contrast front image is used as an input image, the image quality improvement engine may use as learning data an image obtained by OCTA imaging using an OCT imaging device with higher performance than the OCT imaging device used for OCTA imaging of the input image, or an image obtained in an OCTA imaging process that requires more labor than the OCTA imaging process of the input image.

Although not described in the present embodiment, a high-quality image generated from multiple images and used as output data for training data can be generated from multiple images that have already been aligned. The alignment process may involve, for example, selecting one of the multiple images as a template, determining the similarity with other images while changing the position and angle of the template, determining the amount of misalignment with the template, and correcting each image based on the amount of misalignment. Any other existing alignment process may also be performed.

When aligning a three-dimensional image, the three-dimensional image may be decomposed into multiple two-dimensional images, and the two-dimensional images may be aligned and integrated together to align the three-dimensional image. The two-dimensional image may be decomposed into one-dimensional images, and the one-dimensional images may be aligned together to align the two-dimensional image. The alignment may be performed on the data used to generate the images, rather than on the images themselves.

In addition, in this embodiment, when the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement unit 404, the process proceeds to step S540, and the image quality improvement process by the image quality improvement unit 404 is started. In response to this, the output unit 405 may display the determination result by the image quality improvement possibility determination unit 403 on the display unit 20, and the image quality improvement unit 404 may start the image quality improvement process in response to an instruction from the examiner. At this time, the output unit 405 may display the input image and the imaging conditions such as the imaging part acquired for the input image, together with the determination result, on the display unit 20. In this case, the image quality improvement process is performed after the examiner determines whether the determination result is correct or not, so that the image quality improvement process based on an erroneous determination can be reduced.

In addition, instead of making a judgment by the image quality improvement possibility judgment unit 403, the output unit 405 may display the input image and the imaging conditions such as the imaging part acquired for the input image on the display unit 20, and the image quality improvement unit 404 may start the image quality improvement process in response to an instruction from the examiner.

Second Embodiment
Next, an image processing apparatus according to a second embodiment will be described with reference to Figs. 4 and 7. In the first embodiment, the image quality improvement unit 404 includes one image quality improvement engine. In contrast, in this embodiment, the image quality improvement unit includes multiple image quality improvement engines that perform machine learning using different teacher data, and generates multiple high-image-quality images for an input image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with two or more image quality improvement engines that have undergone machine learning using different training data. Here, a method for creating a training data group according to this embodiment will be described. Specifically, first, a group of pairs of original images as input data and superimposed images as output data, in which various imaging sites are imaged, is prepared. Next, the pair groups are grouped for each imaging site to create a training data group. For example, a training data group is created in such a way that a first training data group is made up of a pair group acquired by imaging a first imaging site, and a second training data group is made up of a pair group acquired by imaging a second imaging site.

Then, separate image quality improvement engines are made to perform machine learning using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, and a second image quality improvement engine corresponding to a machine learning model trained with the second training data.

Since each of these image quality improvement engines uses different teacher data to train the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine provides a high degree of image quality improvement for the input image acquired by shooting the first shooting part, and a low degree of image quality improvement for the image acquired by shooting the second shooting part. Similarly, the second image quality improvement engine provides a high degree of image quality improvement for the input image acquired by shooting the second shooting part, and a low degree of image quality improvement for the image acquired by shooting the first shooting part.

Since each piece of training data is composed of pairs grouped by the imaging location, the image quality tendencies of the images constituting the pair groups are similar. Therefore, the image quality improvement engine can improve image quality more effectively than the image improvement engine according to the first embodiment for corresponding imaging locations. Note that the imaging conditions for grouping pairs of training data are not limited to the imaging location, and may be the imaging angle of view, image resolution, or a combination of two or more of these.

Below, a series of image processing according to this embodiment will be described with reference to FIG. 7. FIG. 7 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S710 and S720 is similar to steps S510 and S520 according to the first embodiment, and therefore a description thereof will be omitted. Note that if the image quality of the input image is to be unconditionally improved, after the processing of step S720, the processing of step S730 may be omitted and the processing may proceed to step S740.

When the shooting conditions of the input image are acquired in step S720, the process proceeds to step S730. In step S730, the image quality improvement feasibility determination unit 403 uses the group of shooting conditions acquired in step S720 to determine whether any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

If the image quality improvement capability determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S760. On the other hand, if the image quality improvement capability determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S740. Note that, depending on the settings and implementation form of the image processing device 400, step S740 may be performed, as in the first embodiment, even if the image quality improvement engines have determined that some shooting conditions cannot be handled.

In step S740, the image quality improvement unit 404 selects an image quality improvement engine that performs image quality improvement processing from the image quality improvement engine group based on the shooting conditions of the input image acquired in step S720 and the teacher data information of the image quality improvement engine group. Specifically, for example, for an image shooting part in the shooting condition group acquired in step S720, an image quality improvement engine that has teacher data information related to the same image shooting part or surrounding image shooting parts and has a high degree of image quality improvement is selected. In the above example, when the image shooting part is the first image shooting part, the image quality improvement unit 404 selects the first image quality improvement engine.

In step S750, the image quality improvement unit 404 uses the image quality improvement engine selected in step S740 to generate a high-quality image by improving the image quality of the input image. After that, in step S760, if a high-quality image was generated in step S750, the output unit 405 outputs the high-quality image and displays it on the display unit 20. On the other hand, if it is determined in step S730 that image quality improvement processing is not possible, the input image is output and displayed on the display unit 20. Note that when displaying the high-quality image on the display unit 20, the output unit 405 may display that the high-quality image was generated using the image quality improvement engine selected by the image quality improvement unit 404.

As described above, the image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines that have been trained using different learning data. Here, each of the plurality of image quality improvement engines has been trained using different learning data for at least one of the imaging part, imaging angle of view, front images at different depths, and image resolution. The image quality improvement unit 404 generates a high-quality image using an image quality improvement engine that corresponds to at least one of the imaging part, imaging angle of view, front images at different depths, and image resolution of the input image.

With this configuration, the image processing device 400 according to this embodiment can generate high-quality images more effectively.

In this embodiment, the image quality improvement unit 404 selects an image quality improvement engine to be used for image quality improvement processing based on the shooting conditions of the input image, but the image quality improvement engine selection process is not limited to this. For example, the output unit 405 may display the shooting conditions of the acquired input image and the image quality improvement engine group on the user interface of the display unit 20, and the image quality improvement unit 404 may select an image quality improvement engine to be used for image quality improvement processing in response to an instruction from the examiner. The output unit 405 may display information on the teacher data used to learn each image quality improvement engine along with the image quality improvement engine group on the display unit 20. The display format of the information on the teacher data used to learn the image quality improvement engine may be arbitrary, and for example, the image quality improvement engine group may be displayed using a name related to the teacher data used for learning.

The output unit 405 may also display the image quality improvement engine selected by the image quality improvement unit 404 on the user interface of the display unit 20 and accept instructions from the examiner. In this case, the image quality improvement unit 404 may determine, in response to instructions from the examiner, whether or not to ultimately select the image quality improvement engine as the image quality improvement engine to be used for image quality improvement processing.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Third Embodiment
Next, an image processing device according to a third embodiment will be described with reference to Figs. 4 and 7. In the first and second embodiments, the imaging condition acquisition unit 402 acquires an imaging condition group from the data structure of an input image, etc. In contrast, in this embodiment, the imaging condition acquisition unit estimates an imaging part or imaging area of an input image based on the input image using an imaging location estimation engine.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment is the same as the image processing device 400 according to the second embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the second embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first and second embodiments, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The imaging condition acquisition unit 402 according to this embodiment is equipped with an imaging location estimation engine that estimates the imaging part or imaging area depicted in the input image acquired by the acquisition unit 401. The imaging location estimation method provided by the imaging location estimation engine according to this embodiment performs estimation processing using a machine learning algorithm.

In this embodiment, training of a machine learning model related to a method for estimating an imaging location using a machine learning algorithm uses teacher data consisting of pairs of input data, which is an image, and output data, which is imaging site labels or imaging area labels corresponding to the input data. Here, the input data refers to an image having specific imaging conditions that is assumed to be the processing target (input image). The input data is preferably an image acquired from an imaging device having the same image quality tendency as the imaging device 10, and is preferably the same model with the same settings as the imaging device 10. The types of imaging site labels and imaging area labels, which are output data, may be imaging sites and imaging areas at least partially included in the input data. For example, in the case of OCT, the types of imaging site labels, which are output data, may be "macular region", "optic disc", "macular region and optic disc", and "others", etc.

The shooting location estimation engine according to this embodiment can output the location of the shooting part or shooting area depicted in the input image by learning using such teacher data. In addition, the shooting location estimation engine can also output the probability of the shooting part or shooting area for each shooting part label or shooting area label of the required level of detail. By using the shooting location estimation engine, the shooting condition acquisition unit 402 can estimate the shooting part or shooting area of the input image based on the input image and acquire it as the shooting condition for the input image. Note that when the shooting location estimation engine outputs the probability of the shooting part or shooting area for each shooting part label or shooting area label, the shooting condition acquisition unit 402 acquires the shooting part or shooting area with the highest probability as the shooting condition of the input image.

Next, as in the second embodiment, a series of image processing steps according to this embodiment will be described with reference to the flow diagram in FIG. 7. Note that the processing steps S710 and steps S730 to S760 according to this embodiment are similar to those steps in the second embodiment, and therefore will not be described. Note that if the image quality of the input image is to be unconditionally improved, after the processing step S720, the processing step S730 may be omitted and the processing may proceed to step S740.

When the input image is acquired in step S710, the process proceeds to step S720. In step S720, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image acquired in step S710.

Specifically, the imaging condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. Furthermore, if the imaging condition group does not include information on the imaging part or imaging area, the imaging condition acquisition unit 402 inputs the input image to an imaging location estimation engine and estimates which imaging part was imaged to acquire the input image. Specifically, the imaging condition acquisition unit 402 inputs the input image to an imaging location estimation engine, evaluates the probability output for each imaging part label group, and sets and acquires the imaging part with the highest probability as the imaging condition for the input image.

If no imaging conditions other than the imaging part and imaging area are stored in the input image, the imaging condition acquisition unit 402 can acquire a group of imaging information including a group of imaging conditions from the imaging device 10 or an image management system (not shown).

The subsequent processing is similar to the series of image processing in the second embodiment, so a detailed explanation is omitted.

As described above, the imaging condition acquisition unit 402 according to this embodiment functions as an estimation unit that estimates at least one of the imaging part and imaging area of the input image. The imaging condition acquisition unit 402 includes an imaging location estimation engine that uses images labeled with the imaging part and imaging area as learning data, and estimates the imaging part and imaging area of the input image by inputting the input image to the imaging location estimation engine.

As a result, the image processing device 400 according to this embodiment can obtain the shooting conditions for the shooting part and shooting area of the input image based on the input image.

In this embodiment, the shooting condition acquisition unit 402 estimates the shooting part and shooting area of the input image using the shooting location estimation engine when the shooting condition group does not include information on the shooting part and shooting area. However, the situation in which the shooting part and shooting area are estimated using the shooting location estimation engine is not limited to this. The shooting condition acquisition unit 402 may estimate the shooting part and shooting area using the shooting location estimation engine even when information on the shooting part and shooting area included in the data structure of the input image is insufficient at the required level of detail.

In addition, the imaging condition acquisition unit 402 may estimate the imaging part and imaging area of the input image using the imaging location estimation engine, regardless of whether the data structure of the input image includes information on the imaging part and imaging area. In this case, the output unit 405 may display the estimation result output from the imaging location estimation engine and information on the imaging part and imaging area included in the data structure of the input image on the display unit 20, and the imaging condition acquisition unit 402 may determine these imaging conditions according to the examiner's instructions.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Fourth Embodiment
Next, an image processing device according to a fourth embodiment will be described with reference to Figures 4, 5, 8 and 9. In this embodiment, an image quality improvement unit enlarges or reduces an input image so that the input image has an image size that the image quality improvement engine can handle. The image quality improvement unit also reduces or enlarges an output image from the image quality improvement engine so that the image size of the output image becomes the image size of the input image, thereby generating a high-image quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to the image quality improvement engine according to the first embodiment. However, in this embodiment, as the teacher data used for learning the image quality improvement engine, a group of pairs of input data and output data, which are composed of a group of images in which the input data images and the output data images are enlarged or reduced to a certain image size, are used.

Now, referring to FIG. 8, the teacher data of the image quality improvement engine according to this embodiment will be described. As shown in FIG. 8, for example, consider the case where there is a low-quality image Im810 and a high-quality image Im820 that are smaller than a certain image size set for the teacher data. In this case, the low-quality image Im810 and the high-quality image Im820 are each enlarged so that they become the certain image size set for the teacher data. Then, the enlarged low-quality image Im811 and the enlarged high-quality image Im821 are paired, and the pair is used as one piece of teacher data.

As in the first embodiment, the input data for the training data uses an image with specific shooting conditions that are assumed to be the processing target (input image), and the specific shooting conditions are a predetermined shooting part, shooting method, and shooting angle of view. In other words, unlike the first embodiment, the specific shooting conditions in this embodiment do not include image size.

The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image and generate a high-image quality image. At this time, the image quality improvement unit 404 generates a deformed image by enlarging or reducing the input image so that it has a certain image size set for the teacher data, and inputs the deformed image to the image quality improvement engine. The image quality improvement unit 404 also reduces or enlarges the output image from the image quality improvement engine so that it has the image size of the input image, generating a high-image quality image. Therefore, the image quality improvement unit 404 according to this embodiment can use the image quality improvement engine to improve the image quality of even an input image of an image size that could not be handled in the first embodiment, and generate a high-image quality image.

Next, a series of image processing steps according to this embodiment will be described with reference to Figs. 5 and 9. Fig. 9 is a flow diagram of the image quality improvement processing according to this embodiment. Note that the processing steps S510, S520, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to shooting conditions other than image size, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S520, as in the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

If the image quality improvement feasibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in FIG. 9 is started. In the image quality improvement process according to this embodiment, first, in step S910, the image quality improvement unit 404 enlarges or reduces the input image to a certain image size set for the training data, and generates a deformed image.

Next, in step S920, the image quality improvement unit 404 inputs the generated deformed image to an image quality improvement engine to obtain a high-image-quality deformed image.

Then, in step S930, the image quality improvement unit 404 reduces or enlarges the high-quality transformed image to the image size of the input image to generate a high-quality image. Once the image quality improvement unit 404 generates a high-quality image in step S930, the image quality improvement process according to this embodiment ends, and the process proceeds to step S550. The process of step S550 is similar to step S550 in the first embodiment, and therefore a description thereof will be omitted.

As described above, the image quality improvement unit 404 according to this embodiment adjusts the image size of the input image to an image size that the image quality improvement engine can handle, and inputs the image to the image quality improvement engine. The image quality improvement unit 404 also generates a high-quality image by adjusting the output image from the image quality improvement engine to the original image size of the input image. As a result, the image processing device 400 according to this embodiment can use the image quality improvement engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images suitable for image diagnosis.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Fifth embodiment
Next, an image processing device according to a fifth embodiment will be described with reference to Figures 4, 5, 10 and 11. In this embodiment, an image quality improvement unit generates a high-quality image by image quality improvement processing using an image quality improvement engine with a certain resolution as a reference.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, in this embodiment, the training data used for training the image quality improvement engine is different from the training data in the first embodiment. Specifically, the image group constituting the pair group of input data and output data of the training data is enlarged or reduced to an image size such that the resolution of the image group becomes a certain resolution, and then padding is performed to obtain a sufficiently large and constant image size. Here, the resolution of the image group refers to, for example, the spatial resolution of the imaging device or the resolution for the imaging area.

Now, referring to FIG. 10, the teacher data of the image quality improvement engine according to this embodiment will be described. As shown in FIG. 10, for example, consider the case where there is a low-quality image Im1010 and a high-quality image Im1020, both of which have a resolution lower than the fixed resolution set for the teacher data. In this case, the low-quality image Im1010 and the high-quality image Im1020 are each enlarged so that they have the fixed resolution set for the teacher data. Furthermore, the enlarged low-quality image Im1010 and the high-quality image Im1020 are each padded so that they have the fixed image size set for the teacher data. Then, the low-quality image Im1011 and the high-quality image Im1021 that have been enlarged and padded are paired, and the pair is used as one piece of teacher data.

The fixed image size set for the training data is the maximum possible image size when the image expected to be processed (input image) is enlarged or reduced to a fixed resolution. If the fixed image size is not large enough, when the image input to the image quality improvement engine is enlarged, the image size may become too large for the machine learning model to handle.

In addition, the area to be padded is filled with a fixed pixel value, filled with nearby pixel values, or mirror padded according to the characteristics of the machine learning model so as to effectively improve image quality. As in the first embodiment, the input data uses images with specific shooting conditions that are expected to be processed, and the specific shooting conditions are a predetermined shooting part, shooting method, and shooting angle of view. In other words, unlike the first embodiment, the specific shooting conditions in this embodiment do not include image size.

The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image to generate a high-quality image. At this time, the image quality improvement unit 404 generates a deformed image by enlarging or reducing the input image so that it has a certain resolution set for the teacher data. The image quality improvement unit 404 also pads the deformed image so that it has a certain image size set for the teacher data to generate a padded image, and inputs the padded image to the image quality improvement engine.

The image quality improvement unit 404 also trims the high-quality padded image output from the image quality improvement engine by the area where padding was performed, to generate a high-quality deformed image. The image quality improvement unit 404 then reduces or enlarges the generated high-quality deformed image to the image size of the input image, to generate a high-quality image.

For this reason, the image quality improvement unit 404 according to this embodiment can generate a high-quality image by improving the image quality using the image quality improvement engine, even for input images with image sizes that could not be handled in the first embodiment.

Next, a series of image processing steps according to this embodiment will be described with reference to Figs. 5 and 11. Fig. 11 is a flow diagram of the image quality improvement processing according to this embodiment. Note that the processing steps S510, S520, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to shooting conditions other than image size, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S520, as in the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

If the image quality improvement feasibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in FIG. 11 is started. In the image quality improvement process according to this embodiment, first, in step S1110, the image quality improvement unit 404 enlarges or reduces the input image to a certain resolution set for the training data, and generates a deformed image.

Next, in step S1120, the image quality improvement unit 404 performs padding on the generated transformed image so that the image size is the same as the image size set for the training data, to generate a padded image. At this time, the image quality improvement unit 404 fills the area to be padded with a fixed pixel value, fills with a neighboring pixel value, or performs mirror padding in accordance with the characteristics of the machine learning model so as to effectively improve the image quality.

In step S1130, the image quality improving unit 404 inputs the padding image to an image quality improving engine to obtain a high-image-quality padding image.
Next, in step S1140, the image quality improving unit 404 performs trimming on the high-image-quality padded image by the area that was padded in step S1120, and generates a high-image-quality transformed image.

Then, in step S1150, the image quality improvement unit 404 reduces or enlarges the high-quality transformed image to the image size of the input image to generate a high-quality image. Once the image quality improvement unit 404 generates a high-quality image in step S1130, the image quality improvement process according to this embodiment ends, and the process proceeds to step S550. The process of step S550 is similar to step S550 in the first embodiment, and therefore a description thereof will be omitted.

As described above, the image quality improvement unit 404 according to this embodiment adjusts the image size of the input image so that the resolution of the input image becomes a predetermined resolution. Furthermore, for the input image whose image size has been adjusted, the image quality improvement unit 404 generates a padded image by padding so that the adjusted image size becomes an image size that can be handled by the image quality improvement engine, and inputs the padded image to the image quality improvement engine. The image quality improvement unit 404 then trims the output image from the image quality improvement engine by the amount of the padded area. The image quality improvement unit 404 then generates a high-quality image by adjusting the image size of the trimmed image to the original image size of the input image.

As a result, the image quality improvement unit 404 of this embodiment can use the image quality improvement engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images. Furthermore, by using an image quality improvement engine that has learned using training data based on resolution, it may be possible to improve the image quality of input images more efficiently than the image quality improvement engine of the fourth embodiment, which simply processes images of the same image size.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Sixth Embodiment
Next, an image processing apparatus according to a sixth embodiment will be described with reference to Figures 4, 5, 12 and 13. In this embodiment, an image quality improving unit improves the image quality of an input image for each region of a certain image size, thereby generating a high-quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, in this embodiment, the teacher data used for training the image quality improvement engine is different from the teacher data in the first embodiment. Specifically, pairs of input data, which is a low-quality image, and output data, which is a high-quality image, constituting the teacher data are constituted by rectangular area images of a certain image size in which the positional relationship in the low-quality image and the high-quality image corresponds. Note that the rectangular area is an example of a partial area, and does not have to be rectangular and may be any shape.

Now, referring to FIG. 12, the teacher data of the image quality improvement engine according to this embodiment will be described. As shown in FIG. 12, consider a case where one of the pairs constituting the teacher data includes, for example, an original image Im1210, which is a low-image quality image, and an overlaid image Im1220, which is a high-image quality image. In this case, in the first embodiment, the input data of the teacher data is Im1210, and the output data is Im1220.

In contrast, in this embodiment, the rectangular area image R1211 of the original image Im1210 is used as input data, and the rectangular area image R1221, which is the same shooting area as the rectangular area image R1211 in the superimposed image Im1220, is used as output data. Then, the rectangular area image R1211, which is the input data, and the rectangular area image R1221, which is the output data, form a pair of teacher data (hereinafter, the first rectangular area image pair). Here, the rectangular area image R1211 and the rectangular area image R1221 are images of a certain image size. Note that the original image Im1210 and the superimposed image Im1220 may be aligned by any method. Also, the corresponding positional relationship between the rectangular area image R1211 and the rectangular area image R1221 may be specified by any method such as template matching. Note that, depending on the design of the image quality improvement engine, the image size and the number of dimensions of the input data and the output data may differ. For example, if the processing target is an OCT image, and the input data is part of a B-scan image (two-dimensional image), the output data may be part of an A-scan image (one-dimensional image).

The fixed image size for the rectangular area images R1211 and R1221 can be determined, for example, from the common divisor of the pixel counts of the corresponding dimensions for the image size group of the image assumed to be processed (input image). In this case, it is possible to prevent the positional relationship of the rectangular area images output by the image quality improvement engine from overlapping. Specifically, for example, consider a case where the image assumed to be processed is a two-dimensional image, and the first image size of the image size group is 500 pixels wide and 500 pixels high, and the second image size is 100 pixels wide and 100 pixels high. Here, the fixed image size for the rectangular area images R1211 and R1221 is selected from the common divisor of each side. In this case, for example, the fixed image size is selected from 100 pixels wide and 100 pixels high, 50 pixels wide and 50 pixels high, 25 pixels wide and 25 pixels high, etc.

When the image to be processed is three-dimensional, the number of pixels is determined for width, height, and depth. Note that multiple rectangular areas can be set for one pair of a low-quality image corresponding to the input data and a high-quality image corresponding to the output data. For this reason, for example, rectangular area image R1212 of original image Im1210 is set as input data, and rectangular area image R1222, which is the same shooting area as rectangular area image R1212 in superimposed image Im1220, is set as output data. Then, a pair of teacher data is formed by rectangular area image R1212, which is the input data, and rectangular area image R1222, which is the output data. This makes it possible to create a rectangular area image pair other than the first rectangular area image pair.

By creating a large number of pairs of rectangular area images while changing the images of rectangular areas to images with different coordinates, the group of pairs that make up the training data can be enriched, and efficient image quality improvement can be expected by an image quality improvement engine that has trained using the training pairs. However, pairs that do not contribute to improving the image quality of the machine learning model can be prevented from being added to the training data. For example, if a rectangular area image created from a high-quality image that is output data that makes up a pair has image quality that is not suitable for diagnosis, the image output by an image quality improvement engine that has trained using such training data may also have image quality that is not suitable for image diagnosis. Therefore, pairs that include such high-quality images can be removed from the training data.

In addition, for example, if the average luminance or luminance distribution of a pair of rectangular area images created from a low-quality image and a high-quality image differs significantly, such pairs can be removed from the training data. If learning is performed using such training data, the image quality improvement engine may output an image that is unsuitable for image diagnosis, with a luminance distribution that differs significantly from the input image.

Furthermore, for example, consider a case where the structure and position of the object depicted in a pair of rectangular area images, one created from a low-image quality image and the other created from a high-image quality image, are significantly different. In this case, an image quality improvement engine that has learned using such training data may output an image that is not suitable for image diagnosis, in which the object is depicted in a structure and position significantly different from that in the input image. For this reason, such pairs can be removed from the training data.

As in the first embodiment, images with specific shooting conditions that are expected to be processed are used as input data for the training data, and the specific shooting conditions are a predetermined shooting area, shooting method, and shooting angle of view. In other words, unlike the first embodiment, the specific shooting conditions in this embodiment do not include image size.

The image quality improvement unit 404 according to the present embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image and generate a high-quality image. At this time, the image quality improvement unit 404 divides the input image into a group of rectangular area images of a certain image size set for the teacher data that are continuous without gaps. The image quality improvement unit 404 improves the image quality of each of the divided rectangular area image groups using the image quality improvement engine to generate a group of high-quality rectangular area images. After that, the image quality improvement unit 404 arranges and combines the generated high-quality rectangular area images according to the positional relationship of the input image to generate a high-quality image. Here, during learning, as long as the positional relationship between the input data and the output data, which are paired images, corresponds to each other, each rectangular area may be cut out (extracted) from any location in the low-quality image and the high-quality image. On the other hand, during image quality improvement, the input image may be divided into a group of continuous rectangular area images without gaps. In addition, the image size of each pair of images during learning and the image size of each rectangular area image during image quality improvement may be set to correspond to each other (for example, to be the same). This can improve learning efficiency while preventing problems such as unnecessary calculations and missing parts that result in no image.

In this way, the image quality improvement unit 404 of this embodiment improves the image quality of the input image in units of rectangular regions, and by combining the improved image quality images, it is possible to improve the image quality of images of image sizes that could not be handled in the first embodiment, thereby generating a high-image quality image.

Next, a series of image processing steps according to this embodiment will be described with reference to Figs. 5, 13, and 14. Fig. 13 is a flow diagram of the image quality improvement process according to this embodiment. Note that the processes of steps S510, S520, and S550 according to this embodiment are similar to those of the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to shooting conditions other than image size, after the process of step S520, the process of step S530 may be omitted and the process may proceed to step S540.

In step S520, as in the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

If the image quality improvement feasibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

When the process proceeds to step S540, the image quality improvement process according to the present embodiment shown in FIG. 13 is started. This will be described with reference to FIG. 14. In the image quality improvement process according to the present embodiment, first, in step S1310, as shown in FIG. 14(a), the input image is divided into a group of rectangular area images of a certain image size (size shown in R1411) set for the teacher data, which are continuous without gaps. Here, FIG. 14(a) shows an example in which the input image Im1410 is divided into a group of rectangular area images R1411 to R1426 of a certain image size. Note that, as described above, depending on the design of the image quality improvement engine, the image sizes and number of dimensions of the input image and the output image of the image quality improvement engine may differ. In this case, the division positions of the input image can be adjusted by overlapping or separating them so that there are no defects in the combined high-image quality image generated in step S1320. FIG. 14(b) shows an example in which the division positions are overlapped. In FIG. 14B, R1411' and R1412' show overlapping areas. Although not shown to avoid complication, R1413 to R1426 also have similar overlapping areas R1413' to R1426'. The rectangular area size set for the teacher data in the case of FIG. 14B is the size shown in R1411'. Since there is no data in the periphery (top, bottom, left, right ends) of the outside of the input image Im1410, the area is filled with a constant pixel value, filled with a neighboring pixel value, or mirror padded. In addition, depending on the image quality improvement engine, the accuracy of image quality improvement may decrease in the periphery (top, bottom, left, right ends) of the inside of the image due to filter processing. Therefore, as shown in FIG. 14B, rectangular area images may be set by overlapping the division positions, and a part of the rectangular area images may be trimmed and synthesized as the final image. The size of the rectangular area is set according to the characteristics of the image quality improvement engine. Although OCT tomographic images are shown in Fig. 14(a) and (b), the input image (Im1450) may be a front image such as an OCTA En-Face image, as shown in Fig. 14(c) and (d), and similar processing is possible. The size of the rectangular area image is set appropriately depending on the target image and the type of image quality improvement engine.

Next, in step S1320, the image quality improvement unit 404 uses an image quality improvement engine to improve the image quality of the rectangular area images R1411 to R1426, or, if an overlapping area is set, the rectangular area images R1411' to R1426', to generate a group of high-image-quality rectangular area images.

Then, in step S1330, the image quality improving unit 404 arranges each of the generated high-quality rectangular area images in the same positional relationship as each of the rectangular area images R1411 to R1426 divided from the input image, and combines them to generate a high-quality image. If an overlapping area is set, the rectangular area images R1411' to R1426' are arranged in the same positional relationship as each of them, and then the rectangular area images R1411 to R1426 are cut out and combined to generate a high-quality image. Note that the overlapping area may be used to correct the luminance values of the rectangular area images R1411' to R1426'. For example, a reference rectangular area image may be arbitrarily set. Then, by measuring the luminance values of the same coordinate points in adjacent rectangular images that have an overlapping area with the reference rectangular image, the difference (ratio) in luminance values between adjacent images can be found. Similarly, by calculating the difference (ratio) in luminance values in the overlapping areas in all images, correction can be performed to eliminate unevenness in luminance values overall. Note that it is not necessary to use the entire overlapping area for brightness value correction, and a portion of the overlapping area (a few peripheral pixels) may not be used.

As described above, the image quality improvement unit 404 according to this embodiment divides the input image into a plurality of rectangular area images (third images) R1411 to R1426 of a predetermined image size. The image quality improvement unit 404 then inputs the divided rectangular area images R1411 to R1426 to an image quality improvement engine to generate a plurality of fourth images, and integrates the plurality of fourth images to generate a high-quality image. Note that if the rectangular area groups overlap in positional relationship during integration, the pixel values of the rectangular area groups can be integrated or overwritten.

As a result, the image quality improvement unit 404 of this embodiment can use the image quality improvement engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images. Furthermore, if the teacher data is created from multiple images obtained by dividing low-quality images and high-quality images into a predetermined image size, a large amount of teacher data can be created from a small number of images. Therefore, in this case, the number of low-quality images and high-quality images used to create the teacher data can be reduced.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Seventh embodiment
Next, an image processing apparatus according to a seventh embodiment will be described with reference to Figures 15 to 17. In this embodiment, an image quality evaluation unit selects the image with the highest image quality from among multiple high-image-quality images output from multiple image quality improvement engines in response to an instruction from an examiner.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

Figure 15 shows a schematic configuration of an image processing device 1500 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image quality evaluation unit 1506 is provided in the image processing device 1500 according to this embodiment. Note that the image processing device 1500 may be composed of multiple devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in Figure 4 and the description will be omitted.

Furthermore, the image processing device 1500 may be connected to the imaging device 10, the display unit 20, and other devices (not shown) via any circuit or network, similar to the image processing device 400 according to the first embodiment. Furthermore, these devices may be connected to any other device via a circuit or network, or may be configured integrally with any other device. Note that, although these devices are separate devices in this embodiment, some or all of these devices may be configured integrally.

The image quality improvement unit 404 according to this embodiment is equipped with two or more image quality improvement engines that have undergone machine learning using different teacher data. Here, a method for creating a teacher data group according to this embodiment will be described. Specifically, first, a group of pairs of input data, which is a low-quality image, and output data, which is a high-quality image, captured under various shooting conditions is prepared. Next, a teacher data group is created by grouping the pair groups according to any combination of shooting conditions. For example, a teacher data group is created in such a way that a first teacher data group is composed of a pair group acquired by a first combination of shooting conditions, and a second teacher data group is composed of a pair group acquired by a second combination of shooting conditions.

Then, machine learning is performed in separate image quality improvement engines using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, and a second image quality improvement engine corresponding to a machine learning model trained with the first training data.

Since each of these image quality improvement engines uses different teacher data to train the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine provides a high degree of image quality improvement for input images captured and acquired under the first combination of shooting conditions, and a low degree of image quality improvement for images captured and acquired under the second combination of shooting conditions. Similarly, the second image quality improvement engine provides a high degree of image quality improvement for input images captured and acquired under the second shooting conditions, and a low degree of image quality improvement for images captured and acquired under the first shooting conditions.

Since each piece of training data is composed of pairs grouped according to a combination of shooting conditions, the image quality tendencies of the images constituting the pair groups are similar. Therefore, the image quality improvement engine can improve image quality more effectively than the image improvement engine according to the first embodiment, so long as the combination of shooting conditions corresponds. Note that the combination of shooting conditions for grouping pairs of training data may be any combination, and may be, for example, a combination of two or more of the shooting part, shooting angle of view, and image resolution. Furthermore, training data may be grouped based on a single shooting condition, as in the second embodiment.

The image quality assessment unit 1506 selects the highest quality image from the multiple high quality images generated by the image quality improvement unit 404 using multiple image quality improvement engines, in accordance with the examiner's instructions.

The output unit 405 can display the high-quality image selected by the image quality evaluation unit 1506 on the display unit 20 or output it to another device. The output unit 405 can display multiple high-quality images generated by the image quality improvement unit 404 on the display unit 20, and the image quality evaluation unit 1506 can select the high-quality image with the highest image quality in response to instructions from the examiner who checks the display unit 20.

This allows the image processing device 1500 to output the highest quality image in accordance with the examiner's instructions from among multiple high quality images generated using multiple image quality improvement engines.

Below, a series of image processing according to this embodiment will be described with reference to Figs. 16 and 17. Fig. 16 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S1610 and S1620 according to this embodiment is similar to the processing of steps S510 and S520 in the first embodiment, and therefore a description thereof will be omitted. Note that, when improving the image quality of an input image unconditionally with respect to the shooting conditions, after the processing of step S1620, the processing of step S1630 may be omitted and the processing may proceed to step S1640.

In step S1620, as in the first embodiment, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, and the process proceeds to step S1630. In step S1630, as in the second embodiment, the image quality improvement feasibility determination unit 403 uses the acquired group of shooting conditions to determine whether or not any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

If the image quality improvement capability determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S1660. On the other hand, if the image quality improvement capability determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S1640. Note that, depending on the settings and implementation form of the image processing device 400, step S1640 may be performed, as in the first embodiment, even if the image quality improvement engines have determined that some shooting conditions cannot be handled.

In step S1640, the image quality improvement unit 404 inputs the input image acquired in step S1610 to each of the image quality improvement engines, and generates a group of high-image-quality images.

In step S1650, the image quality assessment unit 1506 selects the image with the highest image quality from the group of high-image-quality images generated in step S1640. Specifically, first, the output unit 405 displays the group of high-image-quality images generated in step S1640 on the user interface of the display unit 20.

Here, FIG. 17 shows an example of the interface. The interface displays input image Im1710, and high-quality images Im1720, Im1730, Im1740, and Im1750 output by each of the image quality improvement engines. The examiner operates any input device (not shown) to specify the image with the highest image quality, that is, the image most suitable for image diagnosis, from the image group (high-quality images Im1720 to Im1750). Note that since an input image that has not been improved in image quality by the image quality improvement engine may be more suitable for image diagnosis, the input image may be added to the image group that is the subject of the examiner's specification.

The image quality assessment unit 1506 then selects the high-quality image indicated by the examiner as the image with the highest quality.

In step S1660, the output unit 405 displays the image selected in step S1650 on the display unit 20 or outputs it to another device. However, if it is determined in step S1630 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. Note that if an input image is specified by the examiner or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

As described above, the image quality improvement unit 404 according to this embodiment generates multiple high-quality images from an input image using multiple image quality improvement engines, and the output unit 405 of the image processing device 1500 outputs at least one of the multiple high-quality images in response to the examiner's instructions. In particular, in this embodiment, the output unit 405 outputs the image with the highest image quality in response to the examiner's instructions. This allows the image processing device 1500 to output the high-quality image with the highest image quality in response to the examiner's instructions from among the multiple high-quality images generated using the multiple image quality improvement engines.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1500, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Eighth embodiment
Next, an image processing device according to an eighth embodiment will be described with reference to Figures 15 and 16. In this embodiment, an image quality evaluation unit uses an image quality evaluation engine to select an image with the highest image quality from among a plurality of high-image-quality images output from a plurality of image quality improvement engines.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 1500 according to the seventh embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the seventh embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the seventh embodiment, the configuration shown in FIG. 15 is indicated using the same reference numerals and description will be omitted.

The image quality evaluation unit 1506 according to this embodiment is equipped with an image quality evaluation engine that evaluates the image quality of an input image. The image quality evaluation engine outputs an image quality evaluation index for the input image. The image quality evaluation processing method for calculating the image quality evaluation index in the image quality evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm. The input data of the pairs that constitute the teacher data for training the machine learning model are image groups consisting of a group of low-image-quality images and a group of high-image-quality images captured in advance under various shooting conditions. In addition, the output data of the pairs that constitute the teacher data for training the machine learning model are, for example, a group of image quality evaluation indices set for each of the image groups of the input data by an examiner performing image diagnosis.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 16. Note that the processing steps S1610, S1620, S1630, and S1660 according to this embodiment are similar to those steps in the seventh embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S1620, the processing step S1630 may be omitted and the processing may proceed to step S1640.

In step S1630, as in the seventh embodiment, if the image quality improvement capability determination unit 403 determines that one of the image quality improvement engines can handle the input image, the process proceeds to step S1640. Note that, as in the first embodiment, depending on the settings and implementation form of the image processing device 400, step S1640 may be performed even if the image quality improvement engine determines that some shooting conditions cannot be handled by the image quality improvement engine.

In step S1640, the image quality improvement unit 404 inputs the input image acquired in step S1610 to each of the image quality improvement engines, and generates a group of high-image-quality images.

In step S1650, the image quality evaluation unit 1506 selects the image with the highest image quality from the group of high-quality images generated in step S1640. Specifically, the image quality evaluation unit 1506 first inputs the group of high-quality images generated in step S1640 to the image quality evaluation engine. The image quality evaluation engine calculates an image quality evaluation index for each input high-quality image based on learning. The image quality evaluation unit 1506 selects the high-quality image with the highest image quality evaluation index calculated from the calculated image quality evaluation indexes. Note that since an input image that has not been improved in quality by the image improvement engine may be more suitable for image diagnosis, the image quality evaluation unit 1506 may also input the input image to the image quality evaluation engine and add the image quality evaluation index for the input image to the selection. Step S1660 is similar to step S1660 in the seventh embodiment, and therefore a description thereof will be omitted.

As described above, the image processing device 1500 according to this embodiment further includes an image quality evaluation unit 1506 that evaluates the image quality of a high-quality image. The image quality improvement unit 404 generates multiple high-quality images from an input image using multiple image quality improvement engines, and the output unit 405 of the image processing device 1500 outputs at least one of the multiple high-quality images according to the evaluation result by the image quality evaluation unit 1506. In particular, the image quality evaluation unit 1506 according to this embodiment includes an image quality evaluation engine that uses an evaluation value by a predetermined evaluation method as learning data. The image quality evaluation unit 1506 selects the high-quality image that has the highest evaluation result using the image quality evaluation engine by the image quality evaluation unit 1506 from the multiple high-quality images. The output unit 405 outputs the high-quality image with the highest evaluation value selected by the image quality evaluation unit 1506.

As a result, the image processing device 1500 according to this embodiment can easily output the high-quality image most suitable for image diagnosis from among multiple high-quality images based on the output of the image quality evaluation engine.

In this embodiment, the image quality evaluation unit 1506 selects a high-quality image with the highest image quality evaluation index among the image quality evaluation indices output by the image quality evaluation engine, and the output unit 405 displays the selected high-quality image on the display unit 20. However, the configuration of the image quality evaluation unit 1506 is not limited to this. For example, the image quality evaluation unit 1506 may select high-quality images with the top several image quality evaluation indices among the image quality evaluation indices output by the image quality evaluation engine, and the output unit 405 may display the selected high-quality image on the display unit 20. In addition, the output unit 405 may display the image quality evaluation index output by the image quality evaluation engine together with the corresponding high-quality image on the display unit 20, and the image quality evaluation unit 1506 may select the image with the highest image quality in accordance with the examiner's instructions.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1500, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Ninth embodiment
Next, an image processing device according to a ninth embodiment will be described with reference to Figures 18 and 19. In this embodiment, an authenticity evaluation unit uses an authenticity evaluation engine to evaluate whether or not the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

Figure 18 shows a schematic configuration of an image processing device 1800 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image processing device 1800 according to this embodiment is provided with an authenticity evaluation unit 1807. Note that the image processing device 1800 may be composed of multiple devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the configuration shown in Figure 4 is indicated using the same reference numerals and description will be omitted.

Furthermore, the image processing device 1800 may be connected to the imaging device 10, the display unit 20, and other devices (not shown) via any circuit or network, similar to the image processing device 400 according to the first embodiment. Furthermore, these devices may be connected to any other device via a circuit or network, or may be configured integrally with any other device. Note that, although these devices are separate devices in this embodiment, some or all of these devices may be configured integrally.

The authenticity evaluation unit 1807 is equipped with an authenticity evaluation engine. The authenticity evaluation unit 1807 uses the authenticity evaluation engine to evaluate whether the high-quality image generated by the image quality improvement engine has been sufficiently improved in quality. The authenticity evaluation processing method in the authenticity evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm.

The training data for training the machine learning model includes pairs of high-quality images taken in advance under various shooting conditions and labels indicating that the images were taken and acquired with the target imaging device (hereinafter, genuine labels). The training data also includes pairs of high-quality images generated by inputting low-quality images into an image-enhancing engine with poor image-enhancing accuracy and labels indicating that the images were not taken and acquired with the target imaging device (hereinafter, counterfeit labels).

An authenticity evaluation engine that has been trained using such teacher data cannot reliably evaluate whether an input image is an image that was captured and acquired by a camera device, but it can evaluate whether the image resembles an image that was captured and acquired by a camera device. Using this characteristic, the authenticity evaluation unit 1807 can input a high-quality image generated by the image quality improvement unit 404 to the authenticity evaluation engine, and evaluate whether the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality.

When the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved, the output unit 405 displays the high-quality image on the display unit 20. On the other hand, when the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved, the output unit 405 displays the input image on the display unit 20. Note that, when displaying the input image, the output unit 405 can display on the display unit 20 that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved or that the image being displayed is the input image.

A series of image processing steps according to this embodiment will be described below with reference to FIG. 19. FIG. 19 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S1910 to S1940 according to this embodiment are similar to the processing steps S510 to S540 in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S1920, the processing step S1930 may be omitted and the processing may proceed to step S1940.

In step S1940, when the image quality improvement unit 404 generates a group of high-quality images, the process proceeds to step S1950. In step S1950, the authenticity evaluation unit 1807 inputs the high-quality images generated in step S1940 to an authenticity evaluation engine, and performs an authenticity evaluation based on the output of the authenticity evaluation engine. Specifically, if the authenticity evaluation engine outputs a genuine label (genuine), the authenticity evaluation unit 1807 evaluates that the generated high-quality images have been sufficiently improved in image quality. On the other hand, if the authenticity evaluation engine outputs a counterfeit label (fake), the authenticity evaluation unit 1807 evaluates that the generated high-quality images have not been sufficiently improved in image quality.

In step S1960, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality, the output unit 405 causes the display unit 20 to display the high-quality image. On the other hand, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved in quality, the output unit 405 causes the display unit 20 to display the input image.

As described above, the image processing device 1800 according to this embodiment further includes an authenticity evaluation unit 1807 that evaluates the image quality of a high-quality image, and the authenticity evaluation unit 1807 includes an authenticity evaluation engine that evaluates the authenticity of an image. The authenticity evaluation engine includes a machine learning engine that uses as learning data images generated by an image quality improvement engine that has lower (worse) image quality improvement processing accuracy than the image quality improvement engine of the image quality improvement unit 404. The output unit 405 of the image processing device 1800 outputs a high-quality image when the output from the authenticity evaluation engine of the authenticity evaluation unit is true.

As a result, with the image processing device 1800 according to this embodiment, the examiner can efficiently check high-quality images that have been sufficiently improved.

In addition, the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine may be trained in coordination to improve the efficiency and accuracy of both engines.

In this embodiment, the image quality improvement unit 404 generates one high-quality image, and the authenticity evaluation unit 1807 evaluates the generated high-quality image, but the evaluation by the authenticity evaluation unit 1807 is not limited to this. For example, as in the second embodiment, when the image quality improvement unit 404 generates multiple high-quality images using multiple image quality improvement engines, the authenticity evaluation unit 1807 may be configured to evaluate at least one of the multiple high-quality images generated. In this case, for example, the authenticity evaluation unit 1807 may evaluate all of the multiple high-quality images generated, or may evaluate only the image of the multiple high-quality images specified by the examiner.

Furthermore, the output unit 405 may display on the display unit 20 the result of the judgment made by the authenticity evaluation unit 1807 as to whether or not the high-quality image has been sufficiently improved in quality, and output the high-quality image according to the examiner's instructions.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1800, as in the first embodiment. The output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Tenth embodiment
Next, an image processing device according to a tenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improvement unit divides a three-dimensional input image into a plurality of two-dimensional images, inputs the images to an image quality improvement engine, and combines output images from the image quality improvement engine to generate a three-dimensional high-quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires a three-dimensional image composed of a group of structurally continuous two-dimensional images. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a group of OCT B-scan images (tomographic images). Also, for example, it is a three-dimensional CT volume image composed of a group of axial tomographic images.

The image quality improvement unit 404 is equipped with an image quality improvement engine, as in the first embodiment. Note that the pair group of input data and output data, which is the teacher data for the image quality improvement engine, is composed of a group of two-dimensional images. The image quality improvement unit 404 divides the acquired three-dimensional image into multiple two-dimensional images, and inputs each two-dimensional image to the image quality improvement engine. This allows the image quality improvement unit 404 to generate multiple two-dimensional high-quality images.

The output unit 405 combines the multiple two-dimensional high-quality images generated by the image quality improvement unit 404 for each two-dimensional image of the three-dimensional image, and outputs a three-dimensional high-quality image.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S510 to S530 and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. However, in step S510, the acquisition unit 401 acquires a three-dimensional image. Note that, if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. The image quality improvement possibility determination unit 403 may make the determination based on the shooting conditions of the three-dimensional image, or may make the determination based on the shooting conditions for the multiple two-dimensional images that make up the three-dimensional image. In step S540, the image quality improvement unit 404 divides the acquired three-dimensional image into multiple two-dimensional images. The image quality improvement unit 404 inputs each of the multiple divided two-dimensional images to the image quality improvement engine to generate multiple two-dimensional high-quality images. The image quality improvement unit 404 combines the multiple generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

In step S550, the output unit 405 displays the generated high-quality three-dimensional image on the display unit 20. Note that the display format of the high-quality three-dimensional image may be arbitrary.

As described above, the image quality improvement unit 404 according to this embodiment divides a three-dimensional input image into multiple two-dimensional images and inputs them to the image quality improvement engine. The image quality improvement unit 404 combines the multiple two-dimensional high-quality images output from the image quality improvement engine to generate a three-dimensional high-quality image.

As a result, the image quality improvement unit 404 according to this embodiment can improve the image quality of a three-dimensional image using an image quality improvement engine that has been trained using teacher data of two-dimensional images.

The output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Eleventh embodiment
Next, an image processing device according to an eleventh embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improvement unit divides a three-dimensional input image into a plurality of two-dimensional images, improves the image quality of the plurality of two-dimensional images in parallel by a plurality of image quality improvement engines, and generates a high-quality three-dimensional image by combining output images from the image quality improvement engines.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the tenth embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the tenth embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first and tenth embodiments, the same reference numerals are used for the configuration shown in FIG. 4, and description thereof will be omitted.

The image quality improvement unit 404 according to this embodiment is provided with a plurality of image quality improvement engines similar to those in the tenth embodiment. The plurality of image quality improvement engines provided in the image quality improvement unit 404 may be implemented in two or more device groups via a circuit or network so as to enable distributed processing, or may be implemented in a single device.

As in the tenth embodiment, the image quality improvement unit 404 divides the acquired three-dimensional image into multiple two-dimensional images. The image quality improvement unit 404 uses multiple image quality improvement engines to improve the image quality of the multiple two-dimensional images in a shared manner (in parallel), generating multiple two-dimensional high-quality images. The image quality improvement unit 404 combines the multiple two-dimensional high-quality images output from the multiple image quality improvement engines based on the three-dimensional image to be processed, generating a three-dimensional high-quality image.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S510 to S530 and S550 according to this embodiment are similar to those steps in the tenth embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may make this determination based on the shooting conditions of the three-dimensional image, or may make this determination based on the shooting conditions for the multiple two-dimensional images that make up the three-dimensional image.

In step S540, the image quality improvement unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality improvement unit 404 inputs each of the divided two-dimensional images into a plurality of image quality improvement engines, and performs image quality improvement processing in parallel to generate a plurality of two-dimensional high-quality images. The image quality improvement unit 404 combines the generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

In step S550, the output unit 405 displays the generated high-quality three-dimensional image on the display unit 20. Note that the display format of the high-quality three-dimensional image may be arbitrary.

As described above, the image quality improvement unit 404 according to this embodiment includes multiple image quality improvement engines. The image quality improvement unit 404 divides a three-dimensional input image into multiple two-dimensional images, and generates multiple two-dimensional high-quality images using the multiple image quality improvement engines in parallel. The image quality improvement unit 404 generates a three-dimensional high-quality image by integrating the multiple two-dimensional high-quality images.

As a result, the image quality improvement unit 404 according to this embodiment can improve the image quality of a three-dimensional image using an image quality improvement engine that has been trained using teacher data of two-dimensional images. Furthermore, compared to the tenth embodiment, it is possible to improve the image quality of a three-dimensional image more efficiently.

The training data for the multiple image quality improvement engines may be different training data depending on the processing target to be processed by each image quality improvement engine. For example, a first image quality improvement engine may learn from training data for a first imaging region, and a second image quality improvement engine may learn from training data for a second imaging region. In this case, each image quality improvement engine can improve the image quality of two-dimensional images with greater accuracy.

In addition, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. In addition, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. In other words, a high-quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing may be used.

Twelfth embodiment
Next, an image processing apparatus according to a twelfth embodiment will be described with reference to Figs. 5 and 20. In this embodiment, an acquisition unit 401 acquires an input image from an image management system 2000, not from an image capture device.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device 400 according to the first embodiment, the same reference symbols are used for the configuration shown in FIG. 4, and the description will be omitted.

Figure 20 shows a schematic configuration of an image processing device 400 according to this embodiment. The image processing device 400 according to this embodiment is connected to an image management system 2000 and a display unit 20 via any circuit or network. The image management system 2000 is a device and system that receives and stores images captured by any imaging device or images that have been processed. The image management system 2000 can also transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. The image management system can include, for example, a picture archiving and communication system (PACS).

The acquisition unit 401 according to this embodiment can acquire an input image from the image management system 2000 connected to the image processing device 400. In addition, the output unit 405 can output a high-quality image generated by the image quality improvement unit 404 to the image management system 2000.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S520 to S540 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S510, the acquisition unit 401 acquires an image stored in the image management system 2000 as an input image from the image management system 2000 connected via a circuit or network. The acquisition unit 401 may acquire the input image in response to a request from the image management system 2000. Such a request may be issued, for example, when the image management system 2000 saves an image, before transmitting the saved image to another device, or when displaying the saved image on the display unit 20. The request may also be issued, for example, when a user operates the image management system 2000 to request high image quality processing, or when a high image quality image is used for the image analysis function provided by the image management system 2000.

The processing of steps S520 to S540 is the same as that in the first embodiment. When the image quality improvement unit 404 generates a high-quality image in step S540, the processing proceeds to step S550. In step S550, if a high-quality image was generated in step S540, the output unit 405 outputs the high-quality image to the image management system 2000 as an output image. If a high-quality image was not generated in step S540, the input image is output to the image management system 2000 as an output image. Depending on the settings and implementation of the image processing device 400, the output unit 405 may process the output image so that it can be used by the image management system 2000, or convert the data format of the output image.

As described above, the acquisition unit 401 according to this embodiment acquires an input image from the image management system 2000. Therefore, the image processing device 400 according to this embodiment can output a high-quality image suitable for image diagnosis based on the image stored in the image management system 2000, without increasing the invasiveness or labor of the photographer or subject. The output high-quality image can be stored in the image management system 2000, or displayed on a user interface provided by the image management system 2000. The output high-quality image can be used for the image analysis function provided by the image management system 2000, or can be transmitted via the image management system 2000 to other devices connected to the image management system 2000.

The image processing device 400, the image management system 2000, and the display unit 20 may be connected to other devices (not shown) via circuits or networks. In addition, although these devices are separate devices in this embodiment, some or all of these devices may be configured as an integrated unit.

Furthermore, as in the first embodiment, the output unit 405 may output the generated high-quality image to the image management system 2000 or another device connected to the image processing device 400.

Thirteenth embodiment
Next, an image processing apparatus according to a thirteenth embodiment will be described with reference to Figures 4, 5, 21A, and 21B. In this embodiment, an image quality improving unit receives a plurality of images as input images and generates one high-quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 in this embodiment acquires multiple images from the imaging device 10 or another device as input data to be processed.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. The teacher data may also be similar to that of the first embodiment. The image quality improvement unit 404 inputs each of the multiple images acquired by the acquisition unit 401 to the image quality improvement engine, and performs a process of overlaying the multiple high-quality images output to generate a final high-quality image. Note that the image quality improvement unit 404 may align the multiple high-quality images by any method before performing a process of overlaying the multiple high-quality images.

The output unit 405 displays the final high-quality image generated by the image quality improvement unit 404 on the display unit 20. The output unit 405 may display a plurality of input images on the display unit 20 together with the final high-quality image. The output unit 405 may display the generated plurality of high-quality images on the display unit 20 together with the final high-quality image and the input images.

Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5 and Fig. 21A. Fig. 21A is a flow diagram of the image quality improvement processing according to this embodiment. Note that the processing steps S510 to S530 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described.

However, in step S510, the acquisition unit 401 acquires multiple images, and in steps S520 and S530, the shooting conditions are acquired for each of the multiple images, and it is determined whether or not the image quality can be improved by the image quality improvement engine. If the image quality of the input image is to be improved unconditionally in terms of the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540. Furthermore, if it is determined that some of the multiple images cannot be handled by the image quality improvement engine, the image may be excluded from subsequent processing.

If the image quality improvement possibility determination unit 403 determines in step S530 that the multiple input images can be handled by the image quality improvement engine, the process proceeds to step S540. When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in FIG. 21A is started. In the image quality improvement process according to this embodiment, first, in step S2110, the image quality improvement unit 404 inputs each of the multiple input images to the image quality improvement engine to generate a group of high-image-quality images.

Next, in step S2120, the image quality improvement unit 404 performs an overlay process on the generated high-quality images to generate a single final high-quality image. The overlay process may be performed by an averaging process such as an additive average or any other existing process. When overlaying, the image quality improvement unit 404 may align the multiple high-quality images by any method before overlaying them. Once the image quality improvement unit 404 has generated the final high-quality image, the process proceeds to step S550.

In step S550, the output unit 405 causes the display unit 20 to display the generated final high-quality image.

As described above, the image quality improvement unit 404 according to this embodiment generates one final high-quality image from multiple input images. Since the image quality improvement by the image quality improvement engine is based on the input image, for example, if a lesion or the like is not displayed properly in a certain input image, the pixel value will be low in the high-quality image obtained by improving the image quality of the input image. On the other hand, there are cases where the lesion or the like is displayed properly in another input image capturing the same location, and the pixel value is high in the high-quality image obtained by improving the image quality of the other input image. Therefore, by superimposing these high-quality images, it becomes possible to properly display the areas with low or high pixel values, and a high-quality image with high contrast can be generated. In addition, by making the number of input images smaller than the number required for conventional superimposition, the cost of longer imaging time, etc., as in the conventional case, can be reduced.

This effect is particularly noticeable when using input images that use motion contrast data such as OCTA.

Motion contrast data is the detection of temporal changes in the subject during time intervals when the same location on the subject is repeatedly photographed. For example, in some time intervals, only slight movement of the subject may be detected. In contrast, when photographs are taken at a different time interval, the movement of the subject may be detected as greater movement. Therefore, by overlaying high-quality images of the motion contrast images from each case, it is possible to interpolate motion contrast that did not occur or was only slightly detected at a particular time. Therefore, this type of processing makes it possible to generate a motion contrast image in which the contrast of more of the subject's movements is enhanced, allowing the examiner to grasp the condition of the subject more accurately.

Therefore, when an image that depicts an area that changes over time, such as an OCTA image, is used as the input image, a specific area of the subject can be imaged in more detail by overlaying high-quality images taken at different times.

In this embodiment, a high quality image is generated from each of a plurality of input images, and the high quality images are then superimposed to generate a single final high quality image; however, the method of generating a single high quality image from a plurality of input images is not limited to this. For example, in another example of the image quality improvement process of this embodiment shown in FIG. 21B, when the image quality improvement process is started in step S540, in step S2130, the image quality improvement unit 404 superimposes the input images to generate a single superimposed input image.

Then, in step S2140, the image quality improvement unit 404 inputs one superimposed input image to an image quality improvement engine to generate one high-quality image. Even with this type of image quality improvement process, as with the image quality improvement process described above, it becomes possible to appropriately display areas with low or high pixel values in multiple input images, and to generate a high-quality image with high contrast. This process can also have a remarkable effect when a motion contrast image such as the OCTA image described above is used as the input image.

When performing this high image quality processing, the image quality engine uses superimposed images of the same number of input images as the multiple input images to be processed as input data for the teacher data of the image quality engine. This allows the image quality engine to perform appropriate image quality processing.

In addition, in the image quality improvement process according to this embodiment and the other image quality improvement processes described above, the process of combining the high-image-quality image group or the input image group is not limited to overlay. For example, a single image may be generated by applying a MAP estimation process to these image groups. Also, a single image may be generated by synthesizing the high-image-quality image group or the input image group.

When generating a single image by synthesizing a group of high-quality images or a group of input images, for example, an image with a wide range of gradations in the high-luminance region and an image with a wide range of gradations in the low-luminance region may be used as the input image. In this case, for example, an image with a wide range of gradations in the high-luminance region and an image with a wide range of gradations in the low-luminance region are synthesized to generate a high-quality image. This makes it possible to generate an image that can express a wider range of brightness (dynamic range). In this case, the input data of the training data for the high-quality engine can be an image with a wide range of gradations in the high-luminance region or a low-quality image with a wide range of gradations in the low-luminance region, which are to be processed. Also, the output data of the training data for the high-quality engine can be a high-quality image corresponding to the input data.

Also, an image having a wide range of gradations in high-luminance areas and an image having a wide range of gradations in low-luminance areas may be synthesized, and the synthesized image may be enhanced in image quality by the image quality enhancement engine. In this case, an image capable of expressing a wider range of brightness may be generated. In this case, the input data of the training data for the image quality enhancement engine may be an image obtained by synthesizing a low-quality image having a wide range of gradations in high-luminance areas and a low-quality image having a wide range of gradations in low-luminance areas, which are to be processed. Furthermore, the output data of the training data for the image quality enhancement engine may be a high-quality image corresponding to the input data.

In these cases, the image quality improvement engine can be used to improve the image quality of images that can express a wider range of brightness, making it possible to process fewer images than before and providing images suitable for image analysis at a lower cost.

In addition, any method may be used to capture an image having a wide range of gradations in the high-luminance region and an image having a wide range of gradations in the low-luminance region, such as shortening or lengthening the exposure time of the imaging device. Furthermore, the method of dividing the range of gradations is not limited to low-luminance and high-luminance regions, and may be any method.

In addition, in the image quality improvement process according to this embodiment, multiple input images may be processed in parallel using multiple image quality improvement engines. Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing. In other words, a high-quality image obtained by performing at least one of the following processing groups or imaging methods may be used: overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

<Fourteenth embodiment>
Next, an image processing apparatus according to a fourteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving section uses a medium quality image generated from a plurality of low quality images as an input image and generates a high quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires a medium quality image obtained by superimposing multiple low quality images as input data to be processed from the image capture device 10 or another device. When superimposing the low quality images, any positioning process may be performed.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, the image quality improvement engine of this embodiment is designed to input a medium image quality image, which is of medium image quality, and output a high image quality image. A medium image quality image is a composite image generated by superimposing a group of low image quality images. A high image quality image is an image of higher quality than a medium image quality image. Also, for the group of pairs that constitute the teacher data used to train the image quality improvement engine, the input data that constitutes each pair is a medium image quality image generated in the same manner as the medium image quality image, and the output data is a high image quality image.

The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improvement unit 404. The output unit 405 may cause the display unit 20 to display the input image together with the high-quality image, and in this case, the output unit 405 may display on the display unit 20 that the input image is an image generated from a plurality of low-quality images.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S520 to S550 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described.

In step S510, the acquisition unit 401 acquires a medium quality image as an input image from the image capture device 10 or another device. The acquisition unit 401 may acquire a medium quality image generated by the image capture device 10 as an input image in response to a request from the image capture device 10. Such a request may be issued, for example, when the image capture device 10 generates an image, before or after the image generated by the image capture device 10 is saved in a storage device provided in the image capture device 10, when the saved image is displayed on the display unit 20, when a high quality image is used for image analysis processing, etc.

The subsequent processing is similar to that in the first embodiment, so a detailed explanation is omitted.

As described above, the acquisition unit 401 according to this embodiment acquires, as an input image, a medium quality image that is an image generated using multiple images of a specific part of the subject. In this case, since the input image is a clearer image, the high quality engine can generate a high quality image with higher accuracy. Note that the number of low quality images used to generate the medium quality image may be less than the number of images used to generate a conventional overlaid image.

Note that the medium quality image is not limited to an image obtained by superimposing multiple low quality images, but may be, for example, an image obtained by applying a MAP estimation process to multiple low quality images, or an image obtained by synthesizing multiple low quality images. When synthesizing multiple low quality images, images with different gradations may be synthesized.

In addition, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. In addition, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. In other words, a high-quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing may be used.

<Fifteenth embodiment>
Next, an image processing device according to a fifteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving unit performs image quality improvement according to the first embodiment and the like as well as image size increase (size increase) of an input image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires an image of low image size (low-size image) as an input image. Note that a low-size image is an image that comprises fewer pixels than an image of high image size (high-size image) output by a high-image-quality engine described later. Specifically, for example, when the image size of a high-size image is 1024 pixels wide, 1024 pixels high, and 1024 pixels deep, the image size of a low-size image is 512 pixels high, 512 pixels deep, and so on. In this regard, in this specification, "enhancing the image size" refers to a process of increasing the number of pixels per image and enlarging the image size.

The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine, as in the first embodiment. However, the image quality improvement engine of this embodiment is configured to reduce noise and enhance contrast for an input image, and to increase the image size of the input image. Therefore, the image quality improvement engine of this embodiment is configured to input a low-size image and output a high-size image.

In relation to this, for the group of pairs that constitute the training data for the image quality improvement engine, the input data that constitutes each pair is a low-size image, and the output data is a high-size image. Note that the high-size images used for the output data can be obtained from a device with higher performance than the imaging device that obtained the low-size images, or can be obtained by changing the settings of the imaging device. Furthermore, if a group of high-size images already exists, the group of low-size images to be used as input data may be obtained by reducing the group of high-size images to the image size of the images expected to be obtained from the imaging device 10. Furthermore, for the high-size images, low-size images that are superimposed are used, as in the first embodiment, etc.

Note that the image size enlargement of the input image by the image quality improvement unit 404 according to this embodiment is different from simple image enlargement, since it is obtained as teacher data from a device with higher performance than the image capture device 10 or by changing the settings of the image capture device 10. Specifically, the image size enlargement process of the input image by the image quality improvement unit 404 according to this embodiment can reduce degradation in resolution compared to when the image is simply enlarged.

With this configuration, the image quality improvement unit 404 according to this embodiment can reduce noise and enhance contrast from the input image, and generate a high-quality image with a large image size.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S520, S530, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S510, the acquisition unit 401 acquires a low-size image from the imaging device 10 or another device as input data to be processed. The acquisition unit 401 may acquire a low-size image generated by the imaging device 10 as an input image in response to a request from the imaging device 10. Such a request may be issued, for example, when the imaging device 10 generates an image, before or after the image generated by the imaging device 10 is saved in a storage device provided in the imaging device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

The processing of steps S520 and S530 is the same as that in the first embodiment, and therefore a description thereof will be omitted. In step S540, the image quality improvement unit 404 inputs the input image to an image quality improvement engine, and generates an image that has been subjected to noise reduction and contrast enhancement as a high-quality image and has a large image size. The processing thereafter is the same as that in the first embodiment, and therefore a description thereof will be omitted.

As described above, the image quality improvement unit 404 according to this embodiment generates a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the image size has been increased. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis without increasing the invasiveness or effort of the photographer or subject.

In this embodiment, a high-quality image is generated by performing high-quality processing and high-resolution processing according to the first embodiment, etc., using one high-quality engine, but the configuration for performing these processes is not limited to this. For example, the image quality improvement unit may include an image quality improvement engine that performs image quality processing according to the first embodiment, etc., and another image quality improvement engine that performs high image size processing.

In this case, the image quality improvement engine that performs the image quality improvement process according to the first embodiment, etc. can use a machine learning model that has been trained in the same way as the image quality improvement engine according to the first embodiment, etc. Furthermore, as input data for the teacher data of the image quality improvement engine that performs the high image size process, the high image quality image generated by the image quality improvement engine according to the first embodiment, etc. is used. Furthermore, as output data for the teacher data of the image quality improvement engine, the high image quality image generated by the image quality improvement engine according to the first embodiment, etc. for an image acquired by a high-performance imaging device is used. This allows the image quality improvement engine that performs the high image size process to generate a final high image quality image by increasing the image size of the high image quality image that has been subjected to the image quality improvement process according to the first embodiment, etc.

The high image quality engine may perform high image size processing before the high image quality processing engine according to the first embodiment, etc. In this case, the training data for the high image quality engine performing high image size processing is configured as a group of pairs of input data, which is a low size image acquired by the imaging device, and output data, which is a high size image. The training data for the high image quality engine performing high image quality processing according to the first embodiment, etc. is configured as a group of pairs of input data, which is a high size image, and output data, which is an image in which the high size image is superimposed.

Even with this configuration, the image processing device 400 can generate a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the image size has been enlarged.

In this embodiment, the high image quality processing according to the first embodiment and the like has been described in terms of a configuration in which the superimposed image is used as output data of the teacher data, but as with the first embodiment, the output data is not limited to this. In other words, a high image quality image obtained by performing at least one of a group of processes or imaging methods, such as superimposition processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400.

<Sixteenth embodiment>
Next, an image processing apparatus according to a sixteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving section improves spatial resolution in addition to the image quality improvement according to the first embodiment.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires a low spatial resolution image as an input image. Note that a low spatial resolution image is an image with a lower spatial resolution than the high spatial resolution image output by the image quality improvement unit 404.

The image quality improvement unit 404 is equipped with an image quality improvement engine, as in the first embodiment. However, the image quality improvement engine of this embodiment is configured to reduce noise and enhance contrast of the input image, as well as to increase the spatial resolution of the input image. Therefore, the image quality improvement engine of this embodiment is configured to input a low spatial resolution image and output a high spatial resolution image.

In relation to this, for the group of pairs that make up the training data for the image quality improvement engine, the input data that makes up each pair is a low spatial resolution image, and the output data is a high spatial resolution image. Note that the high spatial resolution image can be obtained from a device with higher performance than the imaging device that acquired the low spatial resolution image, or can be obtained by changing the settings of the imaging device. Also, for the high spatial resolution image, a low spatial resolution image is overlaid, as in the first embodiment, etc.

With this configuration, the image quality improvement unit 404 according to this embodiment can reduce noise and enhance contrast from the input image, and generate a high-quality image with high spatial resolution.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S520, S530, and S550 according to this embodiment are similar to those in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

In step S510, the acquisition unit 401 acquires a low spatial resolution image from the imaging device 10 or another device as input data to be processed. The acquisition unit 401 may acquire a low spatial resolution image generated by the imaging device 10 as an input image in response to a request from the imaging device 10. Such a request may be issued, for example, when the imaging device 10 generates an image, before or after the image generated by the imaging device 10 is saved in a storage device provided in the imaging device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

The processing of steps S520 and S530 is the same as that in the first embodiment, and therefore a description thereof will be omitted. In step S540, the image quality improvement unit 404 inputs the input image to an image quality improvement engine, and generates an image with high spatial resolution and noise reduction as a high-quality image. The processing thereafter is the same as that in the first embodiment, and therefore a description thereof will be omitted.

As described above, the image quality improvement unit 404 according to this embodiment generates a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the spatial resolution has been improved. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis, without increasing the invasiveness or effort of the photographer or subject.

In this embodiment, a high-quality image is generated by performing high-quality image processing and high-resolution processing according to the first embodiment, etc., using one high-quality image engine, but the configuration for performing these processes is not limited to this. For example, the image quality improvement unit may include an image quality improvement engine that performs high-quality image processing according to the first embodiment, etc., and another image quality improvement engine that performs high-resolution processing.

In this case, the image quality improvement engine that performs the image quality improvement process according to the first embodiment, etc. can use a machine learning model that has been trained in the same way as the image quality improvement engine according to the first embodiment, etc. Furthermore, as input data of the teacher data of the image quality improvement engine that performs the high resolution process, the high quality image generated by the image quality improvement engine according to the first embodiment, etc. is used. Furthermore, as output data of the teacher data of the image quality improvement engine, the high quality image generated by the image quality improvement engine according to the first embodiment, etc. for an image acquired by a high performance imaging device is used. In this way, the image quality improvement engine that performs the high spatial resolution process can generate a final high quality image with high spatial resolution for the high quality image that has been subjected to the image quality improvement process according to the first embodiment, etc.

The high spatial resolution processing by the image quality improvement engine may be performed before the high image quality processing by the image quality improvement processing engine according to the first embodiment, etc. In this case, the training data for the image quality improvement engine performing the high spatial resolution processing is configured as a group of pairs of input data, which is a low spatial resolution image acquired by the imaging device, and output data, which is a high spatial resolution image. The training data for the image quality improvement engine performing the image quality improvement processing according to the first embodiment, etc. is configured as a group of pairs of input data, which is a high spatial resolution image, and output data, which is an image in which the high spatial resolution image is superimposed.

Even with this configuration, the image processing device 400 can generate a high-quality image in which at least one of noise reduction and contrast enhancement is performed compared to the input image, and the spatial resolution is improved.

In this embodiment, the high image quality processing according to the first embodiment and the like has been described in terms of a configuration in which the superimposed image is used as output data of the teacher data, but as with the first embodiment, the output data is not limited to this. In other words, a high image quality image obtained by performing at least one of a group of processes or imaging methods, such as superimposition processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

The image quality improvement unit 404 may use an image quality improvement engine to perform the image quality improvement process according to the fifteenth embodiment in addition to the high spatial resolution process. In this case, at least one of noise reduction and contrast enhancement is performed compared to the input image, and an image with a larger image size and higher spatial resolution compared to the input image can be generated as a high-quality image. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis without increasing the invasiveness or effort of the photographer or subject.

Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400.

Seventeenth embodiment
Next, an image processing apparatus according to a seventeenth embodiment will be described with reference to Figures 22 and 23. In this embodiment, an analysis unit performs image analysis on a high-quality image generated by an image quality improvement unit.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

Figure 22 shows a schematic configuration of an image processing device 2200 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image processing device 2200 according to this embodiment is provided with an analysis unit 2208. Note that the image processing device 2200 may be composed of multiple devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the configuration shown in Figure 4 is indicated using the same reference numerals and description will be omitted.

The analysis unit 2208 applies a predetermined image analysis process to the high-quality image generated by the image quality improvement unit 404. In the field of ophthalmology, the image analysis process includes any existing image analysis process such as retinal layer segmentation, layer thickness measurement, optic nerve three-dimensional shape analysis, lamina cribrosa analysis, vascular density measurement of OCTA images, and corneal shape analysis for images acquired by OCT. In addition, the image analysis process is not limited to analysis processes in the field of ophthalmology, and also includes any existing analysis process in the field of radiology, such as diffusion tensor analysis and VBL (Voxel-based Morphometry) analysis.

The output unit 405 can display the high-quality image generated by the image quality improvement unit 404 on the display unit 20, and can also display the analysis results of the image analysis process by the analysis unit 2208. The output unit 405 can display only the image analysis results by the analysis unit 2208 on the display unit 20, or can output the image analysis results to the imaging device 10, an image management system, or other devices. The display form of the analysis results can be any form depending on the image analysis process performed by the analysis unit 2208, and can be displayed as, for example, an image, a numerical value, or a character. The display form of the analysis results can be a form in which the analysis results obtained by analyzing the high-quality image are superimposed on the high-quality image with any transparency. In other words, the display form of the analysis results can be an image (for example, a two-dimensional map) obtained by blending the analysis results obtained by analyzing the high-quality image and the high-quality image with any transparency.

The series of image processing steps according to this embodiment will be described below with reference to FIG. 23, using an OCTA En-Face image as an example. FIG. 23 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S2310 to S2340 according to this embodiment are similar to the processing steps S510 to S540 in the first embodiment, and therefore will not be described here. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S2320, the processing step S2330 may be omitted, and the processing may proceed to step S2340.

In step S2340, the image quality improvement unit 404 improves the image quality of the OCTA En-Face image, and the process proceeds to step S2350. In step S2350, the analysis unit 2208 performs image analysis on the high-image-quality image generated in step S2340. As an image analysis of the high-image-quality OCTA En-Face image, any binarization process can be applied to detect blood vessel-equivalent areas (blood vessel regions) from the image. The area density can be analyzed by determining the proportion of the image occupied by the detected blood vessel-equivalent areas. In addition, by thinning the binarized blood vessel-equivalent areas to an image with a line width of 1 pixel, the proportion of blood vessels that do not depend on the thickness (also called skeleton density) can be determined. These images may be used to analyze the area and shape (circularity, etc.) of the avascular zone (FAZ). As a method of analysis, the above-mentioned numerical values may be calculated from the entire image, or a numerical value may be calculated for a specified region of interest (ROI) based on the instructions of the examiner (user) using a user interface (not shown). The setting of the ROI is not necessarily specified by the examiner, but a predetermined region may be automatically specified. Here, the various parameters described above are examples of analysis results related to blood vessels, and any parameters related to blood vessels may be used. The analysis unit 2208 may perform multiple image analysis processes. That is, an example of analysis of an En-Face image of OCTA is shown here, but not only this, but also retinal layer segmentation, layer thickness measurement, optic disc three-dimensional shape analysis, lamina cribrosa analysis, etc. may be performed on an image acquired by OCT at the same time. In relation to this, the analysis unit 2208 may perform some or all of multiple image analysis processes in response to instructions from the examiner via an arbitrary input device.

In step S2360, the output unit 405 displays the high-quality image generated by the image quality improvement unit 404 and the analysis results by the analysis unit 2208 on the display unit 20. The output unit 405 may output the high-quality image and the analysis results to separate display units or devices. The output unit 405 may also display only the analysis results on the display unit 20. Furthermore, when the analysis unit 2208 outputs multiple analysis results, the output unit 405 may output some or all of the multiple analysis results to the display unit 20 or another device. For example, the analysis results regarding blood vessels in the OCTA En-Face image may be displayed on the display unit 20 as a two-dimensional map. Furthermore, values indicating the analysis results regarding blood vessels in the OCTA En-Face image may be superimposed on the OCTA En-Face image and displayed on the display unit 20.

As described above, the image processing device 2200 according to this embodiment further includes an analysis unit 2208 that performs image analysis on the high-quality image, and the output unit 405 causes the display unit 20 to display the analysis results by the analysis unit 2208. In this way, the image processing device 2200 according to this embodiment uses high-quality images for image analysis, thereby improving the accuracy of the analysis.

In addition, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 2200, as in the first embodiment. In addition, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. In other words, a high-quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing may be used.

<Eighteenth embodiment>
Next, an image processing device according to an eighteenth embodiment will be described with reference to Fig. 4. In this embodiment, an example will be described in which a high image quality unit generates a high image quality image by adding noise to an image during learning and learning the noise component.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires an image as input data to be processed from the image capture device 10 or another device. An example of the configuration of a CNN in the image quality improvement unit according to this embodiment will be described with reference to FIG. 24. FIG. 24 shows an example of a machine learning model configuration in the image quality improvement unit 404. The configuration shown in FIG. 24 is composed of a plurality of layers that process and output a group of input values. As shown in FIG. 24, the types of layers included in the configuration include a convolution layer, a downsampling layer, an upsampling layer, and a merger layer. The convolution layer is a layer that performs convolution processing on a group of input values according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The number of dimensions of the kernel size of the filter may also be changed according to the number of dimensions of the input image. The downsampling layer is a process that reduces the number of output value groups to be less than the number of input value groups by thinning out or synthesizing the input value groups. Specifically, there is a Max Pooling process, for example. The upsampling layer is a process that increases the number of output value groups to be more than the number of input value groups by duplicating the input value groups or adding values interpolated from the input value groups. Specifically, there is a linear interpolation process, for example. The synthesis layer is a layer that inputs value groups such as the output value group of a certain layer and the pixel value group constituting an image from multiple sources and performs a process of synthesizing them by connecting or adding them. In such a configuration, the value group output from the pixel value group constituting the input image Im2410 through the convolution processing block and the pixel value group constituting the input image Im2410 are synthesized in the synthesis layer. After that, the synthesized pixel value group is formed into a high-quality image Im2420 in the last convolution layer. Although not shown in the figure, as an example of a modification of the CNN configuration, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer.

The image quality improvement engine of this embodiment inputs a low-quality image obtained by adding a first noise component to an image obtained from the image capture device 10 or another device, and as output data, trains an image obtained by adding a second noise component to an image obtained from the image capture device 10 or another device as a high-quality image. That is, in this embodiment, the teacher image used during learning is a common image for both the low-quality image and the high-quality image, and the noise components in each image are different. Since the same image is used, alignment is not required when pairing images.

Noise components that are added include Gaussian noise and modeled noise specific to the target image. However, the first and second noises are different noises. Different noises mean that the spatial locations (pixel positions) to which the noise is added are different, or that the noise values are different. For example, in the case of OCT, noise specific to the target image can be estimated based on data captured without a model eye or a test eye being placed, and used as a noise model. In the case of OCTA, noise that appears in the avascular zone (FAZ) or noise that appears in an image captured of a model eye that reproduces blood flow in a schematic manner can be used as a noise model.

In the case of Gaussian noise, the standard deviation or variance value is defined as the noise magnitude, and noise is randomly added to the image based on these values. The average value as a whole may not change as a result of adding random noise. In other words, the average value of the noise added to each pixel of one image is set to 0. Here, it is not necessary to make the average value 0, and it is sufficient to add noise of different patterns to the input data and the output data. In addition, it is not necessary to add noise to both the input data and the output data, and noise may be added to either one of them. Here, when noise is not added, for example, a false image of blood vessels may occur in the image after the image quality is improved, but it is also possible to think that this occurs when the difference between the images before and after the image quality is improved is relatively large. For this reason, the difference between the images before and after the image quality is improved may be reduced. At this time, during learning, two images obtained by adding noise of different patterns to a low image quality image and a high image quality image may be paired images, or two images obtained by adding noise of different patterns to a high image quality image may be paired images.

The output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the display unit 20. The output unit 405 may also display the input image on the display unit 20 together with the high-quality image.

The subsequent processing is similar to that in the first embodiment, so a detailed explanation is omitted.

In this embodiment, a high-quality image is generated using an image in which a first noise component and a second noise component different from the first noise component are added to a low-quality image obtained from the imaging device 10 or another device, but the configuration for performing these processes is not limited to this. For example, the image to which noise is added may be a high-quality image that has been subjected to the overlay process shown in the first embodiment, with the first and second noise components added. In other words, a configuration may be used in which an image in which the first noise component is added to the overlay process image is learned as a low-quality image, and an image in which the second noise component is added to the overlay process image is learned as a high-quality image.

Furthermore, in this embodiment, an example of learning using the first and second noise components has been described, but this is not limiting. For example, a configuration may be used in which the first noise component is added only to the image to be considered as a low-quality image, and learning is performed without adding any noise component to the image to be considered as a high-quality image. The image in this case may be an image obtained from the imaging device 10 or another device, or an image obtained by superimposing the image may be used as the target.

Furthermore, the size of the noise component may be dynamically changed for each type of input image or rectangular area image being learned. Specifically, adding a large noise value increases the effect of noise removal, while adding a small noise value decreases the effect of noise removal. Therefore, for example, the noise component value to be added may be reduced for dark images and increased for bright images, and learning may be performed by adjusting the noise to be added according to the conditions and type of the entire image or rectangular area image.

In this embodiment, the image capture conditions are not specified, but learning is performed using images with various capture ranges and different numbers of scans, frontal images of different capture sites, and images of different depths.

The above describes images obtained from the image capture device 10 or other devices, noise images obtained by adding noise to those images, superimposition processed images, and images obtained by adding noise to superimposition processed images. However, these combinations are not limited to those described above, and low-quality images and high-quality images may be combined in any manner.

Nineteenth embodiment
Next, an image processing device according to a nineteenth embodiment will be described with reference to Figs. 25 and 26. In this embodiment, an image quality improvement unit includes a plurality of image quality improvement engines, and generates a plurality of high-image-quality images for an input image. An example in which a synthesis unit 2505 synthesizes the plurality of high-image-quality images output from the plurality of image quality improvement engines will be described.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 in this embodiment acquires images as input data to be processed from the imaging device 10 or other devices.

The image quality improvement unit 404 according to this embodiment is equipped with multiple image quality improvement engines, as in the second embodiment. Here, each of the multiple image quality improvement engines has been trained using different learning data for at least one of the imaging part, imaging angle of view, front images at different depths, noise components, and image resolution. The image quality improvement unit 404 generates a high-quality image using multiple image quality improvement engines that correspond to at least one of the imaging part, imaging angle of view, front images at different depths, noise components, and image resolution of the input image.

Figure 26 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S2610 and S2620 according to this embodiment are similar to the processing steps S510 and S520 in the first embodiment, and therefore will not be described. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S2620, the processing step S2630 may be omitted and the processing may proceed to step S2640.

In step S2620, as in the first embodiment, once the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, the process proceeds to step S2630. In step S2630, as in the second embodiment, the image quality improvement feasibility determination unit 403 uses the acquired group of shooting conditions to determine whether or not any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

If the image quality improvement capability determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S2660. On the other hand, if the image quality improvement capability determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S2640. Note that, depending on the settings and implementation form of the image processing device 400, step S2640 may be performed, as in the first embodiment, even if the image quality improvement engines determine that some shooting conditions cannot be handled.

In step S2640, the image quality improvement unit 404 inputs the input image acquired in step S2610 to each of the image quality improvement engines, and generates a group of high-image-quality images.

In step S2650, the synthesis unit 2405 synthesizes some high-quality images from the group of high-quality images generated in step S2640. Specifically, for example, the results of two high-quality images are synthesized: a first image quality improvement engine that has trained using paired images of a low-quality image acquired from the imaging device 10 as shown in the first embodiment and a high-quality image obtained by performing an averaging process or the like on a group of images acquired by capturing low-quality images multiple times; and a second image quality improvement engine that has trained using paired images in which noise has been added as shown in the 18th embodiment. As a synthesis method, synthesis can be performed using averaging, weighted averaging, or the like.

In step S2660, the output unit 405 displays the image synthesized in step S2650 on the display unit 20 or outputs it to another device. However, if it is determined in step S2630 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. Note that if an input image is specified by the examiner or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

<Twentieth embodiment>
Next, an image processing device according to a twentieth embodiment will be described with reference to Fig. 4. In this embodiment, an example will be described in which an image quality improvement unit uses an output result of a first image quality improvement engine and a second image quality improvement engine generates a high image quality image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 in this embodiment acquires images as input data to be processed from the imaging device 10 or other devices.

The image quality improvement unit 404 according to this embodiment is provided with multiple image quality improvement engines similar to those of the first embodiment. The image quality improvement unit of this embodiment is provided with a first image quality improvement engine that has learned output data from low image quality images acquired as input data from the image capture device 10 or other devices, and medium image quality images generated from the multiple low image quality images. It further includes a second image quality improvement engine that has learned output data from images output from the first image quality improvement engine, and images of higher image quality than the medium image quality images. Note that the medium image quality images are the same as those of the fourteenth embodiment, and therefore will not be described here.

The output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the display unit 20. The output unit 405 may display the input image on the display unit 20 together with the high-quality image, and in this case, the output unit 405 may display on the display unit 20 that the input image is an image generated from a plurality of low-quality images.

Next, a series of image processing steps according to this embodiment will be described with reference to FIG. 5. Note that the processing steps S510 to S530 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described.

In step S540, the image quality improvement unit 404 uses an image quality improvement engine to improve the image quality of the input image and generate a high-image quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to a first image quality improvement engine, which generates a first high-image quality image with improved image quality. The first high-image quality image is then input to a second image quality improvement engine to obtain a second high-image quality image. The image quality improvement engine generates a high-image quality image that is similar to an overlay process using the input image, based on a machine learning model that has been machine-learned using training data. For this reason, the image quality improvement engine can generate a high-image quality image with reduced noise and enhanced contrast compared to the input image.

The subsequent processing is similar to that in the first embodiment, so a detailed explanation is omitted.

In this embodiment, high-quality images are generated using a first high-quality engine that has learned a pair of low-quality images and medium-quality images obtained from the imaging device 10 or another device, and a second high-quality engine that has learned a pair of first high-quality images and high-quality images, but the configuration for performing these processes is not limited to this. For example, the image pair learned by the first high-quality engine may be an engine that learns noise as described in the 18th embodiment, and the second high-quality engine may learn a pair of the first high-quality image and a high-quality image. As an opposite configuration, the first high-quality engine may learn a pair of low-quality images and medium-quality images, and the second high-quality engine may learn an image with noise added to the first high-quality image.

Furthermore, both the first image quality improvement engine and the second image quality improvement engine may be engines that learn noise as described in the 18th embodiment. In this case, for example, the first image quality improvement engine learns a pair of images in which the first and second noises are added to a high-quality image generated by the superimposition processing image, and the second image quality improvement engine learns a pair of images in which the first and second noises are added to the first high-quality image generated by the first image quality improvement engine. Note that, although two image quality improvement engines have been described in this embodiment, this is not limiting, and a configuration in which a third and fourth are further linked and processed may be used. By cleaning up the images used for learning, a network that is more likely to generate smoother and sharper images is constructed.

<Twenty-first embodiment>
Next, an image processing device according to a twenty-first embodiment will be described with reference to Figs. 4 and 27. In the first embodiment, the image quality improvement unit 404 includes one image quality improvement engine. In contrast, in this embodiment, the image quality improvement unit includes multiple image quality improvement engines that perform machine learning using different teacher data, and generates multiple high-image-quality images for an input image.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment is the same as the image processing device 400 according to the second embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing devices according to the first and second embodiments. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first and second embodiments, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The image quality improvement unit 404 according to this embodiment is equipped with two or more image quality improvement engines that have been subjected to machine learning using different teacher data. Here, a method for creating a teacher data group according to this embodiment will be described. First, a pair group of original images as input data and superimposed images as output data, which are taken with images having various shooting ranges and different numbers of scans, is prepared. Taking OCT and OCTA as an example, for example, a first image group pair taken with 300 A scans and 300 B scans in a 3×3 mm range and a second image group pair taken with 500 A scans and 500 B scans in a 10×10 mm range are taken. At this time, the scan density differs by two times between the first image group pair and the second image group pair. Therefore, these image groups are grouped separately. Then, if there is an image group taken with 600 A scans and 600 B scans in a 6×6 mm range, it is grouped in the same group as the first image group. In other words, images with the same or nearly the same scan density (with an error of about 10%) are grouped together.

Next, a training data group is created by grouping the pairs by scan density. For example, a training data group is created in such a way that a first training data group is made up of pairs acquired by shooting at a first scan density, and a second training data group is made up of pairs acquired by shooting at a second scan density.

Then, separate image quality improvement engines are made to perform machine learning using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, and a second image quality improvement engine corresponding to a machine learning model trained with the second training data.

Since each of these image quality improvement engines uses different teacher data to train the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine provides a high degree of image quality improvement for input images acquired at a first scan density, but a low degree of image quality improvement for images acquired at a second scan density. Similarly, the second image quality improvement engine provides a high degree of image quality improvement for input images acquired at a second scan density, but a low degree of image quality improvement for images acquired at a first scan density.

On the other hand, there are cases where it is not possible to collect a sufficient number of images with various shooting ranges and different scan densities as training data during learning. In such cases, for such image groups, an image quality improvement engine that has learned noise components is prepared, as shown in the 18th embodiment.

The image quality improvement engine, which has learned about noise components, is less affected by the scan density at the time of shooting, so it is applied when an image with a scan density that has not been learned is input.

Since each piece of training data is composed of pairs grouped by scan density, the image quality tendencies of the images constituting the pair groups are similar. Therefore, the image quality improvement engine can improve image quality more effectively than the image improvement engine according to the first embodiment, as long as the image quality is compatible with the corresponding scan density. Note that the shooting conditions for grouping pairs of training data are not limited to scan density, but may be the shooting site, images at different depths in frontal images, or a combination of two or more of these.

Below, a series of image processing according to this embodiment will be described with reference to FIG. 27. FIG. 27 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S2710 and S2720 is similar to steps S510 and S520 according to the first embodiment, and therefore description thereof will be omitted.

When the shooting conditions of the input image are acquired in step S2720, the process proceeds to step S2730. In step S2730, the image quality improvement feasibility determination unit 403 uses the group of shooting conditions acquired in step S2720 to determine whether any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

If the image quality improvement possibility determination unit 403 determines that the shooting conditions are not met, the process proceeds to step S2770. On the other hand, if the image quality improvement possibility determination unit 403 determines that the shooting conditions are met, the process proceeds to step S2740.

In step S2740, the image quality improvement unit 404 selects an image quality improvement engine from the group of image quality improvement engines to perform image quality improvement processing based on the shooting conditions of the input image acquired in step S2720 and the teacher data information of the group of image quality improvement engines. Specifically, for example, for the scan density among the group of shooting conditions acquired in step S2720, an image quality improvement engine that has teacher data information regarding the scan density and provides a high level of image quality improvement is selected. In the above example, when the scan density is the first scan density, the image quality improvement unit 404 selects the first image quality improvement engine.

On the other hand, in step S2770, the image quality improvement unit 404 selects an image quality improvement engine that has learned the noise components.

In step S2750, the image quality improvement unit 404 uses the image quality improvement engine selected in steps S2740 and S2770 to generate a high-image quality image by improving the image quality of the input image. After that, in step S2760, the output unit 405 outputs the high-image quality image in step S2750 and displays it on the display unit 20. Note that when displaying the high-image quality image on the display unit 20, the output unit 405 may display that the high-image quality image was generated using the image quality improvement engine selected by the image quality improvement unit 404.

As described above, the image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines that have been trained using different learning data. Here, each of the plurality of image quality improvement engines has been trained using different learning data for at least one of the imaging region, imaging angle of view, front images at different depths, and image resolution. Furthermore, for data for which sufficient correct answer data (output data) could not be collected, learning was performed using noise components. The image quality improvement unit 404 generates a high-quality image using an image quality improvement engine corresponding to at least one of these.

With this configuration, the image processing device 400 according to this embodiment can generate high-quality images more effectively.

<Twenty-second embodiment>
Next, an image processing device according to a 22nd embodiment will be described with reference to Figs. 30 to 32. In this embodiment, a wide-angle image generating unit generates a wide-angle image (panoramic image) using a plurality of high-quality images generated by an image quality improving unit.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

Figure 31 (a) is a flow diagram of a series of image processing according to this embodiment. In step S3110, the acquisition unit 401 acquires multiple images (at least two) as input data from the imaging device 10 or another device. The multiple images are images of different positions of the same subject (such as the subject's eye), and are images of places where the images do not completely overlap with the subject, but where the images overlap partially. Taking the case of imaging the subject's eye as an example, images of different positions of the same subject's eye can be obtained by changing the position of the fixation light during imaging and having the subject's eye gaze at the fixation light. Note that when imaging an image, it is desirable to change the position of the fixation light so that at least about 20% of the overlapping areas between adjacent images are in the same place. Figure 32 (a) shows an example of an En-Face image of OCTA captured by changing the position of the fixation light so that adjacent images partially overlap. Figure 32 (a) shows an example of a case where the position of the fixation light is changed and different positions are captured five times. Note that while five images are shown as an example in Figure 32, the number is not limited to five and can be two or more.

The process of step S3120 in this embodiment is similar to the process of step S520 in the first embodiment, and therefore will not be described. If the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S3130 may be omitted after the process of step S3120, and the process may proceed to step S3140.

In step S3120, as in the first embodiment, once the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, the process proceeds to step S3130. In step S3130, as in the first embodiment, the image quality improvement feasibility determination unit 403 uses the acquired group of shooting conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 is capable of handling the input image.

If the image quality improvement capability determination unit 403 determines that the image quality improvement engine cannot handle multiple input images, the process proceeds to step S3160. On the other hand, if the image quality improvement capability determination unit 403 determines that the image quality improvement engine can handle multiple input images, the process proceeds to step S3140. Note that, depending on the settings and implementation form of the image processing device 400, step S3140 may be performed, as in the first embodiment, even if the image quality improvement engine determines that it cannot handle some shooting conditions.

In step S3140, the image quality improvement unit 404 performs processing on the multiple input images acquired in step S3110 to generate multiple high-image-quality images.

In step S3150, the wide-angle image generating unit 3005 synthesizes several high-quality images from the group of high-quality images generated in step S3140. Specifically, an OCTA En-Face image is used as an example for explanation. Although the multiple images do not completely overlap, adjacent images are OCTA En-Face images that are taken so that some areas of the images overlap with each other. Therefore, the wide-angle image generating unit 3005 detects overlapping areas from the multiple OCTA En-Face images and performs alignment using the overlapping areas. By transforming the OCTA En-Face images based on the alignment parameters and synthesizing the images, it is possible to generate an OCTA En-Face image with a wider range than a single OCTA En-Face image. At this time, since the En-Face images of the multiple OCTAs to be input have been made high quality in step S3140, the En-Face image of the wide-angle OCTA output in step S3150 has already been made high quality. FIG. 32(b) shows an example of the En-Face image of the wide-angle OCTA generated by the wide-angle image generating unit 3005. FIG. 32(b) is an example generated by aligning the five images shown in FIG. 32(a). FIG. 32(c) shows the positional correspondence between FIG. 32(a) and FIG. 32(b). As shown in FIG. 32(c), Im3220 to Im3250 are arranged around Im3210. Note that the En-Face image of the OCTA can generate En-Face images of multiple OCTAs by setting different depth ranges from three-dimensional motion contrast data. Therefore, although an example of a surface layer image with a wide angle of view is shown in FIG. 32, the present invention is not limited to this. For example, the En-Face image (Im2910) of the surface layer OCTA shown in FIG. 29 may be used for alignment, and the En-Face images of the OCTA in other depth ranges may be deformed using the parameters obtained there. Alternatively, a color image may be used as the input image for alignment, and a composite color image may be generated in which the En-Face of the surface layer OCTA is set to the RG component of the RGB components, and the En-Face image of the OCTA to be aligned is set to the B component. Then, the En-Face image of the composite color OCTA in which layers of multiple depth ranges are combined into one image may be aligned. As a result, by extracting only the B component from the aligned color OCTA En-Face image, a wide-angle OCTA En-Face image in which the target OCTA En-Face image has been aligned can be obtained. Note that the target for improving image quality is not limited to a two-dimensional OCTA En-Face image, but may also be three-dimensional OCT or three-dimensional motion contrast data itself. In this case, alignment may be performed using three-dimensional data to generate a wide range of three-dimensional data. A high-quality wide-angle image can be generated by extracting any cross section (any plane in the XYZ directions is possible) or any depth range (range in the Z direction) from the wide range of three-dimensional data.

In step S3160, the output unit 405 displays the image synthesized from the multiple images in step S3150 on the display unit 20 or outputs it to another device. However, if it is determined in step S3130 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. Note that if an input image is specified by the examiner or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

In this embodiment, a single high-quality wide-angle image is generated by generating high-quality images from multiple input images and aligning the high-quality images, but the method of generating a single high-quality image from multiple input images is not limited to this. For example, in another example of the image quality improvement process of this embodiment shown in FIG. 31(b), a single wide-angle image may be generated first, and image quality improvement process may be performed on the wide-angle image to finally generate a single high-quality wide-angle image.

This process will be explained using Figure 31 (b), but the explanation of the process similar to that of Figure 31 (a) will be omitted.

In step S3121, the wide-angle image generating unit 3005 synthesizes the multiple images acquired in step S3110. The wide-angle image generation is the same as that described in step S3150, except that the input image is an image acquired from the image capture device 10 or another device, and is an image before it is subjected to high image quality processing.

In step S3151, the image quality improvement unit 404 processes the high-quality image generated by the wide-angle image generation unit 3005 to generate a single high-quality wide-angle image.

With this configuration, the image processing device 400 according to this embodiment can generate high-quality images with a wide angle of view.

Regarding the above-mentioned first to twenty-second embodiments, the display of the high-quality image on the display unit 20 by the output unit 405 is basically performed automatically in response to the generation of the high-quality image by the image quality improvement unit 404 and the output of the analysis result by the analysis unit 2208. However, the display of the high-quality image may be performed in response to an instruction from the examiner. For example, the output unit 405 may display on the display unit 20 an image selected in response to an instruction from the examiner from among the high-quality image generated by the image quality improvement unit 404 and the input image. In addition, the output unit 405 may switch the display on the display unit 20 from the captured image (input image) to a high-quality image in response to an instruction from the examiner. That is, the output unit 405 may change the display of the low-quality image to the display of the high-quality image in response to an instruction from the examiner. In addition, the output unit 405 may change the display of the high-quality image to the display of the low-quality image in response to an instruction from the examiner. Furthermore, the image quality improvement unit 404 may start the image quality improvement process by the image quality improvement engine (input of an image to the image quality improvement engine) in response to an instruction from the examiner, and the output unit 405 may display the high-image quality image generated by the image quality improvement unit 404 on the display unit 20. In contrast, when an input image is photographed by the imaging device 10, the image quality improvement engine may automatically generate a high-image quality image based on the input image, and the output unit 405 may display the high-image quality image on the display unit 20 in response to an instruction from the examiner. Note that these processes can also be performed similarly for the output of the analysis result. That is, the output unit 405 may change the display of the analysis result of the low-image quality image to the display of the analysis result of the high-image quality image in response to an instruction from the examiner. Also, the output unit 405 may change the display of the analysis result of the high-image quality image to the display of the analysis result of the low-image quality image in response to an instruction from the examiner. Of course, the output unit 405 may change the display of the analysis result of the low-image quality image to the display of the low-image quality image in response to an instruction from the examiner. The output unit 405 may change the display of the low-quality image to a display of the analysis result of the low-quality image in response to an instruction from the examiner. The output unit 405 may change the display of the analysis result of the high-quality image to a display of the high-quality image in response to an instruction from the examiner. The output unit 405 may change the display of the high-quality image to a display of the analysis result of the high-quality image in response to an instruction from the examiner. The output unit 405 may change the display of the analysis result of the low-quality image to a display of another type of analysis result of the low-quality image in response to an instruction from the examiner. The output unit 405 may change the display of the analysis result of the high-quality image to a display of another type of analysis result of the high-quality image in response to an instruction from the examiner. Here, the display of the analysis result of the high-quality image may be a display in which the analysis result of the high-quality image is superimposed on the high-quality image with an arbitrary transparency. The display of the analysis result of the low-quality image may be a display in which the analysis result of the low-quality image is superimposed on the low-quality image with an arbitrary transparency. At this time, the change to the display of the analysis result may be, for example, a change to a state in which the analysis result is superimposed on the displayed image with an arbitrary transparency. The display of the analysis result may be changed to an image (e.g., a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency. Furthermore, the image processing device may be configured to start the processing by the shooting location estimation engine, the image quality evaluation engine, the authenticity evaluation engine, and the evaluation unit in response to an instruction from the examiner. Note that, with respect to the above-mentioned first to twenty-second embodiments, the display mode in which the output unit 405 displays the high-quality image on the display unit 20 may be arbitrary. For example, the output unit 405 may display the input image and the high-quality image side by side, or may display them by switching between them. Furthermore, the output unit 405 may display the input image and the high-quality image in order according to the shooting site, the shooting date and time, the facility where the shooting was performed, and the like. Similarly, the output unit 405 may display the image analysis results using the high-quality image in order according to arbitrary shooting conditions of the high-quality image and the input image corresponding to the high-quality image. Furthermore, the output unit 405 may display the image analysis results using the high-quality image in order for each analysis item.

<Twenty-third embodiment>
Next, an image processing apparatus according to a twenty-third embodiment will be described with reference to Fig. 4, Fig. 29 and Fig. 33. In this embodiment, learning is performed using teacher data consisting of a pair group of output data, which is a high-quality image corresponding to input data. At that time, one image quality improvement engine is generated using a plurality of high-quality output data generated by a plurality of image quality improvement engines.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the first embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the first embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 will be indicated using the same reference numerals and description will be omitted.

The acquisition unit 401 according to this embodiment acquires an image as input data to be processed from the image capture device 10 or another device. The generation of the image quality improvement engine in the image quality improvement unit 404 according to this embodiment will be described with reference to FIG. 29 and FIG. 33. First, the first learning in this embodiment will be described with reference to FIG. 33(a). FIG. 33(a) shows an example of a group of pairs of multiple input data and output data and multiple image quality improvement engines. Im3311 and Im3312 show a group of pairs of input data and output data. For example, this pair is the group of pairs of the surface layer (Im2910) shown in FIG. 29. Then, 3313 shows an image quality improvement engine that has been learned using the group of pairs of Im3311 and Im3312. Note that the learning in FIG. 33(a) may be a method of using a high-quality image generated by superimposition processing as described in the first embodiment, or a method of learning noise components as described in the 18th embodiment. Or a combination of these may be used. Im3321 and Im3322 are pairs of input data and output data, for example, the pair group of the deep layer (Im2920) shown in FIG. 29. Then, 3323 indicates an image quality improvement engine that has been trained using the pair group of Im3321 and Im3322. Similarly, Im3331 and Im3332 are pairs of input data and output data, for example, the pair group of the outer layer (Im2930) shown in FIG. 29. Then, 3333 indicates an image quality improvement engine that has been trained using the pair group of Im3331 and Im3332. That is, in FIG. 33(a), training is performed for each image. Therefore, for example, in the case of the noise component described in the 18th embodiment, training can be performed with noise parameters suitable for each image. At this time, the image quality improvement engine can include a machine learning engine obtained using training data in which noise according to the state of at least a part of the area of the medical image is added to the at least a part of the area. Here, the noise according to the state may be, for example, noise of a magnitude according to the pixel value of at least a part of the region. The noise according to the state may be, for example, small noise when at least a part of the region has few features (for example, a small pixel value, a low contrast, etc.). The noise according to the state may be, for example, large noise when at least a part of the region has many features (for example, a large pixel value, a high contrast, etc.). The image quality improvement engine may include a machine learning engine obtained using learning data including a plurality of front images to which noise of different magnitudes is added for at least two depth ranges among a plurality of depth ranges. At this time, for example, in a depth range corresponding to a front image with few features (for example, a small pixel value), a front image to which small noise is added may be used as learning data. For example, in a depth range corresponding to a front image with many features (for example, a large pixel value), a front image to which large noise is added may be used as learning data. Note that, in a depth range corresponding to a front image with medium features, a front image to which medium noise is added may be used as learning data. Here, the multiple depth ranges may have parts of two adjacent depth ranges overlapping each other in the depth direction.

Next, image inference in this embodiment will be described with reference to FIG. 33(b). In FIG. 33(b), an image is generated using image quality improvement engines 3313-3333 that have been trained in FIG. 33(a). For example, when low-quality surface image Im3310 is input to image quality improvement engine 3313 that has been trained using multiple surface images, it outputs high-quality surface image Im3315. Also, when low-quality deep image Im3320 is input to image quality improvement engine 3323 that has been trained using multiple deep images, it outputs high-quality deep image Im3325. Similarly, when low-quality outer layer image Im3330 is input to image quality improvement engine 3333 that has been trained using multiple outer layer images, it outputs high-quality outer layer image Im3335.

Next, the second learning in this embodiment will be described with reference to FIG. 33(c). FIG. 33(c) shows how one image quality improvement engine 3300 is trained using a plurality of image pair groups of different types. Im3310 shows a low-quality surface image, Im3315 shows a pair group of high-quality surface images, Im3320 shows a low-quality deep image, Im3325 shows a pair group of high-quality deep images, Im3330 shows a low-quality outer layer image, and Im3335 shows a pair group of high-quality outer layer images. That is, the image quality improvement engine 3300 is generated using teacher data consisting of a pair group of output data, which is a high-quality image generated using the image quality improvement engine learned in the first learning, and low-quality input data. As a result, the image quality improvement engine 3300 can generate high-quality images suitable for image diagnosis, in which noise has been reduced and contrast has been increased, from various types of input images.

The output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the display unit 20. The output unit 405 may also display the input image on the display unit 20 together with the high-quality image.

The subsequent processing is similar to that in the first embodiment, so a detailed explanation is omitted.

In this embodiment, the En-Face image of OCTA is described using three layers of different depths, but the type of image is not limited to this, and the type may be increased by generating En-Face images of OCTA with different depth ranges set by changing the reference layer and offset value. The type of image is not limited to differences in the depth direction, but may be different for each part. For example, it may be different shooting locations such as the anterior eye and the posterior eye. Furthermore, the image is not limited to the En-Face image of OCTA, but may be an En-Face image of brightness generated from OCT data. Then, in the first learning, learning may be performed separately, and in the second learning, these En-Face images of OCTA and the En-Face image of brightness may be learned together. Furthermore, in addition to En-Face images, different shooting devices such as tomographic images, SLO images, fundus photos, and fluorescent fundus photos may be used.

Although an example has been described in which the second learning results in one high image quality engine, it is not necessary that there is only one. Any high image quality engine may be configured to learn using a group of pairs of output data of the high image quality engine generated in the first learning and low image quality input data. Furthermore, in the second learning, FIG. 33(c) shows an example in which learning is performed simultaneously using multiple image pair groups of different types, but this is not limited to this and transfer learning may also be used. For example, after learning using a group of pairs of surface images Im3310 and Im3315, the network may be used to learn a group of pairs of deep images Im3320 and Im3325, thereby finally generating high image quality engine 3300.

With this configuration, the image quality improvement unit 404 according to this embodiment can generate high-quality images more effectively for various types of images.

<Twenty-fourth embodiment>
Next, an image processing device according to the 24th embodiment will be described with reference to Fig. 34. In this embodiment, an example will be described in which the output unit 405 displays the processing result in the image quality improvement unit 404 on the display unit 20. Note that, in this embodiment, the description will be given using Fig. 34, but the display screen is not limited to this. The image quality improvement process can also be applied to a display screen that displays multiple images obtained at different dates and times side by side, such as in follow-up observation. In addition, the image quality improvement process can also be applied to a display screen, such as an imaging confirmation screen, on which the examiner checks whether imaging was successful immediately after imaging.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

The output unit 405 can display on the display unit 20 multiple high-quality images generated by the image quality improvement unit 404 and low-quality images that have not been improved in quality. This makes it possible to output low-quality images and high-quality images according to the examiner's instructions.

An example of the interface 3400 is shown below with reference to FIG. 34. 3400 indicates the entire screen, 3401 indicates a patient tab, 3402 indicates a photography tab, 3403 indicates a report tab, and 3404 indicates a setting tab, and the diagonal line in the report tab 3403 indicates the active state of the report screen. In this embodiment, an example of displaying a report screen is described. Im3405 is an SLO image, and Im3406 displays an OCTA En-Face image shown in Im3407 superimposed on the SLO image Im3405. Here, the SLO image is a front image of the fundus acquired by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown). Im3407 and Im3408 show OCTA En-Face images, Im3409 shows a luminance En-Face image, and Im3411 and Im3412 show tomographic images. 3413 and 3414 show the boundary lines of the upper and lower ranges of the OCTA En-Face images shown in Im3407 and Im3408, respectively, superimposed on the tomographic image. Button 3420 is a button for specifying the execution of image quality improvement processing. Of course, as described below, button 3420 may also be a button for instructing the display of a high-quality image.

In this embodiment, the image quality improvement process is performed by specifying the button 3420, or the execution is determined based on information saved (stored) in the database. First, an example in which the display of a high-quality image and a low-quality image are switched by specifying the button 3420 in response to an instruction from the examiner will be described. Note that the image to be subjected to the image quality improvement process will be described as an OCTA En-Face image. When the examiner specifies the report tab 3403 to transition to the report screen, the low-quality OCTA En-Face images Im3407 and Im3408 are displayed. After that, when the examiner specifies the button 3420, the image quality improvement unit 404 performs image quality improvement process on the images Im3407 and Im3408 displayed on the screen. After the image quality improvement process is completed, the output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the report screen. In addition, Im3406 displays an image that has been subjected to image quality improvement processing because Im3407 is superimposed on the SLO image Im3405. Then, the display of the button 3420 is changed to an active state, and a display is made so that it is clear that the image quality improvement processing has been performed. Here, the execution of processing in the image quality improvement unit 404 does not need to be limited to the timing when the examiner designates the button 3420. Since the types of the OCTA En-Face images Im3407 and Im3408 to be displayed when the report screen is opened are known in advance, the image quality improvement processing may be performed when transitioning to the report screen. Then, the output unit 405 may display the high image quality image on the report screen at the timing when the button 3420 is pressed. Furthermore, the types of images to be subjected to image quality improvement processing in response to an instruction from the examiner or when transitioning to the report screen do not need to be two types. Processing may be performed on images that are likely to be displayed, such as multiple OCTA En-Face images of the superficial layer (Im2910), deep layer (Im2920), outer layer (Im2930), and choroidal vascular network (Im2940) shown in FIG. 29. In this case, the images obtained by performing the image quality improvement processing may be temporarily stored in memory or in a database.

Next, a case where the image quality improvement process is performed based on information saved (stored) in the database will be described. If the state of performing the image quality improvement process is saved in the database, when the screen transitions to the report screen, the high-quality image obtained by performing the image quality improvement process is displayed by default. The button 3420 is displayed in an active state by default, so that the examiner can understand that the high-quality image obtained by performing the image quality improvement process is displayed. If the examiner wants to display the low-quality image before the image quality improvement process, the examiner can display the low-quality image by selecting the button 3420 to cancel the active state. If the examiner wants to return to the high-quality image, the examiner selects the button 3420. The execution or non-execution of the image quality improvement process on the database is specified by hierarchy, such as common to all data saved in the database, and for each shooting data (each examination). For example, if the state of performing the image quality improvement process on the entire database is saved, and the examiner saves a state in which the image quality improvement process is not performed on individual shooting data (individual examination), the shooting data will be displayed without the image quality improvement process the next time it is displayed. A user interface (e.g., a save button) not shown may be used to save the execution status of the image quality improvement process for each piece of imaging data (each examination). Furthermore, when transitioning to other imaging data (other examination) or other patient data (e.g., changing to a display screen other than the report screen in response to an instruction from the examiner), the execution status of the image quality improvement process may be saved based on the display status (e.g., the status of button 3420). In this way, if the execution or non-execution of image quality improvement process is not specified on an imaging data unit (examination unit), processing is performed based on information specified for the entire database, and if it is specified on an imaging data unit (examination unit), processing can be performed individually based on that information.

In this embodiment, an example is shown in which Im3407 and Im3408 are displayed as OCTA En-Face images, but the OCTA En-Face image to be displayed can be changed according to the examiner's specifications. Therefore, the following describes how to change the image when the execution of image quality improvement processing is specified (button 3420 is active).

The image is changed using a user interface (e.g., a combo box) not shown. For example, when the examiner changes the type of image from the superficial layer to the choroidal vascular network, the image quality improvement unit 404 performs image quality improvement processing on the choroidal vascular network image, and the output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the report screen. That is, the output unit 405 may change the display of the high-quality image of the first depth range to the display of a high-quality image of a second depth range that is at least partially different from the first depth range, in response to an instruction from the examiner. At this time, the output unit 405 may change the display of the high-quality image of the first depth range to the display of a high-quality image of the second depth range by changing the first depth range to the second depth range in response to an instruction from the examiner. Note that, for an image that is likely to be displayed at the time of transition to the report screen as described above, if a high-quality image has already been generated for the image, the output unit 405 may display the generated high-quality image. The method of changing the type of image is not limited to the above, and it is also possible to generate an En-Face image of OCTA with a different depth range set by changing the reference layer and offset value. In that case, when the reference layer or offset value is changed, the image quality improvement unit 404 executes image quality improvement processing on an arbitrary En-Face image of OCTA, and the output unit 405 displays a high-quality image on the report screen. The reference layer and offset value can be changed using a user interface (e.g., a combo box or a text box) not shown. In addition, the generation range of the En-Face image of OCTA can be changed by dragging (moving the layer boundary) either of the boundary lines 3413 and 3414 superimposed and displayed on the tomographic images Im3411 and Im3412. When the boundary line is changed by dragging, the execution command of the image quality improvement processing is executed continuously. Therefore, the image quality improvement unit 404 may always process the execution command, or may execute it after the layer boundary is changed by dragging. Alternatively, the execution of the high image quality processing may be commanded continuously, but the previous command may be canceled when the next command arrives, and the latest command may be executed. Note that the high image quality processing may take a relatively long time. Therefore, regardless of the timing at which the command is executed, it may take a relatively long time until the high image quality image is displayed. Therefore, from the time when the depth range for generating the En-Face image of the OCTA is set according to the instruction from the examiner until the high image quality image is displayed, the En-Face image of the OCTA (low image quality image) corresponding to the set depth range may be displayed. That is, when the depth range is set, the En-Face image of the OCTA (low image quality image) corresponding to the set depth range is displayed, and when the high image quality processing is completed, the display of the En-Face image of the OCTA (the low image quality image) may be changed to the display of the high image quality image. Also, from the time when the depth range is set until the high image quality image is displayed, information indicating that the high image quality processing is being executed may be displayed. Note that these can be applied not only when the execution of image quality improvement processing has already been specified (button 3420 is active), but also, for example, when the execution of image quality improvement processing is instructed in response to an instruction from the examiner and before a high-quality image is displayed.

In this embodiment, an example has been shown in which different layers are displayed as OCTA En-Face images in Im3407 and Im3408, and low-image-quality and high-image-quality images are switched between for display, but this is not limiting. For example, a low-image-quality OCTA En-Face image may be displayed side-by-side in Im3407, and a high-image-quality OCTA En-Face image may be displayed side-by-side in Im3408. When images are switched for display, it is easy to compare areas that have changed because the images are switched in the same location, and when displayed side-by-side, it is easy to compare the entire images because the images can be displayed simultaneously.

Next, the execution of the high image quality processing during screen transition will be described using Figures 34(a) and (b). Figure 34(b) is an example of a screen in which the En-Face image Im3407 of OCTA in Figure 34(a) is enlarged and displayed. In Figure 34(b), the button 3420 is displayed in the same manner as in Figure 34(a). The screen transition from Figure 34(a) to Figure 34(b) is performed, for example, by double-clicking the En-Face image Im3407 of OCTA, and the screen transition from Figure 34(b) to Figure 34(a) is performed by pressing the close button 3430. Note that the method of screen transition is not limited to the method shown here, and a user interface not shown may be used. If the execution of the high image quality processing is specified at the time of screen transition (button 3420 is active), the state is maintained even during screen transition. That is, when a high-quality image is displayed on the screen of FIG. 34(a) and the screen of FIG. 34(b) is transitioned to the screen of FIG. 34(b), the high-quality image is also displayed on the screen of FIG. 34(b). And the button 3420 is made active. The same applies when transitioning from FIG. 34(b) to FIG. 34(a). In FIG. 34(b), the button 3420 can be designated to switch the display to a low-quality image. Regarding the screen transition, the transition is not limited to the screen shown here, and if the transition is to a screen that displays the same shooting data, such as a display screen for follow-up observation or a display screen for panoramic images, the transition is performed while maintaining the display state of the high-quality image. That is, on the display screen after the transition, an image corresponding to the state of the button 3420 on the display screen before the transition is displayed. For example, if the button 3420 on the display screen before the transition is in an active state, the high-quality image is displayed on the display screen after the transition. Also, for example, if the active state of the button 3420 on the display screen before the transition is released, the low-quality image is displayed on the display screen after the transition. When the button 3420 on the display screen for follow-up observation is activated, multiple images obtained at different dates and times (different examination dates) displayed side by side on the display screen for follow-up observation may be switched to high-quality images. That is, when the button 3420 on the display screen for follow-up observation is activated, multiple images obtained at different dates and times may be collectively reflected. An example of the display screen for follow-up observation is shown in FIG. 38. When the tab 3801 is selected in response to an instruction from the examiner, the display screen for follow-up observation is displayed as shown in FIG. 38. At this time, the examiner can change the depth range of the En-Face image by selecting it from the default depth range set (3802 and 3803) displayed in the list box. For example, the retinal surface layer is selected in the list box 3802, and the retinal deep layer is selected in the list box 3803. The analysis result of the En-Face image of the retinal surface layer is displayed in the upper display area, and the analysis result of the En-Face image of the retinal deep layer is displayed in the lower display area. That is, when a depth range is selected, the analysis results of a plurality of En-Face images of the selected depth range are collectively changed for a plurality of images of different dates and times. At this time, when the display of the analysis results is deselected, the display may be collectively changed to a parallel display of a plurality of En-Face images of different dates and times. Then, when the button 3420 is designated in response to an instruction from the examiner, the display of the plurality of En-Face images is collectively changed to a display of a plurality of high-quality images. Also, when the display of the analysis results is in a selected state, when the button 3420 is designated in response to an instruction from the examiner, the display of the analysis results of a plurality of En-Face images is collectively changed to a display of the analysis results of a plurality of high-quality images. Here, the display of the analysis results may be a display in which the analysis results are superimposed on the image with an arbitrary transparency. At this time, the change to the display of the analysis results may be, for example, a change to a state in which the analysis results are superimposed on the displayed image with an arbitrary transparency. Also, the change to the display of the analysis results may be, for example, a change to a display of an image (e.g., a two-dimensional map) obtained by blending the analysis results and the image with an arbitrary transparency. In addition, the type of layer boundary and the offset position used to specify the depth range can be changed collectively from a user interface such as 3805 and 3806. The depth ranges of a plurality of En-Face images at different dates and times may be changed collectively by displaying a tomographic image together and moving the layer boundary data superimposed on the tomographic image according to an instruction from the examiner. In this case, a plurality of tomographic images at different dates and times may be displayed side by side, and when the above movement is performed on one tomographic image, the layer boundary data may be moved similarly on the other tomographic images. In addition, the image projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a user interface such as a context menu. In addition, the selection button 3807 may be selected to display a selection screen, and an image selected from the image list displayed on the selection screen may be displayed. In addition, the arrow 3804 displayed at the top of FIG. 38 is a mark indicating that the examination is currently selected, and the reference examination (Baseline) is the examination selected during Follow-up imaging (the image on the leftmost side of FIG. 38). Of course, a mark indicating the reference examination may be displayed on the display unit. Also, when the "Show Difference" checkbox 3808 is specified, a measurement value distribution (map or sector map) for the reference image is displayed on the reference image. Furthermore, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement distribution calculated for the image displayed in the area is displayed in the area corresponding to the other examination dates. As the measurement result, a trend graph (a graph of the measurement value for the image on each examination date obtained by measuring the change over time) may be displayed on the report screen. That is, time series data (e.g., a time series graph) of multiple analysis results corresponding to multiple images at different dates and times may be displayed. In this case, analysis results for dates and times other than the multiple dates and times corresponding to the multiple images displayed may also be displayed as time series data in a state that can be distinguished from the multiple analysis results corresponding to the multiple images displayed (e.g., the color of each point on the time series graph differs depending on whether the image is displayed or not). Also, the regression line (curve) of the trend graph and the corresponding formula may be displayed on the report screen.

In this embodiment, the En-Face image of OCTA has been described, but this is not limiting. The image related to the display, image quality improvement, image analysis, and other processes according to this embodiment may be an En-Face image of luminance. Furthermore, it may be a different image such as a tomographic image, an SLO image, a fundus photograph, or a fluorescent fundus photograph, in addition to the En-Face image. In this case, the user interface for executing the image quality improvement process may be one that instructs the execution of image quality improvement process on multiple images of different types, or one that instructs the execution of image quality improvement process by selecting any image from multiple images of different types.

With this configuration, the output unit 405 can display the image processed by the image quality improvement unit 404 according to this embodiment on the display unit 20. At this time, as described above, when at least one of the multiple conditions related to the display of the high-quality image, the display of the analysis result, the depth range of the displayed front image, etc. is selected, the selected state may be maintained even if the display screen is changed. Also, as described above, when at least one of the multiple conditions is selected, the selected state of the at least one may be maintained even if another condition is changed to a selected state. For example, when the display of the analysis result is in a selected state, the output unit 405 may change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image in response to an instruction from the examiner (for example, when the button 3420 is designated). Also, when the display of the analysis result is in a selected state, the output unit 405 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image in response to an instruction from the examiner (for example, when the designation of the button 3420 is released). In addition, the output unit 405 may change the display of the analysis result of the low-quality image to the display of the low-quality image in response to an instruction from the examiner (for example, when the designation of the display of the analysis result is released) when the display of the high-quality image is in a non-selected state. In addition, the output unit 405 may change the display of the analysis result of the low-quality image to the display of the analysis result of the low-quality image in response to an instruction from the examiner (for example, when the designation of the display of the analysis result is released) when the display of the high-quality image is in a selected state. In addition, the output unit 405 may change the display of the analysis result of the high-quality image to the display of the high-quality image in response to an instruction from the examiner (for example, when the designation of the display of the analysis result is released) when the display of the high-quality image is in a selected state. In addition, the output unit 405 may change the display of the analysis result of the high-quality image to the display of the analysis result of the high-quality image in response to an instruction from the examiner (for example, when the display of the analysis result is specified) when the display of the high-quality image is in a selected state. In addition, a case will be considered where the display of the high-quality image is in a non-selected state and the display of the first type of analysis result is in a selected state. In this case, the output unit 405 may change the display of the first type of analysis result of the low image quality image to the display of the second type of analysis result of the low image quality image in response to an instruction from the examiner (for example, when the display of the second type of analysis result is specified). Also, consider a case where the display of the high image quality image is in a selected state and the display of the first type of analysis result is in a selected state. In this case, the output unit 405 may change the display of the first type of analysis result of the high image quality image to the display of the second type of analysis result of the high image quality image in response to an instruction from the examiner (for example, when the display of the second type of analysis result is specified). In addition, in the display screen for follow-up observation, as described above, these display changes may be configured to be reflected collectively on multiple images obtained at different dates and times. Here, the display of the analysis result may be a display in which the analysis result is superimposed on the image with an arbitrary transparency. At this time, the change to the display of the analysis result may be, for example, a change to a state in which the analysis result is superimposed on the displayed image with an arbitrary transparency. Additionally, the change to the display of the analysis results may be, for example, a change to the display of an image (e.g., a two-dimensional map) obtained by blending the analysis results and the image with an arbitrary transparency.

<Twenty-fifth embodiment>
Next, an image processing apparatus according to a twenty-fifth embodiment will be described with reference to Fig. 35. In this embodiment, the processing determination unit 3506 will be described.

Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the processing determination unit 3506 according to this embodiment will be described below.

The processing determination unit 3506 determines whether the image quality improvement process in the image quality improvement unit 404 should be performed by the GPU (Graphics Processing Unit) or the CPU.

The processing determination unit 3506 determines whether the environment is sufficient to perform high image quality processing using machine learning, such as the GPU name of the GPU mounted on the device that executes the processing of the high image quality unit 404, the GPU driver, the memory size of the GPU, etc. If the processing determination unit 3506 determines that the GPU can be used, the high image quality unit 404 performs processing using the GPU. If the processing determination unit 3506 determines that the GPU cannot be used, the high image quality unit 404 performs processing using the CPU. If the processing determination unit 3506 determines that the GPU cannot be used, the output unit 405 displays on the display unit 20 that processing will be performed by the CPU rather than the GPU, since the CPU takes longer to process than the GPU. Note that if the GPU is available, a display may be made to process using the GPU. As a method of displaying on the display unit 20, a message may be displayed, or only words such as GPU and CPU may be displayed. Note that if the image quality improvement unit 404 is processed by the CPU and the processing takes a long time (for example, several tens of seconds to several minutes or more), the image quality improvement process may be disabled, and, for example, button 3420 shown in FIG. 34 may be hidden. By hiding the user interface for executing the image quality improvement process, the function cannot be used. When the function is disabled, a message indicating that it is disabled may be displayed on the display unit 20.

The processing determination unit 3506 is not limited to only determining whether it is a GPU or a CPU, but can also determine whether or not to execute the high image quality processing itself. For example, a case where license registration is required to execute the high image quality processing will be described. The processing determination unit 3506 determines whether or not a license has been registered, and if a license has been registered, it displays, for example, button 3420 shown in FIG. 34 as indicating that the high image quality processing can be executed. If a license has not been registered, button 3420 shown in FIG. 34 is hidden, thereby making it impossible to use. Note that if license registration is required to execute the high image quality processing, it is executed before the above-mentioned GPU processing or CPU processing determination.

The processing determination unit 3506 may not only automatically determine whether to execute processing, but may also determine processing based on instructions from the examiner. For example, if the examiner uses a user interface (not shown) to specify CPU execution, the processing determination unit 3506 determines that processing should be performed by the CPU rather than the GPU. In that case, the processing determination unit 3506 does not need to check the GPU installed in the device, and the image quality improvement unit 404 performs processing using the CPU.

The processing by the processing determination unit 3506 described above does not need to be performed every time the image quality improvement unit 404 performs processing, but may be performed when the image processing device is started. Alternatively, the determination may be performed periodically (for example, once a day).

With this configuration, the processing determination unit 3506 according to this embodiment determines whether or not image quality improvement processing can be performed. Then, it becomes possible to select an appropriate environment and perform machine learning processing.

(Variation 1)
In the various embodiments described above, the efficiency and accuracy of the image quality improvement engine and the authenticity evaluation engine may be improved by training the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine so as to compete with each other. Here, the network trained so that multiple models compete with each other is, for example, a generative adversarial network (GAN). In this case, the machine learning model of the image quality improvement engine corresponds to a generative model (Generator) that generates an image. In addition, the machine learning model of the authenticity evaluation engine corresponds to a discriminator model (Discriminator) that identifies whether the generated image is genuine or not. For example, the machine learning model of the image quality improvement engine is trained so that when an image that is a correct answer for image quality improvement is evaluated by the authenticity evaluation engine, a genuine label is output. Then, the machine learning model of the authenticity evaluation engine is trained so that when an image generated by the image quality improvement engine is evaluated by the authenticity evaluation engine, a counterfeit label is output. As a result, repeated training is performed until the images generated by the image enhancement engine become indistinguishable from the correct images for image enhancement, thereby improving the efficiency and accuracy of the image enhancement engine and the authenticity assessment engine.

The image quality improvement engine may be a trained model obtained by training training data including at least one image generated by a generative adversarial network. In this case, the image quality improvement engine may be configured to allow the examiner to select whether or not to use at least one image generated by the generative adversarial network as training data for training.

(Variation 2)
In the various embodiments and modifications described above, the high-quality image generated by the high-quality engine and the input image may be synthesized and output. For example, when the pixel value of the input image is low (the image is dark), the high-quality engine may reduce the pixel value as a noise component. Therefore, based on the brightness of the input image, the ratio of synthesis between the image generated by the high-quality engine and the input image may be changed and output. That is, the output unit 405 (display control unit) may output a synthetic image obtained by synthesizing the input image and the high-quality image (second image) at a ratio obtained using information on at least a part of the region of the input image (first image). At this time, the ratio of synthesis of the two images may be determined by using the pixel value (brightness of at least a part of the region) in at least a part of the region of the input image as the above information. At this time, for example, the lower the pixel value in the input image (darker), the higher the ratio of synthesis of the input image with the high-quality image. Also, for example, the higher the pixel value in the input image (brighter), the lower the ratio of synthesis of the input image with the high-quality image. Specifically, the synthesis ratio is changed based on the statistical value of pixel values of the entire image (average, median, mode, minimum, maximum, variance, standard deviation, etc.). For example, when the statistical value of pixel values of the input image is lower than a first threshold, the image generated by the high image quality engine and the input image are synthesized at a ratio of 0.5:0.5 (average of the two images) to output a synthetic image. Alternatively, when the statistical value of pixel values of the input image is higher than a second threshold, the image generated by the high image quality engine and the input image are synthesized at a ratio of 0.9:0.1 (weighted average of the two images) to output a synthetic image. Note that the synthesis ratio between the first threshold and the second threshold changes smoothly. The statistical value calculated from the input image may be obtained for the entire image, or may be divided into several regions to obtain local statistical values. When dividing the image into several regions, the ratio value may be corrected to a smooth value so that the synthesis ratio does not change suddenly in adjacent regions. Furthermore, instead of dividing the image into regions, a smoothing filter such as a Gaussian filter may be used to blur the image, and the pixel-by-pixel value may be compared with the first and second thresholds to determine the ratio of pixel-by-pixel synthesis. Note that the image for calculating the statistical value of pixel values is not limited to the input image. For example, when the input image is an OCTA image, the statistical value of pixel values may be calculated using an enface or projection image of luminance.

Furthermore, the ratio of synthesis of two images may be changed not by the statistical value of pixel values of the input image but by the difference between the input image and the high-quality image generated by the high-quality engine. That is, the ratio of synthesis of two images may be determined by using the difference value of pixel values in at least some areas of the two images corresponding to each other as the above information. Specifically, when the difference between the input image and the high-quality image is large, the ratio of the input image may be increased. That is, when the high-quality image has been overly denoised, the ratio of the input image is increased and synthesis is performed to generate a natural high-quality image. Note that when calculating the difference value, it is possible to judge based on not only simple difference information but also structural difference information. For example, a filter such as a Hessian filter may be used to extract only linear structures. This makes it possible to extract only noise with a certain degree of continuity, such as blood vessels, without detecting random noise as a difference. Furthermore, it is possible to simply label the noise components and extract only noise with a certain degree of size. Similarly, when changing the ratio of synthesis based on the difference, it may be calculated for the entire image, or it may be divided into several areas to calculate local difference values.

Furthermore, the proportion of parts and images to be synthesized may be determined by recognizing the parts and images. In this regard, for example, an OCTA image of the superficial layer will be described. In the OCTA image of the superficial layer, since there are no blood vessels in the FAZ (avascular area of the fovea), the FAZ may be dark in the OCTA image. For this reason, it is possible to increase the proportion of high-quality images in the input image for the FAZ. In other words, the proportion of images with more reduced noise is increased. On the other hand, if there is a dark area in a position other than the FAZ, it is difficult to determine whether the area is an avascular area (NPA: Nonperfusion Area) or an area where blood vessels actually exist but the brightness is reduced due to a shadow or the like. Therefore, it is possible to decrease the proportion of high-quality images in the input image. In other words, the proportion of images in which the originally existing low-brightness area may have disappeared from the image is decreased. In this way, not only the brightness and difference change of the image, but also the proportion of parts to be synthesized may be changed. Next, the case of recognizing an image will be described. The appearance and brightness of the OCTA image changes depending on the depth of the surface, deep, and outer layers. Therefore, the type of the target image may be recognized and the ratio may be changed depending on the type of layer. The image may be recognized using the position information of the boundary line when generating the layer, or may be automatically recognized from the image. In other words, the composition ratio may be changed depending on the depth from which the OCTA image was generated, rather than being determined only by the brightness of the image. For example, the OCTA image of the surface layer is generally bright, and the OCTA image of the outer layer is generally dark. Therefore, the first and second thresholds of the composition ratio based on the statistical value of the pixel values and the corresponding ratios may be different values for the OCTA image of the surface layer and the OCTA image of the outer layer. For example, if the first threshold is lower in the surface layer, the image generated by the high image quality engine and the input image are composed at a ratio of 0.5:0.5, but if the first threshold is lower in the outer layer, the image generated by the high image quality engine and the input image may be composed at a ratio of 0.7:0.3.

Note that while the image synthesis described above is a process of synthesizing pixel values themselves, it is also possible to change the opacity of the images. In other words, the synthesis ratio may be an alpha blend value. For example, if the input image ratio is 0.3, an image may be displayed in which the alpha value of the image generated by the high image quality engine is 1 and the alpha value of the input image is 0.3. In this case, it is preferable to always display the image generated by the high image quality engine, and to change the alpha value of the input image to display it semi-transparently.

When outputting an image that combines an image generated by the image quality improvement engine with an input image, the image quality improvement engine may output an image with a ratio that is automatically determined as described above. The ratio of combining the two images may be configured to be changeable in response to instructions from the examiner using a user interface (not shown). In this case, the user interface may be configured to allow the ratio to be changed using a slider bar or by inputting a number into a text box, or multiple images with different ratios may be presented for selection.

The ratio of combining the input image and the high-quality image may be determined from information about at least a portion of the region of the input image using a trained model obtained by training with training data in which a medical image is used as input data and information about the ratio of combining the medical image and a high-quality medical image obtained by improving the quality of the medical image is used as correct answer data (output data). In this case, the information about the ratio may be, for example, a ratio value set (changed) in response to instructions from the examiner. In addition, the trained model may be, for example, one obtained by training with training data including input data that is a set of a medical image and a high-quality medical image obtained by improving the quality of the medical image. In this case, the trained model can be obtained by machine learning using the training data.

Here, the machine learning includes, for example, deep learning consisting of a multi-layered neural network. In addition, for example, a convolutional neural network (CNN) can be used for at least a part of the multi-layered neural network. In addition, a technology related to an autoencoder (autoencoder) may be used for at least a part of the multi-layered neural network. In addition, a technology related to backpropagation (backpropagation method) may be used for learning. However, the machine learning is not limited to deep learning, and any model that can extract (express) the feature amount of the learning data such as an image by itself by learning may be used. In addition, the machine learning is not limited to such a model, and may be a model that learns the feature amount obtained in advance using a medical image before learning as the learning data. For example, the machine learning may be a support vector machine, an adaboost, a random forest, a Bayesian network, etc. In addition, the learned model may be updated by additional learning in which the ratio value set (changed) according to the instruction from the examiner is used as the learning data. For example, if the examiner tends to set the ratio of the input image to the high-quality image high when the input image is relatively dark, the trained model will be additionally trained to have such a tendency. This allows, for example, customization of the trained model to obtain a synthesis ratio that suits the examiner's preferences. At this time, a button for determining whether or not to use the set (changed) ratio value as training data for additional training in response to an instruction from the examiner may be displayed on the display screen. In addition, the ratio determined using the trained model may be set as a default value, and the ratio value may be changed from the default value in response to an instruction from the examiner. In addition, the image quality improvement engine may be a trained model obtained by additionally training training data including at least one high-quality image generated by the image quality improvement engine. At this time, it may be configured to be selectable whether or not to use the high-quality image as training data for additional training in response to an instruction from the examiner.

The composite image output by combining the image generated by the image quality improvement engine shown in this modified example with the input image can be output instead of the high quality image described in the various embodiments above, and the same is true for display screens such as follow-up observation and panoramic images. That is, the composite images in this modified example may be acquired at multiple positions, and a wide-angle image may be generated using the multiple composite images. The wide-angle image generated using the multiple composite images may be displayed on a display screen for panoramic images. The composite images in this modified example may be acquired at different dates and times, and the multiple composite images may be displayed side by side on a display screen for follow-up observation. It is also possible to perform analysis processing such as vascular analysis processing on the composite image in this modified example.

(Variation 3)
In the report screen in the various embodiments and modifications described above, the analysis results such as the layer thickness of the desired layer and various blood vessel densities may be displayed. In addition, the values (distributions) of parameters related to the target area including at least one of the optic disc, macular region, blood vessel region, nerve fiber bundle, vitreous region, macular region, choroid region, sclera region, lamina cribrosa region, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, blood vessel wall, blood vessel inner wall boundary, blood vessel outer boundary, ganglion cells, corneal region, iridocorneal region, Schlemm's canal, etc. may be displayed as the analysis results. In this case, for example, by analyzing a medical image to which various artifact reduction processes have been applied, it is possible to display a highly accurate analysis result. Note that the artifacts may be, for example, a false image region caused by light absorption by the blood vessel region, a projection artifact, a band-shaped artifact in a front image caused in the main scanning direction of the measurement light due to the state of the subject's eye (movement, blinking, etc.), etc. Moreover, the artifact may be, for example, any defective region that occurs randomly on a medical image of a predetermined part of a subject for each shooting. Moreover, the parameter value (distribution) for a region including at least one of the various artifacts (defective regions) described above may be displayed as an analysis result. Moreover, the parameter value (distribution) for a region including at least one of abnormal parts such as drusen, neovascularization, white spots (hard white spots), and pseudodrusen may be displayed as an analysis result. Moreover, the analysis result may be displayed as an analysis map, a sector showing a statistical value corresponding to each divided region, or the like. The analysis result may be generated using a trained model (analysis result generation engine, trained model for generating analysis result) obtained by learning the analysis result of a medical image as learning data. In this case, the trained model may be obtained by learning using training data including a medical image and the analysis result of the medical image, or training data including a medical image and the analysis result of a medical image of a different type from the medical image, or the like. The trained model may be obtained by learning using training data including input data that is a set of multiple medical images of different types of a predetermined part, such as a luminance front image and a motion contrast front image. Here, the luminance front image corresponds to a luminance En-Face image, and the motion contrast front image corresponds to an OCTA En-Face image. The trained model may be configured to display an analysis result obtained using a high-quality image generated by a high-quality engine. The input data included in the training data may be a high-quality image generated by a high-quality engine, or a set of a low-quality image and a high-quality image. The training data may be data in which information including at least one of an analysis value (e.g., average value or median value) obtained by analyzing an analysis region, a table including the analysis value, an analysis map, and the position of an analysis region such as a sector in an image, is labeled as correct answer data (for supervised learning) on the input data. The trained model may be configured to display an analysis result obtained by a trained model for generating an analysis result in response to an instruction from an examiner.

In addition, various diagnostic results such as glaucoma and age-related macular degeneration may be displayed on the report screen in the various embodiments and modified examples described above. At this time, for example, by analyzing a medical image to which various artifact reduction processes as described above have been applied, a highly accurate diagnostic result can be displayed. In addition, the diagnostic result may be displayed by displaying the position of the identified abnormal part on the image, or the state of the abnormal part by characters or the like. In addition, the classification result of the abnormal part, etc. (for example, Curtin classification) may be displayed as the diagnostic result. In addition, the diagnostic result may be generated using a trained model (diagnosis result generation engine, trained model for generating diagnostic result) obtained by learning the diagnostic result of the medical image as learning data. In this case, the trained model may be obtained by learning using training data including a medical image and a diagnostic result of the medical image, or training data including a medical image and a diagnostic result of a type of medical image different from the medical image. In addition, the diagnostic result obtained by using a high-quality image generated by the high-quality engine may be displayed. The input data included in the learning data may be a high-quality image generated by a high-quality image engine, or a set of a low-quality image and a high-quality image. The learning data may be data in which the input data is labeled with information including at least one of the following: diagnosis, type and condition (degree) of the lesion (abnormal site), position of the lesion in the image, position of the lesion relative to the region of interest, findings (image findings, etc.), grounds for the diagnosis (positive medical support information, etc.), and grounds for denying the diagnosis (negative medical support information). The system may be configured to display the diagnosis result obtained by the trained model for generating the diagnosis result in response to an instruction from the examiner.

In addition, in the report screen in the various embodiments and modifications described above, object recognition results (object detection results) and segmentation results such as the above-mentioned areas of interest, artifacts, and abnormal areas may be displayed. In this case, for example, a rectangular frame or the like may be superimposed around the object on the image. Also, for example, a color or the like may be superimposed on the object in the image. Note that the object recognition results and segmentation results may be generated using a trained model obtained by learning training data in which information indicating object recognition or segmentation is used as correct answer data and labeled on a medical image. Note that the above-mentioned analysis result generation and diagnosis result generation may be obtained by using the above-mentioned object recognition result and segmentation result. For example, the analysis result generation and diagnosis result generation process may be performed on the area of interest obtained by the object recognition or segmentation process.

The above-mentioned trained model may be a trained model obtained by training with training data including input data that includes a set of multiple medical images of different types of a specific part of the subject. In this case, the input data included in the training data may be, for example, data that includes a set of a motion contrast front image and a luminance front image (or a luminance tomographic image) of the fundus. In addition, the input data included in the training data may be, for example, data that includes a set of a tomographic image (B-scan image) of the fundus and a color fundus image (or a fluorescent fundus image). In addition, the multiple medical images of different types may be anything obtained by different modalities, different optical systems, or different principles. In addition, the above-mentioned trained model may be a trained model obtained by training with training data including input data that includes a set of multiple medical images of different parts of the subject. In this case, the input data included in the training data may be, for example, data that includes a set of a tomographic image (B-scan image) of the fundus and a tomographic image (B-scan image) of the anterior part of the eye. In addition, the input data included in the learning data may be, for example, data including a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic disc of the fundus. The input data included in the learning data may be, for example, a set of a tomographic image of the anterior eye and a color fundus image. The above-mentioned learned model may be a learned model obtained by learning with learning data including input data including a set of a plurality of medical images of a specific part of the subject with different shooting angles of view. The input data included in the learning data may be, for example, a panoramic image, which is a combination of a plurality of medical images obtained by time-dividing a specific part into a plurality of regions. In this case, by using a wide-angle image such as a panoramic image as learning data, there is a possibility that the feature amount of the image can be accurately acquired because it has a larger amount of information than a narrow-angle image, and therefore the results of each process can be improved. Furthermore, the input data included in the learning data may be a set of multiple medical images of a specific part of a subject taken at different dates and times.

In addition, the display screen on which at least one of the above-mentioned analysis results, diagnosis results, object recognition results, and segmentation results is displayed is not limited to a report screen. Such a display screen may be displayed on at least one of the following display screens: an image capture confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before image capture (a display screen on which various live moving images are displayed). For example, by displaying at least one of the above results obtained using the above-mentioned trained model on the image capture confirmation screen, the examiner can confirm the results with high accuracy even immediately after image capture. In addition, the change in display between the low-quality image and the high-quality image may be, for example, a change in display between the analysis results of the low-quality image and the analysis results of the high-quality image.

(Variation 4)
In the preview screen in the various embodiments and modifications described above, the learned model may be used for at least one frame of the live video. In this case, when multiple live videos of different parts or different types are displayed on the preview screen, the learned model corresponding to each live video may be used. As a result, even for live videos, for example, the processing time can be shortened, so that the examiner can obtain highly accurate information before starting to capture. Therefore, for example, failures in re-capturing can be reduced, so that the accuracy and efficiency of diagnosis can be improved. Note that the multiple live videos may be at least one video, such as a video of the anterior part of the eye for alignment in the XYZ directions, a frontal video of the fundus for focus adjustment of the fundus observation optical system or OCT focus adjustment, and a tomographic video of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length).

In addition, the moving images to which the learned model can be applied are not limited to live moving images, but may be, for example, moving images stored (saved) in a storage unit. At this time, for example, a moving image obtained by aligning at least one frame of a tomographic moving image of the fundus stored (saved) in the storage unit may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, a reference frame may first be selected based on a condition such that the vitreous body is present as much as possible on the frame. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) generated sequentially by the high-quality engine for at least one frame of the moving image may be configured to be displayed continuously. Here, during various adjustments, there is a possibility that the subject to be photographed, such as the retina of the subject's eye, has not yet been well imaged. For this reason, there is a possibility that a high-quality image cannot be obtained with high accuracy due to a large difference between the medical image input to the learned model and the medical image used as learning data. Therefore, when the evaluation value of the image quality evaluation of the tomographic image (B scan) exceeds a threshold value, the display of the high-quality moving image (continuous display of high-quality frames) may be automatically started. When the evaluation value of the image quality evaluation of the tomographic image (B scan) exceeds a threshold value, the image quality improvement button may be changed to a state (active state) that the examiner can specify. Also, a different image quality improvement engine may be prepared for each imaging mode having a different scanning pattern, etc., and a high image quality improvement engine corresponding to the selected imaging mode may be selected. Also, one image quality improvement engine obtained by learning learning data including various medical images obtained in different imaging modes may be used. As the method of alignment between the above-mentioned frames, the same method may be applied for the alignment in the X direction and the alignment in the Z direction (depth direction), or all different methods may be applied. Also, alignment in the same direction may be performed multiple times using different methods, for example, after rough alignment, precise alignment may be performed. In addition, the alignment method may be, for example, (coarse alignment in the Z direction) using retinal layer boundaries obtained by segmenting a tomographic image (B-scan image), (precise alignment in the X and Z directions) using correlation information (similarity) between a reference image and multiple regions obtained by dividing a tomographic image, (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image), (X direction) alignment using a two-dimensional front image, etc. Also, it may be configured so that rough alignment is performed on a pixel-by-pixel basis, and then precise alignment is performed on a sub-pixel basis.

(Variation 5)
In the various embodiments and modified examples described above, when a trained model is undergoing additional training, it may be difficult to output (infer/predict) using the trained model undergoing additional training itself. For this reason, it is preferable to prohibit the input of medical images to the trained model undergoing additional training. In addition, another trained model that is the same as the trained model undergoing additional training may be prepared as a spare trained model. In this case, it is preferable to allow input of medical images to be executed to the spare trained model during additional training. Then, after the additional training is completed, the trained model after the additional training is evaluated, and if there is no problem, the spare trained model may be replaced with the trained model after the additional training. In addition, if there is a problem, the spare trained model may be used.

In addition, a trained model obtained by training for each imaging site may be selectively used. Specifically, a plurality of trained models including a first trained model obtained using training data including a first imaging site (lungs, examined eye, etc.) and a second trained model obtained using training data including a second imaging site different from the first imaging site may be prepared. Then, one of the plurality of trained models may be selected. In this case, a control means may be provided that searches for data (for example, via a network from a server of an external facility such as a hospital or research institute) in which the imaging site corresponding to the selected trained model (header information or information manually input by the examiner) and the image of the imaging site are paired in response to an instruction from the examiner, and performs training using the searched data as training data for the selected trained model as additional learning. This allows efficient additional learning for each imaging site using the image of the imaging site corresponding to the trained model.

In addition, when acquiring training data for additional learning via a network from a server at an external facility such as a hospital or research institute, it is desirable to reduce deterioration in reliability due to tampering or system trouble during additional learning. Therefore, the validity of the training data for additional learning may be detected by checking the consistency using a digital signature or hashing. This makes it possible to protect the training data for additional learning. In this case, if the validity of the training data for additional learning cannot be detected as a result of checking the consistency using a digital signature or hashing, a warning to that effect is issued and additional learning using that training data is not performed. The server may be installed anywhere and may take any form, such as a cloud server, fog server, edge server, etc.

(Variation 6)
In the various embodiments and modifications described above, the instructions from the examiner may be instructions by voice or the like, in addition to manual instructions (e.g., instructions using a user interface or the like). At this time, for example, a machine learning engine including a voice recognition engine (voice recognition model, trained model for character recognition) obtained by machine learning may be used. The manual instructions may be instructions by character input or the like using a keyboard, touch panel, or the like. At this time, for example, a machine learning engine including a character recognition engine (character recognition model, trained model for character recognition) obtained by machine learning may be used. The instructions from the examiner may be instructions by gesture or the like. At this time, a machine learning engine including a gesture recognition engine (gesture recognition model, trained model for gesture recognition) obtained by machine learning may be used. The instructions from the examiner may be a gaze detection result of the examiner on the monitor, or the like. The gaze detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by shooting around the monitor. At this time, the pupil detection from the moving image may use an object recognition engine as described above. The instructions from the examiner may be instructions by brain waves, weak electrical signals flowing through the body, or the like. In such a case, for example, the learning data may be learning data in which character data or voice data (waveform data) indicating an instruction to display the results of the processing of various learned models as described above is used as input data, and the execution command for actually displaying the results of the processing of various learned models on the display unit is used as the correct answer data. In addition, the learning data may be learning data in which character data or voice data indicating an instruction to display a high-quality image obtained by a learned model for high image quality is used as input data, and the execution command for displaying the high-quality image and the execution command for changing the button 3420 to an active state are used as the correct answer data. Of course, the learning data may be anything as long as the instruction content indicated by the character data or voice data corresponds to the execution command content. In addition, the voice data may be converted into character data using an acoustic model, a language model, or the like. In addition, a process of reducing noise data superimposed on the voice data may be performed using waveform data obtained by multiple microphones. In addition, the device may be configured to be able to select between an instruction by character or voice and an instruction by a mouse, touch panel, or the like according to an instruction from an examiner. Furthermore, the instructions by text or voice may be configured to be turned on or off in response to an instruction from the examiner.

Here, the machine learning includes deep learning as described above, and at least a part of the multi-layered neural network can be, for example, a recurrent neural network (RNN). Here, as an example of the machine learning engine according to this modification, an RNN, which is a neural network that handles time-series information, will be described with reference to FIG. 36. Also, a long short-term memory (hereinafter, LSTM), which is a type of RNN, will be described with reference to FIG. 37. FIG. 36(a) shows the structure of an RNN, which is a machine learning engine. The RNN 3520 has a loop structure in the network, and inputs data x ^t 3510 at time t and outputs data h ^t 3530. Since the RNN 3520 has a loop function in the network, it is possible to take over the state at the current time to the next state, and therefore it can handle time-series information. FIG. 36(b) shows an example of input and output of a parameter vector at time t. The data x ^t 3510 includes N pieces of data (Params1 to ParamsN). The data h ^t 3530 output from the RNN 3520 includes N pieces of data (Params1 to ParamsN) corresponding to the input data. However, since the RNN cannot handle long-term information during error backpropagation, the LSTM may be used. The LSTM can learn long-term information by including a forget gate, an input gate, and an output gate. Here, the structure of the LSTM is shown in FIG. 37(a). In the LSTM 3540, the information that the network takes over at the next time t is the internal state c ^t-1 of the network called a cell and the output data h ^t-1 . The lowercase letters (c, h, x) in the figure represent vectors. Next, the details of the LSTM 3540 are shown in FIG. 37(b). In FIG. 37(b), FG indicates a forgetting gate network, IG indicates an input gate network, and OG indicates an output gate network, each of which is a sigmoid layer. Therefore, a vector in which each element has a value between 0 and 1 is output. The forgetting gate network FG determines how much past information to retain, and the input gate network IG determines which value to update. CU is a cell update candidate network and is an activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gate network OG selects cell candidate elements and selects how much information to transmit at the next time. Note that the above-mentioned LSTM model is a basic form, so it is not limited to the network shown here. The connection between networks may be changed. Instead of LSTM, QRNN (Quasi Recurrent Neural Network) may be used. Furthermore, the machine learning engine is not limited to a neural network, and boosting, support vector machines, etc. may be used. In addition, when the examiner's instructions are input by characters or voice, a technology related to natural language processing (e.g., sequence to sequence) may be applied. In addition, a dialogue engine (dialogue model, trained model for dialogue) that responds to the examiner by outputting characters or voice may be applied.

(Variation 7)
The various embodiments and modifications described above include at least each of the following, and also include at least various combinations of each of the following to the extent that there is no technical contradiction. Note that the machine learning described below can be applied to various learning methods such as those described above. Also, at least some of the regions described below are, for example, the partial regions described above, such as rectangular regions.

First, the high image quality engine may include a machine learning engine that performs machine learning to generate a two-dimensional high-image quality image using a two-dimensional medical image of a predetermined part of the subject. In this case, the high image quality engine may be a trained model obtained by training training data including at least a part of the area of the two-dimensional medical image. For example, the high image quality engine may be a trained model obtained by training training data including at least a part of the area of a first front image in a first depth range of the subject's eye. In addition, as another high image quality engine, another trained model obtained by training training data including at least a part of the area of a second front image in a second depth range of the subject's eye, which is at least a part of the area of a second front image in a second depth range that is different from the first depth range, may be generated. That is, when the feature amount of the second front image and the feature amount of the first front image are relatively different, not only the first trained model may be generated, but also the second trained model may be generated. As a result, for example, multiple trained models can be selectively used according to multiple medical images. Therefore, multiple medical images whose feature amounts are relatively different from each other can be accurately trained. If these features are relatively similar, it is not necessary to generate a second trained model; it is sufficient to generate a common trained model obtained by training the first front image and the second front image as training data.

The high image quality engine may also include a machine learning engine that performs machine learning to generate a high-quality three-dimensional image using a three-dimensional medical image of a specific part of the subject. In this case, the high image quality engine may be a trained model obtained by training training data including at least a part of the three-dimensional medical image. Here, consider a case where the three-dimensional medical image is composed of a plurality of two-dimensional medical images at different positions. In this case, for example, the B-scan image is a tomographic image of the XZ plane, and the different positions are in the Y direction. In this case, the data input to the training data or trained model may be a three-dimensional medical image composed of a plurality of two-dimensional medical images whose positional deviations in the XZ direction have been corrected (aligned). In addition, when a high-quality three-dimensional image is generated from a three-dimensional medical image using a trained model, it takes more processing time than a two-dimensional medical image, so for example, it may be configured to be processed by a server capable of high-speed processing. In this case, it may be configured to transmit medical image data obtained by the imaging device from the client to the server, and after processing using the trained model in the server, the processed data may be transmitted from the server to the client. The server may be in any form, such as a cloud server, a fog server, an edge server, etc., regardless of the installation location. As for the method of aligning the above-mentioned multiple two-dimensional medical images, the method of aligning in the X direction and the method of aligning in the Z direction (depth direction) may be the same method, or all different methods may be applied. The alignment in the same direction may be performed multiple times using different methods, for example, after performing a rough alignment, a precise alignment may be performed. As the alignment method, for example, (coarse in the Z direction) alignment using a retinal layer boundary obtained by segmenting a tomographic image (B-scan image), (precise in the X direction or Z direction) alignment using correlation information (similarity) between a reference image and multiple regions obtained by dividing the tomographic image, (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image), (X direction) alignment using a two-dimensional front image, etc. Alternatively, the alignment may be performed roughly on a pixel-by-pixel basis, and then precisely on a sub-pixel basis.

The high image quality engine may also be a trained model obtained by training trained data including two-dimensional medical images of a plurality of ranges in which at least a portion of the range is different in three-dimensional medical image data of a predetermined part of the subject. For example, the high image quality engine may be a trained model obtained by training trained data including at least a portion of a first front image of a first depth range of the subject's eye and at least a portion of a second front image of a second depth range in which at least a portion of the range is different from the first depth range. That is, the high image quality engine may be a trained model obtained by training trained data including a plurality of medical images obtained using three-dimensional medical image data of a predetermined part of the subject, the plurality of medical images having different feature amounts. As a result, the high image quality engine can obtain, for example, a feature amount having a high degree of abstraction as a training result for a plurality of different feature amounts. For example, even if a medical image has a feature amount different from the plurality of feature amounts, the image quality can be relatively high-precision if the extracted feature amount having a high degree of abstraction is within an applicable range. For example, a trained model obtained by training data including at least a portion of a first front image in a first depth range and at least a portion of a second front image in a second depth range can be used to generate a high-quality image with high accuracy from at least a portion of a third front image in a third depth range that is at least partially different from the first and second depth ranges. In this case, for example, the first depth range is a superficial layer in which relatively thick blood vessels are distributed, and the second depth range is a deep layer in which relatively thin blood vessels are distributed (or no blood vessels are distributed). Therefore, the feature amount of the first front image and the feature amount of the second front image are different from each other.

The set of input data and correct answer data (output data) included in the learning data may be a set of low-quality images and high-quality images. For example, the high-quality image may be obtained by superimposing multiple low-quality images. In this case, by superimposing, a part that is not commonly captured in multiple low-quality images but is captured in one of them may be visualized in the high-quality image. In other words, a part that does not exist in the low-quality image may appear in the high-quality image. In such a case, the image quality improvement engine may not only obtain the image quality improvement feature as the learning result, but may also obtain a feature that newly generates a non-existent part, for example, there is a possibility that false blood vessels will be generated in an area in the high-quality image where blood vessels do not actually exist.

Therefore, the set of input data and correct answer data included in the learning data may be a plurality of medical images in which the difference between the parts present on the images is relatively small. For example, it may be a set of a high-quality image and a high-quality image to which noise has been added, or a set of a plurality of high-quality images to which different noises have been added. In this case, the noise may be noise to the extent that the feature amount of the parts present on the images is not lost. In addition, the noise pattern in which whether or not noise is added to each pixel is randomly determined may be different for each image. Note that the size of the noise added may be different for each image. In addition, it may be a set of a plurality of medical images in which the number of overlapping sheets is different from each other. In this way, the high-quality engine can, for example, accurately obtain the feature amount of the high-quality image as a learning result. Therefore, by using the high-quality engine, it is possible to accurately generate a high-quality image from an input low-quality image. In this case, the input low-quality image may be a medical image to which various artifact reduction processes as described above have been applied. Note that the reduction process may be applied to the high-quality image. In addition, the reduction process may be configured to be selectable according to an instruction from the examiner.

Here, different noises may be added to at least some areas of a plurality of medical images having different feature amounts. For example, when a moderate amount of noise is added to a relatively dark medical image in comparison with a relatively bright medical image, there is a possibility that a part present in the relatively dark medical image may be lost. Therefore, for example, the magnitude of the noise added to at least some areas of the relatively dark medical image may be smaller than the magnitude of the noise added to at least some areas of the relatively bright image. Furthermore, the high-quality images used as learning data may be high-quality images generated by another image quality improvement engine.

Even if the various techniques described above are used, when the low-quality image is a relatively dark medical image, the high-quality image engine may regard it as noise, etc., and therefore it is possible that parts that existed in the low-quality image may be partially lost in the high-quality image. In this case, for example, a composite image may be obtained by combining the low-quality image and the high-quality image in a ratio according to the brightness of the images, etc. In this way, for example, even if a part is lost in the high-quality image, it is possible to restore such a part to the high-quality image, since it exists in the low-quality image.

The high-quality image or composite image may be stored in the storage unit in response to an instruction from the examiner. At this time, after an instruction from the examiner to save the high-quality image or composite image, when registering the file name, a file name including information (e.g., characters) indicating that the image is generated by processing using a trained model for image quality improvement (image quality improvement processing) may be displayed in an editable state in response to an instruction from the examiner as a recommended file name in any part of the file name (e.g., the first part, the last part) in response to an instruction from the examiner. Also, in various display screens such as a report screen, as described above, when a high-quality image is displayed on the display unit, a display indicating that the displayed image is a high-quality image generated by processing using a trained model for image quality improvement may be displayed together with the high-quality image. In this case, the user can easily identify that the displayed high-quality image is not the image itself obtained by shooting by the display, thereby reducing misdiagnosis and improving diagnostic efficiency. Note that the display indicating that the image is a high-quality image generated by processing using a trained model for image quality improvement may be any type of display that allows the input image and the high-quality image generated by the processing to be distinguished from each other. In addition, not only for the processing using the trained model for image quality improvement, but also for the processing using various trained models as described above, a display indicating that the result is generated by the processing using the trained model of that type may be displayed together with the result. At this time, the display screen such as the report screen may be stored in the storage unit in response to an instruction from the examiner. For example, the report screen may be stored in the storage unit as one image in which the high-quality image or synthetic image, etc., and a display indicating that these images are high-quality images generated by the processing using the trained model for image quality improvement are arranged side by side. In addition, for the display indicating that the high-quality image is generated by the processing using the trained model for image quality improvement, a display indicating what kind of training data the trained model for image quality improvement has been trained with may be displayed on the display unit. The display may include a display of an explanation of the types of input data and correct answer data of the training data, any display regarding correct answer data such as the imaging site included in the input data and correct answer data, etc. In addition, not only for the processing using the trained model for image quality improvement, but also for the processing using various trained models as described above, a display indicating what kind of training data the trained model of that type has been trained with may be displayed on the display unit. Also, information (e.g., text) indicating that the image was generated by processing using a trained model for image quality improvement may be displayed or saved in a state where it is superimposed on the high-quality image or the composite image. In this case, the location to be superimposed on the image may be anywhere as long as it is an area (e.g., an edge of the image) that does not overlap with an area in which a site of interest to be photographed is displayed. Also, the non-overlapping area may be determined, and the information may be superimposed on the determined area.

In addition, when the button 3420 is set to be in an active state (high image quality processing is on) as the default display screen of the report screen, a report image corresponding to the report screen including a high image quality image or a composite image, etc. may be configured to be sent to the server in response to an instruction from the examiner. In addition, when the button 3420 is set to be in an active state as the default, a report image corresponding to the report screen including a high image quality image or a composite image, etc. may be (automatically) sent to the server at the end of the examination (for example, when the shooting confirmation screen or the preview screen is changed to the report screen in response to an instruction from the examiner). At this time, a report image generated based on various settings in the default settings (for example, settings related to at least one of the depth range for generating an En-Face image on the initial display screen of the report screen, whether or not an analysis map is superimposed, whether or not a high image quality image is displayed, whether or not a display screen for follow-up observation is displayed, etc.) may be configured to be sent to the server.

In addition, among the various trained models as described above, an image obtained by a first type of trained model (e.g., a high-quality image, an image showing an analysis result such as an analysis map, an image showing an object recognition result, an image showing a segmentation result) may be input to a second type of trained model different from the first type. At this time, a result by processing the second type of trained model (e.g., an analysis result, a diagnosis result, an object recognition result, a segmentation result) may be generated. In addition, among the various trained models as described above, an image to be input to a second type of trained model different from the first type may be generated from the image input to the first type of trained model using a result by processing the first type of trained model (e.g., an analysis result, a diagnosis result, an object recognition result, a segmentation result). At this time, the generated image is likely to be an image suitable for processing by the second type of trained model. Therefore, the accuracy of the image (e.g., a high-quality image, an image showing an analysis result such as an analysis map, an image showing an object recognition result, or an image showing a segmentation result) obtained by inputting the generated image into the second type of trained model can be improved. In addition, a similar image search may be performed using an external database stored on a server or the like, using the analysis result or diagnosis result obtained by processing the trained model as a search key. Note that, in cases where multiple images stored in a database are managed with the feature values of each of the multiple images already attached as additional information by machine learning or the like, a similar image search engine (similar image inspection model, trained model for similar image search) using the image itself as a search key may be used.

Other Embodiments
The present invention can also be realized by a process in which a program that realizes one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. The present invention also includes inventions that have been modified within the scope of the present invention and inventions equivalent to the present invention. In addition, the above-mentioned embodiments can be appropriately combined within the scope of the present invention.

Claims (10)

a designation means for designating a part of a depth range of a predetermined part of a subject in three-dimensional medical image data of the predetermined part in response to an instruction from an examiner;
an acquisition means for acquiring a first image, which is a medical image of the predetermined portion corresponding to the specified partial depth range, by using the three-dimensional medical image data;
an image quality improvement unit that uses an image quality improvement engine including a machine learning engine obtained using learning data including a plurality of medical images corresponding to a plurality of depth ranges of a predetermined part of a subject to generate a second image from the first image, the second image having higher image quality than the first image, and generates a composite image by combining pixel values of corresponding pixels in the first image and the second image at a ratio that can be changed in response to an instruction from an examiner ;
A medical image processing device comprising: The medical image processing device of claim 1, wherein the image quality improvement engine includes a machine learning engine obtained using learning data including a plurality of medical images obtained by adding noise of different magnitudes to at least a portion of each of a plurality of medical images corresponding to the plurality of depth ranges according to information regarding pixel values of at least a portion of each of the plurality of depth ranges. 3. The medical image processing apparatus according to claim 1, further comprising a wide-angle image generation unit that generates a wide-angle image using a plurality of first images obtained by photographing different positions of the specified part in a direction intersecting a depth direction of the specified part so that partial areas of the plurality of adjacent medical images corresponding to the specified partial depth range overlap, the plurality of second images obtained from the plurality of first images using the image quality improvement engine and a plurality of the first images by combining pixel values of each other for each corresponding pixel in the plurality of second images and the first images in the ratio. 4. The medical image processing device according to claim 1, wherein the image quality improvement engine includes a machine learning engine obtained using learning data including an image obtained by OCTA imaging using an OCT imaging device with higher performance than an OCT imaging device used for OCTA imaging of the first image, or an image obtained by an OCTA imaging process that requires more labor than the OCTA imaging process of the first image. 5. The medical image processing device according to claim 1, wherein the image quality improvement unit divides the first image into a plurality of two-dimensional images, inputs the images to the image quality improvement engine, and generates the second image by integrating the plurality of output images from the image quality improvement engine. The image quality improvement engine includes a machine learning engine obtained by using learning data including a plurality of medical images having corresponding positional relationships as paired images;
The medical image processing apparatus according to claim 5 , wherein the image quality improving unit divides the first image into the plurality of two-dimensional images with an image size corresponding to an image size of the paired images and inputs the divided images to the image quality improving engine. The medical image processing device according to claim 5 or 6, wherein the image quality improvement engine includes a machine learning engine obtained using learning data including images of a plurality of partial regions set so that adjacent partial regions partially overlap with each other for an area including a medical image and its external periphery. The medical image processing apparatus according to claim 1 , wherein the image quality improvement engine includes a machine learning engine obtained by using learning data including medical images obtained by superimposition processing. Specifying a depth range of a predetermined portion of the subject's three-dimensional medical image data in response to an instruction from an examiner;
acquiring a first image, which is a medical image of the predetermined portion corresponding to the specified partial depth range, using the three-dimensional medical image data;
generating a second image having higher image quality than the first image from the first image using an image quality improvement engine including a machine learning engine obtained using learning data including a plurality of medical images corresponding to a plurality of depth ranges of a predetermined part of a subject , and generating a composite image by combining pixel values of corresponding pixels in the first image and the second image at a ratio that can be changed in response to an instruction from an examiner ;
A medical image processing method comprising: A program that, when executed by a processor, causes the processor to perform each step of the medical image processing method according to claim 9 .

Images