JP7703349B2 - Image processing device, image processing method, and program - Google Patents

Description

The disclosed technology relates to an image processing device, an image processing method, and a program.

As a device for obtaining a tomographic image of a subject, a device using optical coherence tomography (OCT device) is known. By using a medical tomographic imaging device such as an OCT device, it is possible to observe the state inside the retinal layer in three dimensions, and such a medical tomographic imaging device is useful for diagnosing ophthalmic retinal diseases such as AMD. In recent years, OCT used in clinical settings is broadly divided into two methods, SD-OCT (Spectral Domain OCT) and SS-OCT (Swept Source OCT), as a method for acquiring images at high speed. In SD-OCT, a broadband light source is used and an interferogram is acquired with a spectrometer. In contrast, in SS-OCT, a high-speed wavelength swept light source is used as the light source, and spectral interference is measured with a single channel photodetector.

Recently, OCT angiography (OCTA), which uses no contrast agent to image blood vessels, has been attracting attention among both types of OCT. In OCTA, motion contrast data is generated from OCT images acquired by OCT. Here, motion contrast data refers to data obtained by repeatedly capturing images of the same cross section of the object to be measured using OCT, and detecting changes in the object over time between these captures. For example, the data is calculated from the difference, ratio, or correlation of changes in the phase, vector, or intensity of the complex OCT signal over time.

In addition, it is becoming common for OCTA images to be displayed as two-dimensional OCTA front images by projecting three-dimensional motion contrast data calculated from the acquired three-dimensional OCT images onto a two-dimensional plane. In this regard, Patent Document 1 discloses a technology for generating a two-dimensional front image by specifying the depth range of the motion contrast data to be projected. Furthermore, Patent Document 2 discloses a technology for generating a high-resolution image from a low-resolution image using an artificial intelligence engine obtained by machine learning using low-resolution and high-resolution images.

JP 2017-6179 A JP 2018-5841 A

Generally speaking, the more training data there is, the better. Also, it is better for the inference process (estimation process) to be performed quickly.

Therefore, one of the objectives of the disclosed technology is to use medical images with an image size suitable for learning and inference (estimation) as input data for a trained model.

In addition to the above objective, the present invention also has another objective to achieve effects that cannot be obtained by conventional techniques, which are derived from the configurations shown in the detailed description of the invention described below.

An image processing device relating to at least one embodiment of the disclosed technology includes an image processing unit having a trained model obtained by training using training data including input data that is a medical image having a first image size, and the image processing unit inputs a first image, which is a medical image having a second image size larger than the first image size, as input data to the trained model, and outputs a second image having the second image size as output data.

According to at least one embodiment of the disclosed technology, medical images with an image size suitable for learning and inference (estimation) can be used as input data for a trained model.

1 is a block diagram showing a schematic configuration of an OCT apparatus according to a first embodiment. 1 is a diagram illustrating a schematic configuration of an imaging device according to a first embodiment. FIG. 2 is a block diagram showing a schematic configuration of an image quality improving unit according to the first embodiment. 1 shows an example of the configuration of a neural network related to an image quality improvement engine. 1 shows an example of learning data related to image quality improvement processing. 1 shows an example of an input image for image quality improvement processing. FIG. 4 is a flowchart showing a schematic flow of image processing according to the first embodiment. FIG. 4 is a flowchart showing an example of a flow of image processing according to the first embodiment. FIG. 4 is a flowchart showing an example of a flow of image processing according to the first embodiment. FIG. 4 is a flowchart showing an example of a flow of image processing according to the first embodiment. FIG. 4 is a flowchart showing an example of a flow of image processing according to the first embodiment. FIG. 4 is a flowchart showing an example of a flow of image processing according to the first embodiment. FIG. 4 is a flowchart showing an example of a flow of image processing according to the first embodiment. FIG. 10 is a flowchart showing a schematic flow of image processing according to a second embodiment. 13 shows an example of a machine learning model related to variant example 8. 13 shows an example of a machine learning model related to variant example 8. 1 shows an example of a plurality of subset regions obtained by dividing an input image. An example of Self-Attention used in the Transformer is shown below.

Below, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. In addition, the same reference symbols are used in the drawings to indicate elements that are identical or functionally similar. In the following, the eye axis direction is described as Z, the horizontal direction of the fundus plane is described as X, and the vertical direction of the fundus plane is described as Y.

In the following, a machine learning model refers to a learning model based on a machine learning algorithm. Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. Another example is deep learning, which uses a neural network to generate features and connection weighting coefficients for learning. Any of the above algorithms that can be used can be used as appropriate and applied to the following embodiments and modified examples. Furthermore, teacher data refers to learning data, and is composed of a pair of input data and output data. Furthermore, correct answer data refers to the output data of the learning data (teacher data).

Note that a trained model refers to a machine learning model that follows any machine learning algorithm, such as deep learning, and that has been trained (learned) in advance using appropriate teacher data (learning data). However, although a trained model has been obtained in advance using appropriate learning data, this does not mean that no further learning is performed, and additional learning can also be performed. Additional learning can also be performed after the device is installed at the site of use.

(First embodiment)
An image processing system according to a first embodiment of the present invention will be described in detail below with reference to Figures 1 to 13. In this embodiment, as an example of an image processing system, an OCT device including an image processing device that processes a tomographic image of a subject acquired by OCT will be described.

(Configuration of image processing device)
The configuration of an image processing device 101 according to this embodiment and connections with other devices will be described with reference to FIG. 1. The image processing device 101 is connected to an image capture device 100, an external storage device 102, an output unit 103, and an input unit 104. These connections may be wired or wireless. These connections may also be made via a network. For example, the image processing device 101 may be connected to the image capture device 100 via a network such as the Internet. Also, for example, the external storage device 102 may be placed on a network such as the Internet, so that data can be shared by a plurality of image processing devices.

The image processing device 101 is provided with the following functional blocks: an acquisition unit 101-1, an imaging control unit 101-2, an image processing unit 101-4, a display control unit 101-5, and a storage unit 101-3. The image processing device 101 can be configured using a general computer including a processor, memory, etc., but may also be configured as a computer dedicated to the OCT device. Here, the image processing device 101 may be a computer built into (internal) the OCT device, or a separate (external) computer to which the OCT device is communicatively connected. The image processing device 101 may also be, for example, a personal computer, and a desktop PC, a notebook PC, or a tablet PC (portable information terminal) may also be used. The processor may be a CPU (Central Processing Unit). The processor may be, for example, a Micro Processing Unit (MPU), a Graphical Processing Unit (GPU), or a Field-Programmable Gate Array (FPGA).

Each function of the image processing device 101 may be realized by a processor such as a CPU or MPU executing a software module stored in the memory unit 101-3. The processor may be, for example, a GPU or FPGA. Each function may also be configured by a circuit that performs a specific function, such as an ASIC. For example, the image processing unit 101-4 may be realized by dedicated hardware such as an ASIC, and the display control unit 101-5 may be realized by a dedicated processor such as a GPU that is different from the CPU. The memory unit 101-3 may be configured by any storage medium, such as an optical disk such as a hard disk, or a memory.

The acquisition unit 101-1 is a functional block that acquires signal data such as SLO images, tomographic images, and anterior eye images obtained by photographing a subject using the imaging device 100, images such as SLO images, tomographic images, and anterior eye observation images, and patient information. The acquisition unit 101-1 can also generate images such as SLO images, tomographic images, and anterior eye observation images using the acquired signal data. The acquisition unit 101-1 is provided with a tomographic image generation unit 101-11 and a motion contrast data generation unit 101-12. The acquisition unit 101-1 may acquire various data including images from an external device (not shown), such as a server connected to the image processing device 101. Here, the image processing device 101 and the external device (not shown) may be connected via any network such as the Internet. The acquisition unit 101-1 may also acquire various data stored in the storage unit 101-3.

The tomographic image generating unit 101-11 performs signal processing on the signal data (interference signal) of the acquired tomographic image to generate a tomographic image, and stores the generated tomographic image in the storage unit 101-3. Note that any known method may be used to generate a tomographic image from the interference signal.

The motion contrast data generating unit 101-12 generates motion contrast data based on multiple tomographic images of approximately the same position (corresponding regions in the subject) generated by the tomographic image generating unit 101-11. The method of generating the motion contrast data is described below.

First, the tomographic image generating unit 101-11 generates multiple tomographic images from multiple interference signals acquired by capturing approximately the same position of the subject multiple times. Here, a group of scans of the measurement light when capturing approximately the same position of the subject multiple times to generate motion contrast data is called one cluster. Also, multiple tomographic images corresponding to multiple interference signals captured multiple times at the same position of the subject are called one cluster (one group) of tomographic images. More specifically, the tomographic image generating unit 101-11 generates one cluster of tomographic images by performing wave number transformation, fast Fourier transform (FFT), and absolute value transformation (amplitude acquisition) on the multiple interference signals acquired by the acquiring unit 101-1. Note that the method of generating the tomographic images is not limited to this, and any other known method of performing image processing may be used.

Next, the alignment unit 101-41 of the image processing unit 101-4, which will be described later, aligns the tomographic images belonging to the same cluster and performs overlay processing. After that, the image feature acquisition unit 101-44 of the image processing unit 101-4, which will be described later, acquires layer boundary data from the overlaid tomographic images. In this embodiment, a deformable model is used as a method for acquiring layer boundaries, but any known layer boundary acquisition method may be used. Note that the layer boundary acquisition process is not essential, and the layer boundary acquisition process can be omitted, for example, if the motion contrast image is generated only in three dimensions and a two-dimensional motion contrast image projected in the depth direction is not generated.

The motion contrast data generating unit 101-12 calculates motion contrast data between adjacent tomographic images in the same cluster. In this embodiment, the motion contrast data generating unit 101-12 calculates the decorrelation value Mxz as the motion contrast data based on the following formula (1).

Here, Axz indicates the amplitude (of the complex data after FFT processing) at the position (x, z) of the tomographic image data A, and Bxz indicates the amplitude at the same position (x, z) of the tomographic data B. 0≦Mxz≦1, and the greater the difference between the two amplitude values, the closer to 1 the value is. The motion contrast data generating unit 101-12 performs decorrelation calculation processing as shown in formula (1) between any temporally adjacent tomographic images (belonging to the same cluster). Note that the tomographic images to be subjected to decorrelation calculation processing do not necessarily have to be temporally adjacent as long as they are images within a specified period. The motion contrast data generating unit 101-12 generates an image having the average of the obtained (number of tomographic images per cluster - 1) motion contrast data values as the final motion contrast image.

Note that here, the motion contrast data is calculated based on the amplitude of the complex data after FFT processing, but the method of calculating the motion contrast data is not limited to the above. For example, the motion contrast data may be calculated based on the phase information of the complex data, or based on both the amplitude and phase information. Alternatively, the motion contrast data may be calculated based on the real part and imaginary part of the complex data.

In addition, in this embodiment, a decorrelation value is calculated as the motion contrast data, but the calculation method of the motion contrast data is not limited to this, and any known calculation method may be used. For example, the motion contrast data may be calculated based on the difference between two values, or based on the ratio of two values. Furthermore, the motion contrast data can be calculated, for example, as the variance value between two tomographic images or the corresponding interference signals, or the value obtained by dividing the maximum value by the minimum value (maximum value/minimum value). Any known calculation method may be used for these calculations.

In addition, when controlling the scanning means so that the measurement light scans approximately the same position multiple times, the time interval between one scan (one B-scan) and the next scan (next B-scan) may be changed (determined). This allows the vascular region to be visualized with high accuracy, even if the blood flow speed differs depending on the state of the blood vessels, for example.

At this time, for example, the time interval may be changed in response to an instruction from an operator (examiner). Also, for example, in response to an instruction from an operator, any one of a plurality of motion contrast images corresponding to a plurality of pre-set time intervals may be selected. Furthermore, for example, the time interval when the motion contrast data is acquired may be associated with the motion contrast data and stored in the storage unit 101-3. Also, for example, the display control unit 101-5 may cause the output unit 103 to display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data. Furthermore, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. At this time, for example, the time interval may be determined (output) from the motion contrast image using a machine learning model. Such a machine learning model can be obtained, for example, by learning learning data in which a plurality of motion contrast images corresponding to a plurality of time intervals are used as input data, and the difference from the plurality of time intervals to the time interval when a desired motion contrast image is acquired is used as correct answer data.

Furthermore, here, the pixel value of the final motion contrast image is obtained by calculating the average value of the multiple decorrelation values obtained, but the final pixel value is not limited to this. For example, an image having a pixel value that is the median or maximum value of the multiple decorrelation values obtained may be generated as the final motion contrast image.

The imaging control unit 101-2 is a functional block that performs imaging control for the imaging device 100. For example, the imaging control unit 101-2 can perform drive control of the light source, scanning unit, and focusing lens drive unit included in the imaging device 100. The imaging control unit 101-2 can control the alignment operation of the stage unit 100-2 of the imaging device 100, which will be described later. Furthermore, the imaging control unit 101-2 can also instruct the imaging device 100 regarding the setting of imaging parameters and the start or end of imaging.

The storage unit 101-3 can store programs for implementing various application software including an operating system (OS), device drivers for peripheral devices, and programs for performing the processes described below. The storage unit 101-3 can also store information acquired by the acquisition unit 101-1 and various images processed by the image processing unit 101-4. For example, the storage unit 101-3 can store medical images such as tomographic images acquired by the acquisition unit 101-1, and store images whose image quality has been improved by the image quality improvement unit 101-47 described below.

The image processing unit 101-4 is a functional block that performs image processing on various images such as tomographic images, motion contrast images, SLO images, and anterior eye observation images. The image processing unit 101-4 includes an alignment unit 101-41, a synthesis unit 101-42, a correction unit 101-43, an image feature acquisition unit 101-44, a projection unit 101-45, an analysis unit 101-46, and an image quality improvement unit 101-47.

The alignment unit 101-41 is a functional block that performs alignment processing between images. For example, the alignment unit 101-41 can align tomographic images that belong to the same cluster and perform overlay processing. Note that the alignment processing and overlay processing may be performed using any known method.

The synthesis unit 101-42 is a functional block that synthesizes a single image from a number of two-dimensional images. The synthesis unit 101-42 includes, for example, a synthesis method designation unit 101-421, a same-modality image synthesis unit 101-422, and a different-modality image synthesis unit 101-423.

The synthesis method designation unit 101-421 designates the type of images to be synthesized (e.g., tomographic image/motion contrast image/tomographic image and motion contrast image) and the synthesis processing method (e.g., overlay/stitching/side-by-side display). The same-modality image synthesis unit 101-422 performs synthesis processing, for example, between tomographic images or between motion contrast images. The different-modality image synthesis unit 101-423 performs synthesis processing between images obtained by different types of modalities, such as between tomographic images and motion contrast images.

The correction unit 101-43 is a functional block that performs processing to suppress projection artifacts that occur in motion contrast images. Here, the projection artifact refers to a phenomenon in which motion contrast in superficial retinal blood vessels is reflected on the deeper layer side (deep retina, outer retina, choroid), resulting in a high decorrelation value in the deeper layer area where no blood vessels actually exist. For example, the correction unit 101-43 performs processing to reduce projection artifacts in the generated motion contrast data. Therefore, the correction unit 101-43 corresponds to an example of a processing means that performs processing to reduce projection artifacts on the generated motion contrast data.

The image feature acquisition unit 101-44 acquires layer boundary data from tomographic images, motion contrast tomographic images, etc. Specifically, the image feature acquisition unit 101-44 performs image segmentation processing on the tomographic images, etc., extracts the layer structure in the tomographic slice of the subject's eye, and acquires layer boundary data such as boundary positions. Note that any known method may be used as the method for image segmentation processing.

The projection unit 101-45 is a functional block that projects or accumulates a tomographic image or a motion contrast image in a set depth range to generate a luminance en-face image (luminance En-Face image) or an OCTA en-face image. The depth range can be set using two reference planes based on the operator's instructions or layer boundary data acquired by the image feature acquisition unit 101-44. The depth range that is set may be any depth range. For example, the depth ranges of the retinal surface layer and the outer retinal layer can be set, and two types of synthesized OCTA en-face images can be generated by the synthesis unit 101-42.

Here, as a method for projecting data corresponding to a depth range set based on two reference planes onto a two-dimensional plane, for example, a method can be used in which a representative value of the data within the depth range is set as a pixel value on the two-dimensional plane. The representative value can include values such as the average, median, or maximum value of pixel values within the depth range of the area surrounded by the two reference planes. Specifically, for example, maximum intensity projection (MIP) or average intensity projection (AIP) can be selected as a projection method.

The depth range of the front image may be, for example, a range including a predetermined number of pixels in the deeper or shallower direction based on one of the two layer boundaries related to the detected retinal layer. The depth range of the En-Face image may be, for example, a range changed (offset) from the range between the two layer boundaries related to the detected retinal layer according to an instruction from the operator. Here, the depth range for generating the front image can be changed by the operator selecting from a default depth range set displayed in a selection list (not shown). The operator can also change the projection range by changing the type and offset position of the layer boundary used to specify the projection range from the user interface (UI) or by operating the input unit 104 to move the layer boundary data superimposed on the tomographic image.

The generated luminance front image and OCTA front image can be output by the output unit 103. The motion contrast image output to the output unit 103 is not limited to the OCTA front image, but may be a three-dimensional motion contrast image rendered three-dimensionally or a motion contrast tomographic image corresponding to a tomographic image. When the generated motion contrast image or the like is displayed by the output unit 103, the above-mentioned projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a UI such as a context menu. For example, the motion contrast image after the projection artifact suppression processing may be displayed as a three-dimensional image.

The analysis unit 101-46 is a functional block that performs analysis processing of various images such as tomographic images and motion contrast images. The analysis unit 101-46 includes an emphasis unit 101-461, an extraction unit 101-462, a measurement unit 101-463, and a comparison unit 101-464.

The highlighting unit 101-461 can perform highlighting processing to highlight data of any region in the image, for example, in response to an instruction from the operator. The extraction unit 101-462 can extract features and regions in the image used by the analysis unit 101-46. For example, the extraction unit 101-462 can obtain the layer boundaries of the retina and choroid, the boundaries of the anterior and posterior surfaces of the lamina cribrosa, the positions of the fovea and the center of the optic disc, and the like, from a tomographic image. The extraction unit 101-462 can also extract a vascular region from an OCTA front image. The measurement unit 101-463 can, for example, calculate measurement values to be analyzed from various images. For example, the measurement unit 101-463 can calculate the thickness of a specific layer, or calculate measurement values such as vascular density using the extracted vascular region or vascular centerline data obtained by thinning the vascular region. The comparison unit 101-464 can compare multiple images, such as multiple tomographic images and multiple motion contrast images. Furthermore, the comparison unit 101-464 can also compare the analysis results of multiple images, such as the measurement values calculated by the measurement unit 101-463.

The image quality improvement unit 101-47 is a functional block that improves the image quality of various images. Therefore, the image quality improvement unit 101-47 is an example of an image quality improvement unit that improves the image quality of medical images such as tomographic images and OCTA front images. The configuration and functions of the image quality improvement unit 101-47 will be described in detail later.

The display control unit 101-5 can control the display of the output unit 103 connected to the image processing device 101. The display control unit 101-5 can, for example, cause the output unit 103 to display a tomographic image of the subject's eye, various data such as patient information, etc.

The external storage device 102 can store programs for tomography, patient information, image information, etc. For example, the external storage device 102 can store patient information (patient's name, age, sex, etc.) and information on the examined eye (left eye, right eye, axial length, etc.) in association with captured images (tomographic images, SLO images, OCTA images, etc.), composite images, shooting parameters, image data and measurement data from previous examinations, parameters set by the operator, etc.

The input unit 104 includes, for example, a mouse, a keyboard, a touch panel, etc., and the operator can give instructions to the image processing device 101 and the imaging device 100 via the input unit 104. The output unit 103 can output data such as various images processed by the image processing device 101. The output unit 103 can be configured with any display, etc., and can display, for example, patient information and various images related to the subject based on the control of the display control unit 101-5. In this case, the output unit 103 can function as an example of a display unit that displays various images and information. The output unit 103 may also be equipped with a touch UI, etc.

Note that some components of the OCT device may be configured as separate devices, or may be configured as an integrated device. For example, the output unit 103 may be configured as a touch panel display integrated with the input unit 104.

(Configuration of the imaging device)
The imaging device 100 is a device that images the subject's eye in order to obtain a tomographic image, an SLO image, or the like of the subject's eye. Hereinafter, the configuration of the measurement optical system and the spectroscope in the imaging device 100 according to this embodiment will be described with reference to Figs. 1 and 2. In this embodiment, an imaging device including an optical system of SD-OCT (Spectral Domain OCT) is used as the imaging device 100. However, the imaging device may be configured using an imaging device including an optical system of SS-OCT or TD-OCT (Time Domain OCT), for example.

As shown in FIG. 1, the imaging device 100 is provided with a measurement optical system 100-1, a stage unit 100-2, and a base unit 100-3. The measurement optical system 100-1 is an optical system for acquiring an anterior eye image, an SLO fundus image of the subject's eye, and a tomographic image. The stage unit 100-2 holds the measurement optical system 100-1 so that it can move back and forth and left and right, and can move the measurement optical system 100-1 under the control of the imaging control unit 101-2. The base unit 100-3 has a built-in spectroscope 230, which will be described later.

The configuration of the measurement optical system 100-1 will be described with reference to FIG. 2. In the measurement optical system 100-1, an objective lens 201 is placed facing the test eye 200, and a first dichroic mirror 202 and a second dichroic mirror 203 are arranged on its optical axis. The optical path from the objective lens 201 is branched by these dichroic mirrors into an optical path 250 for the OCT optical system, an optical path 251 for the SLO optical system and fixation lamp, and an optical path 252 for anterior eye observation, for each wavelength band.

In this embodiment, an optical path 252 for anterior eye observation is arranged in the reflection direction of the first dichroic mirror 202, and an optical path 250 for the OCT optical system and an optical path 251 for the SLO optical system and fixation lamp are arranged in the transmission direction of the first dichroic mirror 202. In addition, an optical path 250 for the OCT optical system is arranged in the reflection direction of the second dichroic mirror 203, and an optical path 251 for the SLO optical system and fixation lamp is arranged in the transmission direction of the second dichroic mirror 203. However, the arrangement of each optical path relative to the dichroic mirror may be reversed.

The optical path 251 for the SLO optical system and fixation lamp includes an SLO scanning unit 204, lenses 205 and 206, a mirror 207, a third dichroic mirror 208, an APD (Avalanche Photodiode) 209, an SLO light source 210, and a fixation lamp 211. The mirror 207 is configured using a prism on which a perforated mirror or a hollow mirror is vapor-deposited, and separates the illumination light from the SLO light source 210 and the return light of the illumination light from the subject's eye 200. The third dichroic mirror 208 branches the optical path from the third dichroic mirror 208 into the optical path of the SLO light source 210 and the optical path of the fixation lamp 211 for each wavelength band. In this embodiment, the optical path of the SLO light source 210 is arranged in the reflection direction of the third dichroic mirror 208, and the fixation lamp 211 is arranged in the transmission direction of the third dichroic mirror 208. However, the arrangement of each optical path relative to the dichroic mirror may be reversed.

The SLO scanning unit 204 scans the illumination light emitted from the SLO light source 210 on the subject's eye 200, and is composed of an X scanner that scans in the X direction and a Y scanner that scans in the Y direction. The SLO scanning unit 204 is controlled by the imaging control unit 101-2. In this embodiment, the X scanner is a polygon mirror that performs high-speed scanning, and the Y scanner is composed of a galvanometer mirror. However, the configuration of the SLO scanning unit 204 is not limited to this, and the X scanner and Y scanner may be configured using any deflection mirror depending on the desired configuration.

The lens 205 is driven in the optical axis direction by a motor (not shown) to focus the SLO optical system and the fixation lamp 211. The motor for driving the lens 205 is controlled by the imaging control unit 101-2.

The SLO light source 210 generates light with a wavelength of about 780 nm. The APD 209 detects the return light from the subject's eye 200. The fixation lamp 211 generates visible light to encourage the subject to fixate.

The illumination light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208, passes through the mirror 207, and passes through the lenses 206 and 205 to reach the SLO scanning unit 204, where it is scanned on the subject's eye 200 by the SLO scanning unit 204. The return light from the subject's eye 200 travels the same path as the illumination light, is reflected by the mirror 207, and is guided to the APD 209.

The APD 209 generates signal data of an SLO fundus image based on the returned light and outputs it to the image processing device 101. The acquisition unit 101-1 of the image processing device 101 generates an SLO image based on the signal data output from the APD 209. An SLO image of the fundus of the test eye 200 can be acquired by scanning the illumination light emitted from the SLO light source 210 over the fundus of the test eye 200. In contrast, an SLO image of the anterior segment of the test eye 200 can also be acquired by scanning the anterior segment of the test eye with the illumination light emitted from the SLO light source 210.

The light emitted from the fixation lamp 211 passes through the third dichroic mirror 208 and mirror 207, passes through the lenses 206 and 205, and is formed into a predetermined shape at any position on the subject's eye 200 by the SLO scanning unit 204. By having the subject gaze at the light emitted from the fixation lamp 211, the subject's fixation can be encouraged.

Lenses 212 and 213, a split prism 214, and a CCD 215 for anterior eye observation that detects infrared light are arranged in the optical path 252 for anterior eye observation. The CCD 215 has sensitivity to the wavelength of the irradiation light for anterior eye observation (not shown), specifically, around 970 nm. The data signal output from the CCD 215 is output to the image processing device 101, and the image processing device 101 can generate an anterior eye observation image based on the input signal data.

The split prism 214 is disposed at a position conjugate with the pupil of the test eye 200. The image processing device 101 can detect the distance of the measurement optical system 100-1 in the Z-axis direction (optical axis direction) from the test eye 200 using a split image of the anterior eye based on light that has passed through the split prism 214.

The optical path 250 for the OCT optical system is provided with an OCT optical system, and the OCT optical system is used to capture a tomographic image of the test eye 200. More specifically, the OCT optical system is used to obtain an interference signal for generating a tomographic image. The optical path 250 for the OCT optical system is provided with an XY scanner 216, lenses 217 and 218, and an optical fiber 224.

The XY scanner (OCT scanning unit) 216 is for scanning the measurement light on the test eye 200. The XY scanner 216 is controlled by the imaging control unit 101-2. Although the XY scanner 216 is illustrated as a single mirror in FIG. 2, it is actually a galvanometer mirror that performs scanning in two axial directions, X and Y. However, the configuration of the XY scanner 216 is not limited to this, and may be configured using any deflection mirror depending on the desired configuration. For example, the XY scanner 216 may be configured using any deflection means, such as a MEMS mirror that can scan the measurement light in two-dimensional directions with a single mirror.

The lens 217 is driven in the optical axis direction by a motor (not shown) to focus the measurement light emitted from the optical fiber 224 connected to the optical coupler 219 on the test eye 200. By this focusing, the measurement light returning from the test eye 200 is simultaneously imaged as a spot on the tip of the optical fiber 224 and is incident thereon. The motor for driving the lens 217 is controlled by the imaging control unit 101-2.

Next, the configuration of the optical path from the OCT light source 220, the reference optical system, and the spectrometer will be described. The OCT optical system further includes the OCT light source 220, a reference mirror 221, dispersion compensation glass 222, a lens 223, an optical coupler 219, single-mode optical fibers 224-227 that are connected to and integrated with the optical coupler, and a spectrometer 230. The OCT optical system constitutes a Michelson interferometer with these components.

The OCT light source 220 is a super luminescent diode (SLD), which is a typical low-coherence light source. The central wavelength is 855 nm, and the wavelength bandwidth is about 100 nm. Here, the bandwidth is an important parameter because it affects the resolution in the optical axis direction of the tomographic image obtained. The type of light source selected here is an SLD, but any light source that can emit low-coherent light can be used, such as an amplified spontaneous emission (ASE). Considering that the eye is being measured, near-infrared light is suitable for the central wavelength. In addition, since the central wavelength affects the lateral resolution of the tomographic image obtained, it should be as short as possible. In this embodiment, the central wavelength of the OCT light source 220 is set to 855 nm for both reasons.

The light emitted from the OCT light source 220 is split through the optical fiber 225 into measurement light that is incident on the optical fiber 224 side via the optical coupler 219 and reference light that is incident on the optical fiber 226 side. The measurement light is irradiated to the subject's eye 200, which is the observation target, through the optical path 250 for the OCT optical system, and reaches the optical coupler 219 as return light through the same optical path due to reflection and scattering by the subject's eye 200. On the other hand, the reference light reaches the reference mirror 221 through the optical fiber 226, the lens 223, and the dispersion compensation glass 222 inserted to match the wavelength dispersion of the measurement light and the reference light, and is reflected. The reference light reflected by the reference mirror 221 returns along the same optical path and reaches the optical coupler 219.

The measurement light and reference light are combined by the optical coupler 219 to become interference light. Here, the measurement light and reference light interfere with each other when the optical path length of the measurement light and the optical path length of the reference light are approximately the same. The reference mirror 221 is held adjustable in the optical axis direction by a motor and drive mechanism (not shown) controlled by the imaging control unit 101-2, and it is possible to match the optical path length of the reference light to the optical path length of the measurement light. The interference light is guided to the spectroscope 230 via the optical fiber 227.

In addition, polarization adjustment units 228 and 229 are provided in the optical fibers 224 and 226. The polarization adjustment units 228 and 229 adjust the polarization of the measurement light and the reference light, respectively. The polarization adjustment units 228 and 229 have several sections in which the optical fiber is pulled around in a loop shape. The polarization adjustment units 228 and 229 rotate these looped sections around the longitudinal direction of the optical fiber to twist the optical fiber, thereby adjusting and matching the polarization states of the measurement light and the reference light.

The spectrometer 230 is provided with lenses 232 and 234, a diffraction grating 233, and a line sensor 231. The interference light emitted from the optical fiber 227 is converted into parallel light via the lens 234, then dispersed by the diffraction grating 233, and imaged on the line sensor 231 by the lens 232.

The interference light is measured by the line sensor 231 as intensity information for each wavelength. The intensity information for each wavelength measured by the line sensor 231 is output to the image processing device 101 as signal data of a tomographic image (interference signal). The image processing device 101 can generate a tomographic image of the subject's eye 200 using the received signal data.

In this embodiment, a Michelson interferometer is used as the interferometer, but a Mach-Zehnder interferometer may also be used. For example, depending on the light intensity difference between the measurement light and the reference light, a Mach-Zehnder interferometer can be used when the light intensity difference is large, and a Michelson interferometer can be used when the light intensity difference is relatively small.

(High-quality image processing)
The image quality improvement process for motion contrast data acquired by OCT will be described below in detail, but the terms used in the description of the embodiment will be briefly defined. First, information on three-dimensional volume data related to an interference signal or a tomographic image of brightness based thereon is referred to as OCT data, and information on three-dimensional volume data related to motion contrast is referred to as OCTA data. Furthermore, two-dimensional information that can be extracted from three-dimensional volume data related to an interference signal or a tomographic image of brightness based thereon is referred to as an OCT image, and two-dimensional information that can be extracted from three-dimensional volume data related to motion contrast is referred to as an OCTA image. In particular, an image generated by projecting or accumulating three-dimensional volume data related to an interference signal or a tomographic image of brightness based thereon within a specified depth range is referred to as an OCT front image (brightness En-Face). Furthermore, an image generated by projecting or accumulating three-dimensional volume data related to motion contrast is referred to as an OCTA front image. Furthermore, a two-dimensional image including depth data is referred to as a tomographic image.

(High image quality section)
3 is a block diagram showing a more detailed configuration of the image quality improvement unit 101-47. The image quality improvement unit 101-47 includes an image quality improvement processing unit 301, an area detection unit 302, an ROI setting unit 303, a blend processing unit 304, and a BC adjustment unit 305.

The image quality improvement processing unit 301 performs image quality improvement processing on the medical image acquired by the acquisition unit 101-1 using a trained model for image quality improvement (image quality improvement engine, image quality improvement model) described below, and generates a high-quality image. The image quality improvement processing unit 301 can function as an example of an image quality improvement unit that performs image quality improvement processing on a medical image of a subject (first image) using a trained model obtained by learning using a medical image of the subject as training data, and acquires a high-quality image of the subject (second image). The medical image to be image quality improved may be an OCT image, an OCTA image, a tomographic image, an OCT front image, an OCTA front image, an SLO image, an anterior eye observation image, or an analysis map obtained by analyzing these images.

The region detection unit 302 performs region detection processing on a medical image or a medical image with high image quality using a trained model for region detection (region detection engine, region detection model) described later, and detects at least one region (e.g., a target region). For example, the region detection unit 302 can detect a target region and a region other than the target region. The region detection unit 302 can function as an example of a detection unit that detects a target region in a medical image or a high image quality image of a subject. For example, the region detection unit 302 detects a perfusion region and a non-perfusion region from an OCTA front image using a region detection engine. In fundus diagnosis, by identifying the perfusion region and the non-perfusion region, it is possible to determine whether there is no blood flow where blood vessels should be, or whether some blood flow is observed where there are no blood vessels (such as neovascularization). Note that the target region is not limited to the perfusion region and the non-perfusion region, and may include a region where overcorrection occurs by the high image quality engine. For example, the target region may include a foveal avascular region, an optic disc, etc. in addition to the perfusion region and the non-perfusion region.

The classification of regions may be managed by attribute information (labels) that indicate each region. For example, the region detection unit 302 may generate a label image having attribute information as pixel values corresponding to each pixel position of the medical image. In addition to the attribute information, a value indicating the likelihood (reliability, probability) of each piece of attribute information output from the region detection engine may be stored in association with each pixel of the medical image.

The ROI setting unit 303 sets an ROI (Region Of Interest) based on the region detected by the region detection unit 302. The ROI can be at least one of the regions (target regions) detected by the region detection unit 302. The ROI may be automatically set by the region detection unit 302 as a region having predetermined attribute information, in which case the ROI setting unit 303 may be omitted.

The blending processing unit 304 performs blending, which is a type of image processing, on a set ROI, which is an area having predetermined attribute information, using a medical image and a high-quality image of the medical image. Any known process such as alpha blending may be used as the blending process. The blending processing unit 304 can function as an example of an image processing unit that performs image processing on a target area in a medical image or a high-quality image so as to increase the difference between the pixel values of the target area and the pixel values of areas other than the target area and to lower the pixel values of the target area.

The BC adjustment unit 305 performs a BC (Brightness, Contrast) adjustment process, which is a correction process of at least one of brightness and contrast, on a set ROI, which is an area having predetermined attribute information. Here, the contrast correction process may include any known contrast correction process, and may include, for example, correction using a tone curve or gamma curve or level correction. The BC adjustment unit 305 can also function as an example of an image processing unit that performs image processing on a target area in a medical image or high-quality image so as to increase the difference between the pixel values of the target area and the pixel values of areas other than the target area and to lower the pixel values of the target area.

Note that only one of the blending processing unit 304 and the BC adjustment unit 305 may be provided. Furthermore, when both the blending processing unit 304 and the BC adjustment unit 305 are provided, the image processing device 101 may be configured to be able to select either or both of the blending processing and the BC adjustment processing as the image processing to be executed, in response to an instruction from an operator. Note that, when executing both the blending processing and the BC adjustment processing, the image processing device 101 may be configured to switch between or arrange the images subjected to each image processing in response to an operation of a UI or the like, and display them on the output unit 103.

(Medical image quality improvement engine)
The image quality improvement engine used by the image quality improvement processing unit 301 will be described below with reference to Figures 4 to 7. Note that, although the image quality improvement engine is configured to be stored in the storage unit 101-3 in this embodiment, the image quality improvement engine may be provided in an external device connected to the image processing device 101. In this case, the image quality improvement processing unit 301 can perform image quality improvement processing using the image quality improvement engine provided in the external device.

The image quality improvement engine according to this embodiment is a trained model obtained by performing training (learning) on a machine learning algorithm. In this embodiment, training of the machine learning model on the machine learning algorithm uses learning data consisting of a pair group of input data, which is a low-quality image having specific shooting conditions assumed to be the processing target, and output data, which is a high-quality image corresponding to the input data. Note that the specific shooting conditions specifically include a predetermined shooting part, shooting method, shooting angle of view, image size, etc.

Here, we will briefly explain a general trained model. A trained model is a machine learning model that has been trained (learned) in advance using appropriate training data for an arbitrary machine learning algorithm. The training data consists of one or more pairs of input data and output data (correct answer data). The format and combination of the input data and output data of the pairs that make up the training data may be suitable for the desired configuration, such as one being an image and the other being a numerical value, one being composed of a group of multiple images and the other being a string, or both being images.

Specific examples of such data include training data (hereinafter referred to as "first training data") that is composed of a set of pairs of images acquired by OCT and imaging site labels corresponding to the images. The imaging site labels are unique numbers or character strings that represent the site. Other examples of training data include training data (hereinafter referred to as "second training data") that is composed of a set of pairs of low-quality images with a lot of noise acquired by normal OCT imaging and high-quality images that have been imaged multiple times by OCT and processed to improve image quality.

At this time, when input data is input to the trained model, output data according to the design of the trained model is output. The trained model outputs output data that is likely to correspond to the input data, for example, according to trends trained using the training data. The trained model can also output the likelihood (reliability, probability) of the input data corresponding to each type of output data as a numerical value, for example, according to trends trained using the training data.

Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first learning data, the machine learning model outputs the imaging site label of the imaging site captured in the image, and outputs the probability for each imaging site label. Also, for example, when a noisy, low-quality image acquired by normal OCT imaging is input to a machine learning model trained with the second learning data, the machine learning model outputs a high-quality image equivalent to an image captured multiple times by OCT and processed to improve image quality. Note that the machine learning model can also be configured not to use the output data it outputs as learning data from the perspective of maintaining quality.

The machine learning algorithm also includes deep learning techniques such as a convolutional neural network (CNN). In deep learning techniques, different parameter settings for layers and nodes constituting a neural network may result in different degrees of reproducibility of trends trained using learning data in output data. For example, in a deep learning machine learning model using the first learning data, setting more appropriate parameters may increase the probability of outputting a correct imaging site label. Also, for example, in a deep learning machine learning model using the second learning data, setting more appropriate parameters may result in higher quality images being output.

Specifically, parameters in a CNN can include, for example, the filter kernel size, number of filters, stride value, and dilation value set for the convolutional layer, as well as the number of nodes output by the fully connected layer. The parameter group and the number of training epochs can be set to values that are favorable for the usage form of the trained model based on the training data. For example, a parameter group and the number of epochs can be set based on the training data to output the correct imaging site label with a higher probability or to output higher quality images.

One method for determining such a parameter set and the number of epochs is exemplified below. First, 70% of the pair sets constituting the learning data are set for training, and the remaining 30% are set for evaluation, randomly. Next, the machine learning model is trained using the pair sets for training, and at the end of each epoch of training, a training evaluation value is calculated using the pair sets for evaluation. The training evaluation value is, for example, the average value of a group of values obtained by evaluating the output when the input data constituting each pair is input to the machine learning model being trained, and the output data corresponding to the input data, using a loss function. Finally, the parameter set and the number of epochs when the training evaluation value is the smallest are determined as the parameter set and the number of epochs of the machine learning model. In this way, by dividing the pair sets constituting the learning data into those for training and those for evaluation and determining the number of epochs, it is possible to prevent the machine learning model from overlearning the pair sets for training.

Here, the image quality improvement engine according to the present embodiment is configured as a module that improves the image quality of an input low-image quality image and outputs a high-image quality image. Here, in this specification, improving the image quality refers to converting an input image into an image with image quality more suitable for image diagnosis, and a high-image quality image refers to an image converted into an image with image quality more suitable for image diagnosis. In addition, a low-image quality image is, for example, a two-dimensional image or a three-dimensional image acquired by X-ray photography, CT, MRI, OCT, PET, or SPECT, or a three-dimensional moving image of a CT taken continuously without being set to have a particularly high image quality. Specifically, low-image quality images include, for example, images acquired by low-dose photography using an X-ray photography device or CT, photography using an MRI without using a contrast agent, images acquired by short-time photography using OCT, and OCTA images acquired with a small number of photography sessions.

In addition, when high-quality images with low noise and high contrast are used for image analysis such as vascular analysis processing of images such as OCTA, or area segmentation processing of images such as CT or OCT, the analysis can often be performed more accurately than when low-quality images are used. Therefore, high-quality images output by the image quality engine can be useful not only for image diagnosis, but also for image analysis.

The content of image quality suitable for diagnostic imaging depends on what is being diagnosed with various types of diagnostic imaging. Therefore, it is difficult to generalize, but for example, image quality suitable for diagnostic imaging includes image quality with little noise, high contrast, colors and gradations that make it easy to observe the subject, large image size, and high resolution. It can also include image quality in which non-existent objects and gradations that are drawn during the image generation process are removed from the image.

In the image processing method constituting the image quality improvement method by the image quality improvement processing unit 301 in this embodiment, processing using various machine learning algorithms such as deep learning is performed. Note that in addition to processing using machine learning algorithms, the image processing method may also perform any existing processing such as various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing.

Below, with reference to FIG. 4, an example of the configuration of a CNN related to the image quality improvement engine according to this embodiment will be described. FIG. 4 shows an example of the configuration of an image quality improvement engine. The configuration shown in FIG. 4 is composed of a plurality of layers that are responsible for processing and outputting a group of input values. Note that the types of layers included in this configuration include a convolution layer, a downsampling layer, an upsampling layer, and a merging layer, as shown in FIG. 4.

The convolution layer is a layer that performs convolution processing on input value groups according to parameters such as the set filter kernel size, number of filters, stride value, and dilation value. The number of dimensions of the filter kernel size may also be changed according to the number of dimensions of the input image. The downsampling layer is a process that reduces the number of output value groups to be less than the number of input value groups by thinning or synthesizing the input value groups. Specifically, there is a Max Pooling process, for example. The upsampling layer is a process that increases the number of output value groups to be more than the number of input value groups by duplicating the input value groups or adding values interpolated from the input value groups. Specifically, there is a linear interpolation process, for example. The composition layer is a layer that inputs value groups such as the output value groups of a certain layer or pixel values that make up an image from multiple sources and performs a process of synthesizing them by connecting or adding them.

In this configuration, the pixel values constituting the input image Im410 are output through a convolution processing block, and the pixel values constituting the input image Im410 are synthesized in a synthesis layer. The synthesized pixel values are then shaped into a high-quality image Im420 in the final convolution layer. Although not shown, as an example of a modification to the CNN configuration, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer.

The GPU can perform efficient calculations by processing more data in parallel. For this reason, when performing learning multiple times using a learning model such as deep learning, it is effective to perform the processing using a GPU. Therefore, in this embodiment, a GPU is used in addition to a CPU for processing by the image processing unit 101-4, which is an example of a learning unit. Specifically, when executing a learning program including a learning model, the CPU and GPU work together to perform calculations to perform learning. Note that calculations may be performed only by the CPU or the GPU in the processing of the learning unit. Furthermore, the image quality improvement processing unit 301 may also be realized using a GPU, like the learning unit.

The learning unit may also include an error detection unit and an update unit, both of which are not shown. The error detection unit obtains the error between the correct data and the output data output from the output layer of the neural network in response to the input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the correct data. The update unit updates the connection weighting coefficients between the nodes of the neural network, etc., based on the error obtained by the error detection unit, so as to reduce the error. This update unit updates the connection weighting coefficients, etc., using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the above-mentioned error.

Note that when using some image processing methods, such as image processing using CNN, attention must be paid to image size. Specifically, it should be noted that in order to address issues such as the issue of the peripheral areas of a high-quality image not being of sufficient quality, different image sizes may be required for the input low-quality image and the output high-quality image.

For the sake of clarity, this is not specified in this embodiment, but when a high-image-quality engine is adopted that requires different image sizes for the image input to the high-image-quality engine and the image output, the image size is adjusted appropriately. Specifically, the image size is adjusted for input images, such as images used as learning data for training a machine learning model or images input to the high-image-quality engine, by padding or combining the surrounding shooting area of the input image. Note that the area to be padded is filled with a fixed pixel value, filled with nearby pixel values, or mirror padded according to the characteristics of the high-image-quality technique so that the image quality can be effectively improved.

The image quality improvement method by the image quality improvement processing unit 301 may be implemented using only one image processing method, or may be implemented using a combination of two or more image processing methods. A plurality of image quality improvement methods may be implemented in parallel to generate a plurality of high-quality image groups, and the highest quality image may be selected as the final high-quality image. The highest quality image may be selected automatically using an image quality assessment index, or may be selected according to the examiner's (operator's) instructions by displaying a plurality of high-quality image groups on a UI provided in the output unit 103, etc.

In some cases, an input image that has not been enhanced in image quality may be more suitable for image diagnosis, so the input image may be included in the final image selection. Parameters may also be input to the image enhancement engine along with the low-image quality image. For example, parameters specifying the degree of image enhancement or parameters specifying the image filter size used in the image processing method may be input to the image enhancement engine along with the input image.

Here, the input data of the learning data of the high image quality engine according to this embodiment is a low image quality image acquired using the same model and settings as the image capture device 100. The output data of the learning data of the high image quality engine is a high image quality image acquired using settings related to shooting conditions and image processing that require more labor than the same model. Specifically, the output data can be, for example, a high image quality image (overlapped image) obtained by performing an averaging process or other overlay process on a group of images (original images) acquired by capturing images multiple times.

Here, high-quality images and low-quality images will be described using OCTA motion contrast data as an example. Motion contrast data is data used in OCTA etc., which is obtained by repeatedly capturing images of the same location of the subject and detecting temporal changes in the subject between captures. As described above, an OCTA En-Face image (OCTA front image) can be generated by generating a front image using data from a desired range in the depth direction of the subject out of the calculated motion contrast data (an example of three-dimensional medical image data). Note that hereinafter, the number of times OCT data is repeatedly captured at approximately the same position (approximately the same location) is referred to as NOR (Number Of Repeat).

Regarding the learning data according to this embodiment, two different methods for generating high-quality images and low-quality images by overlay processing will be described with reference to Figs. 5(a) and 5(b). Note that the methods for generating high-quality images and low-quality images are not limited to these, and any known generation method may be used.

A first method relating to an example of generating a high-quality image and a low-quality image will be described with reference to FIG. 5(a). In the first method, a motion contrast image generated from OCT data obtained by repeatedly capturing images of approximately the same position on the subject is used as an example of a high-quality image. In FIG. 5(a), motion contrast image Im510 shows a three-dimensional motion contrast image (three-dimensional motion contrast data). Motion contrast image Im511 shows a two-dimensional motion contrast image (two-dimensional motion contrast data) that constitutes the three-dimensional motion contrast image.

Tomographic images Im501-1 to Im501-3 show OCT tomographic images (B-scan images) for generating motion contrast image Im511. Here, NOR corresponds to the number of OCT tomographic images in tomographic images Im501-1 to Im501-3 in FIG. 5(a), and NOR is 3 in the example shown in the figure. Tomographic images Im501-1 to Im501-3 are captured at a predetermined time interval (Δt). Note that "approximately the same position" refers to one line in the front direction (XY) of the subject's eye, which corresponds to the position of motion contrast image Im511 in FIG. 5(a). Note that the front direction is an example of a direction that intersects with the depth direction.

Since the motion contrast data is data that detects changes over time, it is necessary to set NOR at least twice to generate this data. For example, when NOR is 2, one motion contrast data is generated. When NOR is 3, if motion contrast data is generated using only OCT data from adjacent time intervals (first and second, second and third), two motion contrast data are generated. When motion contrast data is generated using OCT data from distant time intervals (first and third), a total of three motion contrast data are generated. That is, if NOR is increased to three, four, etc., the number of motion contrast data at approximately the same position also increases. By aligning multiple motion contrast images obtained by repeatedly photographing approximately the same position and performing superposition processing such as additive averaging, a high-quality motion contrast image can be generated. Therefore, in order to generate a high-quality motion contrast image, NOR should be set to at least three times, and in order to obtain a higher-quality motion contrast image, NOR can be set to five times or more, for example.

On the other hand, an example of a corresponding low-quality image can be a motion contrast image before superimposition processing such as averaging is performed. In this case, the low-quality image can be used as a reference image when performing superimposition processing such as averaging to generate a high-quality image. When performing the superimposition processing, if the position and shape of the target image are deformed and aligned with respect to the reference image, there will be almost no spatial misalignment between the reference image and the image after superimposition processing. Therefore, a low-quality image and a high-quality image can be easily paired. Note that the target image that has been subjected to image deformation processing for alignment, rather than the reference image, can also be used as the low-quality image.

By using each of the original image group (reference image and target image) as input data and the corresponding overlaid image as output data, multiple pair groups can be generated. For example, when obtaining one overlaid image from a group of 15 original images, a pair of the first original image in the original image group and an overlaid image, and a pair of the second original image in the original image group and an overlaid image can be generated. In this way, when obtaining one overlaid image from a group of 15 original images, 15 pair groups can be generated by one image in the original image group and an overlaid image. Note that three-dimensional high-quality data can be generated by repeatedly capturing images at approximately the same position in the main scanning (X) direction and scanning while shifting them in the sub-scanning (Y) direction.

Next, a second method relating to an example of generating high-quality images and low-quality images will be described with reference to FIG. 5(b). In this second method, a high-quality image is generated by superimposing motion contrast images obtained by photographing approximately the same area of the subject multiple times. Note that the approximately same area refers to an area of 3×3 mm or 10×10 mm in the front direction (X-Y) of the subject's eye. By photographing approximately the same area of the subject multiple times, a three-dimensional motion contrast image (three-dimensional motion contrast data) including the depth direction of the tomographic image can be obtained. When photographing the same area multiple times and performing superimposing processing, the NOR can be set to two or three times to shorten each photographing time.

To generate high-quality three-dimensional motion contrast data, at least two or more pieces of three-dimensional data for the same region are acquired. FIG. 5B shows an example of multiple three-dimensional motion contrast images. Motion contrast images Im520 to Im540 are three-dimensional motion contrast images, similar to motion contrast image Im510 described in FIG. 5A. Using these two or more three-dimensional motion contrast images, alignment processing is performed in the front direction (X-Y) and the depth direction (Z), and data that becomes artifacts in each data is removed, followed by averaging processing. This makes it possible to generate a single high-quality three-dimensional motion contrast image from which artifacts have been removed.

On the other hand, the corresponding low-quality image can be used as reference data when performing overlay processing such as averaging. As explained in the first method, there is almost no spatial misalignment between the reference image and the image after averaging, so a low-quality image and a high-quality image can be easily paired. Note that instead of the reference data, any 3D motion contrast image generated from target data that has been subjected to image deformation processing for alignment may also be used as the low-quality image.

With the first method, the imaging itself is completed in one shot, so the burden on the subject is small. However, the more NORs are used, the longer each imaging session takes. Also, if artifacts such as clouding of the eye or eyelashes appear during imaging, a good image may not necessarily be obtained. With the second method, the imaging is performed multiple times, so the burden on the subject increases slightly. However, the time required for each imaging session is short, and even if an artifact appears in one imaging session, as long as the artifact is not visible in another imaging session, a clean image with fewer artifacts can ultimately be obtained. In consideration of these features, any method can be selected to suit the subject's situation when collecting data.

In this embodiment, the low-quality image and the high-quality image used as learning data are described as motion contrast images, but the images used as learning data are not limited to this. Since OCT data is acquired to generate motion contrast data, it is possible to generate low-quality images and high-quality images using the OCT data in a similar manner. Furthermore, although the description of the tracking process is omitted in this embodiment, in order to capture images of approximately the same position or approximately the same area of the subject's eye, it is also possible to capture images while tracking the subject's eye. The tracking process may be performed by any known method.

When a pair of three-dimensional high-quality data and low-quality data can be obtained, a pair of any two-dimensional image can be generated from them. For example, a high-quality OCTA planar image can be generated by projecting or accumulating the generated high-quality three-dimensional motion contrast image in a desired depth range to generate an arbitrary OCTA front image. The corresponding low-quality image can be an arbitrary OCTA front image generated from reference data when performing superposition processing such as averaging. In this case, there is almost no spatial positional deviation between the reference image and the image after averaging, so that a low-quality image and a high-quality image can be easily paired. Note that an arbitrary motion contrast front image generated from target data that has been subjected to image deformation processing for alignment, rather than the reference data, can also be used as the low-quality image.

An example of such a pair of two-dimensional images used as training data will be described in more detail with reference to Figures 6(a) and 6(b). For example, when the image used as training data is an OCTA front image, as described above, the OCTA front image can be generated by projecting or accumulating the three-dimensional volume data related to the motion contrast in a desired depth range. Here, the depth range is the range in the Z direction shown in Figures 5(a) and 5(b).

Figure 6 (a) shows an example of an OCTA front image. As the OCTA front image used for the learning data, OCTA front images generated in different depth ranges, such as the superficial layer (image Im610), deep layer (image Im620), outer layer (image Im630), and choroidal vascular network (image Im640), can be used. Note that the types of OCTA front images are not limited to this, and the types may be increased by generating OCTA front images with different depth ranges set by changing the reference layer and offset value. When learning, OCTA front images of different depths may be learned separately, multiple images of different depth ranges may be combined and learned (for example, divided into the superficial side and the deep side), or OCTA front images of all depth ranges may be learned together. When using brightness En-Face images generated from OCT data as training data, multiple En-Face images generated from any depth range can be used, just like OCTA front images.

For example, consider a case where the image quality improvement engine includes a trained model obtained using training data including multiple OCTA front images corresponding to different depth ranges of the subject's eye. In this case, the acquisition unit 101-1 can acquire an OCTA front image corresponding to a part of a long depth range including different depth ranges as a first image. That is, an OCTA front image corresponding to a depth range different from the multiple depth ranges corresponding to the multiple OCTA front images included in the training data can be used as an input image during image quality improvement processing. Of course, an OCTA front image having the same depth range as during learning may be used as an input image during image quality improvement processing. In addition, the part of the depth range may be set by the operator pressing an arbitrary button on the UI, or may be set automatically. Note that the above content is not limited to an OCTA front image, and can also be applied to, for example, a luminance En-Face image.

When the image to be processed by the trained model is a tomographic image, learning is performed using the OCT tomographic image, which is a B-scan image, or the tomographic image of the motion contrast data as training data. This will be described with reference to FIG. 6(b). In FIG. 6(b), images Im651 to Im653 are OCT tomographic images (luminance tomographic images). The images in FIG. 6(b) are different because they show tomographic images at different positions in the sub-scanning (Y) direction. For tomographic images, learning may be performed together without considering the difference in position in the sub-scanning direction. However, in the case of images of different photographed parts (for example, the center of the macula or the center of the optic disc), learning may be performed separately for each part, or together without considering the photographed part. Note that image features are significantly different between OCT tomographic images and tomographic images of motion contrast data, so it is better to learn them separately.

As an example of a high-quality image used as output data for the learning data, an overlaid image can be used as described above. The overlaid image that has been overlaid emphasizes pixels that are commonly depicted in the original image group, making it a high-quality image suitable for image diagnosis. In this case, the high-quality image generated is a high-contrast image with a clear difference between low-luminance and high-luminance areas as a result of emphasizing the commonly depicted pixels. In addition, for example, in the overlaid image, random noise that occurs with each capture can be reduced, and areas that were not well depicted in the original image at a certain point in time can be interpolated using other original images.

Furthermore, when the overlaid image is used as output data for the training data, a low-quality image can be generated from the overlaid image to be used as input data for the training data. In this case, for example, the overlaid image can be once downsampled to a lower resolution, and the lower-resolution image can be upsampled using a known method (nearest neighbor method, bilinear method, etc.) to be used as input data for the training data. It is also possible to configure a high-quality engine that improves the sense of resolution by performing training using a group of such image pairs as training data.

In addition, when the input data for a machine learning model needs to be composed of multiple images, the required number of original images can be selected from the original image set and used as input data. For example, when obtaining one overlaid image from a set of 15 original images, if two images are required as input data for the machine learning model, 105 (15C2=105) pair sets can be generated.

Of the pairs that make up the training data, pairs that do not contribute to high image quality can be removed from the training data. For example, if the high-image-quality image that is the output data that makes up a pair of the training data is of an image quality that is not suitable for image diagnosis, the image output by the image quality improvement engine that has learned using the training data may also be of an image quality that is not suitable for image diagnosis. Therefore, by removing pairs whose output data is of an image quality that is not suitable for image diagnosis from the training data, the possibility that the image quality improvement engine will generate an image of an image that is not suitable for image diagnosis can be reduced.

In addition, if the average luminance or luminance distribution of a pair of images differs significantly, the image quality improvement engine trained using the training data may output an image that is not suitable for image diagnosis, with a luminance distribution that differs significantly from that of the low-quality image. For this reason, it is possible to remove from the training data pairs of input data and output data that differ significantly in average luminance or luminance distribution.

Furthermore, if the structure or position of the subject depicted in the paired images differs significantly, the image quality improvement engine that has learned using the learning data may output an image that is not suitable for image diagnosis, in which the subject is depicted in a structure or position that differs significantly from that of the low-quality image. For this reason, it is possible to remove from the learning data pairs of input data and output data in which the structure or position of the depicted subject differs significantly. Furthermore, from the perspective of maintaining quality, the image quality improvement engine can be configured not to use the high-quality images that it outputs as learning data.

By using a high-quality image engine that has learned in this way, the high-quality image processing unit 301 can output a high-quality image in which contrast has been increased and noise has been reduced by overlay processing when a medical image acquired in a single shot is input. Therefore, the high-quality image processing unit 301 can generate a high-quality image suitable for image diagnosis based on a low-quality image that is the input image.

Note that, although an example of using an overlaid image as output data of the learning data has been described here, the output data of the learning data of the image quality improvement engine is not limited to this. The output data of the learning data may be a high-quality image corresponding to the input data, for example, an image with noise reduction suitable for diagnosis, an image with contrast correction, an image with high resolution, an image captured under shooting conditions with more labor, etc. Also, an image obtained by subjecting a low-quality image used as input data to image processing using statistical processing such as maximum a posteriori probability estimation (MAP estimation) processing can be used as output data of the learning data. Note that any known method may be used to generate a high-quality image.

In addition, multiple image quality improvement engines may be provided, each of which performs various image quality improvement processes independently, such as noise reduction, contrast correction, and even resolution improvement. Alternatively, one image quality improvement engine may be provided that performs at least two image quality improvement processes. In these cases, the output data of the learning data may be a high-image quality image according to the desired process. For example, for an image quality improvement engine that performs individual processes, the output data of the learning data may be a high-image quality image that has been subjected to individual processes such as noise reduction. In addition, for an image quality improvement engine that performs multiple image quality improvement processes, the output data of the learning data may be a high-image quality image that has been subjected to noise reduction and contrast correction processes, for example.

(Area detection engine for medical images)
Next, the region detection engine of the medical image used by the region detection unit 302 will be described using a contrast front image as an example. The region detection unit 302 according to this embodiment classifies the region into a perfusion region and a non-perfusion region in the contrast front image to check the presence or absence of blood flow, for example, and detects the region. Hereinafter, the non-perfusion region is referred to as NPA (Non Perfusion Area). In addition, the region detection unit 302 according to this embodiment may detect, for example, a foveal avascular zone (FAZ) or an optic nerve head (ONH) as a region. As information on the detected region, the label of the region can be associated with a pixel in the image. Deep learning that associates a label with all pixels in the image in this way is called semantic segmentation.

In general, in deep learning, the task of creating output data (correct answer data) from training data is called annotation. For training data related to a trained model for area detection, output data can be created by providing labels indicating the area to be classified for each pixel position of the input data or the corresponding image through annotation. An image having information indicating the label of each area generated in this way as a pixel value is called an area label image. The annotation work may be performed entirely manually or partially automatically. The efficiency of the annotation work can be improved by performing deep learning as needed when a certain task is completed and correcting the label while referring to the segmentation results by the area detection engine at the intermediate stage. Note that when creating an area label image, not only the input data but also other information may be used. For example, when creating an area label image that is output data from training data in which the input data is a high-resolution image, the area label image may be created by, for example, referring to an image before it is made high-resolution.

A trained model that has been machine-learned using training data that is a pair group of input data that is a predetermined number of medical images and output data that is a region label image corresponding to the medical images can be used as a region detection engine. In addition, the region detection engine that has been trained in this way can output the certainty (reliability, probability) of the region label for each pixel. Therefore, the region detection unit 302 can output, for example, a label with a higher probability than other labels among the labels output from the region detection engine as a region detection result. In addition, a label with a higher probability than a threshold among the labels can be output as a detection result. At this time, if there are multiple labels with a higher probability than the threshold, all of them may be output, or a label with a higher probability than the other labels may be output as a detection result. Furthermore, the region detection unit 302 may determine the detection result using a machine learning model from the probability of each label obtained using the trained model. The machine learning algorithm used in this case may be a different type of machine learning algorithm from the machine learning algorithm used to obtain the label probability, and may be, for example, a neural network, a support vector machine, an adaboost, a Bayesian network, or a random forest.

A common model may be used for both the image quality enhancement engine and the area detection engine. In this case, for example, a known U-Net model can be used, and by making the image quality enhancement engine and the area detection engine the same, the inference process (estimation process) for each can be performed simply by replacing parameters.

In addition, known U-Net models that can be used in image quality improvement engines and area detection engines perform learning with a specified image data size, so image data input to the trained model must be inferred (estimated) with the same size as during learning. If the input image is larger than the image size input to the trained model, the image quality improvement processing unit 301 divides the input image into multiple subset areas (FIG. 17) and performs inference processing on each area to synthesize them. In addition, when setting a part of the inference area as the valid area, the non-valid area may be set as a margin area, and the subset area may be set to overlap the margin area. Note that when setting a margin area, the input image may be enlarged in advance by the margin (not shown). General processing such as mirroring can be used as the enlargement method.

On the other hand, depending on the type of network, the size of the subset can be changed from the size at the time of learning. In the case of a network including a downsampling layer or an upsampling layer, there are certain constraints, but if the size of the subset to be inferred under the constraints is expanded, the number of inference processes can be reduced and the speed can be increased. At this time, the image quality improvement processing unit 301 divides the input image into subset areas with a size different from that at the time of learning and performs the inference process. Conversely, by setting the size of the image data at the time of learning to be small, it is possible to increase the number of training data. Also, consider a case where the trained model is obtained by training using training data including input data that is a medical image having a first image size. At this time, at the time of inference, a first image that is a medical image having a second image size larger than the first image size is input as input data to the trained model. As a result, the image processing unit can output the second image having the second image size as output data of the trained model. Then, a medical image with an image size suitable for learning and inference (estimation) can be used as input data of the trained model. Specifically, since the image size of the input data of the learning data is smaller than the image size of the input data at the time of inference, the number of learning data can be increased. On the other hand, since the image size of the input data at the time of inference is larger than the image size of the input data of the learning data, the inference process can be accelerated. Of course, a large deviation in the size of the subset area between learning and inference may result in undesirable results, but it is sufficient to check in advance whether the difference in the inference result due to the change in size of the subset area is within an acceptable range. For example, when an input image of 232×232 pixels is inferred for a model in which the input size is 224×224 pixels and margin areas of 12 pixels are set on each of the top, bottom, left and right, inference can be performed using the effective area by setting four subset areas. In contrast, if the input size of the learning model is set to 256×256 and expanded only at the time of inference, one subset area is sufficient and the inference process can be completed in one go. In general, an architecture that makes inference processing more efficient, called batch processing, can be used, so the inference processing may be made more efficient by setting the number of subset areas according to the number of batch processes. In this case, the input data of the trained model during inference may be a plurality of medical images having a second image size obtained by dividing a medical image having a third image size larger than the second image size into a plurality of images. In this case, a plurality of images are output as output data of the trained model. Then, an image of the third image size may be generated by synthesizing the plurality of output images.

Furthermore, the input data for area detection may be the high-quality image (superimposed image) described above, or a high-quality image obtained by applying an image quality improvement engine to an input medical image may be used as input data. In this way, by performing learning according to the input data, an area detection engine optimal for each input data can be constructed.

The well-known UNet model is shown as an example of a region detection engine, but region detection may also be performed using a detection transformer (DETR) that uses an encoder-decoder type transformer. Figure 18 shows an example of self-attention used in the transformer. As shown in Figure 18(a), a query, key, and value are created from the output of the convolution of the input image using one by one convolution, etc. As shown in Figure 18(b), the upper left pixel is focused on, and the dot product of the query and key is calculated and Softmax is taken to determine which pixel should characterize the upper left pixel. Then, the values are added while weighting all pixels according to the result of this Softmax. This series of processes is performed on all pixels, and the result is added to the input and output. In other words, Self-Attention is a mechanism for performing Attention on oneself and reflecting the result on oneself, and by using Self-Attention, a mechanism for characterizing oneself by focusing on features in other locations can be realized. By performing Self-Attention for each pixel in this way, it becomes possible to successfully characterize, for example, each pixel of FAZ / NPA / blood vessel even when it is located at a distant position in the image or when there are too many position variations. In conventional CNNs, in order to capture the features of pixels at distant positions, it is necessary to make the hierarchy deeper, increase the filter size, or use Dilated Convolution to widen the reference range, but there is a problem that the computational cost increases. In this way, by using Self-Attention, computational costs can also be suppressed by constructing a model using a Transformer in object recognition and area detection in image processing. The Transformer learns by giving a unique value to the position of a pixel or pixel region, and as a result, it also learns the positional dependency. Dynamically changing the neural network connections according to the input data by using Attention, which dynamically changes where to focus, is a highly versatile method. In other words, it may be applied to estimating non-perfusion regions from a relationship map between the features of pixels or pixel regions, and the image quality improvement process described above may be applied. In addition, region detection may be performed by combining DETR and the Unet model. In other words, after narrowing down the non-perfusion regions using DETR, semantic segmentation using the Unet model may be performed. Conversely, detailed labeling may be performed using DETR after semantic segmentation processing using the Unet model.

The region detection engine does not necessarily need to be an artificial intelligence engine based on machine learning, but may be configured using a rule-based algorithm based on the structure of the subject's eye, including various conventional filter processes. Therefore, the region detection engine used by the region detection unit 302 may extract the FAZ or NPA region by combining filter processes such as a known Gaussian filter, or may extract the region according to instructions from an operator via a GUI or the like. In addition, the region detection engine may detect regions by classifying them into perfusion regions and non-perfusion regions in a motion contrast image, for example, by threshold processing using the structure of the subject's eye or a predetermined threshold.

(Medical image processing procedure)
Next, a series of steps of image processing by the image processing device 101 according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a flow chart showing an outline of the flow of image processing according to this embodiment.

When image processing according to this embodiment is started, in step S701, the acquisition unit 101-1 acquires a medical image. The acquisition unit 101-1 may acquire a medical image from the imaging device 100 or an external device, or may acquire a medical image generated using signal data acquired from these. In this embodiment, the acquisition unit 101-1 acquires, for example, a contrast front image as the medical image. Note that the medical image is not limited to this, and may be a tomographic image, a luminance En-Face image, an SLO image, an analysis image (analysis map) obtained by analyzing a medical image of a subject, or the like.

In step S702, the image quality improvement processing unit 301 applies image quality improvement processing, which is the first image processing, to the acquired medical image using at least one of the image quality improvement engines described above to acquire an image quality improved medical image. The image quality improvement processing unit 301 according to this embodiment acquires an image quality improved OCTA front image from the OCTA front image using the image quality improvement engine.

Next, in step S703, the area detection unit 302 applies area detection processing to the medical image acquired in step S701 or the high-image-quality medical image acquired in step S702 using the above-mentioned area detection engine. The area detection unit 302 detects at least two areas in the medical image or the high-image-quality medical image by the area detection processing, and acquires information indicating each area. For example, the area detection unit 302 may acquire an area label image in which a label indicating each area is set as information for each pixel for the detected area, or may add a label indicating each area to information for each pixel for the medical image. Note that the area detection unit 302 only needs to be able to output information indicating the detected area, and the information indicating the detected area is not limited to the above-mentioned format. For example, the area detection unit 302 may be label information associated with pixel information of the medical image. Also, as described above, a value indicating the likelihood (reliability, probability) of each attribute output from the area detection engine may be associated with each pixel of the medical image and added to the pixel information. In this embodiment, a region detection engine is used to detect perfusion regions and non-perfusion regions (NPAs) in an OCTA en face image or an image-enhanced OCTA en face image. Note that the pixel positions of the detected regions in a medical image or an image-enhanced medical image may correspond in the medical image and the image-enhanced medical image.

When the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the high-image-quality medical image based on the output information. The ROI may be set for at least one of the regions detected by the region detection unit 302. In this embodiment, the ROI setting unit 303 sets an ROI for the NPA in the high-image-quality OCTA front image. As described above, the ROI may be set by the region detection unit 302.

Finally, in step S704, the blending processing unit 304 or the BC adjustment unit 305 applies the second image processing to the ROI in the high-image-quality medical image so that the difference between the pixel value of the ROI and the pixel value of the area other than the ROI increases. At this time, the blending processing unit 304 or the BC adjustment unit 305 applies the second image processing to the ROI in the high-image-quality medical image so that the pixel value of the ROI is lower than the pixel value of the ROI before the second image processing. For example, the blending processing unit 304 performs blending of the medical image acquired in step S701 and the high-image-quality medical image acquired in step S702 as the second image processing for the ROI in the high-image-quality medical image. The blending processing may be performed using any known method, and the blending ratio may be a predetermined ratio or may be set according to the operator's instructions. By performing this blending process, the image before the overcorrection occurs and the image in which the overcorrection occurs are blended for the area where overcorrection occurs due to the image quality improvement process using the learned model, and the difference between the pixel values of the area where overcorrection occurs and the pixel values of the other areas increases. In other words, by performing this blending process, the pixel values of the area where overcorrection occurs are lowered. In this process, it is possible to lower the pixel values only in the area where overcorrection occurs, without lowering the pixel values of the entire image. Therefore, for example, overcorrection in areas such as the NPA and FAZ can be suppressed.

The BC adjustment unit 305 performs a BC adjustment process as a second image processing on the ROI in the high-image-quality medical image. The BC adjustment process may be performed using any known method, for example, the brightness in the ROI may be set to a predetermined value, or may be increased or decreased by a predetermined value. In addition, the contrast in the ROI may be adjusted using, for example, a tone curve or a gamma curve. Furthermore, the BC adjustment process may set at least one of the brightness and contrast in the ROI to a value according to an instruction from the operator, or may increase or decrease the brightness and contrast by the value. For example, the BC adjustment unit 305 may determine a correction value for brightness or a setting value for a tone curve or the like used for contrast adjustment according to an instruction from the operator. By performing the BC adjustment process, at least one of the brightness and contrast is adjusted so that the difference between the pixel value of the overcorrected area and the pixel value of the other area is widened for the area where overcorrection occurs due to the high-image-quality processing using the learned model. In other words, by performing the BC adjustment process, the pixel value of the area where overcorrection occurs is lowered. This type of processing can lower the pixel values only in areas where overcorrection occurs, without lowering the pixel values of the entire image. This makes it possible to suppress overcorrection in areas such as the NPA and FAZ, for example.

In this embodiment, the blending processing unit 304 performs blending processing of the OCTA front image and the high-image-quality OCTA front image for the NPA set as the ROI in the high-image-quality OCTA front image. In addition, the BC adjustment unit 305 performs BC adjustment processing of the high-image-quality OCTA front image for the NPA set as the ROI in the high-image-quality OCTA front image. Note that, as described above, only one of the processing by the blending processing unit 304 and the processing by the BC adjustment unit 305 may be performed, or both may be performed. The high-image-quality image subjected to the second image processing by the blending processing unit 304 or the BC adjustment unit 305 may be displayed on the output unit 103 by the display control unit 101-5, output to an external device, etc. by the output unit 103, or stored in the storage unit 101-3 or the external storage device 102.

By performing image processing according to this embodiment in this manner, it is possible to detect areas where overcorrection occurs due to image processing of medical images using a trained model, and to apply image processing to the detected areas to reduce the overcorrection. For example, it is possible to detect NPAs in an OCTA frontal image, and to apply blending processing or BC adjustment processing to reduce overcorrection for NPAs in a high-quality OCTA frontal image acquired using a trained model.

As mentioned above, NPA is particularly important in diagnosing fundus images. By identifying NPA, it is possible to determine whether there is no blood flow where blood vessels should be, or whether some blood flow is observed where there should be no blood vessels (neovascularization, etc.). In particular, for FAZ and NPA, by preserving to some extent the original state before image enhancement, it is possible to assist the operator (examiner) in distinguishing between noise and blood vessels in image diagnosis.

On the other hand, by setting the brightness and contrast of the NPA to the desired state and actively removing noise, it is possible to make it look more like an NPA and improve visibility. In this case too, it is possible to assist the operator in distinguishing between noise and blood vessels in image diagnosis.

In general, since a diagnosis is performed in a comprehensive manner including other examinations, the second image processing method may be set in advance or may be selectable as appropriate. Also, the second image processing method may be changed depending on the area to be classified. Furthermore, it is also possible to change the setting depending on the eye to be examined. Also, it may be possible to change the setting for the depth of the OCTA front image, i.e., the shallow layer and the deep layer. These settings may be stored for each user or subject.

As described above, the image processing device 101 according to this embodiment includes an image quality improvement processing unit 301, an area detection unit 302, a blend processing unit 304, and a BC adjustment unit 305. The image quality improvement processing unit 301 performs image quality improvement processing on a first medical image of the subject using an image quality improvement model (image quality improvement engine) obtained by learning using the medical image of the subject as learning data, and obtains a high-quality second medical image of the subject. The area detection unit 302 detects a target area in the medical image of the subject (first image or second image). At least one of the blend processing unit 304 and the BC adjustment unit 305 performs image processing on the target area in the second image so that the difference between the pixel values of the target area and the pixel values of areas other than the target area increases and the pixel values of the target area become lower than the pixel values of the target area before the image processing.

The blending unit 304 performs image processing to blend the first image and the second image so that the difference between the pixel values of the target region and the pixel values of the regions other than the target region increases and the pixel values of the target region become lower. The BC adjustment unit 305 performs image processing to correct at least one of the brightness and the contrast so that the difference between the pixel values of the target region and the pixel values of the regions other than the target region increases and the pixel values of the target region become lower. The medical image of the subject is, for example, a motion contrast image of the subject's eye, and the target region can include, for example, at least one of the nonperfusion region, the foveal vascular region, and the optic disc region. The region detection unit 302 can detect the target region using a trained model obtained by training using the medical image of the subject as training data.

With this configuration, the image processing device according to this embodiment can detect areas such as NPAs where overcorrection occurs due to image processing of medical images using a trained model, and apply image processing to the detected areas to reduce the overcorrection. With this processing, it is possible to lower the pixel values only in areas where overcorrection occurs, without lowering the pixel values of the entire image. This makes it possible to reduce overcorrection due to image processing of medical images using a trained model. This can assist the operator in determining whether something is noise or a blood vessel in image diagnosis.

Here, various modifications are possible to the series of image processing according to this embodiment. Specific examples of the series of image processing according to this embodiment will be described in detail below with reference to Figs. 8 to 13. Note that for each example, a description of processing similar to the image processing shown in Fig. 7 will be omitted.

<First Example of Image Processing Method>
A first example of the image processing method will be described with reference to Fig. 8. In this example, an image processing method for suppressing overcorrection by the image quality improvement engine for FAZ and NPA will be described.

The processing in steps S801 and S802 is similar to that in steps S701 and S702, and therefore will not be described. When a medical image with improved image quality is acquired in step S802, processing proceeds to step S803.

In step S803, the region detection unit 302 uses a region detection engine to detect the regions of the FAZ and NPA in the medical image (input image) acquired in step S801. The FAZ and NPA may be detected separately, or may be detected together as one region. Only one of the regions may be detected, or the regions other than the FAZ and NPA may be detected as a perfusion region. When the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the medical image whose image quality has been improved in step S802 based on the output information. Note that the pixel positions of the detected region in the medical image may correspond in the medical image and the medical image whose image quality has been improved.

Finally, in step S804, the blending processing unit 304 performs blending processing on the detected region in the enhanced medical image using the input image and the enhanced medical image acquired in step S802. Here, the blending processing may be a known alpha blending process or the like. The blending ratio may be a fixed value, or the blending processing unit 304 may change the blending ratio for each pixel depending on the reliability (likelihood, probability) of the attribute information output by the region detection engine. Specifically, for pixels having attribute information of a non-perfusion region, the alpha blending ratio can be set so that the higher the reliability, the higher the blending ratio of the input image.

In addition, when attribute information such as FAZ and NPA is detected separately, the blending ratio of each may be changed according to the distinguished attribute information. Furthermore, the input image may be appropriately composited with respect to areas other than FAZ and NPA, for example, pixels surrounding target areas such as FAZ and NPA. This makes it possible to mitigate abrupt changes at area boundaries.

The blending ratio may be specified by the operator via a UI. For example, the blending ratio corresponding to the suppression strength of overcorrection may be set using a GUI such as a slide bar.

In this example, for FAZ and NPA, the original state before image enhancement can be preserved to some extent. Therefore, in image diagnosis, it is possible to assist the operator in distinguishing between noise and blood vessels.

When detecting a target region in a medical image (first image) that is an input image, the region detection unit 302 can use a region detection engine configured with a trained model as the region detection engine. In this case, as the learning data for the region detection engine, the medical image that is the input image can be used as input data, and a region label image having information indicating the label of each region in the medical image that is the input image as pixel values can be used as output data. The region detection unit 302 can also use a region detection engine configured using a rule-based algorithm based on the structure of the subject. In this case, the region detection unit 302 can also detect target regions such as FAZ and NPA from feature parts in a medical image that has not been subjected to high image quality using a trained model, in other words, has not been overcorrected.

<Second Example of Image Processing Method>
A second example of the image processing method will be described with reference to Fig. 9. In this example, an image processing method for emphasizing image quality improvement processing for NPA will be described.

When the performance of the region detection engine is high, the region detection unit 302 can more appropriately detect regions such as the FAZ and NPA. For example, it is true that there are no blood vessels in the FAZ and NPA. Therefore, when the performance of the region detection engine is high, these regions can be processed by the high image quality engine, and then further darkened by BC adjustment to obtain an image more suitable for diagnosis. The processing of this example, which can obtain an image more suitable for diagnosis using such image processing, is described in more detail below.

The processing in steps S901 and S902 is similar to that in steps S701 and S702, and therefore will not be described. When a medical image with improved image quality is acquired in step S902, processing proceeds to step S903.

In step S903, the region detection unit 302 uses a region detection engine to detect the regions of the FAZ and NPA in the medical image (input image) acquired in step S901. The FAZ and NPA may be detected separately, or may be detected together as one region. Only one of the regions may be detected, or the region other than the FAZ and NPA may be detected as a perfusion region. When the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the medical image whose image quality has been improved in step S902 based on the output information. Note that the pixel positions of the detected region in the medical image may correspond in the medical image and the medical image whose image quality has been improved.

Finally, in step S904, the BC adjustment unit 305 performs BC adjustment processing on the pixels of the detected areas (FAZ and NPA areas) in the medical image whose image quality has been improved using the image quality improvement engine in step S902. The BC adjustment can be performed by correcting the luminance value (brightness) of the pixel value to make it darker. For example, the luminance value may simply be lowered or set to zero. Image processing according to the original pixel value, such as gamma correction, may also be applied. Note that the strength of the correction may be changed for the FAZ and NPA, and if attributes and reliability are held as information indicating the detected area, the BC adjustment may be performed according to the reliability.

The method and strength of the BC adjustment may be specified by the operator via a UI. For example, the method may be selected using a GUI such as a radio button, and the strength may be specified using a GUI such as a slide bar.

According to this example, it is also possible to improve the visibility of areas such as the NPA and FAZ. In this case, too, it is possible to assist the doctor in determining whether an area is noise or a blood vessel in image diagnosis. Furthermore, it is also possible to make it possible to select the suppression processing shown in the first example and the enhancement processing shown in the second example for each attribute of the area. For example, enhancement processing may be used for the FAZ, and suppression processing may be used for the NPA. In this case, it is possible to generate an image that is more suitable for diagnosis according to each area, thereby assisting the doctor in making a decision.

<Third Example of Image Processing Method>
A third example of the image processing method will be described with reference to Fig. 10. In this example, an image processing will be described in which regions such as FAZ and NPA are detected by a region detection engine from a medical image whose image quality has been improved by a high image quality engine, and overcorrection in these regions is suppressed.

The processing in steps S1001 and S1002 is similar to that in steps S701 and S702, and therefore will not be described. When a medical image with improved image quality is acquired in step S1002, processing proceeds to step S1003.

In step S1003, the region detection unit 302 uses a region detection engine to detect the regions of the FAZ and NPA in the medical image whose image quality has been improved in S1002. The FAZ and NPA may be detected separately, or may be detected together as one region. Also, only one of the regions may be detected, or the regions other than the FAZ and NPA may be detected as a perfusion region. After the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the medical image whose image quality has been improved in step S1002 based on the output information.

Finally, in step S1004, the blending processing unit 304 performs blending processing on the detected area in the high-image-quality medical image using the high-image-quality medical image and the medical image acquired in step S1001. The blending processing may be the same as the processing described in the first example.

In this case, as in the first example, the original state before image enhancement can be retained to some extent for FAZ and NPA. Therefore, in image diagnosis, it is possible to assist the operator in distinguishing between noise and blood vessels.

When detecting a target region in a medical image with high image quality, the region detection unit 302 can use a region detection engine configured by a trained model. In this case, as the learning data of the region detection engine, a medical image with high image quality (second image) can be used as input data, and a region label image having information indicating the label of each region in the medical image (first image) that is the input image as pixel values can be used as output data. Here, the high-image-quality medical image that is the input data is a medical image with high image quality that has been improved using a high-image quality engine, and can be a high-image quality image in which overcorrection has occurred in the target region such as NPA. In contrast, the region label image that is the output data may be an image obtained by labeling the medical image before it is improved using the high-image quality engine. In this case, the region detection engine configured by the trained model can output a region label image including a label of the target region with overcorrection from the input high-image quality image according to the learning tendency. By using a high-image quality image as an input, the region detection engine configured by the trained model is expected to be able to more appropriately extract features in the image and output a region label image with higher accuracy.

<Fourth Example of Image Processing Method>
A fourth example of the image processing method will be described with reference to Fig. 11. In this example, an image processing will be described in which regions such as FAZ and NPA are detected by a region detection engine from a medical image whose image quality has been improved by a high image quality engine, and the image quality improvement processing of these regions is emphasized.

The processing in steps S1101 and S1102 is similar to that in steps S701 and S702, and therefore will not be described. When a medical image with improved image quality is acquired in step S1102, processing proceeds to step S1103.

In step S1103, the region detection unit 302 uses a region detection engine to detect the regions of the FAZ and NPA in the medical image whose image quality has been improved in step S1102. The FAZ and NPA may be detected separately, or may be detected together as one region. Also, only one of the regions may be detected, or the regions other than the FAZ and NPA may be detected as a perfusion region. After the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the medical image whose image quality has been improved in step S1102 based on the output information.

Finally, in step S1104, the BC adjustment unit 305 performs BC adjustment processing on the pixels in the detected areas (FAZ and NPA areas) in the medical image with improved image quality. The BC adjustment processing may be the same processing as that described in the second example.

In this example, as in the second example, the visibility of areas such as the NPA and FAZ can also be improved. In this case, too, it is possible to assist the doctor in determining whether an area is noise or a blood vessel in image diagnosis. Furthermore, it is possible to make it possible to select the suppression processing shown in the first and third examples and the enhancement processing shown in the fourth example for each attribute of the area. For example, the FAZ may be subjected to enhancement processing, and the NPA may be subjected to suppression processing. In this case, it is possible to generate an image that is more suitable for diagnosis according to each area, thereby assisting the doctor in making a decision.

<Fifth Example of Image Processing Method>
A fifth example of the image processing method will be described with reference to Fig. 12. In this example, image processing will be described in which regions such as FAZ and NPA are detected using a region detection engine in a medical image before image quality is improved, BC adjustment is performed on those regions, and then image quality improvement processing using an image quality improvement engine is applied.

The processing in step S1201 is the same as that in step S701, and therefore description thereof will be omitted. In step S1202, the region detection unit 302 uses a region detection engine to detect the regions of the FAZ and NPA in the medical image (input image) acquired in step S1201. When the region detection unit 302 outputs information indicating the detected region, the ROI setting unit 303 sets an ROI in the input image based on the output information.

Next, in step S1203, the BC adjustment unit 305 performs BC adjustment processing on the pixels in the detected area (FAZ or NPA area) in the input image. Note that the BC adjustment processing may be the same processing as that described in the second example.

Finally, in step S1204, the image quality improvement processing unit 301 uses the image quality improvement engine to perform image quality improvement processing on the medical image that has been subjected to the BC adjustment processing in step S1203, and obtains a medical image with improved image quality. Note that blending processing and BC adjustment processing based on the detected region may be further added to the medical image after the image quality improvement processing by the image quality improvement processing unit 301. The blending processing and BC adjustment processing may be similar to the processing described in the first and second examples, respectively.

As described above, in this example, the region detection unit 302 detects a target region in the first medical image of the subject. The BC adjustment unit 305 performs BC adjustment processing on the target region in the first medical image of the subject so that the difference between the pixel values of the target region and the pixel values of regions other than the target region increases and the pixel values of the target region become lower than the pixel values of the target region before the BC adjustment processing. The image quality improvement processing unit 301 performs image quality improvement processing on the first image on which the BC adjustment processing has been performed, using an image quality improvement model obtained by learning using the medical image of the subject as learning data, to obtain a high-quality second medical image of the subject.

In this example, the BC adjustment process is performed, and image quality improvement processing is performed on the medical image in which the pixel values of the areas where overcorrection occurs are lowered. In this type of processing, it is possible to lower the pixel values only in the areas where overcorrection occurs, without lowering the pixel values of the entire image. As a result, for example, overcorrection in areas such as the NPA and FAZ can be suppressed, and the visibility of areas such as the NPA and FAZ can be improved. In this case, too, it is possible to assist the doctor in distinguishing between noise and blood vessels in image diagnosis.

<Sixth Example of Image Processing Method>
A sixth example of the image processing method will be described with reference to Fig. 13. In this example, a medical image that has been enhanced in image quality using an image quality enhancement engine is subjected to area detection using an area detection engine. After that, the medical image before the image quality enhancement is subjected to BC adjustment based on the area detected by the area detection described above, and then image processing in which image quality enhancement processing using the image quality enhancement engine is applied will be described.

The processing in steps S1301 and S1302 is similar to that in steps S701 and S702, and therefore will not be described. When a medical image with improved image quality is acquired in step S1302, processing proceeds to step S1303.

In step S1303, the area detection unit 302 uses the area detection engine to perform area detection processing on the image whose image quality has been improved in step S1302. The detected area may be FAZ, NPA, etc., as in the first to fifth examples. By applying the area detection engine to the medical image whose image quality has been improved in this way, it is expected that the detection performance can be improved. After the area detection unit 302 outputs information indicating the detected area, the ROI setting unit 303 sets an ROI in the input image based on the output information. The pixel positions of the detected area in the medical image whose image quality has been improved can be assumed to correspond in the medical image and the medical image whose image quality has been improved.

Next, in step S1304, the BC adjustment unit 305 performs BC adjustment processing on the medical image (input image) acquired in step S1301 based on the area detected in step S1303. Note that the BC adjustment processing may be the same processing as that described in the second example.

Finally, in step S1305, the image quality improvement processing unit 301 uses the image quality improvement engine to perform image quality improvement processing on the medical image that has been subjected to the BC adjustment processing in step S1303, and obtains a medical image with improved image quality. Note that blending processing and BC adjustment based on region classification may be further applied to the image after the image quality improvement processing by the image quality improvement processing unit 301 in step S1305. The blending processing and BC adjustment may be similar to the processing described in the first and second examples, respectively.

As described above, in this example, the image quality improvement processing unit 301 uses an image quality improvement model obtained by learning using the medical image of the subject as learning data to obtain an image in which the first medical image of the subject has been improved in image quality from the first medical image of the subject. The area detection unit 302 detects a target area in the image in which the first medical image of the subject has been improved in image quality. The BC adjustment unit 305 performs BC adjustment processing on the target area in the first medical image of the subject so that the difference between the pixel values of the target area and the pixel values of areas other than the target area increases and the pixel values of the target area are lower than the pixel values of the target area before the BC adjustment processing. Furthermore, the image quality improvement processing unit 301 performs image quality improvement processing on the first image on which the BC adjustment processing has been performed using the image quality improvement model, and obtains a second high-quality medical image of the subject. Note that the area detection unit 302 detects the target area using an image in which the first medical image of the subject has been improved in image quality without the BC adjustment performed by the BC adjustment unit 305.

In this example, the BC adjustment process is performed, and image quality improvement processing is performed on the medical image in which the pixel values of the areas where overcorrection occurs are lowered. In this type of processing, it is possible to lower the pixel values only in the areas where overcorrection occurs, without lowering the pixel values of the entire image. As a result, for example, overcorrection in areas such as the NPA and FAZ can be suppressed, and the visibility of areas such as the NPA and FAZ can be improved. In this case, too, it is possible to assist the doctor in distinguishing between noise and blood vessels in image diagnosis.

As described above, various modifications are possible for image processing using an image quality improvement engine and an area detection engine. These described examples are of many possible modifications and are not intended to limit the interpretation of this disclosure. For example, when observing the progress of medical images, blending processing and BC adjustment processing based on the detected area may be performed under the same conditions, or these conditions may be stored and managed for each subject.

Second Embodiment
An image processing system including an image processing device according to a second embodiment of the present invention will be described in detail with reference to FIG. 14. FIG. 14 is a flow diagram showing a series of image processing steps according to this embodiment. Since the configuration of the image processing system according to this embodiment is similar to that of the image processing system according to the first embodiment, the same reference numerals will be used and the description will be omitted. However, the image quality improvement unit according to this embodiment does not need to include the area detection unit 302, the ROI setting unit 303, the blend processing unit 304, and the BC adjustment unit 305. The image processing system according to this embodiment will be described below, focusing on the differences from the image processing system according to the first embodiment.

In the first embodiment, a method for obtaining a more suitable high-quality image by combining an image quality improvement engine using deep learning with an area detection engine and performing additional partial image correction was described. In contrast, in the present embodiment, a configuration for performing image quality improvement processing using an image quality improvement engine constructed by applying these methods is described.

The machine learning algorithm of the image quality improvement engine according to this embodiment may be the same as the machine learning algorithm of the image quality improvement engine according to the first embodiment. However, the output data of the learning data of the image quality improvement engine according to this embodiment uses high-quality images finally generated by any of the methods described in the first embodiment. These processing flows are applied to a predetermined number of medical images to output high-quality images, and a learning data set is constructed by pairing each of them. It is expected that the image quality improvement engine obtained by learning using such learning data will be able to generate high-quality images with suppressed overcorrection according to the learning tendency. In addition, it is expected that the image processing load during inference can be reduced by performing machine learning collectively, including the partially corrected results.

The processing procedure of the image processing device 101 of this embodiment will be described with reference to FIG. 14. First, in step S1401, the acquisition unit 101-1 acquires a medical image. The processing of step S1401 may be the same as the processing of step S701.

Next, in step S1402, the image quality improvement processing unit 301 uses an image quality improvement engine obtained by learning using the learning data as described above to perform image quality improvement processing on the medical image acquired in step S1401, and acquires a high-image quality image. The high-image quality image acquired in this manner is a high-image quality image in which overcorrection is suppressed according to the learning tendency of the image quality improvement engine.

As described above, the image processing device according to this embodiment includes an acquisition unit 101-1 and an image quality improvement processing unit 301. The acquisition unit 101-1 acquires a first image of a medical image of a subject. The image quality improvement processing unit 301 performs image quality improvement processing on the first image using a first trained model obtained by learning using the medical image of the subject as training data, and a second trained model obtained by learning using the medical image of the subject that has been subjected to blending processing or at least one of brightness and contrast correction processing in a target region of the medical image of the subject, to acquire a second image of the medical image of the subject. In this case as well, a high-quality image in which overcorrection is suppressed can be generated.

In addition, by performing various image analyses using medical images whose image quality has been improved by either the method according to the first or second embodiment, it is possible to perform more accurate analyses. In addition, for disease screening using an artificial intelligence engine, images whose image quality has been improved can be used as input images, and it is expected that more accurate processing can be performed.

[Modification 1]
In the first embodiment, when the region detection unit 302 detects a target region using a trained model, a plurality of front images corresponding to a plurality of depth ranges can be used as training data. For example, a plurality of OCTA front images corresponding to a plurality of depth ranges can be used as input data for training data of a region detection engine using a trained model. Note that, as output data of the training data, a region label image corresponding to the OCTA front image used as input data may be used. By using a common trained model obtained by training such training data, the region detection unit 302 can detect a two-dimensional target region from a front image of a subject corresponding to an arbitrary depth range, for example, a depth range selected according to an instruction from an examiner. Note that, the depth range may include, for example, a superficial layer, a deep layer, an outer layer, and a choroidal vascular network, a different depth range in which a reference layer and an offset value are changed, and the like. In addition, the front image is not limited to an OCTA front image, and may be an En-Face image.

Also, multiple trained models corresponding to the depth ranges may be prepared using training data for each depth range. In this case, the area detection unit 302 may select a trained model corresponding to the depth range selected in response to an instruction from the examiner from among multiple trained models obtained by training using multiple front images of the subject corresponding to multiple depth ranges as respective training data, and detect a two-dimensional target area from the front image of the subject using the selected trained model. Also, the area detection unit 302 may select a trained model corresponding to the depth range of the medical image used in the area detection process from among such multiple trained models, and detect a two-dimensional target area from the front image of the subject using the selected trained model.

Furthermore, a combination of multiple front images with different depth ranges (e.g., multiple front images divided into superficial and deep sides) may be used as training data. In this case, the region detection unit 302 also selects a trained model corresponding to the depth range selected in response to instructions from the examiner or the depth range of the medical image used in the region detection process, and can detect a two-dimensional target region from the front image of the subject using the selected trained model.

Furthermore, a three-dimensional image may be used as input data for the training data of the region detection engine using the trained model. For example, a three-dimensional tomographic image or a three-dimensional motion contrast image may be used as input data for the training data, and a three-dimensional region label image in which the three-dimensional image is labeled may be used as output data for the training data. In this case, the region detection unit 302 can detect a three-dimensional target region from the three-dimensional image of the subject using a trained model obtained by training using the three-dimensional image of the subject as training data.

Similarly, the image quality improvement engine may also be trained using learning data in which a three-dimensional image is used as input data and a high-quality three-dimensional image is used as output data. In this case, the image quality improvement processing unit 301 can obtain a high-quality three-dimensional image from a three-dimensional medical image of a subject using an image quality improvement engine obtained by learning using the three-dimensional medical image of the subject as learning data. Furthermore, the blend processing unit 304 and the BC adjustment unit 305 can be configured to perform various processes on a three-dimensional target area in a three-dimensional medical image. In this case, the same effect as that of the first embodiment can be achieved by processing the three-dimensional medical image. Similarly, a three-dimensional image may be used as learning data for the image quality improvement engine according to the second embodiment. In this case, the same effect as that of the second embodiment can be achieved by processing the three-dimensional medical image.

[Modification 2]
A trained model (trained model for high image quality, trained model for area detection) used for the high image quality processing and the area detection processing may be adjusted (tuned) for each subject, and a trained model dedicated to the subject may be generated. For example, a medical image acquired in a past examination of the subject may be used to perform transfer learning of a general-purpose trained model for generating high image quality medical images and a general-purpose trained model for detecting areas, and a trained model dedicated to the subject may be generated. By linking a trained model dedicated to the subject with the ID of the subject and storing it in an external device such as the storage unit 101-3 or a server, the image processing device 101 can identify and use a trained model dedicated to the subject based on the ID of the subject when performing a current examination of the subject. By using a trained model dedicated to the subject, the accuracy of the high image quality processing and the area detection processing can be improved.

[Modification 3]
The image quality improvement engine may be prepared for each type of image, which is input data. For example, an image quality improvement model for anterior eye images, an image quality improvement model for SLO images, an image quality improvement model for tomographic images, an image quality improvement model for OCTA front images, etc. may be prepared. Furthermore, for OCTA front images and En-Face images, an image quality improvement model may be prepared for each depth range for generating images. For example, an image quality improvement model for superficial layers, an image quality improvement model for deep layers, etc. may be prepared. Furthermore, the image quality improvement model may be learned for images for each imaging site (for example, the center of the macula, the center of the optic disc), or may be learned regardless of the imaging site.

At this time, for example, the fundus OCTA front image may be improved in quality using a high-quality model obtained by learning the fundus OCTA front image as learning data, and the anterior eye OCTA front image may be improved in quality using a high-quality model obtained by learning the anterior eye OCTA front image as learning data. The high-quality model may also be learned regardless of the imaging site. Here, for example, the fundus OCTA front image and the anterior eye OCTA front image may have relatively similar distributions of blood vessels that are the imaging targets. In this way, in multiple types of medical images in which the imaging targets are relatively similar to each other, the feature amounts of each image may be relatively similar. Therefore, for example, the high-quality model obtained by learning the fundus OCTA front image as learning data may be used to improve the quality of not only the fundus OCTA front image but also the anterior eye OCTA front image. In addition, for example, a high image quality model obtained by learning an anterior eye OCTA front image as learning data may be used to improve the image quality of not only the anterior eye OCTA front image but also the fundus OCTA front image. In other words, a high image quality model obtained by learning at least one type of front image, a fundus OCTA front image and an anterior eye OCTA front image, as learning data may be used to improve the image quality of at least one type of front image, a fundus OCTA front image and an anterior eye OCTA front image.

Now consider a case where an OCT device capable of fundus photography can also photograph the anterior eye. In this case, for example, a fundus OCTA front image may be applied to the OCTA En-Face image in the fundus photography mode, and an anterior eye OCTA front image may be applied in the anterior eye photography mode. In this case, when the high image quality button is pressed, for example, in the fundus photography mode, in the display area of the OCTA En-Face image, the display of one of the low image quality fundus OCTA front image and the high image quality fundus OCTA front image may be changed to the other. Also, when the high image quality button is pressed, for example, in the anterior eye photography mode, in the display area of the OCTA En-Face image, the display of one of the low image quality anterior eye OCTA front image and the high image quality anterior eye OCTA front image may be changed to the other.

In addition, in an OCT device capable of photographing the fundus, if the anterior eye can also be photographed, an anterior eye adapter may be attached. Also, the optical system of the OCT device may be configured to be movable a distance approximately equal to the axial length of the subject's eye without using an anterior eye adapter. In this case, the focus position of the OCT device may be configured to be significantly changeable toward the normal vision side to the extent that an image is formed on the anterior eye.

In addition, for example, in the fundus imaging mode, a fundus OCT tomographic image may be applied to the tomographic image, and in the anterior segment imaging mode, an anterior eye OCT tomographic image may be applied. In addition, the image quality improvement process of the fundus OCTA front image and the anterior eye OCTA front image described above can also be applied as image quality improvement process of the fundus OCT tomographic image and the anterior eye OCT tomographic image. At this time, when the image quality improvement button is pressed, for example, in the fundus imaging mode, the display of one of the low image quality fundus OCT tomographic image and the high image quality fundus OCT tomographic image may be changed to the other in the display area of the tomographic image. In addition, when the image quality improvement button is pressed, for example, in the anterior segment imaging mode, the display of one of the low image quality anterior eye OCT tomographic image and the high image quality anterior eye OCT tomographic image may be changed to the other in the display area of the tomographic image.

In addition, for example, in the fundus imaging mode, a fundus OCTA tomographic image may be applied to the tomographic image, and in the anterior eye imaging mode, an anterior eye OCTA tomographic image may be applied. In addition, the image quality improvement processing of the fundus OCTA front image and the anterior eye OCTA front image described above can also be applied as image quality improvement processing of the fundus OCTA tomographic image and the anterior eye OCTA tomographic image. At this time, for example, in the fundus imaging mode, in the display area of the tomographic image, information indicating a vascular region (e.g., motion contrast data above a threshold) in the fundus OCTA tomographic image may be configured to be displayed superimposed on the fundus OCT tomographic image at the corresponding position. In addition, for example, in the anterior eye imaging mode, in the display area of the tomographic image, information indicating a vascular region in the anterior eye OCTA tomographic image may be displayed superimposed on the anterior eye OCT tomographic image at the corresponding position.

In this way, for example, in cases where the feature quantities (appearance of the subject) of multiple types of medical images are considered to be relatively similar to each other, at least one type of medical image of the multiple types of medical images may be configured to be able to be improved in image quality using a high image quality model obtained by learning at least one type of medical image of the multiple types of medical images as learning data. This makes it possible to configure a configuration in which image quality improvement of multiple types of medical images can be executed using, for example, a common learned model (common high image quality model).

The display screen for the fundus photography mode and the display screen for the anterior segment photography mode may have the same display layout, or may have a display layout corresponding to each photography mode. Various conditions such as photography conditions and analysis conditions may be the same or different between the fundus photography mode and the anterior segment photography mode.

Here, the target image for the high image quality processing may be, for example, a plurality of OCTA front images (OCTA En-Face image, motion contrast En-Face image) (corresponding to a plurality of depth ranges). The target image for the high image quality processing may be, for example, one OCTA front image corresponding to one depth range. The target image for the high image quality processing may be, for example, a luminance front image (luminance En-Face image), or an OCT tomographic image or a motion contrast data tomographic image (OCTA tomographic image) that is a B-scan image, instead of an OCTA front image. The target image for the high image quality processing may be not only an OCTA front image, but also various medical images such as a luminance front image and an OCT tomographic image or a motion contrast data tomographic image (OCTA tomographic image) that is a B-scan image. In other words, the target image for the high image quality processing may be, for example, at least one of various medical images displayed on the display screen of the output unit 103. At this time, for example, since the image feature amount may differ for each type of image, a trained model for image quality improvement corresponding to each type of target image for image quality improvement processing may be used. For example, when the image quality improvement button is pressed in response to an instruction from the examiner, not only the image quality of the OCTA front image is improved using a trained model for image quality improvement corresponding to the OCTA front image, but also the image quality of the OCT tomographic image may be improved using a trained model for image quality improvement corresponding to the OCT tomographic image. Also, for example, when the image quality improvement button is pressed in response to an instruction from the examiner, not only the display of the high-quality OCTA front image generated using the trained model for image quality improvement corresponding to the OCTA front image is changed to the display of the high-quality OCT tomographic image generated using the trained model for image quality improvement corresponding to the OCT tomographic image. At this time, a line indicating the position of the OCT tomographic image may be superimposed on the OCTA front image. Also, the above-mentioned line may be configured to be movable on the OCTA front image in response to an instruction from the examiner. Also, when the display of the high image quality button is in an active state, after the line is moved, the display may be changed to a high image quality OCT tomographic image obtained by performing high image quality processing on the OCT tomographic image corresponding to the current position of the line. Also, a high image quality button may be displayed for each target image for high image quality processing, so that high image quality processing can be performed independently for each image.

Also, information indicating the vascular region in the OCTA tomographic image (for example, motion contrast data above a threshold) may be superimposed and displayed on the OCT tomographic image, which is a B-scan image at a corresponding position. At this time, for example, when the OCT tomographic image is improved in image quality, the OCTA tomographic image at the corresponding position may be improved in image quality. Then, information indicating the vascular region in the OCTA tomographic image obtained by improving image quality may be superimposed and displayed on the OCT tomographic image obtained by improving image quality. Note that the information indicating the vascular region may be any identifiable information such as color. Also, the superimposed display and non-display of the information indicating the vascular region may be changed according to an instruction from the examiner. Also, when a line indicating the position of the OCT tomographic image is moved on the OCTA front image, the display of the OCT tomographic image may be updated according to the position of the line. At this time, the OCTA tomographic image at the corresponding position is also updated, so that the superimposed display of the information indicating the vascular region obtained from the OCTA tomographic image may be updated. This allows, for example, the three-dimensional distribution and state of the vascular region to be effectively confirmed while easily checking the positional relationship between the vascular region and the region of interest at any position. In addition, the image quality of the OCTA tomographic image may be improved by an averaging process or the like of a plurality of OCTA tomographic images acquired at corresponding positions, instead of using a trained model for improving image quality. In addition, the OCT tomographic image may be a pseudo OCT tomographic image reconstructed as a cross section at any position in the OCT volume data. In addition, the OCTA tomographic image may be a pseudo OCTA tomographic image reconstructed as a cross section at any position in the OCTA volume data. Note that the arbitrary position may be at least one arbitrary position, and may be configured to be changeable according to an instruction from the examiner. In this case, a plurality of pseudo tomographic images corresponding to a plurality of positions may be reconstructed.

The tomographic image (e.g., OCT tomographic image or OCTA tomographic image) may be displayed only once, or multiple images may be displayed. When multiple tomographic images are displayed, tomographic images acquired at different positions in the sub-scanning direction may be displayed. When multiple tomographic images acquired by, for example, a cross scan are displayed with high image quality, images in different scanning directions may be displayed. When multiple tomographic images acquired by, for example, a radial scan are displayed with high image quality, a partially selected multiple tomographic images (e.g., two tomographic images at positions symmetrical to each other with respect to a reference line) may be displayed. Furthermore, multiple tomographic images may be displayed on a follow-up display screen (follow-up display screen), and instructions for image quality improvement and analysis results (e.g., thickness of a specific layer, etc.) may be displayed by a method similar to the above-mentioned method. At this time, the multiple tomographic images displayed may be multiple tomographic images acquired at different dates and times of a specific part of the eye to be examined, or multiple tomographic images acquired at different times on the same examination day. Also, image quality improvement processing may be performed on the tomographic images based on information stored in a database by a method similar to the above-mentioned method.

Similarly, when the SLO image is displayed with high image quality, for example, the SLO image displayed on the same display screen may be displayed with high image quality. Furthermore, when the luminance front image is displayed with high image quality, for example, the luminance front image displayed on the same display screen may be displayed with high image quality. Furthermore, a plurality of SLO images and luminance front images may be displayed on a display screen for follow-up observation, and instructions for image quality improvement and analysis results (for example, the thickness of a specific layer, etc.) may be displayed by a method similar to the above-mentioned method. Furthermore, image quality improvement processing may be performed on the SLO image and the luminance front image based on information stored in a database by a method similar to the above-mentioned method. Note that the display of the tomographic image, the SLO image, and the luminance front image is an example, and these images may be displayed in any manner according to the desired configuration. Furthermore, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the luminance front image may be displayed with high image quality by a single instruction.

With this configuration, the display control unit 101-5 can display the high-quality image obtained by the image quality improvement process on the output unit 103. Note that when at least one of a plurality of conditions related to the display of high-quality images, the display of analysis results, the depth range of the displayed front image, etc. is selected, the selected condition may be maintained even if the display screen is changed. Note that the display of various high-quality images, the above-mentioned lines, information indicating the vascular region, etc. may be controlled by the display control unit 101-5.

The high image quality model may be used for at least one frame of the live video on the preview screen displayed on the output unit 103 by the display control unit 101-5. In this case, when multiple live video images of different parts or different types are displayed on the preview screen, a trained model corresponding to each live video may be used. For example, an image with high image quality using a high image quality model for an anterior eye image may be used for an anterior eye image used for alignment processing. Similarly, an image with high image quality using a high image quality model for each image may be used for various images used for detection processing of a specific area in various images.

At this time, for example, when the high quality button is pressed in response to an instruction from the examiner, the display of multiple different types of live moving images (for example, an anterior eye image, an SLO image, a tomographic image) may be changed (simultaneously) to a display of a high quality moving image obtained by performing high quality processing on each of them. At this time, the display of the high quality moving image may be a continuous display of high quality images obtained by performing high quality processing on each frame. Also, for example, since the image features may differ for each type of image, a learned model for high quality processing corresponding to each type of target image for high quality processing may be used. For example, when the high quality button is pressed in response to an instruction from the examiner, not only the anterior eye image is high quality processed using a high quality model corresponding to the anterior eye image, but also the SLO image is high quality processed using a high quality model corresponding to the SLO image. Also, for example, when the high quality button is pressed in response to an instruction from the examiner, not only the display of a high quality anterior eye image generated using a high quality model corresponding to the anterior eye image is changed to the display of a high quality SLO image generated using a high quality model corresponding to the SLO image. Also, for example, when the high image quality button is pressed in response to an instruction from the examiner, not only the SLO image is processed to have high image quality using a high image quality model corresponding to the SLO image, but also the tomographic image is processed to have high image quality using a high image quality model corresponding to the tomographic image. Also, for example, when the high image quality button is pressed in response to an instruction from the examiner, not only the display is changed to a high-image quality SLO image generated using a high image quality model corresponding to the SLO image, but also the display may be changed to a high-image quality tomographic image generated using a high image quality model corresponding to the tomographic image. At this time, a line indicating the position of the tomographic image may be superimposed on the SLO image. Also, the above line may be configured to be movable on the SLO image in response to an instruction from the examiner. Also, when the display of the high image quality button is in an active state, after the above line is moved, the display may be changed to a high-image quality tomographic image obtained by processing the tomographic image corresponding to the current position of the line. Also, a high image quality button may be displayed for each target image for high image quality processing, so that each image can be independently processed to have high image quality.

As a result, even for live video images, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of imaging. Therefore, for example, when the operator corrects the alignment position while checking the preview screen, the failure of re-imaging can be reduced, so that the accuracy and efficiency of diagnosis can be improved. In addition, the image processing device 101 may drive and control the above-mentioned scanning means in response to an instruction regarding the start of imaging so that a partial area such as an artifact area obtained by segmentation processing or the like is re-imaging (rescanned) during or at the end of imaging. Note that, depending on the state of the movement of the subject's eye, it may not be possible to capture a good image with one rescan, so the driving control may be performed so that rescanning is repeated a predetermined number of times. At this time, even during the predetermined number of rescans, the rescanning may be configured to be terminated in response to an instruction from the operator (for example, after the cancel imaging button is pressed). At this time, the imaging data until the rescanning is terminated in response to an instruction from the operator may be configured to be saved. Note that, for example, a confirmation dialog may be displayed after the cancel imaging button is pressed, and it may be possible to select whether to save the imaging data or discard the imaging data in response to an instruction from the operator. Also, for example, after the cancel shooting button is pressed, the next rescan is not executed (although the current rescan is executed until completion), and the system may be configured to wait for an instruction (input) from the operator in the confirmation dialog. Also, for example, when information indicating the reliability of the object detection result for the target area (for example, a numerical value indicating a ratio) exceeds a threshold, each adjustment, the start of shooting, etc. may be automatically performed. Also, for example, when information indicating the reliability of the object detection result for the target area (for example, a numerical value indicating a ratio) exceeds a threshold, each adjustment, the start of shooting, etc. may be changed to an executable state (cancellation of the execution prohibition state) in response to an instruction from the examiner.

Here, during auto-alignment, the subject, such as the retina of the subject's eye 200, may not yet be imaged properly. For this reason, there is a possibility that a high-quality image may not be obtained with high accuracy due to a large difference between the medical image input to the trained model and the medical image used as training data. Therefore, when an evaluation value such as an image quality evaluation of a tomographic image (B-scan image) exceeds a threshold, the display of a high-quality moving image (continuous display of high-quality frames) may be automatically started. In addition, when an evaluation value such as an image quality evaluation of a tomographic image exceeds a threshold, the image quality improvement button may be changed to a state (active state) that can be specified by the examiner. The image quality improvement button is a button for specifying the execution of image quality improvement processing. Of course, the image quality improvement button may also be a button for instructing the display of a high-quality image.

Also, a different image quality improvement model may be prepared for each imaging mode with a different scan pattern, etc., and a trained model for image quality improvement corresponding to the selected imaging mode may be selected. Also, a single image quality improvement model obtained by training learning data including various medical images obtained in different imaging modes may be used.

Here, in an ophthalmic device, for example, an OCT device, the scanning pattern of the light beam used for measurement and the imaging site differ for each imaging mode. Therefore, for a trained model using a tomographic image as input data, a trained model may be prepared for each imaging mode, and a trained model corresponding to the imaging mode selected according to an instruction from an operator may be selected. In this case, the imaging modes may include, for example, a retina imaging mode, an anterior eye imaging mode, a vitreous body imaging mode, a macular imaging mode, an optic disc imaging mode, an OCTA imaging mode, etc. In addition, the scan patterns may include a 3D scan, a radial scan, a cross scan, a circle scan, a raster scan, and a Lissajous scan (scanning along a Lissajous curve), etc. In addition, in the OCTA imaging mode, the imaging control unit 101-2 controls the above-mentioned scanning unit so that the measurement light is scanned multiple times in the same region (same position) of the subject's eye. In the OCTA imaging mode, for example, a raster scan, a radial scan, a cross scan, a circle scan, a Lissajous scan, etc. can be set as the scan pattern. Furthermore, for a trained model that uses a tomographic image as input data, training can be performed using tomographic images corresponding to cross sections in different directions as training data. For example, training can be performed using tomographic images of cross sections in the xz direction and tomographic images of cross sections in the yz direction as training data.

The determination of whether or not to execute image quality improvement processing using the image quality improvement model (or display a high-quality image obtained by image quality improvement processing) may be made according to an instruction from an operator using an image quality improvement button provided on the display screen, or according to settings previously stored in the storage unit 101-3. The fact that image quality improvement processing is performed using a trained model (image quality improvement model) may be displayed by displaying an active state of the image quality improvement button, or a message to that effect may be displayed on the display screen. The image quality improvement processing may be performed in the execution state at the time of the previous startup of the ophthalmic apparatus, or may be performed in the execution state at the time of the previous examination for each subject.

In addition, the moving images to which various learned models such as high image quality models can be applied are not limited to live moving images, and may be, for example, moving images stored (saved) in the storage unit 101-3. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus stored (saved) in the storage unit 101-3 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, a reference frame may first be selected based on a condition such that the vitreous body is present as much as possible on the frame. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high image quality image (high image quality frame) generated sequentially by the high image quality engine for at least one frame of the moving image may be configured to be continuously displayed.

As for the above-mentioned method of inter-frame alignment, the method of alignment in the X direction and the method of alignment in the Z direction (depth direction) may be the same or different methods may be applied. In addition, alignment in the same direction may be performed multiple times using different methods. For example, after performing coarse alignment, precise alignment may be performed. In addition, as the alignment method, for example, (coarse Z direction) alignment using retinal layer boundaries obtained by segmenting a tomographic image (B-scan image), (precise X-direction and Z-direction) alignment using correlation information (similarity) between a reference image and a plurality of regions obtained by dividing a tomographic image, (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image), (X direction) alignment using a two-dimensional front image, etc. may be configured so that rough alignment is performed on a pixel-by-pixel basis and then precise alignment is performed on a subpixel-by-subpixel basis.

The high image quality model may be updated by additional learning using the ratio value set (changed) in response to an instruction from the examiner as learning data. For example, if the examiner tends to set the ratio of the input image to the high image quality image high when the input image is relatively dark, the learned model will be additionally learned to have such a tendency. This allows, for example, customization as a learned model that can obtain a synthesis ratio that suits the examiner's preferences. At this time, a button for determining whether or not to use the set (changed) ratio value as learning data for additional learning in response to an instruction from the examiner may be displayed on the display screen. In addition, the ratio determined using the learned model may be set as a default value, and the ratio value may be changed from the default value in response to an instruction from the examiner. In addition, the high image quality model may be a learned model obtained by additional learning of learning data including at least one high image quality image generated using the high image quality model. At this time, it may be configured to be selectable by an instruction from the examiner as to whether or not to use the high image quality image as learning data for additional learning.

[Modification 4]
The image feature acquisition unit 101-44, the extraction unit 101-462, and the area detection unit 302 may generate a label image using a trained model for image segmentation and perform image segmentation processing. Here, the label image refers to a label image in which a region label is attached to each pixel of the tomographic image. Specifically, it is an image in which an arbitrary region is divided by a group of pixel values (hereinafter, label values) that can identify an arbitrary region among the group of regions depicted in the acquired image. Here, the arbitrary region to be identified includes a region of interest, a volume of interest (VOI), etc.

By identifying the coordinate group of pixels having an arbitrary label value from an image, it is possible to identify the coordinate group of pixels depicting a corresponding area in the image, such as the retinal layer. Specifically, for example, if the label value indicating the ganglion cell layer that constitutes the retina is 1, the coordinate group of the pixels in the image having a pixel value of 1 is identified, and the pixel group corresponding to this coordinate group is extracted from the image. This makes it possible to identify the area of the ganglion cell layer in the image.

The image segmentation process may include a process of performing a reduction or enlargement process on the label image. In this case, the image completion process method used to reduce or enlarge the label image uses a nearest neighbor method or the like so as not to erroneously generate undefined label values or label values that should not exist at the corresponding coordinates.

Image segmentation processing is processing that identifies regions called ROIs (Regions of Interest) or VOIs, such as organs or lesions depicted in an image, for use in image diagnosis or image analysis. For example, image segmentation processing can identify a group of regions of layers that make up the retina from an image acquired by OCT imaging of the posterior segment. If no region to be identified is depicted in the image, the number of identified regions is zero. If multiple region groups to be identified are depicted in the image, the number of identified regions may be multiple, or it may be one region that surrounds the group of regions.

The identified region groups are output as information that can be used in other processes. Specifically, for example, the coordinate groups of the pixel groups that make up each of the identified region groups can be output as a numerical data group. Also, for example, coordinate groups indicating rectangular regions, elliptical regions, rectangular parallelepiped regions, ellipsoidal regions, etc. that include each of the identified region groups can be output as a numerical data group. Furthermore, for example, coordinate groups indicating straight lines, curves, planes, curved surfaces, etc. that correspond to the boundaries of the identified region groups can be output as a numerical data group. Also, for example, a label image indicating the identified region groups can be output.

Here, as a machine learning model for image segmentation, for example, a convolutional neural network (CNN) can be used. The CNN used in this modified example can be configured as a U-net type machine learning model having an encoder function consisting of multiple layers including multiple downsampling layers, and a decoder function consisting of multiple layers including multiple upsampling layers. In the U-net type machine learning model, the position information (spatial information) that is obscured in multiple layers configured as an encoder can be used in layers of the same dimension (layers that correspond to each other) in multiple layers configured as a decoder (for example, by using skip connections).

As an example of a modified CNN configuration, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer. Through these steps of the CNN, the features of the captured image can be extracted.

As the machine learning model according to this modification, for example, CNN (U-net type machine learning model), a model combining CNN and LSTM, FCN (Fully Convolutional Network), SegNet, etc. can be used. In addition, a machine learning model that performs object recognition, etc. can be used according to a desired configuration. As the machine learning model that performs object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, a machine learning model that performs object recognition on a region basis can also be used. As the machine learning model that performs object recognition on a region basis, YOLO (You Only Look Once), SSD (Single Shot Detector, or Single Shot MultiBox Detector) can also be used.

The learning data for the machine learning model for image segmentation uses a tomographic image acquired by OCT as input data, and uses a labeled image in which a region is labeled for each pixel of the tomographic image as output data. Examples of labeled images that can be used include images labeled with the internal limiting membrane (ILM), nerve fiber layer (NFL), ganglion cell layer (GCL), photoreceptor inner-outer segment junction (ISOS), retinal pigment epithelium layer (RPE), Bruch's membrane (BM), and choroid. Note that images labeled with other regions, such as the vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner plexiform layer (IPL), inner nuclear layer (INL), cornea, anterior chamber, iris, and crystalline lens, may also be used.

In addition, the input data of the machine learning model for image segmentation is not limited to tomographic images. It may be anterior eye images, SLO images, OCTA images, etc. In this case, the learning data may have various images as input data, and labeled images in which each pixel of the various images is labeled with a region name, etc., as output data. For example, if the input data of the learning data is an SLO image, the output data may be an image labeled with the periphery of the optic disc, Disc, Cup, etc.

The labeled image used as output data may be an image in which a doctor or the like has labeled each region in a tomographic image, or an image in which a label has been added to each region by a rule-based region detection process. However, if machine learning is performed using an inappropriately labeled image as output data of the training data, the image obtained using a trained model trained using the training data may also be an inappropriately labeled image. Therefore, by removing pairs including such labeled images from the training data, it is possible to reduce the possibility that an inappropriately labeled image is generated using a trained model. Here, the rule-based region detection process refers to a detection process that utilizes known regularities, such as the regularities of the shape of the retina.

The image feature acquisition unit 101-44, the extraction unit 101-462, and the area detection unit 302 are expected to perform image segmentation processing using such a trained model for image segmentation, thereby quickly and accurately detecting specific areas in various images. The trained model for image segmentation may also be prepared for each type of image that is the input data. For OCTA frontal images and En-Face images, a trained model may also be prepared for each depth range for generating the image. Furthermore, the trained model for image segmentation may also be trained for images for each imaging site (e.g., center of the macula, center of the optic disc), or may be trained regardless of imaging site.

In addition, for the trained model for image segmentation, additional training may be performed using data manually corrected in response to instructions from an operator as training data. The need for additional training and the decision on whether to send data to a server may also be made in a similar manner. In these cases, it is expected that the accuracy of each process can be improved and processing can be performed according to the examiner's preferences.

Furthermore, when the image processing device 101 detects partial regions (e.g., a region of interest, an artifact region, an abnormal region, etc.) of the test eye 200 using a trained model, it can also perform a predetermined image processing for each detected partial region. As an example, a case will be described where at least two partial regions of the vitreous region, the retina region, and the choroid region are detected. In this case, when performing image processing such as contrast adjustment on the at least two detected partial regions, adjustments suitable for each region can be made by using different image processing parameters for each region. By displaying an image in which adjustments suitable for each region have been made, the operator can more appropriately diagnose diseases, etc. for each partial region. Note that the configuration in which different image processing parameters are used for each detected partial region may be similarly applied to partial regions of the test eye 200 obtained by detecting partial regions of the test eye 200 without using a trained model.

[Modification 5]
The display control unit 101-5 in the above-mentioned various embodiments and modified examples may display the analysis results such as the layer thickness of a desired layer and various blood vessel densities on the report screen of the display screen after the tomographic image is captured. In addition, the display control unit 101-5 may display the value (distribution) of a parameter related to a region of interest including at least one of the optic disc, the macular region, the blood vessel region, the capillary region, the arterial region, the venous region, the nerve fiber bundle, the vitreous region, the macular region, the choroid region, the sclera region, the lamina cribrosa region, the retinal layer boundary, the retinal layer boundary edge, the photoreceptor cell, the blood cell, the blood vessel wall, the blood vessel inner wall boundary, the blood vessel outer boundary, the ganglion cell, the cornea region, the iridocorneal region, the Schlemm's canal, etc., as the analysis result. Here, the region of interest may be, for example, a vortex vein, which is an outlet to the outside of the eye of a blood vessel in the Haller layer (an example of a blood vessel in a partial depth range of the choroid region). In this case, the parameters related to the site of interest may be, for example, the number of vortex veins (for example, the number per region), the distance from the optic disc to each vortex vein, the angle at which each vortex vein is located around the optic disc, etc. This makes it possible to accurately diagnose, for example, various diseases related to Pachychoroid (thickened choroid) (for example, choroidal neovascularization). In addition, for example, by analyzing medical images to which various artifact reduction processes have been applied, the above-mentioned various analysis results can be displayed as accurate analysis results. Note that the artifacts may be, for example, false image regions caused by light absorption by blood vessel regions, projection artifacts, band-like artifacts in front images caused in the main scanning direction of the measurement light due to the state of the subject's eye (movement, blinking, etc.), etc. In addition, the artifacts may be any image loss regions that randomly occur on the medical images of a specific part of the subject for each shooting. The display control unit 101-5 may also cause the output unit 103 to display, as an analysis result, parameter values (distribution) for a region including at least one of the various artifacts (image loss region) described above. Also, the display control unit 101-5 may cause the output unit 103 to display, as an analysis result, parameter values (distribution) for a region including at least one of abnormal sites such as drusen, neovascularization, white spots (hard exudates), and pseudodrusen. Also, a comparison result obtained by comparing the analysis result with a standard value or standard range obtained using a standard database may be displayed.

The analysis results may be displayed as an analysis map, or as sectors showing statistical values corresponding to each divided region. The analysis results may be generated using a trained model (analysis result generation engine, trained model for generating analysis results) obtained by learning using the analysis results of medical images as training data. In this case, the trained model may be obtained by learning using training data including medical images and the analysis results of the medical images, or training data including medical images and the analysis results of a different type of medical image from the medical images.

The learning data for performing image analysis may include a labeled image generated using a trained model for image segmentation processing and an analysis result of a medical image using the labeled image. In this case, the image processing device 101 can function as an example of an analysis result generating unit that generates an analysis result of a tomographic image from the result of image segmentation processing using, for example, a trained model for generating analysis results. Furthermore, the trained model may be obtained by learning using learning data that includes input data that is a set of multiple medical images of different types of a specified area, such as a luminance En-Face image and a motion contrast front image (OCTA En-Face image).

The system may also be configured to display analysis results obtained using high-quality images generated using the high-quality model. In this case, the input data included in the training data may be high-quality images generated using a trained model for image quality improvement, or a set of low-quality images and high-quality images. The training data may be images that have been enhanced using a trained model and at least partially corrected manually or automatically.

The learning data may be data in which input data is labeled (annotated) with information including at least one of analytical values (e.g., average values, median values, etc.) obtained by analyzing the analysis region, a table including analytical values, an analysis map, and the position of the analysis region such as a sector in an image, as correct answer data (for supervised learning). Note that the system may be configured to display the analysis results obtained using the trained model for generating analysis results in response to an instruction from an operator.

In addition, the display control unit 101-5 in the above-mentioned embodiment and modified example may display various diagnosis results such as diabetic retinopathy, glaucoma, age-related macular degeneration, etc., on the report screen of the display screen. At this time, for example, by analyzing the medical image to which the above-mentioned various artifact reduction processes have been applied, it is possible to display a diagnosis result with high accuracy. In addition, the diagnosis result may display the position of the identified abnormal part, etc. on the image, or the state of the abnormal part, etc., may be displayed by characters, etc. Furthermore, the classification result of the abnormal part, etc. (for example, Curtin classification) may be displayed as the diagnosis result. In addition, as the classification result, for example, information indicating the certainty of each abnormal part (for example, a numerical value indicating a ratio) may be displayed. In addition, information necessary for the doctor to confirm the diagnosis may be displayed as the diagnosis result. The above necessary information may be, for example, advice such as additional imaging. For example, when an abnormal part is detected in the blood vessel region in the OCTA image, it may be displayed that an additional fluorescent imaging using a contrast agent that can observe the blood vessels in more detail than OCTA will be performed. In addition, the diagnosis result may be information regarding the subject's future medical treatment policy, etc. The diagnosis result may include at least one of the following: a diagnosis name, a type and condition (degree) of a lesion (abnormal site), a position of the lesion in an image, a position of the lesion relative to a region of interest, findings (image findings, etc.), a basis for the diagnosis name (positive medical support information, etc.), and a basis for denying the diagnosis name (negative medical support information). In this case, for example, a diagnosis result that is more likely than a diagnosis result such as a diagnosis name input in response to an instruction from an examiner may be displayed as medical support information. In addition, when multiple types of medical images are used, for example, a type of medical image that may be the basis for the diagnosis result may be displayed in an identifiable manner. In addition, the basis for the diagnosis result may be a map (attention map, activation map) that visualizes the features extracted by the trained model, for example, a color map (heat map) that shows the features in color. In this case, for example, the heat map may be superimposed on the medical image that is the input data. The heat map can be obtained, for example, using Gradient-weighted Class Activation Mapping (Grad-CAM) or Guided Grad-CAM, which are techniques for visualizing areas that contribute greatly to the output value of a predicted (estimated) class (areas with large gradients).

The diagnosis result may be generated using a trained model (a diagnosis result generation engine, a trained model for generating diagnosis results) obtained by learning using the diagnosis results of medical images as training data. The trained model may also be obtained by learning using training data including medical images and diagnostic results for those medical images, or training data including medical images and diagnostic results for a type of medical image different from those medical images.

The learning data may also include a labeled image generated using a region recognition engine or a trained model for segmentation processing, and a diagnosis result of a medical image using the labeled image. In this case, the image processing device 101 can function as an example of a diagnosis result generating unit that generates a diagnosis result of a tomographic image from the result of image segmentation processing, for example, using a trained model for generating a diagnosis result.

Furthermore, the system may be configured to display a diagnosis result obtained using a high-quality image generated using the high-quality engine. In this case, the input data included in the learning data may be a high-quality image generated using the high-quality engine, or a set of a low-quality image and a high-quality image. Note that the learning data may be an image that has been improved in quality using a trained model and at least a portion of which has been manually or automatically corrected.

The learning data may be data obtained by labeling (annotating) the input data with information including at least one of the following as correct answer data (in supervised learning): diagnosis name, type and condition (degree) of lesion (abnormal area), position of lesion in image, position of lesion relative to the region of interest, findings (image findings, etc.), grounds for diagnosis name (positive medical support information, etc.), grounds for denying diagnosis name (negative medical support information), etc. In addition, the system may be configured to display the diagnosis result obtained using the trained model for generating diagnosis result in response to instructions from the examiner.

A trained model may be prepared for each piece of information or each type of information used as input data, and a diagnostic result may be obtained using the trained model. In this case, statistical processing may be performed on the information output from each trained model to determine the final diagnostic result. For example, the proportion of information output from each trained model may be added for each type of information, and the information with a higher total proportion than the other information may be determined as the final diagnostic result. Note that the statistical processing is not limited to calculating the total, and may also be calculating an average or median. For example, the diagnostic result may be determined using information with a higher proportion (the information with the highest proportion) than the other information among the information output from each trained model. Similarly, the diagnostic result may be determined using information with a proportion equal to or greater than a threshold among the information output from each trained model.

In addition, the system may be configured to be capable of judging (approving) whether the determined diagnostic result is good or bad according to an instruction (selection) from an operator. In addition, the diagnostic result may be determined from the information output from each trained model according to an instruction (selection) from an operator. At this time, for example, the display control unit 101-5 may display the information output from each trained model and its ratio on the output unit 103 in an order. Then, the system may be configured such that the operator selects, for example, information with a higher ratio than other information, and the selected information is determined as the diagnostic result. Furthermore, the diagnostic result may be determined using a machine learning model from the information output from each trained model. In this case, the machine learning algorithm may be a different type of machine learning algorithm from the machine learning algorithm used to generate the diagnostic result, and for example, a neural network, a support vector machine, an adaboost, a Bayesian network, or a random forest may be used.

The learning of the various trained models described above may be not only supervised learning (learning with labeled training data) but also semi-supervised learning. Semi-supervised learning is a method in which, for example, after multiple classifiers (classifiers) each perform supervised learning, unlabeled training data is identified (classified), and the unlabeled training data is automatically labeled (annotated) according to the reliability of the identification result (classification result) (for example, identification results with a probability equal to or greater than a threshold), and learning is performed with the labeled training data. Semi-supervised learning may be, for example, co-training (Co-Training or Multiview). In this case, the trained model for generating a diagnosis result may be, for example, a trained model obtained by semi-supervised learning (for example, co-training) using a first classifier that identifies medical images of normal subjects and a second classifier that identifies medical images containing specific lesions. Note that the purpose is not limited to diagnosis, and may be, for example, imaging support. In this case, the second classifier may be, for example, a classifier that identifies medical images including partial regions such as a site of interest or an artifact region.

The display control unit 101-5 according to the various embodiments and modifications described above may display object recognition results (object detection results) and segmentation results of partial regions such as the above-mentioned attention area, artifact area, and abnormal area on the report screen of the display screen. At this time, for example, a rectangular frame or the like may be superimposed around the object on the image. Also, for example, a color or the like may be superimposed on the object in the image. Note that the object recognition results and segmentation results may be generated using a trained model (object recognition engine, trained model for object recognition, segmentation engine, trained model for segmentation) obtained by learning learning data in which information indicating object recognition and segmentation is used as correct answer data to label (annotate) medical images. Note that the above-mentioned analysis result generation and diagnosis result generation may be obtained by using the above-mentioned object recognition result and segmentation result. For example, the analysis result generation and diagnosis result generation process may be performed on the attention area obtained by the object recognition and segmentation process.

When detecting abnormal areas, the image processing device 101 may use a Generative Adversarial Network (GAN) or a Variational Auto-Encoder (VAE). For example, a DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning to generate medical images and a classifier obtained by learning to distinguish between new medical images generated by the generator and real medical images can be used as a machine learning model.

When DCGAN is used, for example, a discriminator encodes an input medical image to make it a latent variable, and a generator generates a new medical image based on the latent variable. The difference between the input medical image and the new medical image generated can then be extracted (detected) as an abnormal area. When VAE is used, for example, an encoder encodes an input medical image to make it a latent variable, and a decoder decodes the latent variable to generate a new medical image. The difference between the input medical image and the new medical image generated can then be extracted as an abnormal area.

Furthermore, the image processing device 101 may detect abnormal areas using a convolutional auto-encoder (CAE). When using a CAE, the same medical images are trained as input data and output data during learning. As a result, when a medical image containing an abnormal area is input to the CAE during estimation, a medical image without an abnormal area is output according to the learning tendency. After that, the difference between the medical image input to the CAE and the medical image output from the CAE can be extracted as the abnormal area.

In these cases, the image processing device 101 can generate information on the difference between a medical image obtained using a generative adversarial network or an autoencoder and a medical image input to the generative adversarial network or an autoencoder as information on an abnormal part. As a result, the image processing device 101 can be expected to detect an abnormal part quickly and accurately. For example, even if it is difficult to collect a large number of medical images including abnormal parts as training data in order to improve the detection accuracy of abnormal parts, medical images of normal subjects, which are relatively easy to collect in large numbers, can be used as training data. For this reason, for example, learning for detecting abnormal parts with high accuracy can be efficiently performed. Here, the autoencoder includes a VAE, a CAE, and the like. In addition, at least a part of the generating unit of the generative adversarial network may be composed of a VAE. As a result, for example, it is possible to generate a relatively clear image while reducing the phenomenon of generating similar data. For example, the image processing device 101 can generate information on the difference between a medical image obtained from various medical images using a generative adversarial network or an autoencoder and a medical image input to the generative adversarial network or the autoencoder as information on an abnormal part. Furthermore, for example, the display control unit 101-5 can cause the output unit 103 to display information regarding the difference between a medical image obtained from various medical images using a generative adversarial network or an autoencoder and the medical image input to the generative adversarial network or the autoencoder as information regarding abnormal areas.

In particular, the trained model for generating a diagnosis result may be a trained model obtained by training with training data including input data that is a set of multiple medical images of different types of a specific part of a subject. In this case, the input data included in the training data may be, for example, input data that is a set of a motion contrast front image and a luminance front image (or a luminance tomographic image) of the fundus. In addition, the input data included in the training data may be, for example, input data that is a set of a tomographic image (B-scan image) of the fundus and a color fundus image (or a fluorescent fundus image). The multiple medical images of different types may be anything obtained using different modalities, different optical systems, or different principles.

In particular, the trained model for generating diagnostic results may be a trained model obtained by training using training data including input data consisting of a set of multiple medical images of different parts of the subject. In this case, the input data included in the training data may be, for example, input data consisting of a set of a tomographic image (B-scan image) of the fundus and a tomographic image (B-scan image) of the anterior segment. In addition, the input data included in the training data may be, for example, input data consisting of a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic disc of the fundus.

The input data included in the learning data may be a plurality of medical images of different parts and different types of the subject. In this case, the input data included in the learning data may be, for example, a set of a tomographic image of the anterior eye and a color fundus image. The above-mentioned learned model may be a learned model obtained by learning with learning data including input data that is a set of a plurality of medical images of a specific part of the subject with different shooting angles of view. The input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a specific part into multiple regions, such as a panoramic image. In this case, by using a wide-angle image such as a panoramic image as the learning data, it is possible to accurately obtain the feature amount of the image because it has more information than a narrow-angle image, and the processing result can be improved. The input data included in the learning data may be a set of a plurality of medical images of a specific part of the subject at different dates and times.

In addition, the display screen on which at least one of the above-mentioned analysis results, diagnosis results, object recognition results, and segmentation results is displayed is not limited to a report screen. Such a display screen may be displayed on at least one of the following display screens: an image capture confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before image capture (a display screen on which various live moving images are displayed). For example, by displaying at least one of the above results obtained using the above-mentioned trained model on the image capture confirmation screen, the operator can confirm the highly accurate results even immediately after image capture.

In addition, for example, when a specific object is recognized, a frame surrounding the recognized object may be superimposed on the live video image. At this time, if information indicating the certainty of the object recognition result (for example, a numerical value indicating a ratio) exceeds a threshold, the frame surrounding the object may be highlighted, for example, by changing the color. This allows the examiner to easily identify the object on the live video image.

In addition, a trained model for generating correct answer data for generating correct answer data such as labeling (annotation) may be used to generate correct answer data used in training the various trained models described above. In this case, the trained model for generating correct answer data may be obtained by additionally learning (sequentially) correct answer data obtained by the examiner through labeling (annotation). In other words, the trained model for generating correct answer data may be obtained by additionally learning training data in which data before labeling is used as input data and data after labeling is used as output data. In addition, in a plurality of consecutive frames such as a moving image, the result of a frame determined to have low accuracy of the result may be corrected taking into account the results of object recognition, segmentation, etc. of the previous and following frames. In this case, the trained model may be configured to additionally learn the corrected result as correct answer data in response to an instruction from the examiner. In addition, for example, for a medical image with low accuracy of the result, the examiner may be configured to additionally learn a labeled (annotated) image as input data while checking a color map (heat map) showing the feature amounts in color, which is an example of a map (attention map, activation map) that visualizes the feature amounts extracted by the trained model on the medical image. For example, if the areas to be noted in a heat map on a layer immediately before outputting the results of the trained model are different from the examiner's intention, additional learning may be performed on a medical image in which the areas that the examiner thinks should be noted are labeled (annotated). This allows the trained model to additionally train the features of a partial area on a medical image that has a relatively large influence on the output results of the trained model by prioritizing (weighting) them over other areas.

Here, the various trained models described above can be obtained by machine learning using training data. Machine learning includes, for example, deep learning consisting of a multi-layered neural network. In addition, for example, a convolutional neural network can be used for at least a part of the multi-layered neural network. In addition, a technology related to an autoencoder (autoencoder) may be used for at least a part of the multi-layered neural network. In addition, a technology related to backpropagation (backpropagation method) may be used for learning. In addition, a method (dropout) of randomly inactivating each unit (each neuron or each node) may be used for learning. In addition, a method (batch normalization) of normalizing data transmitted to each layer of a multi-layered neural network before an activation function (for example, a ReLu function) is applied may be used for learning. However, the machine learning is not limited to deep learning, and any learning using a model that can extract (express) the features of training data such as images by itself through learning may be used. Here, the machine learning model refers to a learning model using a machine learning algorithm such as deep learning. A trained model is a model that has been trained (learned) in advance using appropriate training data for a machine learning model based on an arbitrary machine learning algorithm. However, a trained model is not one that does not undergo further learning, but rather is capable of undergoing additional learning. Training data is composed of a pair of input data and output data (correct answer data). Here, training data may be referred to as teacher data, or correct answer data may be referred to as teacher data.

Note that the GPU can perform efficient calculations by processing more data in parallel. For this reason, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing with the GPU. Therefore, in this modified example, a GPU is used in addition to a CPU for processing by the image processing device 101, which is an example of a learning unit (not shown). Specifically, when a learning program including a learning model is executed, the CPU and the GPU work together to perform calculations to perform learning. Note that the processing of the learning unit may be performed only by the CPU or the GPU. In addition, the processing unit (estimation unit) that performs processing using the various learned models described above may also use a GPU like the learning unit. In addition, the learning unit may include an error detection unit and an update unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network according to the input data input to the input layer and the correct answer data. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the correct answer data. Furthermore, the update unit updates the connection weighting coefficients between the nodes of the neural network based on the error obtained by the error detection unit so as to reduce the error. This update unit updates the connection weighting coefficients, for example, using the backpropagation method. The backpropagation method is a technique for adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the above-mentioned error.

As a machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., a U-net type machine learning model having an encoder function consisting of multiple layers including multiple downsampling layers and a decoder function consisting of multiple layers including multiple upsampling layers can be applied. In a U-net type machine learning model, the position information (spatial information) made ambiguous in the multiple layers configured as an encoder can be used in layers of the same dimension (layers corresponding to each other) in the multiple layers configured as a decoder (for example, by using skip connections).

Furthermore, as the machine learning model used for the above-mentioned object recognition, segmentation, high image quality, etc., for example, FCN (Fully Convolutional Network) or SegNet can be used. Furthermore, a machine learning model that performs object recognition on a region-by-region basis may be used according to a desired configuration. As the machine learning model that performs object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, as the machine learning model that performs object recognition on a region-by-region basis, YOLO (You Only Look Once) or SSD (Single Shot Detector, or Single Shot MultiBox Detector) can also be used.

The machine learning model may be, for example, a capsule network (CapsNet). Here, in a typical neural network, each unit (each neuron or each node) is configured to output a scalar value, so that, for example, spatial information related to the spatial positional relationship (relative position) between features in an image is reduced. This allows learning to be performed in such a way that the effects of local distortion and translation of the image are reduced. On the other hand, in a capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is maintained. This allows learning to be performed in such a way that, for example, the spatial positional relationship between features in an image is taken into account.

[Modification 6]
In the preview screen in the various embodiments and modifications described above, the various trained models described above may be used for at least one frame of the live video. In this case, when multiple live videos of different parts or different types are displayed on the preview screen, the trained models corresponding to each live video may be used. This allows, for example, the processing time to be shortened even for live videos, so that the examiner can obtain highly accurate information before the start of imaging. This allows, for example, failures in re-imaging to be reduced, thereby improving the accuracy and efficiency of diagnosis.

The multiple live video images may be, for example, video images of the anterior eye for alignment in the XYZ directions, and frontal video images of the fundus for focus adjustment of the fundus observation optical system or OCT focus adjustment. The multiple live video images may also be, for example, tomographic video images of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length). When such a preview image is displayed, the image processing device 101 may be configured to perform the above-mentioned various adjustments so that the area detected using the above-mentioned learned model for object recognition or the learned model for segmentation satisfies a predetermined condition. For example, the image processing device 101 may be configured to perform various adjustments such as OCT focus adjustment so that a value (e.g., contrast value or intensity value) related to a vitreous area or a predetermined retinal layer such as RPE detected using the learned model for object recognition or the learned model for segmentation exceeds a threshold value (or becomes a peak value). In addition, for example, the OCT coherence gate may be adjusted so that a specific retinal layer, such as the vitreous region or RPE, detected using a trained model for object recognition or a trained model for segmentation is located at a specific position in the depth direction.

In these cases, the image processing device 101 can use the learned model to perform high-quality processing on the moving image to generate a high-quality moving image. In addition, the imaging control unit 101-2 can drive and control the optical member for changing the imaging range of the reference mirror 221 so that a partial area such as a site of interest obtained by segmentation processing or the like is located at a predetermined position in the display area while a high-quality moving image is displayed. In such a case, the imaging control unit 101-2 can automatically perform alignment processing so that a desired area is located at a predetermined position in the display area based on highly accurate information. The optical member for changing the imaging range may be, for example, an optical member for adjusting the coherence gate position, and specifically, may be the reference mirror 221 that reflects the reference light. In addition, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length, and the optical member may be, for example, a mirror for changing the optical path length of the measurement light (not shown). In addition, the optical member for changing the imaging range may be, for example, the stage unit 100-2. In addition, the imaging control unit 101-2 may drive and control the scanning means in response to an instruction regarding the start of imaging so that a partial area such as an artifact area obtained by segmentation processing or the like is re-imaged (rescanned) during or at the end of imaging. In addition, for example, when information indicating the likelihood of the object recognition result regarding the site of interest (for example, a numerical value indicating a ratio) exceeds a threshold, various adjustments and the start of imaging may be automatically performed. In addition, for example, when information indicating the likelihood of the object recognition result regarding the site of interest (for example, a numerical value indicating a ratio) exceeds a threshold, each adjustment, the start of imaging, etc. may be changed to an executable state (cancellation of the execution prohibition state) in response to an instruction from the examiner.

In addition, the moving images to which the various learned models described above can be applied are not limited to live moving images, and may be, for example, moving images stored (saved) in the storage unit 101-3. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus stored (saved) in the storage unit 101-3 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, a reference frame may first be selected based on a condition such that the vitreous body is present as much as possible on the frame. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) generated sequentially by the learned model for image quality improvement for at least one frame of the moving image may be configured to be continuously displayed.

As for the above-mentioned method of inter-frame alignment, the method of alignment in the X direction and the method of alignment in the Z direction (depth direction) may be the same or different methods may be applied. In addition, alignment in the same direction may be performed multiple times using different methods. For example, after performing coarse alignment, precise alignment may be performed. In addition, as the alignment method, for example, (coarse Z direction) alignment using retinal layer boundaries obtained by segmenting a tomographic image (B-scan image), (precise X-direction and Z-direction) alignment using correlation information (similarity) between a reference image and a plurality of regions obtained by dividing a tomographic image, (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image), (X direction) alignment using a two-dimensional front image, etc. may be configured so that rough alignment is performed on a pixel-by-pixel basis and then precise alignment is performed on a subpixel-by-subpixel basis.

Here, while various adjustments are being made, it is possible that the subject, such as the retina of the subject's eye, may not yet be imaged properly. For this reason, there is a possibility that a high-quality image cannot be obtained with high accuracy due to a large difference between the medical image input to the trained model and the medical image used as training data. Therefore, it is possible to configure the system to automatically start displaying high-quality moving images (continuous display of high-quality frames) when an evaluation value of the image quality evaluation of the tomographic image (B-scan) exceeds a threshold value. In addition, it is possible to configure the system to change the image quality improvement button to a state (active state) that the examiner can specify when an evaluation value of the image quality evaluation of the tomographic image (B-scan) exceeds a threshold value.

For example, a different trained model for improving image quality may be prepared for each imaging mode with a different scan pattern, etc., and a trained model for improving image quality corresponding to the selected imaging mode may be selected. Also, a single trained model for improving image quality obtained by training training data including various medical images obtained in different imaging modes may be used.

[Modification 7]
In the above-described embodiment and modified example, when various trained models are undergoing additional learning, it may be difficult to output (estimate/predict) using the trained model itself during additional learning. For this reason, it is preferable to configure the trained model during additional learning to be prohibited from inputting medical images other than the training data. In addition, another trained model that is the same as the trained model before the additional learning may be prepared as a spare trained model. In this case, it is preferable to configure the spare trained model to be able to input medical images other than the training data during additional learning. Then, after the additional learning is completed, the trained model after the additional learning is evaluated, and if there is no problem, the spare trained model may be replaced with the trained model after the additional learning is performed. In addition, if there is a problem, the spare trained model may be used.

In addition, as an evaluation of the trained model after additional learning, for example, a trained model for classification for classifying high-quality images obtained by the trained model for image quality improvement from other types of images may be used. The trained model for classification may be, for example, a trained model obtained by learning training data including multiple images including high-quality images and low-quality images obtained by the trained model for image quality improvement as input data and data in which the types of these images are labeled (annotated) as correct answer data. In this case, the type of image in the input data at the time of estimation (prediction) may be displayed together with information indicating the likelihood of each type of image included in the correct answer data at the time of learning (for example, a numerical value indicating a ratio). In addition to the above images, the input data for the trained model for classification may include high-quality images that have been subjected to high contrast, noise reduction, etc. by superimposing multiple low-quality images (for example, averaging multiple low-quality images obtained by positioning). In addition, the trained model after the additional learning is performed may be evaluated by, for example, comparing multiple high-quality images obtained from the same image using the trained model after the additional learning and the trained model before the additional learning (backup trained model), or comparing the analysis results of the multiple high-quality images. At this time, for example, it may be determined whether the comparison result of the multiple high-quality images (an example of a change due to additional learning) or the comparison result of the analysis results of the multiple high-quality images (an example of a change due to additional learning) is within a predetermined range, and the determination result may be displayed.

In addition, the trained model obtained by training for each imaging site may be selectively used. Specifically, a plurality of trained models including a first trained model obtained using training data including a first imaging site (e.g., an anterior eye segment, a posterior eye segment, etc.) and a second trained model obtained using training data including a second imaging site different from the first imaging site may be prepared. The image processing device 101 may have a selection means for selecting one of the plurality of trained models. In this case, the image processing device 101 may have a control means for performing additional training on the selected trained model. In response to an instruction from the examiner, the control means can search for data in which the imaging site corresponding to the selected trained model and the image of the imaging site form a pair, and perform training using the searched data as training data as additional training on the selected trained model. The imaging site corresponding to the selected trained model may be obtained from information in the header of the data or may be manually input by the examiner. In addition, the search for data may be performed via a network from a server or the like of an external facility such as a hospital or a research institute. This allows for efficient additional learning for each imaging part using images of the imaging part that correspond to the trained model.

The selection means and control means may be configured by a software module executed by a processor such as the CPU or MPU of the image processing device 101. The selection means and control means may also be configured by a circuit that performs a specific function such as an ASIC, or an independent device.

When acquiring learning data for additional learning from a server of an external facility such as a hospital or research institute via a network, it is useful to reduce the deterioration of reliability due to tampering or system trouble during additional learning. Therefore, the validity of the learning data for additional learning may be detected by confirming the consistency by digital signature or hashing. This makes it possible to protect the learning data for additional learning. In this case, if the validity of the learning data for additional learning cannot be detected as a result of confirming the consistency by digital signature or hashing, a warning to that effect is issued and additional learning using the learning data is not performed. The server may be installed anywhere and may take any form, such as a cloud server, a fog server, or an edge server. When configuring a network within a facility, a site including a facility, or a region including multiple facilities to be capable of wireless communication, the reliability of the network may be improved by configuring it to use radio waves in a dedicated wavelength band allocated exclusively to the facility, site, region, etc. The network may also be configured using wireless communication that is capable of high speed, large capacity, low latency, and multiple simultaneous connections.

In addition, the protection of data by confirming the consistency as described above is not limited to learning data for additional learning, but can be applied to data including medical images. The image management system may be configured so that transactions of data including medical images between servers of multiple facilities are managed by a distributed network. The image management system may be configured so that multiple blocks in which a transaction history and a hash value of a previous block are recorded together are linked in chronological order. In addition, as a technology for confirming the consistency, encryption that is difficult to calculate even using a quantum computer such as a quantum gate method (e.g., lattice encryption, quantum encryption by quantum key distribution, etc.) may be used. Here, the image management system may be a device and system that receives and stores images captured by an imaging device or images that have been processed. In addition, the image management system can transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. The image management system may include, for example, a picture archiving and communication system (PACS). In addition, the image management system has a database that can store various information such as information on the subject associated with the received image and the time of shooting. In addition, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with stored images in response to requests from other devices.

When additional learning is performed on various trained models, the GPU can be used for high-speed processing. The GPU can perform efficient calculations by processing a larger amount of data in parallel, so it is effective to use the GPU for processing when performing multiple learning rounds using a learning model such as deep learning. The additional learning process may be performed by a GPU working together with a CPU, etc.

[Modification 8]
In the various embodiments and modified examples described above, the instructions from the examiner may be instructions by voice or the like in addition to manual instructions (e.g., instructions using a user interface or the like). At this time, for example, a machine learning model including a voice recognition model (voice recognition engine, trained model for voice recognition) obtained by machine learning may be used. The manual instructions may also be instructions by character input or the like using a keyboard, touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (character recognition engine, trained model for character recognition) obtained by machine learning may be used. The instructions from the examiner may also be instructions by gestures or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, trained model for gesture recognition) obtained by machine learning may be used.

In addition, the instructions from the examiner may be the result of gaze detection of the examiner on the display screen of the output unit 103. The gaze detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing the periphery of the display screen of the output unit 103. In this case, the pupil detection from the moving image may use an object recognition engine as described above.In addition, the instructions from the examiner may be instructions based on brain waves, weak electrical signals flowing through the body, etc.

In such a case, for example, the learning data may be learning data in which the input data is character data or voice data (waveform data) indicating an instruction to display the results of the processing of various learned models as described above, and the correct answer data is an execution command for actually displaying the results of the processing of various learned models on the output unit 103. In addition, the learning data may be learning data in which the correct answer data is, for example, an execution command for whether or not to automatically set the shooting parameters and an execution command for changing the button for the command to an active state. In addition, the learning data may be learning data in which the correct answer data is, for example, an execution command for whether or not to perform high-quality image processing to obtain a high-quality image and an execution command for changing the button for the command to an active state. Note that the learning data may be anything as long as the instruction content indicated by the character data or voice data and the execution command content correspond to each other. In addition, the voice data may be converted into character data using an acoustic model, a language model, etc. In addition, a process of reducing noise data superimposed on the voice data may be performed using waveform data obtained by multiple microphones. The system may also be configured to allow a selection between instructions by text, voice, etc. and instructions by a mouse or touch panel, etc., in response to an instruction from the examiner. The system may also be configured to allow a selection between turning on and off instructions by text, voice, etc., in response to an instruction from the examiner.

Here, machine learning includes deep learning as described above, and an RNN can be used for at least a part of a multi-layered neural network, for example. Here, as an example of a machine learning model according to this modification, an RNN, which is a neural network that handles time-series information, will be described with reference to Figs. 15(a) and 15(b). In addition, a long short-term memory (hereinafter, LSTM), which is a type of RNN, will be described with reference to Figs. 16(a) and 16(b).

FIG. 15(a) shows the structure of an RNN, which is a machine learning model. The RNN 1520 has a loop structure in the network, and data x ^t 1510 is input at time t, and data h ^t 1530 is output. Since the RNN 1520 has a loop function in the network, it is possible to take over the current state to the next state, and therefore it is possible to handle time-series information. FIG. 15(b) shows an example of input and output of a parameter vector at time t. The data x ^t 1510 includes N pieces of data (Params1 to ParamsN). In addition, the data h ^t 1530 output from the RNN 1520 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information during backpropagation of errors, LSTM is sometimes used. LSTM can learn long-term information by equipping it with a forget gate, an input gate, and an output gate. Here, the structure of LSTM is shown in FIG. 16(a). In LSTM 1640, the information that the network takes over at the next time t is the internal state c ^t-1 of the network called a cell and output data h ^t-1 . Note that the lowercase letters (c, h, x) in the figure represent vectors.

Next, FIG. 16(b) shows the details of LSTM1640. In FIG. 16(b), the forget gate network FG, the input gate network IG, and the output gate network OG are shown, each of which is a sigmoid layer. As such, they output vectors in which each element has a value between 0 and 1. The forget gate network FG decides how much past information to retain, and the input gate network IG decides which values to update. Also shown in FIG. 16(b) is the cell update candidate network CU, which is an activation function tanh layer. It creates a vector of new candidate values to be added to the cell. The output gate network OG selects the cell candidate elements and selects how much information to convey at the next time.

Note that the above-mentioned LSTM model is a basic form, and is not limited to the network shown here. The connections between networks may be changed. Instead of LSTM, QRNN (Quasi Recurrent Neural Network) may be used. Furthermore, the machine learning model is not limited to a neural network, and boosting, support vector machines, etc. may be used. In addition, when the instructions from the examiner are input by characters or voice, etc., a technology related to natural language processing (e.g., sequence to sequence) may be applied. In this case, as the technology related to natural language processing, for example, a model that is output for each input sentence may be applied. In addition, the above-mentioned various trained models may be applied to the output to the examiner, not limited to the instructions from the examiner. In addition, a dialogue engine (dialogue model, trained model for dialogue) that responds to the examiner by output by characters or voice, etc. may be applied.

In addition, as a technology related to natural language processing, a trained model obtained by pre-training document data by unsupervised learning may be used. In addition, as a technology related to natural language processing, a trained model obtained by further transfer learning (or fine tuning) of a trained model obtained by pre-training according to the purpose may be used. In addition, as a technology related to natural language processing, for example, BERT (Bidirectional Encoder Representations from Transformers) may be applied. In addition, as a technology related to natural language processing, a model capable of extracting (representing) a context (feature amount) by predicting a specific word in a sentence from both the left and right contexts may be applied. In addition, as a technology related to natural language processing, a model capable of determining the relationship (continuity) between two sequences (sentences) in input time series data may be applied. In addition, as a technology related to natural language processing, a model in which a Transformer Encoder is used in the hidden layer and a vector sequence is input and output may be applied.

Here, the instructions from the examiner to which this modified example can be applied may be any instruction related to at least one of changing the display of various images and analysis results as described in the various embodiments and modified examples described above, selecting the depth range for generating an En-Face image, selecting whether to use as learning data for additional learning, selecting a trained model, outputting (displaying, transmitting, etc.) or saving the results obtained using various trained models, etc. Furthermore, the instructions from the examiner to which this modified example can be applied may be instructions not only after shooting, but also before shooting, and may be, for example, instructions regarding various adjustments, instructions regarding setting various shooting conditions, or instructions regarding the start of shooting. Furthermore, the instructions from the examiner to which this modified example can be applied may be instructions regarding changing the display screen (screen transition).

The machine learning model may be a machine learning model that combines a machine learning model for images such as CNN and a machine learning model for time series data such as RNN. In such a machine learning model, for example, the relationship between image features and time series data features can be learned. When the input layer side of the machine learning model is a CNN and the output layer side is an RNN, learning may be performed using learning data in which, for example, medical images are input data and text related to the medical images (e.g., the presence or absence of a lesion, the type of lesion, recommendations for the next examination, etc.) is output data. As a result, for example, medical information related to medical images is automatically explained in text, so that even examiners with little medical experience can easily understand the medical information related to the medical images. In addition, when the input layer side of the machine learning model is an RNN and the output layer side is a CNN, learning may be performed using learning data in which, for example, medical text such as lesions, findings, and diagnoses is input data and medical images corresponding to the medical text are output data. As a result, for example, medical images related to a case that the examiner wants to confirm can be easily searched for.

In addition, a machine translation engine (machine translation model, trained model for machine translation) that machine translates text such as characters and voice into an arbitrary language may be used for instructions from the examiner or output to the examiner. The arbitrary language may be configured to be selectable in response to instructions from the examiner. The arbitrary language may be configured to be automatically selected by using a trained model that automatically recognizes the type of language. The automatically selected language type may be configured to be correctable in response to instructions from the examiner. For example, the above-mentioned natural language processing technology (e.g., Sequence to Sequence) may be applied to the machine translation engine. For example, the machine translation engine may be configured to machine translate a text input to the machine translation engine, and then input the machine-translated text to a character recognition engine or the like. For example, the machine translation engine may be configured to input a text output from the various trained models described above to the machine translation engine, and output the text output from the machine translation engine.

In addition, the above-mentioned various trained models may be used in combination. For example, the configuration may be such that characters corresponding to instructions from the examiner are input to a character recognition engine, and voice obtained from the input characters is input to another type of machine learning engine (e.g., a machine translation engine, etc.). In addition, for example, the configuration may be such that characters output from another type of machine learning engine are input to a character recognition engine, and voice obtained from the input characters is output. In addition, for example, the configuration may be such that voice corresponding to instructions from the examiner is input to a voice recognition engine, and characters obtained from the input voice are input to another type of machine learning engine (e.g., a machine translation engine, etc.). In addition, for example, the configuration may be such that voice output from another type of machine learning engine is input to a voice recognition engine, and characters obtained from the input voice are displayed on the output unit 103. At this time, it may be configured so that output by characters or output by voice as output to the examiner can be selected according to instructions from the examiner. In addition, it may be configured so that input by characters or input by voice as instructions from the examiner can be selected according to instructions from the examiner. In addition, the above-mentioned various configurations may be adopted by selection according to instructions from the examiner.

[Modification 9]
Label images and high-quality images related to the images acquired by the main shooting may be stored in the storage unit 101-3 in response to instructions from the operator. At this time, for example, after an instruction from the operator to save a high-quality image, when registering a file name, a file name including information (e.g., characters) indicating that the image was generated by processing (image quality improvement processing) using a trained model for image quality improvement may be displayed in an editable state in response to instructions from the operator as a recommended file name in any part of the file name (e.g., the first part or the last part). Similarly, for label images and the like, a file name including information that the image was generated by processing using a trained model may be displayed.

In addition, when a high-quality image is displayed on the output unit 103 on various display screens such as a report screen, a display indicating that the displayed image is a high-quality image generated by processing using a high-quality model may be displayed together with the high-quality image. In this case, the operator can easily identify that the displayed high-quality image is not the image itself acquired by shooting, and therefore it is possible to reduce misdiagnosis and improve diagnostic efficiency. The display indicating that the high-quality image is generated by processing using a high-quality model may be in any form as long as it is possible to distinguish the input image from the high-quality image generated by the processing. In addition, for processing using not only the high-quality model but also various learned models as described above, a display indicating that the result is generated by processing using the learned model of that type may be displayed together with the result. For example, when displaying the analysis result of the segmentation result using the learned model for image segmentation processing, a display indicating that the analysis result is based on the result using the learned model for image segmentation may be displayed together with the analysis result.

At this time, the display screen such as the report screen may be stored in the storage unit 101-3 as image data in response to an instruction from the operator. For example, the report screen may be stored in the storage unit 101-3 as a single image in which high-quality images and an indication that these images are images generated by processing using a trained model are arranged side by side.

In addition, regarding the display indicating that the high-quality image was generated by processing using a high-quality model, a display indicating what kind of learning data the high-quality model was trained on may be displayed on the output unit 103. The display may include an explanation of the type of input data and correct answer data of the training data, or any display regarding correct answer data such as the imaging site included in the input data and correct answer data. Note that, for processing using the various trained models described above, such as image segmentation processing, for example, a display indicating what kind of learning data the trained model of that type was trained on may be displayed on the output unit 103.

In addition, information (e.g., text) indicating that the image was generated by processing using a trained model may be displayed or saved in a state where it is superimposed on the image, etc. In this case, the location to be superimposed on the image may be anywhere (e.g., the edge of the image) that does not overlap with the area of interest to be photographed, etc. displayed. Also, the non-overlapping area may be determined, and the information may be superimposed on the determined area. Note that the same processing may be performed not only on images obtained by processing using the high image quality model, but also on images obtained by processing using the various trained models described above, such as image segmentation processing, etc.

In addition, when the default setting is such that the high image quality processing button or the like is in an active state (high image quality processing is on) as the initial display screen of the report screen, a report image corresponding to the report screen including high image quality images or the like may be configured to be sent to the server in response to an instruction from the examiner. In addition, when the default setting is such that the button is in an active state, a report image corresponding to the report screen including high image quality images or the like may be (automatically) sent to the server at the end of the examination (for example, when the shooting confirmation screen or the preview screen is changed to the report screen in response to an instruction from the examiner). At this time, a report image generated based on various settings in the default setting (for example, settings related to at least one of the depth range for generating an En-Face image on the initial display screen of the report screen, whether or not an analysis map is superimposed, whether or not a high image quality image is displayed, whether or not a display screen for follow-up observation is displayed, etc.) may be configured to be sent to the server. Note that the same processing may be performed when the button represents a switch in the image segmentation processing.

[Modification 10]
In the above-described embodiment and modified example, among the various trained models as described above, an image obtained by a first type of trained model (e.g., a high-quality image, an image showing an analysis result such as an analysis map, an image showing a region recognition result, or an image showing a segmentation result) may be input to a second type of trained model different from the first type. At this time, a configuration may be made such that a result (e.g., an analysis result, a diagnosis result, a region recognition result, or a segmentation result) is generated by processing the second type of trained model.

In addition, among the various trained models described above, the results of processing a first type of trained model (e.g., analysis results, diagnosis results, area recognition results, segmentation results) may be used to generate an image to be input to a second type of trained model different from the first type from an image input to the first type of trained model. In this case, the generated image is likely to be an image suitable for processing using the second type of trained model. Therefore, the accuracy of images (e.g., high-quality images, images showing analysis results such as analysis maps, images showing area recognition results, images showing segmentation results) obtained by inputting the generated image into the second type of trained model can be improved.

The system may be configured so that a common image is input to the first type of trained model and the second type of trained model, thereby generating (or displaying) each processing result using these trained models. At this time, the system may be configured so that, for example, in response to an instruction from an examiner, the system may be configured to generate (or display) each processing result using these trained models collectively (in conjunction with each other).

In addition, the type of image to be input (e.g., high-quality image, region recognition result, object recognition result, segmentation result, similar case image), the type of processing result to be generated (or displayed) (e.g., high-quality image, region recognition result, diagnosis result, analysis result, object recognition result, segmentation result, similar case image), the type of input, the type of output (e.g., text, voice, language), etc. may be configured to be selectable in response to an instruction from the examiner. Furthermore, the type of input may be configured to be automatically selectable by using a trained model that automatically recognizes the type of input. Furthermore, the type of output may be configured to be automatically selectable so as to correspond to the type of input (e.g., to be the same type). Furthermore, the automatically selected type may be configured to be correctable in response to an instruction from the examiner. At this time, at least one trained model may be selected in response to the selected type. At this time, if multiple trained models are selected, the method of combining the multiple trained models (e.g., the order in which data is input, etc.) may be determined in response to the selected type. For example, the type of image to be input and the type of processing result to be generated (or displayed) may be configured to be selectable to be different, or if they are the same, information may be output to the examiner to prompt the examiner to select a different type.

In addition, each trained model may be executed anywhere. For example, some of the trained models may be used in a cloud server, and the others may be used in another server such as a fog server or an edge server. When a network is configured to be wirelessly communicated within a facility, a site including a facility, or a region including multiple facilities, the reliability of the network may be improved by configuring the network to use radio waves in a dedicated wavelength band that is limited to the facility, site, region, etc. In addition, a network may be configured using wireless communication that is high-speed, large-capacity, low-latency, and capable of multiple simultaneous connections. With these, for example, surgery on the vitreous body, cataracts, glaucoma, corneal refractive correction, external eye, etc., and treatments such as laser photocoagulation can be supported in real time even if they are remote. At this time, for example, a fog server or edge server that wirelessly receives at least one of the various medical images obtained by the device related to these surgery or treatment may be configured to wirelessly transmit information obtained using at least one of the various trained models to the device related to surgery or treatment. Also, for example, the information wirelessly received by the device related to surgery or treatment may be the amount of movement (vector) of the optical system or optical member as described above, in which case the device related to surgery or treatment may be configured to be automatically controlled. Also, for example, it may be configured as automatic control (semi-automatic control) with the permission of the examiner in order to support the operation by the examiner.

In addition, a similar case image search may be performed using an external database stored in a server or the like, using the analysis results or diagnosis results obtained by the processing of the trained model as described above as a search key. In addition, a similar case image search may be performed using an external database stored in a server or the like, using the object recognition results or segmentation results obtained by the processing of various trained models as described above as a search key. In addition, in cases where multiple medical images stored in a database are managed in a state in which the features of each of the multiple medical images are already attached as additional information by machine learning or the like, a similar case image search engine (similar case image search model, trained model for similar case image search) using the medical image itself as a search key may be used. For example, the image processing device 101 can search for similar case images related to the medical image from various medical images using a trained model for similar case image search (different from the trained model for image quality improvement).

For example, the display control unit 101-5 can display on the output unit 103 similar case images obtained from various medical images using a trained model for similar case image search. At this time, the similar case image is, for example, an image with a feature similar to the feature of the medical image input to the trained model. For example, when a partial region such as an abnormal part is included in the medical image input to the trained model, the similar case image is an image with a feature similar to the feature of the partial region such as an abnormal part. For this reason, not only can learning for accurately searching similar case images be efficiently performed, but when an abnormal part is included in the medical image, the examiner can efficiently diagnose the abnormal part. In addition, multiple similar case images may be searched, and the multiple similar case images may be displayed so that the order of similarity of the feature amounts can be identified. In addition, the trained model for similar case image search may be configured to additionally train using training data including an image selected from the multiple similar case images in response to an instruction from the examiner and the feature amounts of the image.

In addition, the learning data for the various trained models is not limited to data obtained using the ophthalmic device itself that actually takes the photographs, but may be data obtained using the same type of ophthalmic device or data obtained using the same kind of ophthalmic device, depending on the desired configuration.

Note that the various trained models according to the above-described embodiments and modified examples can be provided in the image processing device 101. The trained models may be configured, for example, by software modules executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit that performs a specific function such as an ASIC. These trained models may also be provided in another server device connected to the image processing device 101. In this case, the image processing device 101 can use the trained models by connecting to a server or the like that has the trained models via any network such as the Internet. Here, the server that has the trained models may be, for example, a cloud server, a fog server, or an edge server. Note that, when configuring a network within a facility, a site that includes a facility, or an area that includes multiple facilities to be capable of wireless communication, the reliability of the network may be improved by configuring it to use radio waves in a dedicated wavelength band that is limited to the facility, site, area, or the like. The network may also be configured by wireless communication that is capable of high speed, large capacity, low latency, and multiple simultaneous connections.

[Modification 11]
The medical images processed by the image processing device 101 according to the various embodiments and modified examples described above include images acquired using any modality (imaging device, imaging method). The medical images to be processed may include medical images acquired by any imaging device or the like, or images created by a medical image processing device or a medical image processing method.

Furthermore, the medical image to be processed is an image of a specific part of the subject (test subject), and the image of the specific part includes at least a part of the specific part of the subject. The medical image may also include other parts of the subject. The medical image may be a still image or a moving image, and may be a black and white image or a color image. The medical image may also be an image that represents the structure (shape) of the specific part, or an image that represents its function. Images that represent function include images that represent hemodynamics (blood flow rate, blood flow velocity, etc.), such as OCTA images, Doppler OCT images, fMRI images, and ultrasonic Doppler images. The specific part of the subject may be determined according to the subject to be photographed, and may include any part, such as the human eye (test subject eye), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, the head, chest, legs, and arms.

The medical image may be a tomographic image of the subject, or a frontal image. The frontal image includes, for example, an SLO image of the fundus or anterior part of the eye, a fundus image photographed by fluorescence, and an En-Face image generated using data acquired by OCT (three-dimensional OCT data) of at least a partial range in the depth direction of the subject. The En-Face image may be an OCTA En-Face image (motion contrast frontal image) generated using three-dimensional OCTA data (three-dimensional motion contrast data) of at least a partial range in the depth direction of the subject. Three-dimensional OCT data and three-dimensional motion contrast data are also examples of three-dimensional medical image data.

Here, the motion contrast data is data showing the change between multiple volume data obtained by controlling the measurement light to scan multiple times in the same region (same position) of the test eye. At this time, the volume data is composed of multiple tomographic images obtained at different positions. Then, by obtaining data showing the change between multiple tomographic images obtained at approximately the same position at each different position, the motion contrast data can be obtained as volume data. The motion contrast front image is also called an OCTA front image (OCTA En-Face image) related to OCT angiography (OCTA) that measures the movement of blood flow, and the motion contrast data is also called OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value) between two tomographic images or the corresponding interference signals, and may be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to scan multiple times in the same region (same position) of the test eye. In addition, when controlling the scanning means so that the measurement light is scanned a plurality of times at approximately the same position, the time interval between one scan (one B-scan) and the next scan (next B-scan) may be changed (determined). As a result, even if the blood flow velocity differs depending on the state of the blood vessel, the blood vessel region can be visualized with high accuracy. At this time, for example, the time interval may be changed according to an instruction from the examiner. Also, for example, according to an instruction from the examiner, any one of a plurality of motion contrast images corresponding to a plurality of time intervals set in advance may be selected. Also, for example, the time interval when the motion contrast data is acquired may be associated with the motion contrast data and stored in the storage unit 101-3. Also, for example, the display control unit 101-5 may display the time interval when the motion contrast data is acquired and the motion contrast image corresponding to the motion contrast data on the output unit 103. Also, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. In this case, for example, a machine learning model may be used to determine (output) the time interval from the motion contrast image. Such a machine learning model can be obtained by learning training data in which, for example, a plurality of motion contrast images corresponding to a plurality of time intervals are input data, and the difference between the plurality of time intervals and the time interval when the desired motion contrast image was obtained is used as correct answer data.

The En-Face image is a front image generated by projecting data in the range between two layer boundaries in the XY direction. At this time, the front image is generated by projecting or integrating data corresponding to at least a part of the depth range of volume data (three-dimensional tomographic image) obtained using optical interference and determined based on two reference planes onto a two-dimensional plane. The En-Face image is a front image generated by projecting data corresponding to a depth range determined based on a detected retinal layer from the volume data onto a two-dimensional plane. Note that, as a method for projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a method of setting a representative value of the data within the depth range as a pixel value on the two-dimensional plane can be used. Here, the representative value can include values such as the average value, median value, or maximum value of pixel values within a range in the depth direction of the area surrounded by the two reference planes. The depth range related to the En-Face image may be, for example, a range including a predetermined number of pixels in a deeper or shallower direction based on one of the two layer boundaries related to the detected retinal layer. Furthermore, the depth range of the En-Face image may be, for example, a range that is changed (offset) from the range between two layer boundaries related to the detected retinal layers in response to an instruction from the operator.

The imaging device is a device for capturing images used in diagnosis. The imaging device includes, for example, a device for obtaining an image of a specific part of a subject by irradiating the specific part with light, radiation such as X-rays, electromagnetic waves, or ultrasound, or a device for obtaining an image of a specific part by detecting radiation emitted from a subject. More specifically, the imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscope. The configurations according to the above-mentioned embodiments and modifications can be applied to these imaging devices. In this case, the movement of the subject corresponding to the movement of the subject's eye to be predicted may be, for example, the movement of the face or body, the movement of the heart (heartbeat), etc.

The OCT device may include a time domain OCT (TD-OCT) device or a Fourier domain OCT (FD-OCT) device. The Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a swept wavelength OCT (SS-OCT) device. The OCT device may include a line-OCT device (or an SS-line-OCT device) using line light. The OCT device may include a full field-OCT device (or an SS-full field-OCT device) using area light. The OCT device may include a Doppler-OCT device. The SLO device and the OCT device may include a wavefront compensation SLO (AO-SLO) device or a wavefront compensation OCT (AO-OCT) device using wavefront compensation optics. The SLO device and the OCT device may include a polarized SLO (PS-SLO) device or a polarized OCT (PS-OCT) device for visualizing information related to a polarization phase difference or depolarization. The SLO device and the OCT device may include a pathological microscope SLO device or a pathological microscope OCT device. The SLO device and the OCT device may include a handheld SLO device or a handheld OCT device. The SLO device and the OCT device may include a catheter SLO device or a catheter OCT device. The SLO device and the OCT device may include a head-mounted SLO device or a head-mounted OCT device. The SLO device and the OCT device may include a binocular SLO device or a binocular OCT device. The SLO device and the OCT device may be capable of changing the imaging angle of view by an optically variable magnification configuration. The SLO device may also be capable of acquiring color images or fluorescent images using RGB light sources, with a configuration in which one light receiving element receives light in a time-division manner, or a configuration in which multiple light receiving elements receive light simultaneously.

In addition, in the above-described embodiment and modified example, the image processing device 101 is configured as part of the OCT device, but the image processing device 101 may be configured separately from the OCT device. In this case, the image processing device 101 may be connected to the imaging device 100 of the OCT device via the Internet or the like. In addition, the configuration of the OCT device is not limited to the above-described configuration, and some of the components included in the OCT device, such as an SLO imaging unit, may be configured separately from the OCT device.

In addition, in trained models for voice recognition, character recognition, gesture recognition, etc., it is considered that the slope between successive input data values in a time series is extracted as part of the feature and used in the estimation process. It is expected that such trained models can perform accurate estimation by using the effect of temporal changes in specific numerical values in the estimation process. Also, in trained models for image quality improvement, area recognition, segmentation processing, image analysis, and diagnostic result generation according to the above-mentioned embodiments and modified examples, it is considered that the magnitude of the luminance values of the tomographic image, the order, slope, position, distribution, continuity, etc. of bright and dark areas are extracted as part of the feature and used in the estimation process.

Other Embodiments
The present invention can also be realized by supplying software (programs) that realize one or more functions of the various embodiments and modifications described above to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the programs. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

In this case, the processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). The processor or circuitry may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

The present invention has been described above with reference to embodiments and modifications, but the present invention is not limited to the above embodiments and modifications. The present invention also includes inventions that have been modified within the scope of the present invention, and inventions equivalent to the present invention. In addition, the above-mentioned embodiments and modifications can be combined as appropriate within the scope of the present invention.

Claims (29)

An image processing device comprising: an image processing unit having a trained model obtained by training using training data including input data that is a medical image having a first image size, and an image processing unit that outputs as output data a second image generated by inputting as input data a first image that is a medical image having a second image size larger than the first image size into the trained model. The image processing device according to claim 1, wherein the image processing unit inputs a plurality of medical images having a third image size larger than the second image size obtained by dividing a medical image having the second image size into a plurality of images as input data to the trained model, and outputs a plurality of images corresponding to the plurality of medical images as output data. The image processing device according to claim 2, wherein the image processing unit generates an image of the third image size by synthesizing the output images. The image processing device according to any one of claims 1 to 3, wherein the trained model includes an Encoder-Decoder type Transformer. The image processing device according to any one of claims 1 to 4, wherein the image processing unit performs image processing on a target region in the first image or the second image so that the difference between pixel values of the target region and pixel values of regions other than the target region increases and the pixel values of the target region become lower. The image processing device according to claim 5, wherein the image processing includes a process of blending the first image and the second image so that the difference between the pixel values of the target region and the pixel values of the region other than the target region increases and the pixel values of the target region become lower. The image processing device according to claim 5 or 6, wherein the image processing unit detects a target area in a medical image using the first image or the second image, and performs the image processing on a target area in the second image that corresponds to the detected target area in the first image. The image processing device according to any one of claims 1 to 7, wherein the image processing unit detects a target region using the trained model. The medical image is a frontal image,
The image processing device according to claim 1 , wherein the image processing unit detects a two-dimensional target region using the trained model obtained by learning using a plurality of front images of a subject corresponding to a plurality of depth ranges as training data. The medical image is a frontal image,
The image processing device according to any one of claims 1 to 7, wherein the image processing unit selects a trained model corresponding to a depth range selected in accordance with an instruction from an examiner from among a plurality of trained models obtained by learning using a plurality of front images of a subject corresponding to a plurality of depth ranges as respective training data, and detects a two-dimensional target region using the selected trained model. The medical image is a three-dimensional medical image.
The image processing device according to claim 1 , wherein the image processing unit detects a three-dimensional target region by using the trained model obtained by training using a three-dimensional medical image of a subject as training data. The image processing device according to any one of claims 1 to 7, wherein the image processing unit detects a target region using the first image by rule-based processing based on the structure of the subject. The image processing device according to claim 5, wherein the image processing includes a process of correcting at least one of brightness and contrast so as to increase the difference between pixel values of the target region and pixel values of regions other than the target region. the medical image of the subject is a motion contrast image of the subject's eye;
The image processing device according to claim 5 , wherein the target region includes at least one of an aperfusion region, a foveal vascular region, and an optic disc region. The image processing device according to any one of claims 1 to 14, wherein the image processing unit performs image quality improvement processing on the first image using a first trained model obtained by training using a medical image of the subject as training data, and a second trained model obtained by training using a medical image of the subject that has been subjected to blending processing or at least one of brightness and contrast correction processing in a target region of the medical image of the subject, and outputs a second image of the medical image of the subject. The image processing device according to any one of claims 1 to 15, further comprising a display control unit that causes a live video image of the subject to be displayed on a display unit. The image processing device according to claim 16, wherein the display control unit causes the display unit to display a high-quality image generated using a trained model obtained by training training data including images of a subject, the high-quality image being obtained by inputting a medical image of the subject. The image processing device according to claim 17, wherein the display control unit causes the display unit to display the anterior eye image generated as the high-quality image as the live video image, causes the display unit to display an SLO image generated as the high-quality image, in which a line indicating the position of the tomographic image generated as the high-quality image is superimposed, as the live video image, and causes the display unit to display the tomographic image corresponding to the position of the line on the SLO image as the live video image. The image processing device according to claim 18, wherein the display control unit displays information indicating a vascular region in the cross-sectional image generated as the high-quality image and corresponding to the position of the line, superimposed on the cross-sectional image corresponding to the position of the line. The image processing device according to any one of claims 16 to 19, wherein the display control unit causes the display unit to display analysis results obtained by inputting an image related to the subject, the analysis results being generated using a trained model for generating analysis results obtained by training training data including an image related to the subject. The image processing device according to any one of claims 16 to 20, wherein the display control unit causes the display unit to display a diagnostic result obtained by inputting an image related to the subject, the diagnostic result being generated using a trained model for generating a diagnostic result obtained by training training data including an image related to the subject. The image processing device according to any one of claims 16 to 21, wherein the display control unit causes the display unit to display, as information about an abnormal area, information about a difference between an image obtained by inputting an image about a subject, which is generated using a generative adversarial network or an autoencoder, and the image about the subject input to the generative adversarial network or the autoencoder. The image processing device according to any one of claims 16 to 22, wherein the display control unit causes the display unit to display similar case images obtained by inputting an image related to the subject, the similar case images being searched for using a trained model for similar case image search obtained by training training data including an image related to the subject. The image processing device according to any one of claims 16 to 23, wherein the display control unit causes the display unit to display an object recognition result or a segmentation result generated using a trained model for object recognition or a trained model for segmentation obtained by training training data including an image of a subject, the object recognition result or the segmentation result obtained by inputting an image of the subject. The image processing device according to any one of claims 1 to 24, wherein the operator's instruction to acquire the second image is information obtained using at least one of a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition. An image processing device comprising : an image processing unit having a trained model obtained by training using training data including input data that is a medical image having a first image size during training; and an image processing unit that outputs as output data a second image generated by inputting as input data a first image that is a medical image having a second image size larger than the first image size into the trained model during inference. An image processing method using a trained model obtained by training with training data including input data which is a medical image having a first image size, the image processing method including a step of inputting a first image which is a medical image having a second image size larger than the first image size as input data into the trained model, and outputting a second image which is generated by the input data. An image processing method using a trained model obtained by training using training data including input data that is a medical image having a first image size during training, the image processing method including a step of outputting, as output data, a second image generated by inputting, as input data, a first image that is a medical image having a second image size larger than the first image size, into the trained model during inference. A program that, when executed by a computer, causes the computer to execute each step of the image processing method according to claim 27 or 28.

Images