JP7800364B2 - Fundus image processing device and fundus image processing program - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act is applicable. Announced at the 58th General Meeting of the Japanese Society of Ophthalmology held on September 4, 2022.

The present disclosure relates to a fundus image processing device and a fundus image processing program used to process fundus images of a subject's eye.

In recent years, technologies have been proposed that process fundus images of a subject's eye to obtain various medical information to assist users (e.g., doctors) in treating the subject's eye (diagnosis, examination, treatment, etc.). For example, the image processing device described in Patent Document 1 obtains a probability distribution for identifying tissue in an ophthalmic image by inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm. The image processing device obtains the degree of deviation of the obtained probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating the degree of abnormality in the tissue structure.

Japanese Patent Application Laid-Open No. 2020-18794

When an abnormality such as a disease occurs in the subject's eye, changes may appear in multiple layers in the fundus tissue and/or in the boundaries between adjacent layers (hereinafter sometimes referred to as "layers/boundaries"). Therefore, if users could properly understand the state of layers/boundaries using various medical information such as the degree of discrepancy, the efficiency of their medical treatment could be improved. However, the manner in which changes appear in layers/boundaries often differs depending on the condition of the subject's eye (for example, the nature of the abnormality, such as a disease, occurring in the subject's eye). Therefore, simply mechanically presenting users with a wide range of medical information obtained by processing ophthalmic images is not enough to fully assist them in their medical treatment.

A typical objective of the present disclosure is to provide a fundus image processing device and fundus image processing program that can appropriately present various medical information obtained by processing ophthalmologic images to a user depending on the situation.

A typical embodiment of the present disclosure provides a fundus image processing device that processes fundus images in which multiple layers in the fundus of a subject's eye and the boundaries between layers are included in the imaging range. The control unit of the fundus image processing device includes an image acquisition step of acquiring a three-dimensional image of the fundus captured by a fundus image capturing device, a discrepancy map generation step of inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm to acquire a probability distribution for identifying at least one of the layers and boundaries in the fundus tissue captured in the three-dimensional image, and generating a discrepancy map showing a two-dimensional distribution of the discrepancy of the acquired probability distribution with respect to the probability distribution when the layer or boundary to be identified is accurately identified, and a discrepancy map generation step of generating the three-dimensional image. The system can execute a thickness analysis map generation step of generating a thickness analysis map showing a two-dimensional distribution of analysis results for the thickness of at least one layer of the fundus tissue shown in the image, and a display control step of displaying medical information including at least one of the deviation map and the thickness analysis map on a display unit. In the display control step, if a macular disease mode in which macular disease is treated is selected as the treatment mode in which treatment is performed for the subject's eye, the deviation map is included in the medical information of the subject's eye that is initially displayed on the display unit, and if a glaucoma mode in which glaucoma is treated is selected as the treatment mode, the thickness analysis map is included in the medical information of the subject's eye that is initially displayed on the display unit.

A typical embodiment of the present disclosure provides a fundus image processing program that is executed by a fundus image processing device that processes fundus images in which multiple layers in the fundus of a subject's eye and the boundaries between layers are included in the imaging range. The fundus image processing program is executed by a control unit of the fundus image processing device, and includes an image acquisition step of acquiring a three-dimensional image of the fundus captured by the fundus image capturing device; and a discrepancy map that generates a two-dimensional distribution of the degree of discrepancy between the acquired probability distribution and the probability distribution when the layer or boundary to be identified is accurately identified by inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm. The fundus image processing device can be caused to execute a generation step, a thickness analysis map generation step for generating a thickness analysis map showing a two-dimensional distribution of analysis results for the thickness of at least one layer of the fundus tissue shown in the three-dimensional image, and a display control step for displaying medical information including at least one of the discrepancy map and the thickness analysis map on a display unit, wherein in the display control step, if a macular disease mode for treating macular disease is selected as the examination mode for treating the subject's eye, the discrepancy map is included in the medical information of the subject's eye that is initially displayed on the display unit, and if a glaucoma mode for treating glaucoma is selected as the examination mode, the thickness analysis map is included in the medical information of the subject's eye that is initially displayed on the display unit.

The fundus image processing device and fundus image processing program disclosed herein allow various medical information obtained by processing ophthalmologic images to be presented to the user appropriately according to the situation.

1 is a block diagram showing a schematic configuration of a mathematical model construction device 101, a fundus image processing device 1, and OCT devices 10A and 10B. FIG. 1 is an explanatory diagram for explaining an example of a method for capturing a three-dimensional tomographic image. FIG. 4 is a diagram showing an example of a two-dimensional tomographic image 42. FIG. 4 is a diagram showing an example of a three-dimensional tomographic image 43. FIG. 1 is a diagram showing a schematic structure of layers and boundaries in the fundus. 10 is a flowchart of a mathematical model construction process executed by the mathematical model construction device 101. 10 is a flowchart of fundus image processing executed by the fundus image processing device 1. 10 is a flowchart of a layer/boundary identification process executed during fundus image processing. 4 is a diagram schematically showing the relationship between a two-dimensional tomographic image 42 input to a mathematical model and one-dimensional regions A1 to AN in the two-dimensional tomographic image 42. FIG. 10 is a flowchart of a process for macular diseases executed during fundus image processing. FIG. 10 is a diagram showing an example of an initial display screen in a macular disease mode. FIG. 10 is a diagram showing an example of a display screen in which a thickness analysis map 70 is additionally displayed in a macular disease mode. FIG. 10 is a diagram showing an example of a method for displaying a plurality of deviation maps 60. 10 is a flowchart of a process for glaucoma executed during fundus image processing. FIG. 10 is a diagram showing an example of an initial display screen in a glaucoma mode.

<Overview>
The fundus image processing device exemplified in the present disclosure processes a fundus image in which multiple layers and boundaries between layers in the fundus of the subject's eye are included in the imaging range. The control unit of the fundus image processing device can execute an image acquisition step, a deviation map generation step, a thickness analysis map generation step, and a display control step. In the image acquisition step, the control unit acquires a three-dimensional image of the fundus captured by the fundus image capturing device. In the deviation map generation step, the three-dimensional image is input to a mathematical model trained by a machine learning algorithm to acquire a probability distribution for identifying at least one of the layers and boundaries in the fundus tissue captured in the three-dimensional image, and a deviation map is generated based on the acquired probability distribution. The deviation map shows a two-dimensional distribution of deviation of the acquired probability distribution from the probability distribution when the layer or boundary to be identified is accurately identified. In the thickness analysis map generation step, the control unit generates a thickness analysis map showing a two-dimensional distribution of analysis results for the thickness of at least one layer in the fundus tissue captured in the three-dimensional image. In the display control step, the control unit causes the display unit to display medical information including at least one of the deviation map and the thickness analysis map. Specifically, in the display control step, when a macular disease mode for treating macular diseases is selected as the examination mode for treating the subject's eye, the control unit causes the display unit to initially display the medical information of the subject's eye, including the deviation map. When a glaucoma mode for treating glaucoma is selected as the examination mode, the control unit causes the display unit to initially display the medical information of the subject's eye, including the thickness analysis map.

As a premise, when there are no abnormalities in the structure of layers and boundaries, the mathematical model is likely to accurately identify the layers and boundaries, and the probability distribution when identifying layers and boundaries using the mathematical model is likely to be biased. On the other hand, when there are abnormalities in the structure of layers and boundaries, the probability distribution is less likely to be biased. Therefore, the deviation map, which shows the two-dimensional distribution of the deviation between the probability distribution when layers and boundaries are accurately identified and the actually obtained probability distribution, is likely to show the degree of abnormality in the structure of layers and boundaries.

When a macular disease develops in a subject's eye, it is often accompanied by abnormalities in the structure of layers and boundaries. Therefore, in conventional macular disease treatments, the user must check two-dimensional tomographic images at various positions within the capture range of a three-dimensional tomographic image to determine whether or not a structural abnormality has occurred. In contrast, with the technology disclosed herein, when a macular disease mode is selected, a discrepancy map is initially displayed, which makes it easy to grasp the degree of abnormality in the structure of layers and boundaries in a two-dimensional area. Therefore, the discrepancy map initially displayed on the display unit makes it easy for the user to properly identify locations where structural abnormalities are likely to occur.

Furthermore, when glaucoma develops in the subject's eye, the thickness of at least some layers often becomes thinner (i.e., thinning occurs). According to the technology disclosed herein, when the glaucoma mode is selected, a thickness analysis map showing the two-dimensional distribution of the analysis results for layer thickness is initially displayed. This makes it easier for the user to appropriately treat glaucoma based on the subject's layer thickness using the thickness analysis map initially displayed on the display unit. Through the above processing, various medical information obtained by processing ophthalmic images is presented to the user appropriately according to the situation.

In this disclosure, the initial display screen refers to the screen on which medical information including a map (at least one of a deviation map and a thickness analysis map) is first displayed when the display of medical information about the examinee's eye is started in any of the treatment modes. Therefore, after the treatment mode for the examinee's eye is started, some screen that does not include a map (for example, a startup screen) may be displayed before the initial display screen including the map is displayed.

The specific form of the thickness analysis map initially displayed when the glaucoma mode is selected can be selected as appropriate. For example, in the thickness analysis map generation step, the control unit may generate a normal eye comparison map showing the results of comparing the two-dimensional distribution of thickness of at least one layer in the three-dimensional image to be analyzed with the two-dimensional distribution of thickness of the same layer in a normal eye (e.g., at least one of a percentile map showing the two-dimensional distribution of the percentile of the difference between the two, and a deviation map showing the two-dimensional distribution of the deviation between the two). Furthermore, in the thickness analysis map generation step, the control unit may generate a thickness analysis map showing the two-dimensional distribution of thickness of at least one layer in the three-dimensional image to be analyzed. The control unit may display both the normal eye comparison map and the thickness map on the display unit as the thickness analysis map.

Macular diseases are diseases that cause abnormalities in the central part of the retina at the fundus (the macula and the area surrounding the macula). For example, at least one of diabetic retinopathy, age-related macular degeneration, retinitis pigmentosa, retinal vein occlusion, macular edema, premacular membrane, macular hole, and submacular hematoma may be classified as macular diseases. As an example, in this embodiment, diabetic retinopathy, age-related macular degeneration, and retinitis pigmentosa are included in macular diseases.

The deviation may be output by a mathematical model. Alternatively, the control unit may calculate the deviation based on the probability distribution output by the mathematical model.

The deviation may include the entropy (average information content) of the acquired probability distribution. Entropy represents the degree of uncertainty, disorder, and chaos. In the present disclosure, the entropy of the probability distribution output when layers and boundaries are accurately identified is 0. Furthermore, the more difficult it is to identify layers and boundaries, the greater the entropy. Therefore, using the entropy of the probability distribution as the deviation more appropriately quantifies the degree of abnormality in the structure of layers and boundaries. However, values other than entropy may also be used as the deviation. For example, at least one of the standard deviation, coefficient of variation, and variance, which indicate the degree of dispersion of the acquired probability distribution, may be used as the deviation. The KL divergence, which is a measure of the difference between probability distributions, may also be used as the deviation. The maximum value of the acquired probability distribution may also be used as the deviation.

In the present disclosure, a three-dimensional image is constructed by arranging multiple two-dimensional images. The degree of deviation is obtained for each of the multiple two-dimensional images that make up the three-dimensional image, or for each pixel, column, or row that makes up the image. As a result, the degree of deviation is obtained for the entire fundus tissue depicted in the three-dimensional image when layers and boundaries are identified using a mathematical model. As an example, the deviation map of the present disclosure shows the two-dimensional distribution of the degree of deviation when the fundus tissue is viewed from the front (i.e., along the optical axis of the light used to capture the three-dimensional image). However, the direction in which the two-dimensional distribution of the degree of deviation is shown in the deviation map can be changed as appropriate. Furthermore, when displaying a deviation map of a specific layer, the control unit may display a two-dimensional distribution, such as the average value of the deviation of at least two boundaries included in the specific layer, as the deviation map of the specific layer.

When the glaucoma mode is selected as the treatment mode, the control unit may exclude the discrepancy map from the medical information of the subject's eye to be initially displayed on the display unit. As mentioned above, when glaucoma develops in the subject's eye, at least some layers often become thinned. However, layer thinning caused by glaucoma is often not accompanied by structural abnormalities in the layers or boundaries. If no structural abnormalities in the layers or boundaries occur, the discrepancy map is unlikely to show any notable features. Therefore, when the glaucoma mode is selected, discrepancy maps that are not frequently used in glaucoma treatment are excluded from the initial display, which helps to shorten the time required for glaucoma treatment.

When the glaucoma mode is selected as the treatment mode, the control unit may, in response to instructions input by the user, cause the display unit to display the discrepancy map together with the thickness analysis map, or alternately with the thickness analysis map. In this case, even when the glaucoma mode is selected, the user can appropriately grasp the degree of abnormality in the layer/boundary structure by displaying the discrepancy map as needed.

When the macular disease mode is selected as the treatment mode, the control unit may exclude the layer thickness analysis map from the medical information of the subject's eye that is initially displayed on the display unit. In treatment for macular disease, the layer thickness analysis map may be used to additionally confirm the presence or absence of significant edema, but may not be used in the treatment of macular disease itself. Therefore, when the macular disease mode is selected, excluding the layer thickness analysis map, which is not used very frequently, from the initial display can help shorten the time required for treatment for macular disease.

When the macular disease mode is selected as the treatment mode, the control unit may, in response to instructions input by the user, cause the display unit to display the thickness analysis map together with the deviation map, or alternately with the deviation map. In this case, even when the macular disease mode is selected, the user can appropriately grasp the two-dimensional distribution of layer and boundary thicknesses by displaying the thickness analysis map as needed.

The control unit may further execute an extraction position specification receiving step and a specified tomographic image display step. In the extraction position specification receiving step, the control unit receives instruction input from the user to specify an extraction position for a portion of a two-dimensional tomographic image extending in the depth direction of the fundus tissue from the image region of the three-dimensional image. In the specified tomographic image display step, when the instruction input for specifying the extraction position is received, the control unit extracts a two-dimensional tomographic image extending in the depth direction of the fundus tissue from the specified extraction position and displays it on the display unit. In the extraction position specification receiving step, if the macular disease mode is selected as the diagnosis mode, instruction input for the extraction position may be received on a discrepancy map displayed on the display unit. Also, if the glaucoma mode is selected as the diagnosis mode, instruction input for the extraction position may be received on a thickness analysis map displayed on the display unit.

As mentioned above, when macular disease develops in the subject's eye, it is often accompanied by abnormalities in the structure of layers and boundaries. The discrepancy map makes it easy for the user to grasp the degree of abnormality in the structure of layers and boundaries in a two-dimensional area. Therefore, by specifying the extraction position of the two-dimensional tomographic image on the discrepancy map while selecting the macular disease mode, the user can appropriately specify the extraction position of the two-dimensional tomographic image according to the degree of abnormality in structures that are highly correlated with macular disease.

Furthermore, when glaucoma develops in the subject's eye, the thickness of at least some layers often thins. The thickness analysis map makes it easy to grasp the two-dimensional distribution of layer thickness. Therefore, by specifying the extraction position of the two-dimensional tomographic image on the thickness analysis map while selecting the glaucoma mode, the user can appropriately specify the extraction position of the two-dimensional tomographic image according to the distribution of thickness of layers that are highly correlated with glaucoma.

The method of having the user specify the extraction position of the two-dimensional tomographic image on a map (discrepancy map or thickness analysis map) can also be selected as appropriate. For example, the control unit may have the user specify a line on the two-dimensional map that indicates the extraction position of the two-dimensional tomographic image (i.e., the position where the extracted two-dimensional tomographic image intersects with the map). In this case, the shape of the line may be straight, or may be a shape other than a straight line (e.g., curved or circular). The size and angle of the line may also be changed according to instructions input by the user.

A layer and boundary within the relevant fundus tissue may be associated with each of a plurality of macular diseases that may occur in the subject. In the deviation map generation step, the control unit may generate a deviation map for each of a plurality of regions by acquiring a plurality of probability distributions for identifying each of a plurality of regions among the plurality of layers and boundaries in the fundus tissue depicted in the three-dimensional image. In the display control step, when the macular disease mode is selected, the control unit may selectively display, on the display unit, a deviation map for a region associated with a macular disease specified by the user.

In this case, the user can efficiently use the deviation map depending on the type of macular disease they wish to treat. For example, the user can easily check the deviation map for the area associated with the subject's predicted macular disease simply by specifying the predicted macular disease. The user can also predict the subject's macular disease by switching the specified macular disease and checking the deviation map for the area associated with each macular disease.

The method for prompting the user to specify at least one of multiple macular diseases can also be selected as appropriate. For example, the control unit may prompt the user to specify at least one of multiple macular disease names prepared in advance, or may prompt the user to input the name of the macular disease. The type of macular disease may be specified before the display of medical information begins. In this case, the control unit may set the discrepancy map initially displayed on the display unit in macular disease mode to the discrepancy map for the region associated with the specified macular disease. The type of macular disease may also be specified after the initial display of medical information begins. In this case, the discrepancy map displayed on the display unit may be switched to the discrepancy map for the region associated with the specified macular disease. The control unit may prompt the user to specify a macular disease while the display unit is displaying a discrepancy map for at least one of multiple regions (or all discrepancy maps). The control unit may also prompt the user to specify a macular disease while the display unit is displaying a composite discrepancy map created by aligning and combining multiple discrepancy maps. In this case, areas with high discrepancies can be efficiently identified regardless of the type of macular disease.

The technology for selectively displaying a discrepancy map for a region associated with a macular disease specified by the user on a display unit can also be implemented without combining it with other technologies in this disclosure. In this case, the fundus image processing device can also be expressed as follows: A fundus image processing device processes fundus images in which multiple layers and boundaries between layers of the fundus of a subject's eye are included in the imaging range, and the control unit of the fundus image processing device is capable of executing the following steps: an image acquisition step of acquiring a three-dimensional image of the fundus captured by the fundus image capturing device; a discrepancy map generation step of inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm to acquire multiple probability distributions for identifying each of multiple regions among the multiple layers and boundaries of the fundus tissue captured in the three-dimensional image, and generating a discrepancy map for each of the multiple regions showing a two-dimensional distribution of the discrepancy of the acquired probability distribution from the probability distribution when the layer or boundary to be identified is accurately identified; and a display control step of displaying medical information including the discrepancy map on a display unit; and in the display control step, the control unit selectively displays the discrepancy map for a region associated with a macular disease specified by the user on the display unit.

However, the method for selecting the discrepancy map to be displayed on the display unit can be changed as appropriate. For example, the user may input an instruction to the fundus image processing device via an operation unit or the like to select at least one layer/boundary (or all layers/boundaries) for which the user wishes to check the discrepancy map from among the multiple layers/boundaries. The control unit may cause the display unit to display the discrepancy map for the layer/boundary selected by the user from among the multiple layers/boundaries. The instruction to select a layer/boundary may be received before the display of medical information begins. In this case, the control unit may set the discrepancy map to be initially displayed on the display unit in macular disease mode to the discrepancy map for the selected layer/boundary. Alternatively, the instruction to select a layer/boundary may be received after the initial display of medical information begins. In this case, the discrepancy map displayed on the display unit may be switched to the discrepancy map for the selected layer/boundary.

Furthermore, when the macular disease mode is selected as the treatment mode, the control unit may selectively initially display on the display unit a predetermined number of discrepancy maps with the highest discrepancy rankings from among the multiple discrepancy maps. In this case, the user can easily check the discrepancy maps of the layers/boundaries with the highest discrepancy in the classification by the mathematical model from among the multiple layers/boundaries.

The specific method for initially displaying a predetermined number of deviation maps with the highest deviation rankings can be selected as appropriate. For example, the control unit may initially display a predetermined number of deviation maps in descending order of average deviation values. The control unit may also initially display a predetermined number of deviation maps in descending order of maximum deviation values within the two-dimensional map. The control unit may also initially display a single deviation map with the highest deviation ranking, or may initially display multiple deviation maps in descending order of deviation rankings.

The specific method for displaying multiple deviation maps on the display unit can also be selected as appropriate. For example, the control unit may cause the display unit to display multiple deviation maps separately. Alternatively, the control unit may cause the display unit to display a composite deviation map created by merging multiple deviation maps in an aligned state. The composite deviation map may be generated by superimposing multiple deviation maps, with each deviation map displayed in a different color.

In the display control step, the control unit may cause the display unit to display a portion of a two-dimensional tomographic image of the fundus tissue extending in the depth direction, which is included in the image region of the three-dimensional image. The control unit may cause the color of the portion of the multiple layers and boundaries shown in the two-dimensional tomographic image displayed on the display unit, for which a discrepancy map is displayed, to match the display color of the discrepancy map on the display unit. In this case, the color makes it easy to determine which portion of the multiple layers and boundaries the displayed discrepancy map corresponds to. Therefore, the user can properly determine on the two-dimensional tomographic image the portion for which the discrepancy map is displayed, even when, for example, discrepancy maps for multiple portions are displayed simultaneously.

Note that the technology for matching the color of the area on the two-dimensional tomographic image on which the discrepancy map is displayed with the display color of the discrepancy map can also be implemented without combining it with other technologies in this disclosure. In this case, the fundus image processing device can also be expressed as follows: A fundus image processing device processes fundus images in which multiple layers and boundaries between layers of the fundus of a subject's eye are included in the imaging range, and the control unit of the fundus image processing device is capable of executing the following steps: an image acquisition step for acquiring a three-dimensional image of the fundus captured by a fundus image capturing device; a discrepancy map generation step for inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm to acquire a probability distribution for identifying at least one of the layers and boundaries of the fundus tissue captured in the three-dimensional image, and generating a discrepancy map showing a two-dimensional distribution of the discrepancy of the acquired probability distribution with respect to the probability distribution when the layer or boundary to be identified is accurately identified; and a display control step for displaying medical information including the discrepancy map on a display unit. In the display control step, the control unit displays on the display unit a two-dimensional tomographic image of a portion of the fundus tissue that is included in the image area of the three-dimensional image and extends in the depth direction, and matches the color of the portion of the multiple layers and boundaries captured in the two-dimensional tomographic image where the discrepancy map is displayed with the display color of the discrepancy map.

In the display control step, the control unit may cause the display unit to display a portion of a two-dimensional tomographic image of the fundus tissue extending in the depth direction, which is included in the image region of the three-dimensional image. The control unit may also execute an identification display that allows the user to identify the state of the degree of deviation in the region within the two-dimensional tomographic image. In this case, the user can easily check the state of the degree of deviation in the two-dimensional tomographic image. Therefore, the user can appropriately understand the state of the layers and boundaries shown in the two-dimensional tomographic image according to the degree of deviation.

The specific method for allowing the user to identify the state of the degree of deviation in an area within the two-dimensional tomographic image can be selected as appropriate. For example, the control unit may execute an identification display that indicates areas within the two-dimensional tomographic image where the degree of deviation is equal to or greater than a threshold. The identification display may be added either inside or outside the image area of the two-dimensional tomographic image. The control unit may also allow the user to identify the state of the degree of deviation by assigning a color of a depth corresponding to the magnitude of the degree of deviation to each area within the two-dimensional tomographic image. The control unit may also display an additional display indicating the magnitude of the degree of deviation (for example, a color scale bar showing a color of a depth corresponding to the magnitude of the degree of deviation) outside the image area of the two-dimensional tomographic image.

The technology that allows a user to identify the state of deviation in an area within a two-dimensional tomographic image can also be implemented without combining it with other technologies in this disclosure. In this case, the fundus image processing device can also be expressed as follows: A fundus image processing device processes fundus images in which multiple layers and boundaries between layers in the fundus of a subject's eye are included in the imaging range, and the control unit of the fundus image processing device performs the following steps: an image acquisition step of acquiring a three-dimensional image of the fundus captured by the fundus image capturing device; a deviation degree acquisition step of inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm to acquire a probability distribution for identifying at least one of the layers and boundaries in the fundus tissue captured in the three-dimensional image, and acquiring a distribution of deviation degrees of the acquired probability distribution from the probability distribution when the layer or boundary to be identified is accurately identified; a two-dimensional tomographic image display step of displaying on a display unit a two-dimensional tomographic image of a portion of the fundus tissue extending in the depth direction, which is included in the image region of the three-dimensional image; and a deviation degree identification display step of executing an identification display that allows a user to identify the state of the deviation degree in the region within the two-dimensional tomographic image.

<Embodiment>
(Device configuration)
A typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1 , this embodiment uses a mathematical model construction device 101, a fundus image processing device 1, and OCT devices (fundus image capturing devices) 10A and 10B. The mathematical model construction device 101 constructs a mathematical model by training the mathematical model using a machine learning algorithm. A program realizing the constructed mathematical model is installed in the fundus image processing device 1. The constructed mathematical model identifies (detects) at least one of the layers and boundaries (in this embodiment, a specific number of layers and boundaries) shown in the fundus image based on the input fundus image. The fundus image processing device 1 performs various processes using the results output by the mathematical model. The OCT devices 10A and 10B function as fundus image capturing devices that capture fundus images (in this embodiment, tomographic images of the fundus) of the subject's eye.

As an example, a personal computer (hereinafter referred to as a "PC") is used as the mathematical model construction device 101 of this embodiment. As will be described in detail below, the mathematical model construction device 101 trains a mathematical model using data on a fundus image of the test eye (hereinafter referred to as a "training fundus image") acquired from the OCT device 10A and data indicating multiple layers and boundaries of the test eye from which the training fundus image was captured. As a result, a mathematical model is constructed. However, devices that can function as the mathematical model construction device 101 are not limited to PCs. For example, the OCT device 10A may function as the mathematical model construction device 101. Furthermore, control units of multiple devices (e.g., the CPU of a PC and the CPU of the OCT device 10A) may work together to construct a mathematical model.

Furthermore, a PC is used as the fundus image processing device 1 in this embodiment. However, the device that can function as the fundus image processing device 1 is not limited to a PC. For example, the OCT device 10B or a server may function as the fundus image processing device 1. When the OCT device 10B functions as the fundus image processing device 1, the OCT device 10B can process the captured fundus images while capturing them. Furthermore, a mobile terminal such as a tablet terminal or smartphone may function as the fundus image processing device 1. Control units of multiple devices (for example, the CPU of a PC and the CPU of the OCT device 10B) may work together to perform various processes.

Furthermore, in this embodiment, a CPU is used as an example of a controller that performs various processes. However, it goes without saying that controllers other than a CPU may be used for at least some of the various devices. For example, processing speed may be increased by using a GPU as a controller.

The mathematical model construction device 101 will now be described. The mathematical model construction device 101 is installed, for example, at a manufacturer that provides the fundus image processing device 1 or the fundus image processing program to users. The mathematical model construction device 101 includes a control unit 102 that performs various control processes, and a communication I/F 105. The control unit 102 includes a CPU 103, which is a controller responsible for control, and a storage device 104 that can store programs, data, and the like. The storage device 104 stores a mathematical model construction program for executing the mathematical model construction process (see Figure 6), which will be described later. The communication I/F 105 also connects the mathematical model construction device 101 to other devices (for example, the OCT device 10A and the fundus image processing device 1, etc.).

The mathematical model construction device 101 is connected to an operation unit 107 and a display device 108. The operation unit 107 is operated by a user to input various instructions to the mathematical model construction device 101. The operation unit 107 may be, for example, at least one of a keyboard, a mouse, a touch panel, etc. Note that a microphone or the like for inputting various instructions may be used together with or instead of the operation unit 107. The display device 108 displays various images. The display device 108 may be any of a variety of devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.). Note that "image" in this disclosure includes both still images and moving images.

The mathematical model construction device 101 can acquire fundus image data (hereinafter sometimes simply referred to as "fundus image") from the OCT device 10A. The mathematical model construction device 101 may acquire fundus image data from the OCT device 10A, for example, via at least one of wired communication, wireless communication, a removable storage medium (e.g., a USB memory), etc.

The fundus image processing device 1 will now be described. The fundus image processing device 1 is installed, for example, in a facility where subjects are diagnosed or examined (e.g., a hospital or health checkup facility). The fundus image processing device 1 includes a control unit 2 that performs various control processes, and a communication I/F 5. The control unit 2 includes a CPU 3, which is a controller responsible for control, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a fundus image processing program for executing the fundus image processing (see Figure 7) described below. The fundus image processing program includes a program that realizes the mathematical model constructed by the mathematical model construction device 101. The communication I/F 5 connects the fundus image processing device 1 to other devices (e.g., the OCT device 10B and the mathematical model construction device 101, etc.).

The fundus image processing device 1 is connected to an operation unit 7 and a display device (display unit) 8. Similar to the operation unit 107 and display device 108 described above, various devices can be used for the operation unit 7 and display device 8.

The fundus image processing device 1 can acquire fundus images (in this embodiment, three-dimensional tomographic images of fundus tissue) from the OCT device 10B. The fundus image processing device 1 may acquire fundus images from the OCT device 10B, for example, via at least one of wired communication, wireless communication, a removable storage medium (e.g., USB memory), etc. The fundus image processing device 1 may also acquire programs, etc., that realize the mathematical model constructed by the mathematical model construction device 101 via communication, etc.

The OCT device 10 (10A, 10B) will now be described. The OCT device 10 is an example of a fundus image capturing device that captures tomographic images of the fundus. In this embodiment, a case will be described in which an OCT device 10A that provides fundus images to the mathematical model construction device 101 and an OCT device 10B that provides fundus images to the fundus image processing device 1 are used. However, the number of OCT devices used is not limited to two. For example, the mathematical model construction device 101 and the fundus image processing device 1 may acquire fundus images from multiple OCT devices. Alternatively, the mathematical model construction device 101 and the fundus image processing device 1 may acquire fundus images from a single shared OCT device.

The OCT device 10 can capture two-dimensional and three-dimensional tomographic images of the fundus of the subject's eye. An example of the capture method will be described below. As shown in FIG. 2, the OCT device 10 of this embodiment sets multiple, evenly spaced, linear scanning lines (scan lines) 41 along which spots are scanned within a two-dimensional measurement area 40 that extends in a direction intersecting the optical axis of the OCT measurement light. By scanning the measurement light spots along each scanning line 41, the OCT device 10 can capture two-dimensional tomographic images 42 (see FIG. 3) of cross sections passing through each scanning line 41. The two-dimensional tomographic image 42 may be an averaged image generated by performing an averaging process on multiple two-dimensional tomographic images of the same area. Furthermore, the OCT device 10 can acquire (capture) a three-dimensional image (three-dimensional tomographic image) 43 (see FIG. 4) by arranging multiple two-dimensional tomographic images 42 captured along multiple scanning lines 41 in a direction perpendicular to each two-dimensional image area.

Returning to the explanation of Figure 1, the OCT device 10A connected to the mathematical model construction device 101 can capture at least a two-dimensional tomographic image 42 (see Figure 3) of the fundus of the subject's eye. Furthermore, the OCT device 10B connected to the fundus image processing device 1 can capture a three-dimensional image 43 (see Figure 4) of the fundus of the subject's eye in addition to the two-dimensional tomographic image 42 described above.

(Structure of layers and boundaries of the fundus)
The structure of layers at the fundus of the subject's eye and the boundaries between adjacent layers will be described with reference to Fig. 5. Fig. 5 schematically shows the structure of layers and boundaries at the fundus. The upper side of Fig. 5 is the surface side (superficial layer side) of the retina at the fundus. In other words, the depth of the layers and boundaries increases toward the bottom of Fig. 5. Also, in Fig. 5, the names of boundaries between adjacent layers are enclosed in parentheses.

Explain the layers of the fundus. The fundus is made up of, in order from the surface side (upper side of Figure 5), the ILM (internal limiting membrane), NFL (nerve fiber layer), GCL (ganglion cell layer), IPL (inner plexiform layer), INL (inner nuclear layer), OPL (outer plexiform layer), ONL (outer nuclear layer), ELM (external limiting membrane), and IS/OS (photoreceptor inner/outer segment junction). There are the photoreceptor between inner and outer segments (sometimes called the EZ (Ellipsoid zone)), RPE (retinal pigment epithelium), BM (Bruch's membrane), and choroid.

Furthermore, examples of boundaries that are likely to appear in tomographic images include NFL/GCL (the boundary between NFL and GCL), IPL/INL (the boundary between IPL and INL), OPL/ONL (the boundary between OPL and ONL), RPE/BM (the boundary between RPE and BM), and BM/Choroid (the boundary between BM and Choroid).

In this embodiment, when focusing on all layers of the retina, the layers from the ILM to the RPE/BM are treated as all layers. Furthermore, when diagnosing the subject's eye, the GCC (Ganglion Cell Complex) may be of interest. The GCC includes the NFL, GCL, and IPL. As an example, in this embodiment, the layers from the ILM to the IPL/INL are treated as the GCC.

(Mathematical model construction process)
6, a description will be given of a mathematical model construction process executed by the mathematical model construction device 101. The mathematical model construction process is executed by the CPU 103 in accordance with a mathematical model construction program stored in the storage device 104.

The following describes, as an example, the construction of a mathematical model that analyzes an input two-dimensional tomographic image and outputs the identification results of multiple specific layers/boundaries from among the multiple layers/boundaries shown in the fundus image. In this embodiment, the mathematical model identifies each of the multiple specific layers/boundaries: ILM, NFL/GCL, IPL/INL, OPL/ONL, IS/OS, RPE/BM, and BM. Furthermore, the mathematical model illustrated in this embodiment outputs a probability distribution for identifying the specific layers/boundaries shown in the input fundus image.

In the mathematical model construction process, a mathematical model is constructed by training it using a training dataset. The training dataset includes input data (input training data) and output data (output training data).

As shown in FIG. 6 , the CPU 103 acquires data of a fundus image (a two-dimensional tomographic image in this embodiment) captured by the OCT device 10A as input training data (S1). Next, the CPU 103 acquires data indicating each of a specific number of layers and boundaries of the subject's eye from which the fundus image captured in S1 was captured as output training data (S2). In this embodiment, the output training data includes label data indicating the positions of a specific number of layers and boundaries depicted in the fundus image. The label data may be generated, for example, by the operator operating the operation unit 107 while viewing the layers and boundaries in the fundus image.

Next, the CPU 103 uses a machine learning algorithm to train a mathematical model using the training dataset (S3). Commonly known machine learning algorithms include neural networks, random forests, boosting, and support vector machines (SVMs).

Neural networks are a method of imitating the behavior of biological neuronal networks. Examples of neural networks include feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

Random forest is a method of generating a large number of decision trees by learning from randomly sampled training data. When using random forest, the branches of multiple decision trees that have been trained in advance as classifiers are traced, and the results obtained from each decision tree are averaged (or voted by majority vote).

Boosting is a technique for generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple, weak classifiers.

SVM is a technique for constructing a two-class pattern classifier using linear input elements. For example, SVM learns the parameters of the linear input elements using the criterion of finding a margin-maximizing hyperplane that maximizes the distance from each data point from the training data (hyperplane separation theorem).

A mathematical model refers to, for example, a data structure for predicting the relationship between input data (in this embodiment, data of two-dimensional tomographic images similar to the input training data) and output data (in this embodiment, data of the identification results for multiple specific body parts). A mathematical model is constructed by training using a training dataset. As described above, a training dataset is a set of input training data and output training data. For example, correlation data (e.g., weights) between each input and output is updated through training.

In this embodiment, a multi-layer neural network is used as the machine learning algorithm. The neural network includes an input layer for inputting data, an output layer for generating data for the analysis results to be predicted, and one or more hidden layers between the input and output layers. Multiple nodes (also called units) are arranged in each layer. In particular, this embodiment uses a convolutional neural network (CNN), which is a type of multi-layer neural network.

As an example, the mathematical model constructed in this embodiment outputs a probability distribution for identifying each region, where the random variables are the coordinates (one-dimensional, two-dimensional, three-dimensional, or four-dimensional coordinates) of each of multiple specific layers and boundaries within a region in the fundus image (one-dimensional, two-dimensional, three-dimensional, or four-dimensional region including a time axis). In this embodiment, a softmax function is applied to have the mathematical model output the probability distribution. In detail, the mathematical model constructed in S3 outputs a probability distribution where the random variables are the coordinates of a specific layer or boundary within a one-dimensional region extending in a direction intersecting the specific layer or boundary in the two-dimensional tomographic image (in this embodiment, the depth direction, which is the direction along the optical axis of the OCT measurement light, or the up-down direction in Figure 3). However, the specific method by which the mathematical model outputs the probability distribution for identifying a specific region can be modified as appropriate. For example, the mathematical model may output a probability distribution where the type of specific region in the test eye is the random variable for each region (e.g., pixel) of the input ophthalmic image. Additionally, the ophthalmological images input into the mathematical model may be moving images.

Note that other machine learning algorithms may also be used. For example, generative adversarial networks (GAN), which utilize two competing neural networks, may be employed as the machine learning algorithm.

The processes of S1 to S3 are repeated until construction of the mathematical model is complete (S4: NO). When construction of the mathematical model is complete (S4: YES), the mathematical model construction process ends. In this embodiment, the program and data that realize the constructed mathematical model are incorporated into the fundus image processing device 1.

(Fundus image processing)
7 to 15, fundus image processing performed by the fundus image processing device 1 will be described. The fundus image processing is performed by the CPU 3 of the fundus image processing device 1 in accordance with a fundus image processing program stored in the storage device 4. The CPU 3 may perform fundus image processing, for example, when an instruction to analyze an image file of a fundus image of the subject's eye is input.

As shown in FIG. 7, the CPU 3 acquires a three-dimensional image (three-dimensional tomographic image) of the fundus of the subject's eye (S11). The three-dimensional image is captured by the OCT device 10B and acquired by the fundus image processing device 1. As described above, the three-dimensional image is composed of a combination of multiple two-dimensional images (two-dimensional tomographic images) captured by scanning the measurement light on different scan lines. The CPU 3 may also acquire a signal (e.g., an OCT signal) from the OCT device 10B that serves as the basis for generating the three-dimensional image, and generate the three-dimensional image based on the acquired signal.

The CPU 3 executes layer/boundary identification processing (S12). In the layer/boundary identification processing, the identification results of at least one of the layers/boundaries of the fundus tissue depicted in the three-dimensional image acquired in S11 (in this embodiment, ILM, NFL/GCL, IPL/INL, OPL/ONL, IS/OS, RPE/BM, and BM) are obtained using a mathematical model trained by a machine learning algorithm. The layer/boundary identification processing also obtains the degree of discrepancy when at least one of the layers/boundaries is identified by the mathematical model. The degree of discrepancy is the degree of discrepancy between the probability distribution for the mathematical model to identify the layers/boundaries of the fundus tissue depicted in the three-dimensional image (in this embodiment, each of the multiple layers/boundaries) and the probability distribution when each layer/boundary is accurately identified by the mathematical model. The degree of discrepancy often increases or decreases depending on the degree of abnormality in the structure of the layer/boundary. Therefore, by allowing users to understand the degree of deviation, diagnostic efficiency can be improved.

As shown in FIG. 8, the CPU 3 extracts the Tth two-dimensional tomographic image (the initial value of T is "1") from among the multiple two-dimensional tomographic images constituting the three-dimensional image acquired in S11 (S21). FIG. 9 shows an example of the extracted two-dimensional tomographic image 42. The two-dimensional tomographic image 42 shows multiple layers and boundaries in the fundus of the subject's eye. Additionally, multiple one-dimensional regions A1-AN are set in the two-dimensional tomographic image 42. In this embodiment, the one-dimensional regions A1-AN set in the two-dimensional tomographic image 42 extend along axes that intersect with the layers and boundaries to be identified. In more detail, the one-dimensional regions A1-AN in this embodiment correspond to the respective regions of the multiple (N) A-scans constituting the two-dimensional tomographic image 42 captured by the OCT device 10.

By inputting the Tth two-dimensional tomographic image into the mathematical model, the CPU 3 acquires the probability distribution of coordinates where the Mth (the initial value of M is "1") portion (in this embodiment, the Mth layer/boundary) exists in each of the multiple one-dimensional regions A1-AN as a probability distribution for identifying the Mth layer/boundary (S22). The CPU 3 acquires the identification result of the Mth layer/boundary based on the probability distribution (S23). However, in S23, the identification result of the Mth layer/boundary output by the mathematical model based on the probability distribution may also be acquired. The CPU 3 also acquires the degree of deviation of the probability distribution for the Mth layer/boundary (S24). The degree of deviation acquired in S24 is the difference between the probability distribution acquired in S22 and the probability distribution when the Mth layer/boundary is accurately identified.

In this embodiment, the entropy of the probability distribution P is calculated as the deviation. The entropy is given by the following (Equation 1). The entropy H(P) takes a value of 0≦H(P)≦log(number of events), and the more biased the probability distribution P, the smaller the value becomes. In other words, the smaller the entropy H(P), the higher the identification accuracy of the Mth layer/boundary tends to be.
H(P)=-Σplog(p)...(Math. 1)

Next, CPU 3 determines whether the degrees of deviation have been obtained for all regions (layers/boundaries) to be identified in the Tth two-dimensional tomographic image (S25). If the degrees of deviation have not yet been obtained for some regions (S25: NO), "1" is added to the order M of the region to be identified (S26), and processing returns to S22, where the identification results and degrees of deviation for the next region are obtained (S22-S24). Once the identification results and degrees of deviation for all regions have been obtained (S25: YES), CPU 3 stores the identification results and degrees of deviation obtained for each of the multiple layers/boundaries appearing in the Tth two-dimensional tomographic image in storage device 4 (S27).

Next, CPU 3 determines whether layer/boundary identification results and deviation degrees have been obtained for all two-dimensional tomographic images that make up the three-dimensional tomographic image (S28). If identification results and deviation degrees have not yet been obtained for some two-dimensional tomographic images (S28: NO), "1" is added to the order T of the two-dimensional tomographic image (S29), and processing returns to S21, where the identification results and deviation degrees for the next two-dimensional tomographic image are obtained (S21-S27). Once layer/boundary identification results and deviation degrees have been obtained for all two-dimensional tomographic images (S28: YES), processing returns to fundus image processing (see Figure 7).

Next, the CPU 3 determines whether the macular disease mode or the glaucoma mode is selected as the examination mode for examining the subject's eye (S13). In this embodiment, the user can pre-select (set) either the macular disease mode or the glaucoma mode by operating the operation unit 7. The examination mode may also be automatically selected (set) based on information such as an electronic medical record. The examination mode may also be pre-selected (set) when a three-dimensional image of the fundus of the subject's eye is captured. In this case, the examination mode may be automatically set depending on the image capture method used by the OCT device 10B. For example, if "macular map" is selected as the capture method used by the OCT device 10B, the "macular disease mode" may be set as the examination mode, or if "glaucoma map" is selected as the capture method, the "glaucoma mode" may be set as the examination mode. The examination mode may also be set by selecting one of the examination modes, or by selecting another option (e.g., the type of medical information to be initially displayed). If the macular disease mode is selected (S13: YES), the CPU 3 executes the macular disease processing (see FIG. 10) (S14). If the macular disease mode is not selected and the glaucoma mode is selected (S13: NO), the CPU 3 executes the glaucoma processing (see FIG. 14) (S15).

The macular disease processing will be described in detail with reference to Figures 10 to 13. First, with reference to Figure 11, an example of the initial display screen of the analysis results for the subject's eye being treated when the macular disease mode is selected will be described. In this embodiment, the medical information displayed on the initial display screen in the macular disease mode includes a deviation map 60. The deviation map 60 shows a probability distribution for identifying at least one of the layers and boundaries of the fundus tissue shown in the three-dimensional image acquired in S11, and a two-dimensional distribution of the deviation between the probability distribution when the layers and boundaries are accurately identified (in this embodiment, the two-dimensional distribution as viewed from the direction of the optical axis of the light used to capture the three-dimensional image). As mentioned above, the deviation map 60 is likely to show the degree of abnormality in the layer and boundary structure. In the deviation map 60 shown in Figures 11 to 13, areas with a large degree of deviation are displayed in bright colors.

When a macular disease develops in the subject's eye, it is often accompanied by abnormalities in the structure of layers and boundaries. Therefore, in conventional macular disease treatment, the user must check two-dimensional tomographic images 80 at various positions within the imaging range of the three-dimensional tomographic image to determine whether or not a structural abnormality has occurred. In contrast, in this embodiment, when the macular disease mode is selected, a discrepancy map 60 is initially displayed, which makes it easy to grasp the degree of abnormality in the structure of layers and boundaries in a two-dimensional area. Therefore, the discrepancy map 60, which is initially displayed on the display device 8, makes it easy for the user to appropriately grasp the positions where a structural abnormality is likely to occur.

Furthermore, in this embodiment, when the macular disease mode is selected, the thickness analysis map 70 (see Figures 12 and 15, details will be described later) is excluded from the medical information of the subject's eye to be analyzed that is initially displayed on the display device 8. In the diagnosis of macular disease, the thickness analysis map 70 may be used to additionally confirm the presence or absence of significant edema, etc., but may not be used in the diagnosis of macular disease itself. Therefore, when the macular disease mode is selected, the thickness analysis map 70, which is not used very frequently, is excluded from the initial display, which helps to shorten the time required for the diagnosis of macular disease.

However, in this embodiment, even when the macular disease mode is selected, the thickness analysis map 70 is displayed on the display device 8 together with the deviation map 60 in accordance with instructions input by the user (see Figure 12). Therefore, even when the macular disease mode is selected, the user can appropriately grasp the two-dimensional distribution of layer and boundary thicknesses by displaying the thickness analysis map 70 as needed.

Furthermore, as shown in Figure 11, the medical information displayed on the initial display screen in macular disease mode includes a two-dimensional tomographic image 80 of a portion of the fundus tissue extending in the depth direction, which is included in the image area of the three-dimensional image. Therefore, the user can properly grasp the state of the layers and boundaries of the subject's eye being analyzed from the two-dimensional tomographic image 80 as well. Details of how the two-dimensional tomographic image 80 is displayed will be described later.

Furthermore, in this embodiment, the analysis result display screen displays a two-dimensional fundus observation image 50 obtained by observing the subject's eye being analyzed from the front, and a selected mode display section 55 that indicates the selected examination mode.

As shown in FIG. 10, in the macular disease processing, the CPU 3 identifies, from among multiple layers and boundaries in the subject's eye to be analyzed, a layer/boundary for which a deviation map 60 is to be initially displayed on the analysis result display screen (S31). The CPU 3 generates the deviation map 60 for the layer/boundary identified in S31 based on the deviation obtained in the layer/boundary identification processing (see FIG. 8) (S32). The CPU 3 initially displays the deviation map 60 generated in S32 on the analysis result display screen (S33).

As an example, in this embodiment, layers and boundaries within the relevant fundus tissues are associated with each of multiple macular diseases that may occur in a subject. For example, IPL/INL and OPL/ONL are associated as layers and boundaries associated with diabetic retinopathy. RPE/BM and BM are associated as layers and boundaries associated with age-related macular degeneration. RPE/BM and BM are associated as layers and boundaries associated with retinitis pigmentosa. In this embodiment, the user can pre-specify the type of macular disease for which they wish to treat from among multiple macular diseases. In S31, if a type of macular disease has been specified, CPU 3 identifies the region (layer/boundary) associated with the specified macular disease as the region for which the discrepancy map 60 will be initially displayed. This allows the user to efficiently use the discrepancy map 60 according to the type of macular disease for which they wish to treat.

In addition, in this embodiment, the user can directly select one or more layers/boundaries from among the multiple layers/boundaries for which they wish to check the deviation map 60 via the operation unit 7. In S31, if a layer/boundary has been specified by the user, the CPU 3 identifies the specified layer/boundary as the layer/boundary for which the deviation map 60 will be initially displayed.

The CPU 3 can also selectively initially display on the display unit a predetermined number of deviation maps 60 with the highest deviation rankings from among the multiple deviation maps 60 for multiple layers/boundaries. In this case, the user can easily check the deviation maps 60 for layers/boundaries with the highest deviation rankings in the classification by the mathematical model.

11 and 12, a deviation map 60 of one specific layer/boundary (RPE/BM in FIGS. 11 and 12) is displayed among multiple layers/boundaries in the three-dimensional image to be analyzed. However, as shown in FIG. 13, in S31 to S33, CPU 3 can also display deviation maps 60 for multiple layers/boundaries. In this case, CPU 3 may separately display deviation maps 60A, 60B, and 60C for multiple layers/boundaries. CPU 3 may also display a combined deviation map 60X that combines deviation maps 60A, 60B, and 60C for multiple layers/boundaries in an aligned state. The composite deviation map 60X may be generated by superimposing multiple deviation maps 60A, 60B, and 60C, with the deviation maps 60A, 60B, and 60C for each layer and boundary being represented in different colors.

Next, the CPU 3 extracts a two-dimensional tomographic image 80 at a default position within the imaging range of the three-dimensional image and initially displays it on the display device 8 (S34). The default position for displaying the two-dimensional tomographic image 80 can be set as appropriate. For example, the CPU 3 may specify, among any lines (e.g., straight lines) that can be set on the deviation map 60, the position of the line with the highest deviation as the default position for extracting the two-dimensional tomographic image 80. The CPU 3 may also initially display, on the display device 8, a two-dimensional tomographic image 80 that extends in the depth direction of the fundus from the specified linear default position. The CPU 3 may also specify, as the default position for extracting the two-dimensional tomographic image 80, the position of a line passing through the center of the imaging range of the three-dimensional image (e.g., a straight line that crosses the center position on both sides on a two-dimensional frontal image obtained by viewing the three-dimensional image from the front). Additionally, the CPU 3 may identify a characteristic part of the fundus (for example, the optic disc or macula) using image processing or the like, and set a linear position passing through the identified part as the default position for extracting the two-dimensional tomographic image 80 in S34. Note that the line is not limited to a straight line.

The CPU 3 displays, on the two-dimensional tomographic image, the portion (sometimes called the "corresponding portion") for which the deviation map 60 is displayed among the multiple layers and boundaries in the fundus being analyzed (S35). Therefore, the user can easily and appropriately grasp, on the two-dimensional tomographic image 80 extending in the depth direction, the layer and boundary for which the deviation map 60 is displayed (the layer and boundary indicated by "VT" in Figure 11).

11 and 12, the lines V indicating the positions of the multiple layers/boundaries identified in the layer/boundary identification process (see FIG. 8) are displayed in different colors on the two-dimensional tomographic image 80. The CPU 3 indicates the layer/boundary for which the deviation map 60 is displayed by matching the color of the line VT of the layer/boundary for which the deviation map 60 is displayed on the analysis result display screen with the display color of the deviation map 60 itself. Therefore, the color of the line V indicating each layer/boundary allows the user to easily determine which portion of the multiple layers and boundaries the displayed deviation map 60 corresponds to. Therefore, even when deviation maps 60 for multiple portions are displayed simultaneously (e.g., as shown in FIG. 13), the user can appropriately identify the portion for which the deviation map 60 is displayed on the two-dimensional tomographic image 80.

The CPU 3 allows the user to identify the state of deviation in the area of the two-dimensional tomographic image 80 for the layer/boundary for which the deviation map 60 is displayed on the analysis result display screen (S36). Therefore, the user can easily check the state of deviation in the two-dimensional tomographic image 80. Therefore, the user can appropriately grasp the state of the layer/boundary shown in the two-dimensional tomographic image 80 according to the degree of deviation.

The specific method for allowing the user to identify the state of the degree of deviation in an area within the two-dimensional tomographic image 80 can be selected as appropriate. In the example shown in Figures 11 and 12, the CPU 3 allows the user to identify the state of the degree of deviation by executing an identification display that indicates an area 81 within the two-dimensional tomographic image 80 where the degree of deviation is equal to or greater than a threshold. The CPU 3 may also allow the user to identify the state of the degree of deviation by applying a color of a depth corresponding to the magnitude of the degree of deviation to each area within the two-dimensional tomographic image 80.

The CPU 3 displays the extracted position of the two-dimensional tomographic image 80 displayed on the display device 8 on the discrepancy map 60 (S37). Therefore, the user can diagnose the fundus tissue while checking the extracted position of the displayed two-dimensional tomographic image 80 on the discrepancy map 60. In this embodiment, as shown in Figures 11 and 12, a line P indicating the extracted position of the two-dimensional tomographic image 80 is displayed on the two-dimensional discrepancy map 60 when the fundus tissue is viewed from the front. In this embodiment, a line P indicating the extracted position of the two-dimensional tomographic image 80 is also displayed on the two-dimensional fundus observation image 50. The above processing completes the initial display of the analysis results in the macular disease mode.

Next, the CPU 3 determines whether the user has input an instruction to display the thickness analysis map 70 (S39). If no instruction has been input (S39: NO), the process proceeds directly to the determination in S41. When an instruction to display the thickness analysis map 70 is input via the operation unit 7 or the like (S39: YES), the CPU 3 generates a thickness analysis map 70 showing the two-dimensional distribution of the analysis results for the thickness of a specific layer based on the layer/boundary identification results obtained in the layer/boundary identification process (see FIG. 8), and displays the thickness analysis map 70 together with the deviation map 60 on the analysis result display screen (S40). Therefore, even when the macular disease mode is selected, the user can appropriately grasp the two-dimensional distribution of layer/boundary thickness by displaying the thickness analysis map 70 as needed. It goes without saying that the display and non-display of the thickness analysis map 70 may be switched appropriately depending on the instruction input by the user.

In S40, the CPU 3 displays a thickness analysis map 70 showing the two-dimensional distribution of thickness analysis results for the layers/boundaries for which the deviation map 60 is displayed, among the multiple layers/boundaries in the three-dimensional image being analyzed. Therefore, the user can easily compare the deviation map 60 and the thickness analysis map 70 for the same layer/boundary, making it easier to provide effective medical treatment. Note that, as shown in FIG. 13, when deviation maps 60 for multiple layers/boundaries are displayed on the analysis result display screen, the CPU 3 may display a thickness analysis map 70 for each of the same multiple layers/boundaries separately, or may display a thickness analysis map 70 that combines all the thicknesses of the same multiple layers/boundaries.

The specific form of the thickness analysis map 70 can be selected as appropriate. For example, the CPU 3 may generate and display, as the thickness analysis map 70, a normal eye comparison map showing the results of comparing the two-dimensional distribution of the thickness of at least one layer in the three-dimensional image to be analyzed with the two-dimensional distribution of the thickness of the same layer in a normal eye (for example, at least one of a percentile map showing the two-dimensional distribution of the percentile of the difference between the two, and a deviation map showing the two-dimensional distribution of the deviation between the two). The CPU 3 may also generate and display, as the thickness analysis map 70, a thickness map showing the two-dimensional distribution of the thickness of at least one layer in the three-dimensional image to be analyzed. The CPU 3 may display both the normal eye comparison map and the thickness map as the thickness analysis map 70.

Next, the CPU 3 determines whether the user has input an instruction to specify a position P from which the two-dimensional tomographic image 80 is to be extracted on the deviation map 60 displayed on the display device 8 (S41). If a position P has not been specified (S41: NO), the process proceeds directly to S44. As an example, in this embodiment, the user specifies a position P from which the two-dimensional tomographic image is to be extracted on the deviation map 60 displayed on the display device 8 by operating the operation unit 7 (e.g., by moving and clicking the cursor, or by specifying a position using a touch panel). Note that in this embodiment, the CPU 3 can also accept an instruction to specify a position P from which the two-dimensional tomographic image 80 is to be extracted on the two-dimensional fundus observation image 50 displayed on the display device 8. In the examples shown in FIGS. 11 and 12, the position P from which the two-dimensional tomographic image 80 is to be extracted is specified by specifying the position of a straight line. However, the method for specifying the extraction position P can also be changed. For example, the line used to specify the position P is not limited to a straight line, and may be a circular or curved line, etc.

When a position P for extracting the two-dimensional tomographic image 80 is specified on the deviation map 60 (S41: YES), the CPU 3 displays the specified position P on the deviation map 60 (S42). Therefore, the user can diagnose the fundus tissue of the subject's eye while checking the extraction position P of the displayed two-dimensional tomographic image 80 on the deviation map 60. As described above, in the example shown in FIG. 11 , a line P indicating the extraction position of the two-dimensional tomographic image 80 is displayed on the two-dimensional deviation map 60 when the fundus tissue is viewed from the front. Furthermore, in this embodiment, a line P indicating the extraction position of the two-dimensional tomographic image 80 is also displayed on the two-dimensional fundus observation image 50. Note that when an extraction position P of the two-dimensional tomographic image 80 is specified on the fundus observation image 50, a line P indicating the extraction position of the two-dimensional tomographic image 80 is similarly displayed on the deviation map 60 and the fundus observation image 50.

Furthermore, the CPU 3 extracts from the three-dimensional image a two-dimensional tomographic image 80 extending in the depth direction of the fundus tissue from the specified position P (i.e., a two-dimensional tomographic image 80 that passes through the specified position P and extends in the depth direction), and displays it on the display device 8 (S43). Therefore, the user can check the two-dimensional distribution of the discrepancy in classification for a specific region (in this embodiment, a specific layer/boundary) using the discrepancy map 60, and then directly specify the position of the two-dimensional tomographic image 80 they want to display on the discrepancy map 60. Processing then proceeds to S44.

Next, CPU 3 determines whether the user has input an instruction to specify at least one of multiple macular diseases (S44). If no macular disease has been specified (S44: NO), processing proceeds directly to S46. As described above, in this embodiment, the user can display a deviation map 60 for the layers and boundaries associated with the specified macular disease by specifying at least one macular disease. Specifically, at least one of the layers and boundaries within the associated fundus tissue is associated with each of multiple macular diseases that may occur in the subject. When a macular disease is specified by the user (S44: YES), CPU 3 generates a deviation map 60 for the layer and boundary associated with the specified macular disease from among the multiple layers and boundaries identified by the mathematical model, and displays it on display device 8 (S45). Therefore, the user can easily confirm the deviation map 60 for the area associated with the subject's predicted macular disease simply by specifying the predicted macular disease. Additionally, the user can predict the subject's macular disease by switching between the macular diseases and checking the layer/boundary discrepancy map 60 associated with each macular disease. Note that in S45, the area where the discrepancy map 60 is displayed may also be displayed on the two-dimensional tomographic image 80.

Next, the CPU 3 determines whether the user has input an instruction to specify at least one of the multiple layers/boundaries for which a deviation map 60 is to be displayed (S46). If a layer/boundary has not been specified (S46: NO), the process proceeds directly to S48. If a layer/boundary has been specified by the user (S46: YES), the CPU 3 generates a deviation map 60 for the specified layer/boundary from the multiple layers/boundaries identified by the mathematical model, and displays it on the display device 8 (S47). This allows the user to easily check the deviation map 60 for the desired layer/boundary. Note that in S47 as well, the area for which the deviation map 60 is displayed may be displayed on the two-dimensional tomographic image 80.

The CPU 3 determines whether an instruction to end the display of the analysis result display screen or an instruction to change the treatment mode has been input (S48). If no instruction has been input (S48: NO), processing returns to S39, and steps S39 to S48 are repeated. If an instruction to end the display or an instruction to change the treatment mode has been input (S48: YES), processing returns to fundus image processing (see Figure 7).

The glaucoma processing will be described in detail with reference to Figures 14 and 15. First, with reference to Figure 15, an example of an initial display screen showing analysis results for the subject's eye when the glaucoma mode is selected will be described. In this embodiment, the medical information displayed on the initial display screen in the glaucoma mode includes a thickness analysis map 70. As described above, the thickness analysis map 70 shows a two-dimensional distribution of analysis results for the thickness of at least one layer in the three-dimensional image of the subject. When glaucoma develops in the subject's eye, the thickness of at least some layers often decreases (i.e., thinning occurs). In this embodiment, the thickness analysis map 70 is initially displayed when the glaucoma mode is selected. Therefore, the thickness analysis map 70 initially displayed on the display device 8 makes it easier for the user to appropriately treat glaucoma based on the subject's layer thicknesses.

As mentioned above, the specific form of the thickness analysis map 70 can be selected as appropriate. For example, the CPU 3 may generate and display a normal eye comparison map (e.g., at least one of a percentile map showing a two-dimensional distribution of percentiles of the difference between the two, and a deviation map showing a two-dimensional distribution of deviations between the two) as the thickness analysis map 70. The CPU 3 may also generate and display a thickness map showing a two-dimensional distribution of layer thicknesses as the thickness analysis map 70. The CPU 3 may display both the normal eye comparison map and the thickness map as the thickness analysis map 70.

Furthermore, in this embodiment, when the glaucoma mode is selected, the deviation map 60 (see Figures 11 to 13) is excluded from the medical information of the subject's eye to be analyzed that is initially displayed on the display device 8. Layer thinning caused by glaucoma is often not accompanied by structural abnormalities in the layers and boundaries. If no structural abnormalities in the layers and boundaries occur, the deviation map 60 is unlikely to show any notable features. Therefore, when the glaucoma mode is selected, the deviation map 60, which is not often used in the treatment of glaucoma, is excluded from the initial display, which helps to shorten the time required for treatment of glaucoma.

However, in this embodiment, even when the glaucoma mode is selected, the discrepancy map 60 is displayed on the display device 8 along with the thickness analysis map 70 in accordance with instructions input by the user. Therefore, even when the glaucoma mode is selected, the user can appropriately grasp the degree of abnormality in the layer/boundary structure by displaying the discrepancy map 60 as needed.

In addition, the medical information displayed on the initial display screen in glaucoma mode also includes a two-dimensional tomographic image 80 of a portion of the fundus tissue extending in the depth direction, which is included in the image area of the three-dimensional image. Therefore, the user can properly grasp the state of the layers and boundaries of the subject's eye being analyzed from the two-dimensional tomographic image 80.

Furthermore, the display screen for the analysis results in glaucoma mode also displays a two-dimensional fundus observation image 50 of the subject's eye being analyzed, observed from the front, and a selected mode display section 55 that indicates the selected treatment mode.

As shown in FIG. 14, in the glaucoma processing, the CPU 3 identifies, from among multiple layers and boundaries in the subject's eye to be analyzed, a layer/boundary for which a thickness analysis map 70 is to be initially displayed on the analysis result display screen (S51). The CPU 3 generates the thickness analysis map 70 for the layer/boundary identified in S51 based on the layer/boundary identification results obtained in the layer/boundary identification processing (see FIG. 8) (S52). The CPU 3 initially displays the thickness analysis map 70 generated in S52 on the analysis result display screen (S53).

As an example, in this embodiment, when the glaucoma mode is selected, the layer/boundary on which the thickness analysis map 70 is initially displayed is set in advance. For example, at least one of all layers of the retina (in this embodiment, layers from the ILM to the RPE/BM) or the GCC (layers from the ILM to the IPL/INL) is set as the layer/boundary on which the thickness analysis map 70 is initially displayed. Note that the layer/boundary on which the thickness analysis map 70 is initially displayed may be set according to instructions input by the user.

In addition, in this embodiment, the user can directly select one or more layers/boundaries from among the multiple layers/boundaries for which they wish to check the thickness analysis map 70 via the operation unit 7. In S51, if a layer/boundary has been specified by the user, the CPU 3 identifies the specified layer/boundary as the layer/boundary for which the thickness analysis map 70 will be initially displayed.

Next, the CPU 3 extracts a two-dimensional tomographic image 80 at a default position within the imaging range of the three-dimensional image, and initially displays it on the display device 8 (S54). The method for setting the default position for displaying the two-dimensional tomographic image 80 can be selected as appropriate. For example, the CPU 3 may specify the position of a line passing through the center of the imaging range of the three-dimensional image (e.g., a straight line crossing the center position on the left and right on a two-dimensional frontal image obtained by viewing the three-dimensional image from the front) as the default position for extracting the two-dimensional tomographic image 80. The CPU 3 may also specify a characteristic site on the fundus (e.g., the optic disc or macula) using image processing, and set a linear position passing through the specified site as the default position for extracting the two-dimensional tomographic image 80 in S54. Note that the line is not limited to a straight line.

The CPU 3 displays, on the two-dimensional tomographic image, the portion of the multiple layers and boundaries in the fundus being analyzed where the thickness analysis map 70 is displayed (sometimes referred to as the "corresponding portion") (S55). Therefore, the user can easily and appropriately grasp the layer and boundary where the thickness analysis map 70 is displayed on the two-dimensional tomographic image 80, which extends in the depth direction.

The CPU 3 displays the extracted position of the two-dimensional tomographic image 80 displayed on the display device 8 on the thickness analysis map 70 (S56). Therefore, in glaucoma mode, the user can diagnose the fundus tissue while checking the extracted position of the displayed two-dimensional tomographic image 80 on the thickness analysis map 70. In this embodiment, as shown in FIG. 15, a line P indicating the extracted position of the two-dimensional tomographic image 80 is displayed on the two-dimensional thickness analysis map 70 when the fundus tissue is viewed from the front. Similarly, in this embodiment, a line P indicating the extracted position of the two-dimensional tomographic image 80 is also displayed on the two-dimensional fundus observation image 50. The above processing completes the initial display of the analysis results in glaucoma mode.

Next, the CPU 3 determines whether the user has input an instruction to display the deviation map 60 (S58). If no instruction has been input (S58: NO), the process proceeds directly to the determination in S60. When an instruction to display the deviation map 60 is input via the operation unit 7 or the like (S58: YES), the CPU 3 generates a deviation map 60 showing the two-dimensional distribution of deviations for a specific layer based on the deviations obtained in the layer/boundary identification process (see FIG. 8), and displays this map together with the thickness analysis map 70 on the analysis result display screen (S59). Therefore, even when the glaucoma mode is selected, the user can appropriately grasp the two-dimensional distribution of deviations for layers and boundaries by displaying the deviation map 60 as needed. Needless to say, the display and non-display of the deviation map 60 may be switched as appropriate in response to an instruction input by the user.

In S59, the CPU 3 displays a discrepancy map 60 showing the two-dimensional distribution of discrepancies for layers/boundaries for which thickness analysis maps 70 are displayed, among the multiple layers/boundaries in the three-dimensional image being analyzed. Therefore, the user can easily compare the discrepancy map 60 and thickness analysis map 70 for the same layer/boundary, making it easier to provide effective medical treatment. Note that when thickness analysis maps 70 for multiple layers/boundaries are displayed on the analysis result display screen, the CPU 3 may display the discrepancy maps 60 for each of the same multiple layers/boundaries separately, or may display a composite discrepancy map 60X that combines all of the discrepancy maps 60 for the same multiple layers/boundaries.

Next, the CPU 3 determines whether the user has input an instruction to specify a position P from which the two-dimensional tomographic image 80 is to be extracted on the thickness analysis map 70 displayed on the display device 8 (S60). If the position P has not been specified (S60: NO), the process proceeds directly to S63. As an example, in this embodiment, the user specifies the position P from which the two-dimensional tomographic image is to be extracted on the thickness analysis map 70 displayed on the display device 8 by operating the operation unit 7 (e.g., by moving and clicking the cursor, or by specifying a position using a touch panel). Note that in this embodiment, the CPU 3 can also accept an instruction to specify the position P from which the two-dimensional tomographic image 80 is to be extracted on the two-dimensional fundus observation image 50 displayed on the display device 8. In the example shown in FIG. 15, the position P from which the two-dimensional tomographic image 80 is to be extracted is specified by specifying the position of a straight line. However, the method for specifying the extraction position P can also be changed. For example, the line used to specify the position P is not limited to a straight line, and may be a circular or curved line, etc.

When a position P for extracting the two-dimensional tomographic image 80 is specified on the thickness analysis map 70 (S60: YES), the CPU 3 displays the specified position P on the thickness analysis map 70 (S61). Therefore, the user can diagnose the fundus tissue of the subject's eye while checking the extraction position P of the displayed two-dimensional tomographic image 80 on the thickness analysis map 70. As described above, in the example shown in FIG. 15, a line P indicating the extraction position of the two-dimensional tomographic image 80 is displayed on the two-dimensional thickness analysis map 70 when the fundus tissue is viewed from the front. In this embodiment, a line P indicating the extraction position of the two-dimensional tomographic image 80 is also displayed on the two-dimensional fundus observation image 50. Note that when the extraction position P of the two-dimensional tomographic image 80 is specified on the fundus observation image 50, a line P indicating the extraction position of the two-dimensional tomographic image 80 is similarly displayed on the thickness analysis map 70 and the fundus observation image 50.

Furthermore, the CPU 3 extracts from the three-dimensional image a two-dimensional tomographic image 80 extending in the depth direction of the fundus tissue from the specified position P (i.e., a two-dimensional tomographic image 80 that passes through the specified position P and extends in the depth direction), and displays it on the display device 8 (S62). Therefore, the user can confirm the two-dimensional distribution of the thickness analysis results for a specific region (in this embodiment, a specific layer/boundary) using the thickness analysis map 70, and then directly specify the position of the two-dimensional tomographic image 80 they want to display on the thickness analysis map 70. Processing then proceeds to S63.

Next, the CPU 3 determines whether the user has input an instruction to specify at least one of the multiple layers/boundaries for which a thickness analysis map 70 is to be displayed (S63). If a layer/boundary has not been specified (S63: NO), the process proceeds directly to S65. If a layer/boundary has been specified by the user (S63: YES), the CPU 3 generates a thickness analysis map 70 for the specified layer/boundary from the multiple layers/boundaries identified by the mathematical model, and displays it on the display device 8 (S64). This allows the user to easily confirm the thickness analysis map 70 for the desired layer/boundary. Note that in S64, the area for which the thickness analysis map 70 is displayed may also be displayed on the two-dimensional tomographic image 80.

The CPU 3 determines whether an instruction to end the display of the analysis result display screen or an instruction to change the examination mode has been input (S65). If no instruction has been input (S65: NO), processing returns to S58, and steps S58 to S65 are repeated. If an instruction to end the display or an instruction to change the examination mode has been input (S65: YES), processing returns to fundus image processing (see Figure 7).

Returning to the explanation of Figure 7, when the macular disease processing (S14) or glaucoma table processing (S15) is completed, it is determined whether an instruction to change the treatment mode has been input (S17). If an instruction to change the treatment mode has been input (S17: YES), the process returns to S13, and display control processing of medical information (analysis results) for the changed treatment mode is executed (S14 or S15). If an instruction to change the treatment mode has not been input (S17: NO), an end instruction has been input, and the process ends.

The techniques disclosed in the above embodiments are merely examples. Therefore, it is possible to modify the techniques exemplified in the above embodiments. First, it is possible to execute only some of the techniques exemplified in the above embodiments. Furthermore, in the above embodiment, when an instruction to display the thickness analysis map 70 is input in the macular disease mode (S39: YES), the thickness analysis map 70 is displayed together with the deviation map 60 (i.e., the thickness analysis map 70 is additionally displayed). However, when an instruction to display the thickness analysis map 70 is input in the macular disease mode, the CPU 3 may switch between the thickness analysis map 70 and the deviation map 60 and display them. In this case, the extraction position of the two-dimensional tomographic image may be specified on the displayed deviation map 60 or on the fundus observation image 50. Furthermore, in the above embodiment, when an instruction to display the deviation map 60 is input in the glaucoma mode (S58: YES), the deviation map 60 is displayed together with the thickness analysis map 70 (i.e., the deviation map 60 is additionally displayed). However, when an instruction to display the deviation map 60 is input in the glaucoma mode, the CPU 3 may switch between displaying the deviation map 60 and the thickness analysis map 70. In this case, the extraction position of the two-dimensional tomographic image may be specified on the thickness analysis map 70 that has been switched to and is displayed, or may be specified on the fundus observation image 50.

The process of acquiring a three-dimensional image in S11 of FIG. 7 is an example of an "image acquisition step." The processes of generating a deviation map in S32, S45, and S47 of FIG. 10 and S59 of FIG. 14 are an example of a "deviation map generation step." The processes of generating a thickness analysis map in S40 of FIG. 10 and S52 and S64 of FIG. 14 are an example of a "thickness analysis map generation step." The processes of displaying medical information in S14 and S15 of FIG. 7 are an example of a "display control step." The processes of accepting input of an instruction for the extraction position of the two-dimensional tomographic image 80 in S41 of FIG. 10 and S60 of FIG. 14 are an example of an "extraction position designation acceptance step." The processes of extracting and displaying a two-dimensional tomographic image in S43 of FIG. 10 and S62 of FIG. 14 are an example of a "designated tomographic image display step."

1 Fundus image processing device 3 CPU
4 Storage device 8 Display device 10 (10A, 10B) OCT device 42 Two-dimensional tomographic image 43 Three-dimensional tomographic image 60 Deviation degree map 70 Thickness analysis map 80 Two-dimensional tomographic image

Claims (6)

A fundus image processing device that processes a fundus image in which a plurality of layers in the fundus of an eye to be examined and boundaries between the layers are included in an imaging range,
The control unit of the fundus image processing device
an image acquisition step of acquiring a three-dimensional image of the fundus photographed by the fundus image photographing device;
a deviation map generation step of inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying at least one of a layer and a boundary in the fundus tissue shown in the three-dimensional image, and generating a deviation map showing a two-dimensional distribution of deviation of the obtained probability distribution from a probability distribution when the layer or boundary to be identified is accurately identified;
a thickness analysis map generating step of generating a thickness analysis map showing a two-dimensional distribution of an analysis result of the thickness of at least any layer in the fundus tissue shown in the three-dimensional image;
a display control step of displaying medical information including at least one of the deviation map and the thickness analysis map on a display unit;
is executable,
In the display control step,
When a macular disease mode for performing a macular disease diagnosis is selected as a diagnosis mode for treating the subject's eye, the discrepancy map is included in the medical information of the subject's eye that is initially displayed on the display unit,
A fundus image processing device characterized in that, when a glaucoma mode for performing glaucoma treatment is selected as the treatment mode, the thickness analysis map is included in the medical information of the subject's eye that is initially displayed on the display unit. The fundus image processing device according to claim 1,
In the display control step, the control unit
A fundus image processing device characterized in that, when the glaucoma mode is selected as the diagnosis mode, the deviation map is excluded from the medical information of the subject's eye that is initially displayed on the display unit. The fundus image processing device according to claim 1,
In the display control step, the control unit
A fundus image processing device characterized in that, when the macular disease mode is selected as the diagnostic mode, the thickness analysis map is excluded from the medical information of the subject's eye that is initially displayed on the display unit. The fundus image processing device according to claim 1,
The control unit
an extraction position designation receiving step of receiving an instruction input from a user to designate an extraction position of a part of a two-dimensional tomographic image extending in a depth direction of fundus tissue from an image region of the three-dimensional image;
a designated tomographic image display step of extracting a two-dimensional tomographic image extending in a depth direction of the fundus tissue from the designated extraction position when an instruction input for designating the extraction position is accepted and displaying the extracted two-dimensional tomographic image on the display unit;
Further execute
In the extraction position designation receiving step,
When the macular disease mode is selected as the diagnosis mode, an instruction input of the extraction position is accepted on the deviation map displayed on the display unit, and
A fundus image processing device characterized in that, when the glaucoma mode is selected as the diagnosis mode, it accepts input of instructions for the extraction position on the thickness analysis map displayed on the display unit. The fundus image processing device according to claim 1,
In the display control step, the control unit
a part of a two-dimensional tomographic image of the fundus tissue extending in the depth direction included in an image region of the three-dimensional image is displayed on the display unit;
A fundus image processing device characterized in that the color of the part of the plurality of layers and boundaries shown in the two-dimensional tomographic image where the deviation map is displayed is made to match the display color of the deviation map. A fundus image processing program executed by a fundus image processing device that processes a fundus image in which a plurality of layers in the fundus of an eye to be examined and boundaries between the layers are included in an imaging range,
The fundus image processing program is executed by a control unit of the fundus image processing device,
an image acquisition step of acquiring a three-dimensional image of the fundus photographed by the fundus image photographing device;
a deviation map generation step of inputting the three-dimensional image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying at least one of a layer and a boundary in the fundus tissue shown in the three-dimensional image, and generating a deviation map showing a two-dimensional distribution of deviation of the obtained probability distribution from a probability distribution when the layer or boundary to be identified is accurately identified;
a thickness analysis map generating step of generating a thickness analysis map showing a two-dimensional distribution of an analysis result of the thickness of at least any layer in the fundus tissue shown in the three-dimensional image;
a display control step of displaying medical information including at least one of the deviation map and the thickness analysis map on a display unit;
The fundus image processing device can be caused to execute the above.
In the display control step,
When a macular disease mode for performing a macular disease diagnosis is selected as a diagnosis mode for treating the subject's eye, the discrepancy map is included in the medical information of the subject's eye that is initially displayed on the display unit,
A fundus image processing program characterized in that, when a glaucoma mode for treating glaucoma is selected as the treatment mode, the thickness analysis map is included in the medical information of the test eye that is initially displayed on the display unit.