JP7579889B2 - IMAGE PROCESSING METHOD, PROGRAM, IMAGE PROCESSING APPARATUS, AND OPHTHALMOLOGY SYSTEM - Google Patents

Description

The present invention relates to an image processing method, a program, an image processing device, and an ophthalmic system.

Conventionally, a method has been known in which an enlarged image is displayed in response to a user's operation in order to observe details of a fundus image (see, for example, Patent Document 1).

US Patent Application Publication No. 2020/0069175

The technology disclosed herein provides a novel image processing method.

One embodiment of the present invention is an image processing method including a process for acquiring a wide-angle fundus image captured by an ophthalmic device, a process for extracting a plurality of partial images from the wide-angle fundus image to which an abnormal finding has been added, a selection process for selecting from the plurality of partial images a first partial image showing an abnormal area that is the basis of the abnormal finding, and a display process for displaying the first partial image.

1 is an overall configuration diagram of an information processing system according to a first embodiment. 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment. 1 is a diagram showing a configuration of an ophthalmologic apparatus according to a first embodiment. FIG. 2 is a diagram illustrating a functional configuration of a server according to the first embodiment. FIG. 2 is a diagram showing a configuration of a detection unit according to the first embodiment. 4 is a flowchart showing an outline of processing according to the first embodiment. 4 is a flowchart showing an estimation process according to the first embodiment. 5 is a flowchart showing an output process according to the first embodiment. 10 is a flowchart showing an addition process according to the first embodiment. 13A and 13B are diagrams showing an example of setting a partial region of a wide-angle fundus image and an example of a GUI using the generated partial image. 13A and 13B are diagrams showing an example of setting a partial region of a wide-angle fundus image and an example of a GUI using the generated partial image. FIG. 13 is a diagram illustrating an example of a GUI. FIG. 13 is a diagram illustrating an example of a GUI. FIG. 13 is a diagram illustrating an example of a GUI. FIG. 11 is a diagram illustrating a functional configuration of a server according to the second embodiment. 10 is a flowchart showing an estimation process according to a second embodiment.

First Embodiment
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the drawings, based on a first embodiment which is one of the embodiments of the present invention.

〔composition〕
1 shows a configuration of an information processing system 1 according to an embodiment of the present invention. The information processing system 1 includes a server 10, a terminal 20, and an ophthalmic device 30. The server 10, the terminal 20, and the ophthalmic device 30 are connected to each other via a network 5 so as to be able to transmit and receive data to and from each other.

The network 5 is a wireless or wired communication means, such as the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a public communication network, a dedicated line, etc. Note that although the information processing system 1 according to this embodiment is composed of multiple information management devices, the present invention does not limit the number of these devices. Therefore, the information processing system 1 can be composed of one or more devices as long as they have the following functions:

The server 10 and the terminal 20 are information processing devices provided in medical institutions such as hospitals and clinics, and are primarily operated by medical institution staff to acquire images taken by the ophthalmic device 30, as well as edit and analyze the images.

The ophthalmic device 30 is a device that performs SLO (Scanning Laser Ophthalmoscope) and OCT (Optical Coherence Tomography) (Figure 3). The ophthalmic device 30 has a control device 31 and an imaging device 32.

2 is an example of hardware (hereinafter referred to as "information processing device 100") used to realize the server 10, the terminal 20, and the control device 31 of the ophthalmic device 30. As shown in the figure, the information processing device 100 includes a processor 101, a main memory device 102, an auxiliary memory device 103, an input device 104, an output device 105, and a communication device 106. These are connected to each other so as to be able to communicate with each other via a communication means such as a bus (not shown).

Furthermore, the entire configuration of the information processing device 100 does not necessarily need to be realized by hardware, and all or part of the configuration may be realized by virtual resources such as a cloud server of a cloud system.

The processor 101 is configured using a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The processor 101 reads and executes a program stored in the main memory device 102, thereby realizing the functions of the server 10, the terminal 20, and the control device 31.

The main memory device 102 is a device that stores programs and data, and may be ROM (Read Only Memory), RAM (Random Access Memory), non-volatile semiconductor memory (NVRAM (Non Volatile RAM)), etc.

The auxiliary storage device 103 is, for example, various non-volatile memories (NVRAM: Non-volatile memory) such as SSDs (Solid State Drives) and SD memory cards, hard disk drives, optical storage devices (CDs (Compact Discs), DVDs (Digital Versatile Discs), etc.), storage areas of cloud servers, etc. Programs and data stored in the auxiliary storage device 103 are loaded into the main storage device 102 as needed.

The input device 104 is an interface that accepts the input of information, such as a keyboard, a mouse, a touch panel, a card reader, a voice input device (such as a microphone), a voice recognition device, etc. The information processing device 100 may be configured to accept the input of information between itself and another device via the communication device 106.

The output device 105 is an interface that outputs various types of information, such as a screen display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphics card, etc.), a printing device, etc.), an audio output device (speaker, etc.), a voice synthesis device, etc. The information processing device 100 may be configured to output information between itself and other devices via the communication device 106. The output device 105 corresponds to the display unit in this invention.

The communication device 106 is a wired or wireless communication interface that enables communication with other devices via the network 5, such as a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, etc.

The configuration of the ophthalmic device 30 is shown in Figure 3. The ophthalmic device 30 includes an imaging device 32 and a control device 31. The control device 31 may be provided in the same housing as the imaging device 32, or may be separate from the imaging device 32.

The imaging device 32 operates under the control of the control device 31. The imaging device 32 includes an SLO unit 33, an imaging optical system 34, and an OCT unit 35. The imaging optical system 34 includes an optical scanner 341 and a wide-angle optical system 342. The imaging device 32 captures an image of the subject's eye. The imaging device 32 captures, for example, an image of the fundus of the subject's eye, and obtains a fundus image and a tomographic image (OCT image) described below.

The SLO unit 18 acquires an image of the fundus 12A of the test eye 12. The OCT unit 20 acquires a tomographic image of the test eye 12. Hereinafter, a front view image of the retina created based on the SLO data acquired by the SLO unit 18 will be referred to as an SLO image, and a tomographic image or a front image (en-face image) of the retina created based on the OCT data acquired by the OCT unit 20 will be referred to as an OCT image. Note that an SLO image is sometimes referred to as a two-dimensional fundus image. Also, an OCT image is sometimes referred to as a fundus tomographic image, a posterior segment tomographic image, or an anterior segment tomographic image depending on the part of the test eye 12 that is photographed.

Hereinafter, the image captured by the ophthalmic device 30 may be referred to as the subject's eye image. In addition, if the subject's eye image is an image of the fundus captured using a wide-angle optical system as described below, it may be referred to as a wide-angle fundus image P.

The optical scanner 341 scans the light emitted from the SLO unit 33 in the X and Y directions. The optical scanner 341 may be any optical element capable of deflecting a light beam, such as a polygon mirror or a galvanometer mirror.

The wide-angle optical system 342 includes an objective optical system. The wide-angle optical system 342 provides a wide-angle field of view at the fundus. The SLO system is realized by the control device 31, the SLO unit 33, and the imaging optical system 34 shown in FIG. 3. The SLO system includes the wide-angle optical system 342, and thus allows observation of a wide field of view (FOV: Field of View) at the fundus. The FOV indicates the range that can be photographed by the imaging device 32. The FOV can be expressed as a field of view. In this embodiment, the field of view can be defined by an internal irradiation angle and an external irradiation angle. The external irradiation angle is an irradiation angle that defines the irradiation angle of the light beam irradiated from the UWF ophthalmic device 110 to the subject's eye based on the pupil. The internal irradiation angle is an irradiation angle that defines the irradiation angle of the light beam irradiated to the fundus 12A based on the eyeball center O. The external irradiation angle and the internal irradiation angle correspond to each other. For example, when the external irradiation angle is 120 degrees, the internal irradiation angle corresponds to approximately 160 degrees. In this embodiment, the internal irradiation angle is set to 200 degrees. Here, an SLO fundus image captured at an internal irradiation angle and a photographing angle of view of 160 degrees or more is referred to as a UWF-SLO fundus image.

The wide-angle optical system 342 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric system combining a concave mirror and a lens. By using a wide-angle optical system using an elliptical mirror or a wide-angle lens, it is possible to photograph the retina not only in the center of the fundus but also in the peripheral part of the fundus. When using a system including an elliptical mirror, a configuration using a system using an elliptical mirror described in International Publication WO2016/103484 or International Publication WO2016/103489 may be used. The disclosures of International Publication WO2016/103484 and International Publication WO2016/103489 are each incorporated herein by reference in their entirety.

The SLO unit 33 includes a light source 331B of blue light (B light), a light source 331G of green light (G light), a light source 331R of red light (R light), and a light source 331IR of infrared light (IR light) such as near-infrared light, as well as an optical system 335 that reflects or transmits the light from these light sources and guides it to one optical path. The SLO unit 33 also includes a beam splitter 332, and detection elements 333B, 333G, 333R, and 333IR that detect the B light, G light, R light, and IR light, respectively.

The SLO unit 33 is capable of switching between a light source or a combination of light sources to be emitted, such as a mode that emits B light, R light, and G light, and a mode that emits IR light.

The beam splitter 332 has the function of decomposing the reflected light from the fundus into B light, R light, G light, and IR light, and reflecting each light toward the detection elements 333B, 333G, 333R, and 333IR.

Detection elements 333B, 333G, 333R, and 333IR can detect B light, R light, G light, and IR light, respectively.

The light incident on the imaging optical system 34 from the SLO unit 33 is scanned in the X and Y directions by the optical scanner 341. The scanning light is irradiated onto the fundus via the wide-angle optical system 342. The light reflected by the fundus is incident on the SLO unit 33 via the wide-angle optical system 342 and the optical scanner 341.

The light reflected by the fundus enters the SLO unit 33 via the wide-angle optical system 342 and the optical scanner 341, and is split into B light, R light, G light, and IR light by the beam splitter 332. These lights are detected by the detection elements 333B, 333G, 333R, and 333IR, respectively.

By collecting information on B light, R light, G light, and IR light detected by detection elements 333B, 333G, 333R, and 333IR, the processor 101 of the control device 31 can generate an SLO fundus image.

The OCT system is composed of the control device 31, the OCT unit 35, and the imaging optical system 34 shown in Figure 3. The OCT unit 35 includes a light source 351, a sensor 352, a first optical coupler 353, a reference optical system 354, a collimating lens 355, and a second optical coupler 356.

Light emitted from the light source 351 is branched by the first optical coupler 353. One of the branched lights is collimated by the collimating lens 355 as measurement light and enters the imaging optical system 34. The light that enters the imaging optical system 34 is scanned in the X and Y directions by the optical scanner 341. The scanning light passes through the wide-angle optical system 342 and is irradiated onto the fundus. The measurement light reflected by the fundus enters the OCT unit 35 via the wide-angle optical system 342, and enters the second optical coupler 356 via the collimating lens 355 and the first optical coupler 353.

The other light emitted by the light source 351 and branched by the first optical coupler 353 passes through the reference optical system 354 and enters the second optical coupler 356 as reference light.

The reference light and the measurement light reflected by the fundus are interfered with by the second optical coupler 356 to generate interference light. The interference light is received by the sensor 352. The control device 31 receives a signal from the sensor 352 and generates a tomographic image. Note that, hereinafter, the act of photographing using the OCT system and the image obtained by photographing may be referred to simply as OCT photographing and OCT image, respectively.

[Software configuration]
The main functions (functional configuration) of the server 10 are shown in Fig. 4. As shown in the figure, the server 10 has the functions of a database 114 and a management unit 120. The management unit 120 has the functions of an image processing unit 116 and a detection unit 118 in particular. The database 114 is stored in the main memory device 102 of the server 10. The functions of the management unit 120 are realized by the processor 101 of the server 10 reading and executing a program stored in the main memory device 102 of the server 10.

In addition to the above functions, server 10 also has functions such as an operating system, a file system, a device driver, and a DBMS (DataBase Management System).

The management unit 120 performs the processes executed by the server 10, such as acquiring and managing images. The images acquired and managed by the management unit include images captured by the ophthalmic device 30. The image processing unit 116 mainly generates a GUI and processes images captured by the ophthalmic device 30.

The detection unit 118 has a function of estimating the presence or absence of abnormal findings such as bleeding or retinal detachment in the fundus including the retina or choroid (hereinafter referred to as "abnormal findings") and their details from an image captured by the ophthalmic device 30. In the detection unit 118, the presence or absence of abnormal findings and their details are estimated based on an abnormal area in the image of the subject's eye. In this embodiment, the detection unit 118 is a trained model generated by machine learning.

In detail, the detection unit 118 is a model that performs deep learning to learn image features of abnormal regions in the test eye image. The detection unit 118 constructs a neural network that outputs information indicating the estimated result of the presence or absence of abnormal findings for the input test eye image. For example, the neural network is a deep neural network (DNN).

The detection unit 118 has an input layer that accepts input of the test eye image, an output layer that outputs an estimation result of the presence or absence of abnormal findings, and an intermediate layer that extracts image features of the test eye image (Figure 5). Each of the input layer, output layer, and intermediate layer has nodes (indicated by white circles in the figure), and the nodes of each of these layers are connected by edges (indicated by arrows in the figure). Note that the configuration of the detection unit 118 shown in Figure 5 is an example, and the number of nodes and edges, the number of intermediate layers, etc. can be changed as appropriate.

When the detection unit 118 is a CNN (Convolutional Neural Network), the intermediate layer has a convolution layer that convolves the pixel values of each pixel in the test eye image input from the input layer, and a pooling layer that maps the pixel values, and these layers are used to extract features of the test eye image. The output layer has one or more neurons that output the result of estimating abnormal findings in the input test eye image.

The detection unit 118 can also output the likelihood of the estimation result together with the estimation result. The likelihood is, for example, a probability value output from the output layer of the detection unit 118, and is indicated, for example, by a value between "0" and "1" indicating the degree of reliability of the estimated abnormal findings. By notifying the user of the likelihood, the user can know how accurate the estimation result is.

The detection unit 118 also outputs the severity of the abnormal findings. The severity can be explained as the severity of the symptoms, the grade of the symptoms, the rate of progression, the impact of the symptoms on the human body, and the like. If the symptom is bleeding, for example, the detection unit 118 estimates the severity of the bleeding from the size and amount of bleeding. The detection unit 118 similarly outputs the severity for other abnormal findings such as retinal detachment and neovascularization.

In particular, in this embodiment, the detection unit 118 is described as being a CNN, but the detection unit 118 is not limited to a CNN and may be a neural network other than a CNN or a trained model constructed using another learning algorithm.

The database 114 stores images obtained by photographing fundus tissues including the retina and choroid, such as wide-angle fundus images P and OCT images. Annotations can also be added to the image of the test eye by medical professionals, and the database 114 can store the image of the test eye and the portion of the image to which annotations have been added and the contents of the annotations in association with an image showing a part of the image of the test eye. The stored data is used for learning and re-learning the detection unit 118. The annotations include information indicating the abnormal area of the fundus and details of the abnormal findings such as macular degeneration that are added to the abnormal area. Therefore, when used for learning the detection unit 118, the annotations function as data indicating the correct answer for the abnormal findings and abnormal areas of the input image. In addition, the database 114 stores the patient's medical records such as electronic medical records, including past records of the image of the test eye, patient IDs, and other data. The database 118 also stores information about findings obtained by the detection unit 118 (described later) in association with the images.

For the deep learning of the detection unit 118, a dataset including a large number of images including abnormal regions such as bleeding and neovascularization, and annotations attached to these images is used. The deep learning of the detection unit 118 is performed by having the detection unit 118 learn or re-learn this dataset.

〔process〕
An example of the processing executed by the information processing system 1 will be described below with reference to the flowcharts of Figures 6 to 9. A program stored in the main memory device 102 of the server 10 is started, and the processing of the information processing system 1 is executed by the management unit 120 of the server 10 as follows. Note that, below, the processing executed by the management unit 120 of the server 10 may be simply described as being executed by the "server 10".

The processing performed by the server 10 is broadly composed of four steps, as shown in FIG. 6. First, the server 10 acquires an image of the test eye obtained by the ophthalmic device 30 photographing the retina (S1). In this example, a wide-angle fundus image P is used as the image photographed by the ophthalmic device 30. As described above, the wide-angle fundus image P is photographed by the SLO system of the ophthalmic device 30. The management unit 120 acquires the wide-angle fundus image P stored in the ophthalmic device 30 via the network 5.

Next, the management unit 120 estimates the presence or absence of abnormal findings in the acquired wide-angle fundus image P (S3). Thereafter, the management unit 120 performs an output process of the subject eye image in which the presence or absence of abnormal findings has been estimated (S5).

Next, the management unit 120 performs additional processing (S7). Additional processing includes entering information into the electronic medical record and relearning, and is mainly performed in response to instructions from the user.

(Estimation process)
Details of the estimation process (S3) are shown in FIG 7 and will be described below. In step S31, the management unit 120 inputs a wide-angle fundus image P to the detection unit 118.

Next, the management unit 120 acquires an estimation result of the detection unit 118 regarding the presence or absence of an abnormality (S33). When it is estimated that there is an abnormality, the output of the detection unit 118 includes information that identifies the type of abnormal finding and the abnormal area. An abnormal area is an area where a difference occurs from a normal eye. Examples of abnormal areas captured in the image of the subject's eye include bleeding points, neovascular areas, retinal detachment areas, and non-perfusion areas in the retina or choroid. An abnormal finding is a finding that is given based on an abnormal area, and examples of the types of abnormal findings include the presence or absence and degree of bleeding, neovascularization, retinal detachment, and non-perfusion areas in the retina or choroid. FIG. 10 shows a wide-angle fundus image P including the macula M as an example of an abnormal finding. In the estimation result, an image area estimated to include an abnormal area and an identification result that identifies the type of abnormal finding are acquired.

The detection unit 118 outputs the likelihood of the estimation result and the severity of the abnormal finding together. The management unit 120 then generates abnormal finding information including the image area estimated to include an abnormal area, the identification result that identifies the type of abnormal finding, etc., the likelihood of the estimation result, and the severity of the abnormal finding.

As a result of the above, if an abnormal finding with sufficient certainty is output, the abnormal finding information is associated with and stored in the wide-angle fundus image P. The estimation process generates a wide-angle fundus image P associated with the abnormal finding information.

(Output processing)
Details of the output process (S5) will be described below with reference to Figs. 8, 10, and 11. Fig. 10 shows an example of a wide-angle fundus image P to which an abnormal finding has been added by the estimation process (S3). In the following, bleeding will be used as an example of an abnormal finding for explanation. In the wide-angle fundus image P, a bleeding point B is displayed as an abnormal area.

The management unit 120 uses the image processing unit 116 to extract partial images showing partial regions in the wide-angle fundus image P (S51). The management unit 120 sets partial regions in the wide-angle fundus image P and generates partial images. Multiple partial images are extracted, and the multiple partial images may have overlapping regions or may not overlap each other. If an abnormal finding is given, the region of the partial image is determined so that the abnormal region is included in one of the multiple partial images. If multiple abnormal regions are located apart from each other, the region of the partial image may be determined so that one partial image includes all the abnormal regions, or the region of the partial image may be determined so that multiple partial images include abnormal regions.

10 and 11, frames F1, F2, F3, and F4 shown superimposed on wide-angle fundus image P indicate four partial regions extracted from wide-angle fundus image P. Also shown on the right side of each figure are four partial images D1, D2, D3, and D4 generated as a result of extracting each partial region from wide-angle fundus image P. Partial images D1, D2, D3, and D4 correspond to the regions indicated by frames F1, F2, F3, and F4, respectively.

The image processing unit 116 extracts partial images from the wide-angle fundus image P so that any of the partial regions includes the bleeding point B, which is an abnormal region. In the examples of Figures 10 and 11, the region of the frame F2 includes the bleeding point B, and the other regions (F1, F3, F4) do not include the bleeding point B. As shown as an example in Figure 10, the image processing unit 116 determines the size and arrangement of the regions so that the partial images D1, D2, D3, and D4 all include the entire macula M. At the same time, the image processing unit 116 determines the size and arrangement of the regions so that the bleeding point B is included in any of the regions, in other words, so that the bleeding point B is displayed in any of the partial images. As a result, as shown on the right side of Figure 10, the macula M is displayed in all of the partial images D1, D2, D3, and D4. In addition, the bleeding point B is displayed in the partial image D2.

In the present embodiment, when characteristic fundus structures such as the macula and optic disc are displayed in each partial image, the position of each partial image and abnormal area in the wide-angle fundus image P is easily understood by the user.

In the partial image extraction example of Figure 11, the image processing unit 116 determines the division position so that the macula M is at the center and there is no overlap between the frames F1 to F4, i.e., the areas corresponding to partial images D1 to D4. The image processing unit 116 also adjusts the position and size of the partial images so that bleeding point B is displayed in one of the partial images. As a result, bleeding point B is displayed in partial image D1.

In addition, if there is no abnormal region, the method of setting the partial images is determined according to a predetermined method. For example, the arrangement and size of each image may be determined so that frames F1 to F4, i.e., partial images D1 to D4, all include the entire macula M (Figure 10). Alternatively, the division positions may be determined so that frames F1 to F4, i.e., partial images D1 to D4, are centered on the macula M and the regions do not overlap (Figure 11).

The image processing unit 116 extracts partial images from the wide-angle fundus image P as described above and creates partial images D1, D2, D3, and D4.

Next, the image processing unit 116 performs image processing to clarify the abnormal area (S53). In the example of Figures 10 and 11, in partial image D2 displaying bleeding point B, processing is performed to change the pixel values of bleeding point B and other areas so as to make bleeding point B easier to view. In addition, the image processing unit 116 removes eyelash reflections and other artifacts from partial images D1, D2, D3, D4 and wide-angle fundus image P to make the images easier to view.

The image processing unit 116 further performs a process of setting a method of highlighting the abnormal region (S55). The highlighting method is appropriately selected according to the type, severity, and likelihood of the abnormal finding. For example, when abnormal findings are found in multiple locations, the partial image including the abnormal region with the highest severity is set to be displayed preferentially in the subsequent display process (S59). The image processing unit 116 also performs a process of setting a display method for displaying the partial images in which the abnormal region is displayed among the multiple partial images in a manner that distinguishes them from the partial images in which the abnormal region is not displayed. Specifically, the magnification ratio and display order of the partial images in which the abnormal region is displayed are selected according to the type or likelihood of the abnormal finding. Furthermore, the image processing unit 116 selects an image such as a frame or an icon to be displayed over the part to be highlighted according to the type or likelihood of the abnormal finding, and sets it to be displayed as highlighted display in the subsequent display process (S59). Note that these highlighting methods may be set by accepting a user's input.

The image processing unit 116 creates a GUI (Graphical User Interface) according to the partial image extraction method, abnormal area highlighting method, and highlighting selected up to step S55 (S57). This GUI is transmitted to the terminal 20 via the network 5 and displayed on the output device 105 of the terminal 20 (S59). The GUI accepts instructions from the user operating the terminal 20, changes the display in response to these instructions, and also adds and displays images such as icons and frames.

Examples of the GUI are shown in Figures 10 to 14. As shown in each figure, in the GUI, a wide-angle fundus image P is placed on the left side, and four partial images D1 to D4 are placed to the right of that. Frames F1, F2, F3, and F4 corresponding to partial images D1, D2, D3, and D4 are displayed superimposed on the wide-angle fundus image P. The display of these frames F1 to F4 makes it easy to understand which area of the wide-angle fundus image P each of the partial images D1 to D4 indicates.

In Figure 10, partial image D2, which displays the abnormal area, bleeding point B, is given priority and displayed largest.

Figure 12 shows the GUI when an emphasis method (see S55) is selected that further enlarges and displays partial images including abnormal regions. Furthermore, partial images D1 to D4 are all set to display the macula M (see S51). As shown, the magnification of partial image D2, in which bleeding point B is displayed, is greater than the other partial images D1, D3, and D4. This makes it easier for the user to identify bleeding point B.

In the GUI of FIG. 13, the bleeding point B displayed in the partial image D2 is highlighted by a small frame S (see S55). In response to a user instruction, the image processing unit 116 further enlarges the image within the small frame S and displays it as an enlarged image L. This allows the user to easily confirm the bleeding point B. The user's instruction is given by, for example, clicking the image of the small frame S with the mouse or tapping on the display screen.

It should be noted that the enlarged images L can be displayed not only one at a time, but also multiple at the same time. When there are multiple abnormal regions, the enlarged images L can be displayed sequentially in response to a user instruction. All the enlarged images L can be arranged sequentially and displayed as thumbnails. In this case, it is desirable to arrange the enlarged images L based on the type, severity, or likelihood of the abnormal findings.

The magnification rate or angle of view of the enlarged image L is set arbitrarily. For example, the magnification rate is set to 5 times and the angle of view to 30 degrees, which are easy for a doctor to view. The magnification rate or angle of view may be changed according to a user's instruction. The magnification rate or angle of view may also be set to a value preset by the doctor. The user can also change the position of the partial images D1 to D4 or the range of enlargement by manipulating the position and size of the frames F1 to F4.

In the GUI of FIG. 14, highlighting is performed using icons IC1 to IC4. These highlighting methods differ depending on the type of abnormal finding. In FIG. 14, icons IC1 to IC4, which are displayed differently, are used for four types of abnormal findings: areas that the doctor has determined to be abnormal and added a display, abnormal areas output by the AI, areas that the doctor and the AI have determined to be abnormal, and areas that the doctor has determined not to be abnormal and instructed to change the display on the GUI. In this way, the highlighting method is differentiated depending on the type of abnormal finding. Icons IC1 to IC4 are displayed differently from each other, for example, by changing the line type, shape, color, etc.

In addition, when there are multiple abnormal regions, it is possible to display the enlarged images in order of the severity of the abnormal findings. Furthermore, the display order of the partial images D1 to D4 or the display mode of the enlarged image L is changed depending on the severity.

The highlighting method also changes depending on the severity. For example, areas with more severe abnormalities are highlighted in brighter or more saturated colors.

Moreover, the highlighting as shown in Figures 10 to 14 does not depend on the image division method or the GUI display mode, and any type of highlighting can be selected. It is also possible to combine multiple highlighting methods. For example, in a GUI display in which partial image D2 is enlarged as shown in Figure 11, a small frame S and an enlarged image L may be further displayed superimposed on partial image D2.

The image processing unit 116 can also cooperate with the ophthalmologic device 30 to perform OCT imaging using the OCT system and acquire a tomographic image (S61). An instruction to perform OCT imaging can also be received via a GUI in the display process (S59). For example, the user can indicate the abnormal area on which OCT imaging is to be performed by an operation such as clicking the mouse.

If instructed by the user, the image processing unit 116 identifies the fundus position where the abnormal area is located, and instructs the ophthalmic device 30 to capture an image via the network 5. Based on this instruction, the ophthalmic device 30 performs OCT imaging of the abnormal area using the OCT system. The tomographic image of the retina acquired by OCT imaging is acquired by the server 10, and is then displayed on the output device 105 of the terminal 20. At that time, it may be displayed on the GUI together with the wide-angle fundus image P or the partial images D1 to D4.

OCT imaging may be performed without user instructions. For example, if an abnormal finding is found, the image processing unit 116 identifies the fundus position where the abnormal area is located and instructs the ophthalmic device 30 to perform imaging via the network 5. Based on this instruction, the ophthalmic device 30 performs OCT imaging of the abnormal area using the OCT system.

When OCT imaging is performed, imaging may be performed according to patient information. For example, if a diagnosis such as macular degeneration has been obtained in the electronic medical record stored in the database 114, the server 10 instructs the ophthalmic device 30 to perform OCT imaging of the area near the macula M. Based on this instruction, the ophthalmic device 30 performs OCT imaging of the abnormal area.

(Additional Processing)
The additional process will be described with reference to the flow chart in Fig. 9. The user can confirm the abnormal findings and their regions in the display process (S59), and then add annotations to the estimation results and save them (S71). The saved annotations are stored in the database 114 and are saved in a state in which they are associated with the wide-angle fundus image P and the estimation results.

The saved annotations and wide-angle fundus image P are used for re-learning the detection unit 118 (S73). The detection unit 118 can improve its ability to estimate abnormal findings by learning the wide-angle fundus image P together with annotations indicating the type of abnormal finding and the correct abnormal area, for example.

Second Embodiment
In the first embodiment described above, the detection unit 118 uses a trained model having a CNN. As an embodiment different from the first embodiment, an image processing algorithm may be used instead of a trained model that has undergone machine learning to estimate abnormal findings. This will be described below as a second embodiment.

The software configuration of the server 10 according to the second embodiment is shown in Figure 15. In the second embodiment, the server 10 is provided with a detection unit 119 configured by an image processing algorithm.

In the second embodiment, the configurations of the server 10, the terminal 20, and the ophthalmic device 30 are the same as those of the first embodiment, except for the detection unit 119. The same configurations as those of the first embodiment are given the same reference numbers as those used in the first embodiment, and the description is omitted.

The estimation process (S3) by the detection unit 119 in the second embodiment is described below. In this example, an algorithm for estimating the presence or absence of abnormal findings such as bleeding in the wide-angle fundus image P is described.

In step S35, the detection unit 119 extracts the blood region. Here, the blood region is a region in the wide-angle fundus image P that includes a blood vessel region indicating blood vessels and a bleeding region formed by bleeding from the blood vessels. As a result of the processing in step S35, an image in which the blood region is extracted from the wide-angle fundus image P is obtained.

In step S36, the detection unit 119 extracts only the vascular region.

In the extraction of the regions in steps S35 and S36, a general image analysis method is applied. For example, the wide-angle fundus image P is displayed in 256 gradations, and converted into a binary image by comparing the gradation value of each pixel with a threshold value.

When binarizing, in step S35, a threshold is set to distinguish between blood regions and other regions. In step S36, a threshold is set to distinguish between blood vessel regions and other regions. Various methods can be used to set the threshold, such as the mode method.

In step S37, the detection unit 119 removes noise from the binarized image, particularly around the vascular region. The noise removal process uses, for example, dilation and erosion. Through step S37, an image in which the vascular region is extracted from the wide-angle fundus image P is obtained.

In step S38, the detection unit 119 can obtain an image showing the bleeding area by taking the difference between the image showing the blood area and the image showing the vascular area.

The detection unit 119 estimates the presence or absence of bleeding based on the image thus acquired (S39). If the detection unit 119 determines that bleeding is present, it outputs an abnormal finding of bleeding and an image showing the bleeding area.

The above shows an example of estimation processing for bleeding, but the same processing is also performed for other abnormal findings such as neovascularization. In such cases, the parameters used for extracting the region, such as the threshold setting for binarization, are set appropriately according to the type of abnormal finding.

＜Effects＞
The image processing in the above embodiment is an image processing method including a process (S1) of acquiring a wide-angle fundus image (P) obtained by photographing with an ophthalmic device, a process (S1) of extracting multiple partial images (D1-D4) from the wide-angle fundus image to which abnormal findings have been added, a selection process (S51) of selecting a first partial image from the multiple partial images, which indicates an abnormal area (B) that is the basis of the abnormal findings, and a display process (S59) of displaying the first partial image.
Includes.

The above configuration makes it possible to obtain a display that covers a wide range of abnormal findings in the fundus while allowing the abnormal areas to be easily visually identified.

In the display process (S59), partial areas in which abnormal findings are observed are displayed so as to be distinguishable on the wide-angle fundus image P. In addition, partial images in which abnormal findings are observed among the partial images D1 to D4 can be enlarged and displayed.

This configuration allows the user to easily view details of abnormal findings and abnormal areas.

In addition, the display process (S59) can highlight abnormal areas. This highlighting is changed depending on the type of abnormal finding.

The above configuration allows the user to easily view details of abnormal findings and abnormal areas. In addition, even when there are multiple types of abnormal findings, the user can organize and check the abnormal findings by type.

The above embodiment includes an estimation process (S3) for estimating abnormal findings in the wide-angle fundus image P. The estimation process includes a process (S31) for inputting the wide-angle fundus image P to the detection units 118 and 119 that estimate abnormal findings, and a process for causing the detection units 118 and 119 to estimate abnormal findings.

This type of estimation process can assist doctors and medical professionals in making diagnoses and providing findings. It can also prevent abnormalities from being overlooked.

The above embodiment includes a process (S71) of accepting the addition of annotations regarding abnormal findings to the wide-angle fundus image P, and a process (S73) of re-learning the detection unit 118 using the annotations and the wide-angle fundus image P.

Because the detection unit 118 re-learns using the annotated data, the accuracy of the estimation process of the detection unit 118 can be improved. In other words, the probability of false positives, false negatives, etc. being output can be reduced.

In the estimation process (S3), the severity of the abnormal finding is estimated. In addition, the display method of the first area is changed according to the estimated severity.

The above embodiment further includes a process (S61) for identifying the position of the abnormal area on the fundus, and a process (S61) for performing OCT imaging at the identified position using an OCT system to obtain a tomographic image at the identified position.

With the above configuration, it is possible not only to check the wide-angle fundus image P, but also to accurately and easily obtain a tomographic image when further checking for abnormal findings.

<Modification>
In the above embodiment, the partial regions of the wide-angle fundus image P are set by rectangular regions, but the present invention is not limited to this. For example, the image may be divided by extending a dividing line radially, or may be divided into slits, and any shape may be adopted for the dividing line and the partial region.

The annotation may also include the shape of the partial region setting. For example, the partial region setting shape that the user judges to be optimal may be included in the annotation, and the detection unit 119 may learn and re-learn it. In this case, the detection unit 119 outputs the optimal partial region shape and partial region position corresponding to the abnormal findings estimated. In the partial image extraction process (S51), partial image extraction of the wide-angle fundus image P may be performed according to the partial image extraction method output by the detection unit 119.

The display mode in the display process (S59) and the like is a mode in which abnormal areas are displayed by distinguishing them by color or shape using an image, but the present invention is not limited to such a visual mode. For example, a mode in which display variations are set by sound may be used.

In the above embodiment, an ophthalmic device 30 having an OCT system was used, but the present invention is not limited to this and may be an ophthalmic device that does not have an OCT system or an ophthalmic device that uses another system.

In the above embodiment, multiple terminals are connected to one server 10 to perform the above functions. The present invention does not limit the number of servers or the number of terminals, and for example, the above functions may be realized by only one device. The number of terminals and the number of servers may be further increased. Furthermore, each function does not necessarily have to be realized by a server 10 or the like, and the functions can be shared among multiple devices to achieve the functions. In other words, the present invention does not limit the number of control units or devices, or the sharing of functions among devices.

Reference Signs List 1 Information processing system 10 Server 20 Terminal 30 Ophthalmic device

Claims (16)

A process of acquiring a wide-angle fundus image captured by an ophthalmic device;
A process of extracting a plurality of partial images from the wide-angle fundus image to which an abnormal finding has been added;
a selection process for selecting a first partial image showing an abnormal region that is the basis of the abnormal finding from the plurality of partial images;
a display process for displaying the first partial image;
An image processing method comprising: The image processing method according to claim 1, wherein the display process is a process of displaying a second partial image, which does not show the abnormal region, among the plurality of partial images, simultaneously with the first partial image. The image processing method according to claim 2, wherein the display process is a process of displaying the first partial image and the second partial image in different display forms. The image processing method according to claim 3, wherein the display process displays the first partial image at a size larger than the second partial image. The image processing method according to claim 3 or 4, wherein the display process extracts the abnormal area from the first partial image and displays the abnormal area at a higher display magnification than the second partial image. The image processing method according to any one of claims 1 to 5, wherein the display process is a process of simultaneously displaying a position display image that displays the position of the first partial image in the wide-angle fundus image with the first partial image. The image processing method according to any one of claims 1 to 6, further comprising a process of highlighting the abnormal area in the first partial image. The image processing method according to claim 7, wherein the highlighting is changed depending on the type of the abnormal finding. inputting the wide-angle fundus image into an estimator for estimating abnormal findings;
The image processing method according to claim 1 , further comprising an estimation process for causing the estimator to estimate the abnormal finding in the wide-angle fundus image. The image processing method according to claim 9 , wherein the estimator is a trained model that estimates abnormal findings . A process of receiving an annotation regarding an abnormal finding in the wide-angle fundus image;
The image processing method according to claim 10 , further comprising a process of re-learning the trained model using the annotations and the wide-angle fundus image. The estimation process further includes a process of causing the estimator to estimate a severity of the abnormal finding,
The display process includes:
The image processing method according to claim 9 , further comprising a step of changing a display method of the first partial image depending on the severity. A process of performing an image capture process using an OCT system at a position of the abnormal area in the wide-angle fundus image and acquiring a tomographic image at the position;
The image processing method according to claim 1 , further comprising: A program for causing a computer to execute the image processing method according to any one of claims 1 to 13. An image processing device comprising a processing unit that executes the image processing method according to any one of claims 1 to 13. An ophthalmologic system comprising a processing unit that executes the image processing method according to any one of claims 1 to 13.

Images